Would I trust to let an AI, with zero human input or oversight, to diagnose, come up with treatment plan, and ultimately operate on my l5/s1 disc that's been bugging me for the better part of my adult life?
Would I take a novel drug "discovered" by AI (I mean entirely by AI, no human input, remember we are talking AGI) that promises to cure some chronic neurological disorder?
In both of those cases, they are the biggest hell-no's I can emphatically say.
Until I can say hell yes to that question, we aren't close.
Preempting those who say "Well your doctor/drug companies are probably mostly using/going to be using AI to do that" -- not what we are talking about here, and in both cases, not AGI (and I would probably find a new doctor)
Yes, but it means general the way humans are general. Clearly being a general intelligence shouldn’t require being any better at any individual task than the average human, or even the bottom decile of humans.
Most short-term learning/adaptation is already handled in-context. Modern context windows can hold several books worth of text - plenty for most tasks. Everybody is already using it to adapt models to their projects through skills/instructions/guides etc. ps. I often say that after glossary-skill next must have one is update-skill-skill that threats all .md files as live documents.
Persistent weight adaptation also happens just not in real time - sessions are captured, analyzed, transformed into training data, fed into SFT/RL environments and later contribute to model updates. Takes a bit of time for the whole loop but you can't say it's not present.
There's nothing fundamentally preventing real-time weight updates, ie. LoRA-style online adaptation would be one obvious approach. It's just generally not worth doing at scale. Updating a shared model centrally gives much better data efficiency, batching, evaluation, control etc. than continuously training a separate set of weights for every user/session.
There is some work happening on narrowing that gap, for example Mistral has been pushing efficient LoRA-based customization, continuous pretraining, model adaptation etc.
I also did play a bit with activation steering – it's super cool where you extract profile for some concepts (emotional in my case) and you have effectively toggles to control "brightness/contrast" those areas (enhancing or suppressing those activation regions from profile) injecting to the model those concepts (emotions in my case) – you can do it in real time and it's fun thing to play with.
My Claude admits mistakes and then fixes them on its own all the time.
Often even without my input: "(thinking..) Oh I discovered that I misjudged XYX, let me fix that.. (thinking) (executing scripts) Okay I corrected my mistake, I had accidentily ABC."
I may miss some context because the GP’s link has a paywall. But Altman said, "Let’s say we make an AI that’s really good". What is that supposed to mean? Really good relative to what? Current models are "really good" in many ways but nowhere near AGI. Really good compared to an average human? At everything? We’re talking about AGI, so "it’s a really good programmer/coworker/whatever" is a necessary but nowhere near sufficient condition, obviously. But given the constraints of LLMs, we can cut them some slack and only demand they be human-equivalent at digital tasks rather than walking and cooking. Still, being "really good" at all that seems to me really difficult to measure. But it’s a sufficient but not necessary condition anyway, AGI just means equivalent to human, not equivalent to a really smart human. So I really do wonder what Altman meant there if anything.
"at everything"? Surely you know a friend or two who is not good at almost anything – but you wouldn't hesitate to say that he possesses general intelligence.
What if it can be Einstein, but can't draw a Pelican, write a solid college-level essay, or fold clothes?
The ability to do a ton of book learning in training, and pull in tons of related context at once, is superhuman in some ways, but lags a lot in others.
> What if it can be Einstein, but can’t draw a Pelican, write a solid college-level essay, or fold clothes?
Then it’s an expert system.
Stephen Hawking wasn’t very good at folding clothes.
The ‘General’ part of the term ‘AGI’ seems like a trap to me, because there will always be new workflows to master. Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?
You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.
Meanwhile, building a series of expert systems targeting specific valuable workflows is useful today and seems like it’ll continue to scale to cover huge swathes of economically valuable workflows.
I think that’s the more interesting thing to be measuring. The surface area of useful economic workflows that can be addressed with expert systems built with today’s tech.
Hitting some ‘Artificial Expert Intelligence’ coverage threshold on economically valuable workflows is what will matter for humans well before pure ‘general’ intelligence.
The only important part of 'general' is the ability to learn from experiential data and update your own model. That's what leads to general capability. Humans can't oneshot any task natively, but we can practice for a while until we uncover often novel methods of accomplishing something.
Therefore: the current transformer architecture is fundamentally incapable of AGI because the models have no mutable long-term memory.
You only have weights (large immutable memory), or context (small mutable memory).
Humans have mutable long-term memory: I can learn a new skill, adapt an old skill to new information, or learn new knowledge today that I couldn't perform/didn't know yesterday. I don't have a training cutoff.
Context engineering is an attempt to paper over this limitation. You can get really far with context engineering and huge models, but you will never get to AGI because there are many tasks where humans' mutable long-term memory outperforms.
For example, a human can invent a new musical instrument and then learn how to play the instrument they just invented. That's inference (inventing an instrument) leading to training (neuroplasticity). Humans have the ability to train our NNs with considerably fewer training samples. Everything that you can do with transformers is in one causal direction: training -> inference.
So if we take a huge with enough compute (CPUs, b200s, petabytes of SSDs), we install on it both the Astra, and the toolsuite to incorporate new sensory inputs (threads/sessions), camera, microphone, temp sensors, the lot, into a new version of the model. This model is then swapped for the old model, or traffic slowly brought over, or even adjusting weights in place.
Then my hypothesis is that thing as a whole could achieve AGI.
This feels like a very close approximation on how we humans evolve our brain. By encountering new experiences/sensations, classifying them as negative or positive to us, filling it away in neurons. Or by training motor skills etc. In the end we get more connections between neurons in our brain and we are capable of more.
Bingo, LLM architecture just does not lend itself to becoming AGI. They can get really good, sure, but they will always struggle with novel input and scenarios.
The more training data that is shoved in to them, the more they'll seem to solve novel situations, but in reality it'll be things that exist in the training data.
Aka Star Trek hologram characters aren't sentient, and actually anyone who things droids in Star Wars can think of a weirdo. C3-PO just kept running out of context and trying to revert to it's system prompt.
> Can Astra one-shot level completion on some yet-to-be-released video game? If no, does that mean it’s not yet ‘Generally’ intelligent?
If a model can't learn on their own to play some new game just as well as humans do, it's not AGI.
It's okay if they would take some hours or days of learning (like humans might), but if they can't do it at all during their normal operation, that's not general intelligence
> You won’t get pure ‘general’ intelligence until you find Einstein’s hidden variables and load the state of the entire universe into context.
But humans have general intelligence. AGI is about matching human ability, and we know this is possible in principle because brains exist
And the only reason LLMs can't write essays indistinguishable from human output is because they aren't RLHF'ed to write like humans.
Folding clothes isn't an LLM's job but if you were to insist, they could certainly do it, as any number of videos from robotics labs will attest. That particular future is already here but definitely not evenly-distributed.
Adding sibling comments, I think some people may be overestimating how well the median human can draw a pelican, or create an SVG of a pelican (depending if we’re comparing to an image generation model, or SVG generation).
Most people can't draw a bicycle. There was an artist 10 years ago that asked people to sketch a bike, and then turned these sketches into 3D renders - quite funny.
I can't draw a pelican. Literally my only point of reference would be AI pelican drawings from the test. Otherwise I wouldn't know how to draw one at all.
I would be able to draw an accurate bicycle, but I'm an outlier on that. Most people could not draw one [1].
Can definitely write college level essays and have for a while. The jobs is that when LLMs first started getting popular, but aren’t quite common professors were that some of the worst students in class started writing the best essays. Now everyone complains because they can detect the slop, but most human writing is so bad. But the really good human writing is still much better.
I would maybe argue that Einstein was the most LLM-like of great thinkers.
A lot of his great discoveries were mostly that he was very knowledgeable about the bleeding edge research in a number of disparate areas, and was able to have the aha moment where he could make the connections for how to integrate them.
A lot of other thinkers who created new fields from scratch are probably way harder for an LLM to crack.
That is very aligned with an LLMs ability to have superhuman knowledge in wide areas.
That’s like that scene in the I, Robot movie when Will Smith’s character is asking the robot “can you turn an emtpy canvas into a work of art, or compose a symphony?” and the robot replies “can you?”.
I think they mean improve our understanding of physics with new theoretical results or paradigms. Like if it’s 1899, would Astra develop General and Special relativity on its own?
This might just not be possible at the current time. In 1899, there was an "experimental overhang" in physics -- results that could not be explained theoretically (Michelson-Morley, but also lots and lots of empirical material/spectroscopic properties that we could today calculate using quantum mechanics). The big problem in theoretical physics today is that unifying general relativity and quantum theory has no experimental results you could get at our technological level.
I think you make a fair point, but also remember: new paradigms don't necessarily require confusing / contradictory observations. You could have the simple idea of "what if gravity is an inertial force?" at any period in time and work out the mathematics of this. It would make theoretical predictions which could then be falsified, but then again, who would take it seriously enough to test it if it was maybe say 1850 and not 1899.
A better example is maybe Maxwell's laws. Maxwell wasn't inventing a theory to try and explain confusing results, he was unifying a chaotic, empirical laws from existing experiments. That may be a cleaner example. That knowledge compression into satisfying theoretical framework is likely what is attractive.
You can potentially ask the same thing about like you say -- general relativity / quantum gravity but also likely plenty of other areas that may be like this today. Again going outside my particular area of expertise: standard model physics is in a large important sense empirical; lots of values and numbers that are simply unmotivated by theory or where we don't have a good way to make a principled theoretical choice. That could be a place where these models are able to help.
But right now: I doubt it. This is what everyone is working furiously on right now. How do you close a "science" verification loop? In principle this should be easy right: you have ideation (exploration, sampling with ~high temperature maybe as an analogue) and you have verification (which of these ideas are good) which amounts to rejection sampling in idea space. You have to have a sampler that is good at picking _good_ ideas for efficiency sake and you need a relatively fast and reliable verification step of "is this idea good and worth continuing to explore". But I may oversimplify
This is as good a time as any to note that we might be closing in on a new conceptual revolution in our own time as it relates to holography and an information centric approach to spacetime. Obviously It's the furthest possible thing from a guarantee, but it has much of the enthusiasm and motivation that string theory had previously enjoyed in prior decades.
So it could be a natural experiment for whether AI can contribute to novel physics. Specifically, there's a big question about weather. Something like our informational understanding of black holes where information inside it is equivalent to information on its boundary (which I'm sure I'm not saying correctly), might be generalized to regular space-time. More people should be freaking out with excitement about this and perhaps it's something to which AI can contribute.
Honestly I don't think I have single great article, though some Quanta ones are ok, and the Wikipedia article is okay.
The best thing I can recommend is what I did, which is ask Claude about the significance of (1) quantum computing error correction, and (2) error correction in black hole holography and research convergence between the two.
I have no idea what you're hoping the contribution would be. The AdS/CFT correspondence is 29 years old by now and it doesn't seem to apply to our spacetime, where the cosmological constant seems to be positive rather than negative. There are some puzzling consequences of the holographic principle in that scenario as well (https://arxiv.org/pdf/hep-th/0208013), but the linked articles don't talk about them?
Quanta articles are written for people with no background whatsoever, which makes them impenetrable if you have a bit of background and are trying to figure out what they're about. I don't know how good Claude is compared to that -- whenever I try asking any LLM about something I don't understand, it produces a wall of text, I have no idea whether it's correct or relevant, and I look for a textbook or review paper instead.
The wiki article has a section on 'Energy, matter, and information equivalence', the first Quanta article is almost entirely about the 'deep connection between quantum error correction and the nature of space, time and gravity' and about bringing the same information centric approach from AdS to our spacetime. The second Quanta article is explicitly about about bringing a holographic approach to our non AdS spacetime and cites an Ed Witten paper as the cornerstone of that approach (which one perhaps overly excited MIT physicist describes as 'revolutionary').
AdS is 'old' but the articles aren't suggesting it is new, and our spacetime is not AdS and the articles don't suggest otherwise. The point is that there's a search for a way to fit the holographic approach to our spacetime that's inspired by how AdS helps make sense of black holes. Quanta writing being directed at a lay audience ought to be a good thing, not a bad thing and they do link to papers if that's your jam.
Whether or not your LLM of choice produces indecipherable walls of text, and whether it ties those to sufficiently satisfying citations, I think is just a matter of how you go about the prompting.
You're right, I'm prejudiced against Quanta (often IMO they look for a clean narrative to the point of misleading and/or rely too much on metaphors) and was probably too harsh here. Sorry!
That said, I don't like them because their articles never leave me feeling like I understood something. They never go in an order of simple to complex and constantly try to hook you. (A positive example to contrast would be 3blue1brown, who manages to both hook you and make you understand, even with rather little background.)
Could you share a link to your Claude conversation? If this is a prompting issue, I would be interested in seeing what's possible. Thanks!
There was symbolic AI programs in the 1980’s that “discovered” Kepler’s laws and the resulting solar system model from just tycho brache’s astronomical observations. That was the the very first “new physics” ever.
Do you mean after? People do this!! But I think it’s a bit different. It won’t be apples to apples because the data volume I think is just so much different. Maybe there are good experiments for something like this.
For it to be like a human it wouldn't just need to solve existing phsyics problems, it would need to push the field forward and introduce new paradigms.
My comment wasn't very long, yet you somehow still ignored the main part, "and introduce new paradigms". The point is whether it can do everything humans can, entirely new theoretical frameworks and ideas, such as string theory or dark matter, are not coming out of AI at the moment.
AI could discover candidate novel physics without autonomously operating new physical experiments, and humans or instruments can later independently validate the result. This is analogous to how Einstein developed theories whose predictions were confirmed by experiments and observations only years or decades later.
Do we have any examples of an current day AI system introducing a novel concept or perspective. We've got plenty of counterexamples discovered and some theorems proven, but afaik nothing analogous to a new definition.
It could be a good theoretical physicist. Actually it could be a good experimental physicist as well since senior experimental physicists use grad students for the manual labor.
https://www.nytimes.com/2023/11/20/podcasts/hard-fork-sam-al...
Sam Altman: Let’s say we make an A.I. that is really good, but it can’t go discover novel physics. Would you call that AGI?
Kevin Roose (New York Times): I probably would, yeah. Would you?
Sam Altman: Well, again, I don’t like the term, but I wouldn’t call that done with the mission.