Clearly GPT doesn't "know" things in the way we do- I'd argue that there's probably a little bit of world modeling in there, but piecemeal and crude at best. Only so much you can do with such limited iteration. However, we must acknowledge that there are evidently some tasks that don't require knowledge the way we usually think about it. For a human, being able to talk about a game stems from the same place as our ability to understand one- our knowledge. But for LLMs, you get one without the other. The fact that ChatGPT doesn't "know" what a game is isn't sufficient to explain why it can't play one. It would be in a human, but clearly ChatGPT can perform tasks without knowledge that we'd need knowledge for, since it can talk about games just fine! So why not this one? Your hypothetical perfect sentence predictor could absolutely play a totally novel game, after all. It is a worthwhile question to ask why this particular flawed predictor can do one but not the other.
I’ll give you an example to think about in human terms.
In political discussions, a lot of humans don’t really know what the fuck they’re talking about, but they do know what to say in response to certain stimulus. They know the talking points, the key phrases, the terms, they parrot these back to you when you provoke them to say it. They are basically human sentence predictors. Stuff comes out of their mouth based on what they’ve been trained to think the next word should be. They don’t pause to reflect on some gathered knowledge and then present an observation to the world.
This is basically what GPT is, but with everything. And the only way GPT plays a game, is if the state of the game can sufficiently activate some output that represents the next best move for the game being played, which isn’t really a readily available data set for all games, especially for made up games.
Thank you for that example. I've been thinking a lot about how difficult it is to persuade people to change their mind. Most people seem impervious to incorporating new facts or ideas into their internal narrative (myself included). ChatGPT has made me wonder if that is because much of what we consider "cognition" in humans is really just "human sentence predictors".
We've all been "trained" with various facts and when two people meet who have been trained on substantially different bodies of knowledge/facts/experiences it can be very difficult to find common ground.
FWIW, I asked ChatGPT to give me a short list of cognitive biases and psychological phenomena. Interesting to think how many of these are dependent on our personal "training data":
That is indeed an example, but it still isn't an explanation. Sure, humans can also generate output without understanding- I've done it myself for my undergrad, throwing words together at 2am to make a deadline. I think quite a few people have remarked that LLMs seem to write like a human that isn't paying attention, which squares with what you said.
But the question remains why this is possible! A good enough pure predictor could play novel games or devise new theorems, but GPT absolutely can't. It can however give confident explanations, as well as write passable code and occasionally even do some novel problem solving (simple, impressive only because it comes from a computer, but still there). The question of why it can do some things and not others is interesting, and can't be swept under the rug just by reiterating that it's a predictor.
Does the structure of language really do such a good job of conveying information that GPT can operate on it blindly and get results, Blindsight style? Is composing prose far easier than we expect, leaving the bulk of the model free to do a tiny amount of "reasoning" that we find unjustly impressive because of how well it's presented? Is it handicapped primarily by the fact that it can only carry out extremely short computations, and can't be trained to use chain-of-reasoning to get around the limitation? We have no idea what, if any, inherent limitations predictive models have. We have no idea why GPT-sized models are good at the things they are, and bad at the things they aren't.
