There was an interesting thought from a Microsoft researcher in an episode of This American Life. He had been given early access to GPT4 and found that it had gained an understanding of gravity and balancing. 3.5 would fail to describe a safe order for stacking 3 eggs, a bottle, a book and a nail. But 4 would give robust answers with logical justification added to each construction step for the tower.
The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.
> Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.
I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.
There are many ‘concept models’ that could work well enough. Unless we can inspect the concept model directly (inside the LLM, somehow!) we are left to reason about how the concept is represented via statistics over a set of interactions. So, if, over the course of many interactions, it seems that a model is saying reasonable things we can have more confidence that it has achieved a concept model that is “good enough”. But such an assessment is always in relationship to the test protocol. Claims to generality should be done carefully.
In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.
Yeah I’m setting the bar at the level of a child’s understanding. Because smaller models are well below that, just picking tokens at random. That’s how misunderstanding manifests with LLMs. They don’t say “I don’t know”, they just go off the rails.
The quality bar for different use cases might be such that an understanding of a system on par with a 6 year old is isn’t useful. But right now we clearly have two clusters and I’m drawing the boundary where it’s most useful.
> I take your point, but we should be careful when using phrased like “modeled the concept appropriately” here. That implies correctness.
All models are wrong, but some are useful.
Perhaps we should say "modeled the concept usefully".
> In some ways, my standard here is probably not that different than a rigorous human educational assessment. I’m not talking about standardized tests, I’m talking about adversarial challenges like one would hope to see during a Ph.D. defense.
Usefulness is very contextual obviously. Newton's laws aren't entirely correct but are still useful.
This is the same reasoning that Ilya Sutskever uses. That next token prediction objective causes the model to learn world models. Because being able to predict the next token requires a lot of the knowledge/model of the world.
The researcher remarked (paraphrasing) “The models are built to predict the next token. At some point the only way to improve token prediction is to start modeling the world and building an understanding of how things work.” Since I heard this I see it everywhere with LLMs. Load up a 30B parameter model on your computer and toy around with it. When it fails I now see that it’s reached a concept it doesn’t understand and is in more-or-less Markov Chain mode. Give the same task to GPT4 and it proves it’s modeled the concept appropriately because it can pull the right words out of the ether.