I found this interview with Francois Chollet fascinating, and would be curious to hear what other people make of it.
I think it is impressive that he's managed to devise a benchmark of tasks which are mostly pretty easy for most humans, but which LLMs have so far not been able to make much progress with.
If you don't have time to watch the video, then I think these tweets of his sum up his views quite well:
The point of general intelligence is to make it possible to deal with novelty and uncertainty, which is what our lives are made of. Intelligence is the ability to improvise and adapt in the face of situations you weren't prepared for (either by your evolutionary history or by your past experience) -- to efficiently acquire skills at novel tasks, on the fly.
Meanwhile what the AI of today does is to combine extremely weak generalization power (i.e. ability to deal with novelty and uncertainty) with a dense sampling of everything it might ever be faced with -- essentially, use brute-force scale to *by-pass* the problem of intelligence entirely.
If intelligence is the ability to deal with what you weren't prepared for, then the modern AI strategy is to prepare for everything, so you never need intelligence. This is of course a terrible strategy, because it is impossible to prepare for everything. The problem isn't just scale, the problem is the fact that the real world isn't sampled from a static distribution -- it is ever changing and ever novel.
If his take on things is correct, I am not sure exactly what this implies for AGI timelines. Maybe it would mean that AGI is much further off than we think, because the impressive feats of LLMs that have led us to think it might be close have been overinterpreted. But it seems like it could also mean that AGI will arrive much sooner? Maybe we already have more than enough compute and training data for superhuman AGI, and we are just waiting on that one clever idea. Maybe that could happen tomorrow?
Apologies for not being clear! I'll try and be a bit more clear here, but there's probably a lot of inferential distance here and we're covering some quite deep topics:
So on the first section, I'm going for the latter and taking issue with the term 'automation', which I think speaks to mindless, automatic process of achieving some output. But if digital functionalism were true, and we successful made a digital emulation of a human who contributed to scientific research, I wouldn't call that 'automating science', instead we would have created a being that can do science. That being would be creative, agentic, with the ability to formulate it's own novel ideas and hypotheses about the world. It'd be limited by its ability to sample from the world, design experiments, practice good epistemology, wait for physical results etc. etc. It might be the case that some scientific research happens quickly, and then subsequent breakthroughs happen more slowly, etc.
My opinions on this are also highly influenced by the works of Deutsch and Popper too, who essentially argue that the growth of knowledge cannot be predicted, and since science is (in some sense) the stock of human knowledge, and since what cannot be predicted cannot be automated, scientific 'automation' is in some sense impossible.
Agreed, AI systems are larger than LLMs, and maybe I was being a bit loose with language. On the whole though, I think much of the case by proponents for the importance of working on AI Safety does assume that current paradigm + scale is all you need, or rest on works that assume it. For instance, Davidson's Compute-Centric Framework model for OpenPhil states right in that opening page:
And I get off the bus with this approach immediately because I don't think that's plausible.
As I said in my original comment, I'm working on a full post on the discussion between Chollet and Dwarkesh, which will hopefully make the AGI-sceptical position I'm coming from a bit more clear. If you end up reading it, I'd be really interested in your thoughts! :)