AI's "Scaling Era" is Over: The Real Bottleneck Isn't Compute, It's Ideas
Ilya Sutskever, the co-founder of OpenAI and now head of Safe Superintelligence (SSI), has thrown a wrench into the prevailing narrative of AI development. In a recent interview, Sutskever argued that the industry's relentless focus on scaling—bigger models, more data, and massive compute budgets—has reached its limits. The problem, as he sees it, isn't a lack of computational power, but a dearth of fundamental research and new ideas. This isn't just a minor course correction; it's a potential paradigm shift.
For the last five years or so, the AI world has largely operated under the assumption that more is better. More data, more GPUs, more layers in the neural network – the "scaling" approach has been the dominant strategy. As Sutskever put it, this recipe has been "very low-risk way of investing resources," because it has demonstrably produced results. Tech companies have poured billions into acquiring GPUs and building data centers, effectively betting that brute force is the path to artificial general intelligence (AGI).
But Sutskever now suggests that this era is ending. Data is finite, and organizations already have access to massive amounts of compute. The marginal return on simply throwing more resources at the problem is diminishing. "Is the belief really: 'Oh, it's so big, but if you had 100x more, everything would be so different?' It would be different, for sure. But is the belief that if you just 100x the scale, everything would be transformed? I don't think that's true," Sutskever said.
AI's "Jaggedness" Problem: Smart Tests, Dumb Mistakes
Jaggedness: The Achilles Heel of Scaling
Sutskever points to a critical discrepancy: AI models perform exceptionally well on standardized tests ("evals"), yet their real-world economic impact lags significantly. He calls this problem "jaggedness," where a highly competent model can inexplicably get stuck in basic error loops. He gives the example of asking an AI to fix a bug in code, only to have it introduce another bug, then revert to the original. How is that possible? And this is the part of the report that I find genuinely puzzling.
This "jaggedness" is attributed to two connected issues: reinforcement learning (RL) tunnel vision and evaluation-driven training. Reinforcement learning, while powerful, can make models too narrowly focused. The models are optimized to ace tests rather than master general skills. Sutskever likened it to a student who "will practice 10,000 hours" for competitive programming, excelling narrowly but failing to generalize.
Humans, in contrast, demonstrate superior generalization capabilities. We learn from far fewer examples and are more robust in novel situations. Sutskever argues that this is because humans possess a more effective "value function," shaped by emotions, that guides learning. AI systems, currently lacking this integrated system, struggle to self-correct and learn efficiently.
Beyond Brute Force: Is AI Entering a "Research Era?"
The Return to Research: A New Era for AI?
Sutskever believes that the AI industry is entering a new "age of research." This doesn't mean compute is no longer important. He acknowledges that compute is still necessary for research and can be a "big differentiator." However, the focus must shift from simply acquiring more compute to finding more effective ways of using it. As he stated, it's "back to the age of research again, just with big computers."
One crucial area for research is improving the generalization abilities of AI models. Sutskever emphasizes that current models "somehow just generalize dramatically worse than people." Addressing this gap will require fundamental breakthroughs in machine learning principles.
Sutskever's new venture, Safe Superintelligence (SSI), is betting on this shift. The company has raised $3 billion (a substantial sum, even in the AI world) and claims to have "sufficient compute to prove… that what we are doing is correct." Meta Platforms reportedly attempted to acquire SSI, valuing the startup at $32 billion. This suggests that Sutskever's vision is attracting significant attention from industry players.
One consequence of the "age of scaling" is that it has stifled innovation, with everyone pursuing the same strategies. The solution, according to Sutskever, lies in discovering a fundamental machine learning principle that makes models more productive. This is the big question for the next cycle of AI, and it cannot be solved simply by adding more data or bigger computers. It's about ideas, not just iron.
A $32 Billion Reality Check
Sutskever's argument boils down to this: the low-hanging fruit of scaling has been picked. The industry needs to move beyond simply throwing more resources at the problem and focus on fundamental research. This isn't to say that compute is irrelevant, but it's no longer the primary bottleneck. The real challenge is finding new, more efficient ways to use the compute we already have. The fact that Meta was willing to value SSI at $32 billion—to be more exact, reports suggest the figure was closer to $33 billion—suggests that at least some players in the industry are taking this shift seriously.
OpenAI Co-Founder Ilya Sutskever Says It's Back To Research For Real AI Breakthroughs: 'Now The Scale Is So Big...' - Meta Platforms (NASDAQ:META)
