In recent years, the performance improvement of large language models (LLM) has mainly relied on scale expansion, that is, increasing data volume and computing power. However, this model is gradually reaching a bottleneck. Industry experts pointed out that it is difficult to make significant progress simply by relying on the "bigger and better" strategy, and new technological breakthroughs are imminent. This article will explore the challenges facing the current AI field and how emerging "test-time computing" technologies can bring new directions to the development of AI.
With the rapid development of generative AI, the industry's traditional perception of "bigger is better" is changing. Many top AI scientists recently said that the method of improving AI performance by simply increasing the amount of data and computing power is approaching a bottleneck, and new technological breakthroughs are emerging.
Ilya Sutskever, co-founder of Safe Superintelligence and OpenAI, recently expressed his opinion that traditional pre-training methods have entered the performance platform period. This statement is particularly eye-catching because it was the large-scale pre-training method he advocated in early days that gave birth to ChatGPT. Today, he said that the field of AI has moved from the "era era of scale expansion" to the "era era of miracles and discovery".
Currently, large-scale model training faces multiple challenges: training costs of tens of millions of dollars, the risk of hardware failure caused by system complexity, long test cycles, and limitations on data resources and energy supply. These questions prompted researchers to explore new technological paths.
Among them, the "test-time compute" technology has attracted widespread attention. This approach allows AI models to generate and evaluate multiple solutions in real time during use rather than giving a single answer directly. OpenAI researcher Noam Brown made an image analogy: letting AI think for 20 seconds in a play card, which is comparable to expanding the scale of the model and training time by 100,000 times.
Currently, several top AI laboratories including OpenAI, Anthropic, xAI and DeepMind are actively developing their respective technology versions. OpenAI has applied this technology in its latest model "o1", and Chief Product Officer Kevin Weil said that through these innovative approaches, they have seen a large number of opportunities to improve model performance.
Industry experts believe that this transformation of technology routes may reshape the competitive landscape of the entire AI industry and fundamentally change the demand structure of AI companies for various resources. This marks that AI development is entering a new stage where it focuses more on quality improvement rather than simply scale expansion.
In short, the rise of new technologies such as "test-time computing" marks a new stage in the development of AI, that is, paying more attention to the improvement of model performance rather than simply scale expansion. This will change the competitive landscape of the AI industry and promote AI technology to develop in a more refined and intelligent direction. In the future, breakthroughs in AI technology will rely more on algorithm innovation and understanding of the essence of the model, rather than simple computing power accumulation.