For years, it has been considered something of a holy grail in artificial intelligence: training a model without using text, images, recordings or any other information drawn from the real world. Now, researchers from Tel Aviv University and Stanford University have presented a proof of concept for precisely that approach, known as Zero-Data Pretraining.
The model used in the study, which was trained without any external data sources, improved its ability to predict patterns across a wide range of fields, from language and images to music and genetics.
AI can train on data it creates itself
Almost every familiar artificial intelligence system undergoes a training process in which it is exposed to enormous quantities of information, including books, databases, scientific research, websites, images and recordings, and learns to identify patterns within them.
The new study points to a different possibility: An AI system can develop general learning abilities even when all the information used for training is generated internally, effectively from scratch.
The research, which represents a significant advance in the field, was conducted under the supervision of Dr. Yoav Levine of Tel Aviv University's School of Computer Science and AI as part of Dr. Kfir Dolev's postdoctoral research, in collaboration with researchers from Stanford University.
The researchers introduced an unconventional training method they call a "mutual game," in which two AI models start from scratch and learn together. The first model generates short computer programs that produce sequences of symbols, while the second attempts to predict the next symbol in each sequence. The model generating the exercises receives feedback on how much they contribute to learning, allowing it to continuously improve its selection of challenges to match the developing abilities of the learner.
After training, the researchers found that the second model had acquired the ability to predict and reason about information it had never encountered during training, including text, images, sounds, music, computer code and DNA sequences.
They also discovered that increasing the computational resources devoted to the process consistently improved the model's predictive performance across numerous domains. The model could even infer rules from examples presented during testing, without requiring additional training.
One of the study's most intriguing findings concerned the training material the system generated for itself. Among the sequences it produced were familiar mathematical patterns, including the Fibonacci sequence, a widely recognized pattern in mathematics.
Dr. Yoav Levine
(Photo: University of Tel Aviv)
In other words, the system learned to generate sequences with recognizable structures that the second model could use to improve its learning abilities.
In a separate experiment, the researchers found that pretraining with the new method helped the model learn more quickly when it was subsequently exposed to real-world data.
An answer to one of AI research's biggest questions
The study offers an initial answer to several fundamental questions in artificial intelligence research: Is there a universal component of intelligence that does not depend on external information? Can some aspects of a model's learning ability emerge from general patterns acquired without prior exposure to real-world data?
The results suggest that this may indeed be possible.
"For years, the prevailing assumption was that for an artificial intelligence model to learn anything useful, it first had to be fed large quantities of information from the real world," Levine said. "In this study, we challenged that fundamental assumption and demonstrated that a model can acquire general predictive abilities even when it is trained exclusively on information generated within the learning process itself."
Levine emphasized that the findings remain preliminary and have so far been demonstrated only in relatively small models.
"This is still a proof of concept using small models, and the result opens up fascinating questions: What universal mathematical patterns will emerge as we strengthen intelligence using these methods? How far can we develop intelligence without exposing it to information from the outside world? And can we prepare models to learn more effectively from reality thanks to experiences they have generated for themselves?"
The research comes at a particularly significant moment for the AI industry, as developers of increasingly sophisticated models approach the limits of the high-quality human-generated data available for training. Companies are investing heavily in methods that reduce their dependence on existing datasets, instead using computing power to generate training material and enable models to improve through self-training.
The central question now facing researchers is whether, and how, learning systems that independently develop logical and mathematical structures can be combined with factual knowledge from the real world.
Another major challenge is determining whether the approach can scale beyond small experimental systems to the enormous AI models used in today's most advanced applications.
For now, the findings suggest that some of the foundations of artificial intelligence may be developed without exposing a model to the world itself, potentially opening a new direction for the next generation of AI training.