Abstract:
This talk presents two recent papers that show how synthetic datasets generated by large language models can enable training smaller and more efficient models for specific tasks.

The first paper introduces TinyStories, a dataset of short stories using only very simple words, generated by GPT-3.5/4. TinyStories attempts to preserve the essential elements of natural language, such as grammar, vocabulary, facts, and reasoning, while being more compact and focused than typical corpora. While language models as big as 1B parameters often struggle to produce coherent text beyond one or two sentences, we show that TinyStories can be used to train language models that are much smaller than the state-of-the-art models (below 10 million parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate certain reasoning capabilities.

However, most attempts to create synthetic data using LLMs usually end up in datasets which are very repetitive and seem to lack the diversity which is needed so that a model trained on them would exhibit any ability beyond the memorization of these repeating patterns. The generation of TinyStories relies on the (new) idea of attaining this diversity by injecting randomness into the prompt.

A second paper, based on the same paradigm, presents Phi-1, a new large language model for code, trained using a combination of "textbook quality" data from the web and a dataset of synthetically generated textbooks and exercises. Despite having only 1.3B parameters, it achieves pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP, surpassing models more than 10 times its size. We discuss the implications of these results for the development, analysis and research of language models, especially for low-resource or specialized domains, and the potential of synthetic datasets to improve the performance and efficiency of LLMs.