Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls

Feiyang Kang | Newsha Ardalani | Michael Kuchnik | Youssef Emad | Mostafa Elhoushi | Shubhabrata Sengupta | Shang-Wen Li | Ramya Raghavendra | Ruoxi Jia | Carole-Jean Wu |