bibtext: @inproceedings{
  huang2024unified,
  title={Unified View of Grokking, Double Descent and Emergent Abilities: A Comprehensive Study on Algorithm Task},
  author={Yufei Huang and Shengding Hu and Xu Han and Zhiyuan Liu and Maosong Sun},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=cG1EbmWiSs}
  }
abstract: Recent studies have uncovered intriguing phenomena in deep learning, such as *grokking*, *double descent*, and *emergent abilities* in large language models, which challenge human intuition and are crucial for a deeper understanding of neural models. In this paper, we present a comprehensive study on algorithm task to provide a unified view of these three phenomena, with a focus on the interplay between memorization and generalization. Through extensive experiments spanning a wide range of model sizes and training data quantities, we uncover four distinct training dynamics, each arising from unique combinations of model size and training data quantity, formulating a theoretical framework for further analysis. Utilizing this framework, we establish connections between *double descent* and *grokking* and propose two verifiable predictions regarding the occurrence of *double descent*, both substantiated by our experimental results. Moreover, we expand our experiments to the multi-task learning paradigm, demonstrating how algorithm tasks can be turned into emergent abilities by mixing some pure memorization data. This offers a novel perspective to understand *emergent abilities* in Large Language Models.
title: Unified View of Grokking, Double Descent and Emergent Abilities: A Comprehensive Study on Algorithm Task
authors: Yufei Huang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun