bibtext: @inproceedings{
  du2024chinese,
  title={Chinese Tiny {LLM}: Pretraining a Chinese-Centered Large Language Model},
  author={Xeron Du and Zhouliang Yu and Songyang Gao and Ding Pan and Cheng Yuyang and Ziyang Ma and Ruibin Yuan and Xingwei Qu and Jiaheng Liu and Tianyu Zheng and Xinchen Luo and Guorui Zhou and Wenhu Chen and Ge Zhang},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=RCdoMrg4I0}
  }
abstract: In this study, we introduce $\textbf{CT-LLM}$, a groundbreaking 2B large language model (LLM) that illustrates a pivotal shift towards prioritizing the Chinese language in the development of LLMs. Uniquely initiated from scratch, CT-LLM diverges from the conventional methodology by primarily incorporating Chinese textual data, utilizing an extensive corpus of 1,200 billion tokens, including 800 billion Chinese tokens and 400 billion English tokens. This strategic composition facilitates the model's exceptional proficiency in understanding and processing Chinese, a capability further enhanced through alignment techniques including supervised fine-tuning (SFT) and direct preference optimization (DPO). Demonstrating remarkable performance on the ChineseHardCase Benchmark, CT-LLM not only excels in Chinese language tasks but also showcases its adeptness in English through SFT. This research challenges the prevailing paradigm of training LLMs predominantly on English corpora and then adapting them to other languages, broadening the horizons for LLM training methodologies. By open-sourcing full-process of CT-LLM, we aim to foster further exploration and innovation within both the academic and industrial spheres, paving the way for more inclusive and versatile language models in the future.
title: Chinese Tiny LLM: Pretraining a Chinese-Centered Large Language Model
authors: Xeron Du Zhouliang Yu Songyang Gao Ding Pan Cheng Yuyang Ziyang Ma Ruibin Yuan Xingwei Qu Jiaheng Liu Tianyu Zheng Xinchen Luo Guorui Zhou Wenhu Chen Ge Zhang