bibtext: @inproceedings{
  tao2024crystal,
  title={Crystal: Illuminating {LLM} Abilities on Language and Code},
  author={Tianhua Tao and Junbo Li and Bowen Tan and Hongyi Wang and William Marshall and Bhargav M Kanakiya and Joel Hestness and Natalia Vassilieva and Zhiqiang Shen and Eric P. Xing and Zhengzhong Liu},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=kWnlCVcp6o}
  }
abstract: Large Language Models (LLMs) specializing in code generation (which are also often referred to as code LLMs), e.g., StarCoder and Code Llama, play increasingly critical roles in various software development scenarios. It is also crucial for code LLMs to possess both code generation and natural language abilities for many specific applications, such as code snippet retrieval using natural language or code explanations. The intricate interaction between acquiring language and coding skills complicates the development of strong code LLMs. Furthermore, there is a lack of thorough prior studies on the LLM pretraining strategy that mixes code and natural language. In this work, we propose a pretraining strategy to enhance the integration of natural language and coding capabilities within a single LLM. Specifically, it includes two phases of training with appropriately adjusted code/language ratio. The resulting model, CRYSTAL, demonstrates remarkable capabilities in both domains. Specifically, it has natural language and coding performance comparable to that of Llama 2 and Code Llama, respectively. CRYSTAL exhibits better data efficiency, using 1.4 trillion tokens compared to the more than 2 trillion tokens used by Llama 2 and Code Llama. We verify our pretraining strategy by analyzing the training process and observe consistent improvements in most benchmarks. We also adopted a typical application adaption phase with a code-centric data mixture, only to find out that it did not lead to enhanced performance or training efficiency, underlining the importance of a carefully designed data recipe. To foster research within the community, we commit to open-sourcing every detail of the pretraining, including our training datasets, code, loggings and 136 checkpoints throughout the training.
title: Crystal: Illuminating LLM Abilities on Language and Code
authors: Tianhua Tao Junbo Li Bowen Tan Hongyi Wang William Marshall Bhargav M Kanakiya Joel Hestness Natalia Vassilieva Zhiqiang Shen Eric P. Xing Zhengzhong Liu