bibtext: @inproceedings{
  ostendorff2024llmdatasets,
  title={{LLM}-Datasets: An Open Framework for Pretraining Datasets of Large Language Models},
  author={Malte Ostendorff and Pedro Ortiz Suarez and Lucas Fonseca Lage and Georg Rehm},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=5RdIMlGLXL}
  }
abstract: Large language models have become the cornerstone of today's natural language processing research. To facilitate the training, evaluation, and deployment of language models, the community has developed a series of tools and frameworks and made them openly available. This joint community effort has led to more collaboration, standardization, and overall more progress in language model research. However, one crucial aspect of large language models has been neglected so far: the pretraining datasets. To address this gap, we present an open framework for the collection and systematic compilation of pretraining datasets, called LLM-Datasets. With LLM-Datasets, we make a community-effort and collaborate with experts from the individual languages to collect and systematically compile datasets suitable in terms of data quantity and quality for pretraining language models in a multilingual setting.  The framework provides a unified interface to pretraining datasets enabling the download, text extraction, filtering, and sampling of the pretraining data. It is modular and extensible with new datasets and designed with high-performance-computing requirements in mind that are needed to achieve the scale of today's language models. Users of the framework can focus on the actual data composition and reuse existing datasets from the community while ensuring reproducibility. To showcase LLM-Datasets, we compiled a pretraining dataset with 2.3 trillion tokens for a large language model covering 32 European languages.
title: LLM-Datasets: An Open Framework for Pretraining Datasets of Large Language Models
authors: Malte Ostendorff Pedro Ortiz Suarez Lucas Fonseca Lage Georg Rehm