README.md

---
license: apache-2.0
datasets:
- togethercomputer/RedPajama-Data-1T
language:
- en
base_model:
- KoboldAI/fairseq-dense-125M
---

# Data Scorer

The model to score data for data selection in the paper [Data Selection via Optimal Learning for Language Models](https://arxiv.org/abs/2410.07064). To use the model, follow the instructions [here](https://github.com/microsoft/LMOps/tree/main/data_selection#5-use-the-data-scorer-to-score-examples).

NOTE: you may need to download the [fairseq-125M](https://huggingface.co./KoboldAI/fairseq-dense-125M) to `${PATH_TO_DATA_SELECTION_REPO}/checkpoints/fairseq/125M` to prepare the tokenizer and config.json for the base model.

### Citation

```bibtex
@article{gu2024data,
  title={Data Selection via Optimal Control for Language Models},
  author={Gu, Yuxian and Dong, Li and Wang, Hongning and Hao, Yaru and Dong, Qingxiu and Wei, Furu and Huang, Minlie},
  journal={arXiv preprint arXiv:2410.07064},
  year={2024}
}
```