File size: 968 Bytes
5726922 d9b2b78 5726922 d9b2b78 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
---
license: apache-2.0
datasets:
- togethercomputer/RedPajama-Data-1T
language:
- en
base_model:
- KoboldAI/fairseq-dense-125M
---
# Data Scorer
The model to score data for data selection in the paper [Data Selection via Optimal Learning for Language Models](https://arxiv.org/abs/2410.07064). To use the model, follow the instructions [here](https://github.com/microsoft/LMOps/tree/main/data_selection#5-use-the-data-scorer-to-score-examples).
NOTE: you may need to download the [fairseq-125M](https://huggingface.co./KoboldAI/fairseq-dense-125M) to `${PATH_TO_DATA_SELECTION_REPO}/checkpoints/fairseq/125M` to prepare the tokenizer and config.json for the base model.
### Citation
```bibtex
@article{gu2024data,
title={Data Selection via Optimal Control for Language Models},
author={Gu, Yuxian and Dong, Li and Wang, Hongning and Hao, Yaru and Dong, Qingxiu and Wei, Furu and Huang, Minlie},
journal={arXiv preprint arXiv:2410.07064},
year={2024}
}
``` |