|
--- |
|
license: mit |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
|
|
This model is a fully fine-tuned version of the [Llama-7B](https://huggingface.co./huggyllama/llama-7b) model on synthetically generated arithmetic tasks. It was introduced in [this](https://openreview.net/forum?id=8sKcAWOf2D) paper. It is very similar to [Goat-7B](https://github.com/liutiedong/goat), except it was trained without LoRA. |
|
|
|
For inquiries about checkpoints during the fine-tuning process, kindly reach out to [Nikhil](mailto:[email protected]) via email. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** [Nikhil Prakash](https://nix07.github.io/) |
|
- **Model type:** Autoregressive Decoder-only Language Model |
|
- **License:** MIT License |
|
- **Finetuned from model:** [Llama-7B](https://huggingface.co./huggyllama/llama-7b) |
|
|
|
### Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **Repository:** [Link](https://github.com/Nix07/finetuning/) |
|
- **Paper :** [Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking](https://arxiv.org/abs/2402.14811) |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
```python |
|
from transformers import AutoModel |
|
model = AutoModel.from_pretrained("nikhil07prakash/float-7b") |
|
``` |
|
|
|
## Citation |
|
|
|
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. --> |
|
|
|
**BibTeX:** |
|
|
|
```python |
|
@inproceedings{prakash2023fine, |
|
title={Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking}, |
|
author={Prakash, Nikhil and Shaham, Tamar Rott and Haklay, Tal and Belinkov, Yonatan and Bau, David}, |
|
booktitle={Proceedings of the 2024 International Conference on Learning Representations}, |
|
note={arXiv:2402.14811}, |
|
year={2024} |
|
} |
|
``` |
