|
--- |
|
library_name: peft |
|
datasets: |
|
- cis-lmu/bavarian_to_english |
|
language: |
|
- de |
|
- bar |
|
base_model: |
|
- LSX-UniWue/LLaMmlein_1B |
|
pipeline_tag: text-generation |
|
license: other |
|
--- |
|
|
|
# LLäMmlein 1B |
|
|
|
This is a Bavarian adapter for the German Tinyllama 1B language model which was tuned on a dump of the Bavarian wikipedia, without further optimization. Please don't take it too seriously ;) |
|
Find more details on our [page](https://www.informatik.uni-wuerzburg.de/datascience/projects/nlp/llammlein/) and our [preprint](arxiv.org/abs/2411.11171)! |
|
|
|
## Run it |
|
```py |
|
import torch |
|
from peft import PeftConfig, PeftModel |
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
# script config |
|
base_model_name = "LSX-UniWue/LLaMmlein_1B" |
|
adapter_name = "LSX-UniWue/Betzerl_1B_wiki_preview" |
|
device = "cuda" # or mps |
|
|
|
# load model |
|
config = PeftConfig.from_pretrained(adapter_name) |
|
base_model = model = AutoModelForCausalLM.from_pretrained( |
|
base_model_name, |
|
torch_dtype=torch.bfloat16, |
|
device_map=device, |
|
) |
|
base_model.resize_token_embeddings(32001) |
|
model = PeftModel.from_pretrained(base_model, adapter_name) |
|
tokenizer = AutoTokenizer.from_pretrained(adapter_name) |
|
``` |
