--- license: llama3 base_model: catallama/CataLlama-v0.2-Base tags: - llama - llama-3 - Catalan model-index: - name: CataLlama-v0.2-Instruct-SFT results: [] datasets: - catallama/Catalan-Instruct-V2 language: - ca - en pipeline_tag: text-generation library_name: transformers --- ![](https://huggingface.co./catallama/CataLlama-v0.1-Instruct-DPO/resolve/main/CataLlama-v0.1.png) # CataLlama-v0.2-Instruct-SFT **CataLlama-v0.2-Instruct-SFT** is an instruct fine-tune of [catallama/CataLlama-v0.2-Base](https://huggingface.co./catallama/CataLlama-v0.2-Base) on the [catallama/Catalan-Instruct-V2](https://huggingface.co./datasets/catallama/Catalan-Instruct-V2) dataset. CataLlama-v0.2 was trained on roughly **800 million new tokens** which is almost double compared to CataLlama-v0.1. This new (V2) SFT Dataset was built mostly from scratch and it only retained parts of the V1. On top of the existing instructions in Catalan, **250k additional instructions were translated with ChatGPT-3.5.** All the English instructions existing in the V1 of the dataset were discarded and replaced with high quality instructions scored with [RLHFlow/ArmoRM-Llama3-8B-v0.1](https://huggingface.co./RLHFlow/ArmoRM-Llama3-8B-v0.1) reward model. The model shows improved proficiency with the Catalan language while performing significantly better than CataLlama-v0.1 on all tasks. **This is an instruction fine-tuned model proficient on the following tasks in Catalan** - *Information extraction (suitable for RAG)* - *Named Entity Recognition (NER)* - *Translation from English to Catalan and Catalan to English* - *Summarization - both short form and long form* - *Sentiment analysis* **Model developers** [Laurentiu Petrea](https://www.linkedin.com/in/laurentiupetrea/) based on Llama-3 from Meta. **Model Architecture** CataLlama is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and direct preference optimisation (DPO) to align with human preferences for helpfulness and safety. **License** The model uses the llama-3 license available at: [https://llama.meta.com/llama3/license](https://llama.meta.com/llama3/license) ### Use with transformers See the snippet below for usage with Transformers: **The model follows the same prompt template as Llama-3 Instruct** ```python import transformers import torch model_id = "catallama/CataLlama-v0.2-Instruct-SFT" pipeline = transformers.pipeline( "text-generation", model=model_id, model_kwargs={"torch_dtype": torch.bfloat16}, device_map="auto", ) messages = [ {"role": "user", "content": "Ei com estàs avui?"}, ] prompt = pipeline.tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) outputs = pipeline( prompt, max_new_tokens=1024, do_sample=True, temperature=0.6, top_p=0.9, ) print(outputs[0]["generated_text"][len(prompt):]) ``` ## Training procedure The model was trained **with the same prompt template of Llama-3 Instruct**. The model was trained for two epochs on **8x A100 80GB GPUs using DeepSpeed ZeRO** State-3 without CPU offloading. Then training lasted approximately 8 hours for a total GPU cost of 150€. ### Training hyperparameters The following hyperparameters were used during training: - learning_rate: 2e-05 - distributed_type: multi-GPU - num_devices: 8 - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type: cosine - lr_scheduler_warmup_steps: 100 - num_epochs: 2 ## Intended Use **Note:** This model is not intended to beat benchmarks, but to demonstrate techniques for augmenting LLMs on new languages and preserve rare languages as part of our world heritage. **Intended Use Cases** Llama 3 is intended for commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks. **Out-of-scope** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by the Acceptable Use Policy and Llama 3 Community License. Use in languages other than English**. **Note: Developers may fine-tune Llama 3 models for languages beyond English provided they comply with the Llama 3 Community License and the Acceptable Use Policy.