metadata
license: apache-2.0
Aura-llama
there was an error in the first model config it has been fixed in the new repo TheSkullery/Aura-llama-v1
Now that the cute anime girl has your attention.
Aura-llama is using the methodology presented by SOLAR for scaling LLMs called depth up-scaling (DUS), which encompasses architectural modifications with continued pretraining. Using the solar paper as a base, I integrated Llama-3 weights into the upscaled layers, and In the future plan to continue training the model.
Aura-llama is a merge of the following models to create a base model to work from:
Merged Evals (Has Not Been Finetuned, waiting for tests):
Aura-llama
- Avg: ?
- ARC: ?
- HellaSwag: ?
- MMLU: ?
- T-QA: ?
- Winogrande: ?
- GSM8K: ?
🧩 Configuration
slices:
- sources:
- model: meta-llama/Meta-Llama-3-8B-Instruct
layer_range: [0, 23]
- sources:
- model: meta-llama/Meta-Llama-3-8B-Instruct
layer_range: [7, 31]
merge_method: passthrough
dtype: bfloat16