|
--- |
|
library_name: transformers |
|
tags: [] |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
|
|
|
|
**DistMerge_Llama-3.1-8B-Instruct** is a customized variant of |
|
the **VAGOsolutions/Llama-3.1-SauerkrautLM-8B-Instruct**, which is itself a spectrum fine-tuned version of |
|
**Llama-2.1-8B-Instruct**. This customization is achieved by learning the distribution of all normalization |
|
layer weights from both the original Llama model and its fine-tuned counterpart. A layer-conditional diffusion |
|
based weights generation model that enables sampling for performance enhancement by leveraging the |
|
learned distributions to optimize the merging process is used to generate the normalization layer of |
|
bedio/DistMerge_Llama-3.1-8B-Instruct |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
We trained a diffusion model to learn the distribution of the normalization layers to enable generation weights |
|
that improve the performance. |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. |
|
This model card has been automatically generated. |
|
|
|
- **Developed by:** DeepAuto.ai |
|
- **Shared by [optional]:** the model was shared by from deepauto.ai |
|
- **Model type:** DistMerge_Llama-3.1-8B-Instruct is a customized model by generating diverse weights for |
|
Llama-3.1-SauerkrautLM-8b-Instruct model |
|
- **Language(s) (NLP):** the base model was fine-tuned on German, English. We only use the one provided on Huggingface |
|
- **License:** llama3.1 |
|
Contact: DeepAuto.ai |
|
|
|
|
|
|
|
### Training Procedure |
|
|
|
We employed a latent diffusion process on pretrained model weights, unlocking the ability to generate diverse, |
|
previously unseen neural networks. Remarkably, even within the constraints of one-shot learning, our approach consistently |
|
produces a wide range of weight variations, each offering distinct performance characteristics. These generated weights |
|
not only open opportunities for weight averaging and model merging but also have the potential to significantly enhance |
|
model performance. Moreover, they enable the creation of task-specific weights, tailored to optimize performance for |
|
specialized applications. |
|
|
|
#### Preprocessing [optional] |
|
|
|
- We selected a set of layers and combined their pretrained weights, then trained a Variational Autoencoder (VAE) to encode these weights into the layer dimension. |
|
- We conditionally trained a diffusion model on this set of weights, allowing individual sampling of layer-specific weights. |
|
- All selected layers were encoded into a 1024-dimensional space. This model exclusively contained the sampled weights for layer normalization." |
|
|
|
|
|
#### Training Hyperparameters |
|
|
|
- **Training regime:** The pretrained weights used for training are orriginally in bflot16. |
|
#### Speeds, Sizes, Times [optional] |
|
|
|
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. --> |
|
|
|
[More Information Needed] |
|
|
|
## Evaluation |
|
We evaluate the reconstrution and sampling performance on Winogrande task using lm_eval tools |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
<!-- This should link to a Dataset Card if possible. --> |
|
|
|
[More Information Needed] |
|
|
|
The primary objective of this weight generation process was to demonstrate that by learning only the distribution |
|
of few layers weights9normlaization layers in this case) in an 8-billion-parameter model, it is possible to significantly enhance the |
|
model's capabilities. Notably, this is achieved using a fraction of the computational resources and without the |
|
need for fine-tuning, showcasing the efficiency and potential of this approach. |
