paraschopra/llama-31-8b-base-all-24k-correct_MERGED

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the TIES merge method using meta-llama/meta-llama-3.1-8b as a base.

Models Merged

The following models were included in the merge:

• paraschopra/llama-31-8b-base-120k_minp_plus_train_500_ADAPTOR_MERGED
• meta-llama/meta-llama-3.1-8b-instruct

Configuration

The following YAML configuration was used to produce this model:

   models:
     - model: paraschopra/llama-31-8b-base-120k_minp_plus_train_500_ADAPTOR_MERGED
       parameters:
         weight: 1
         density: 1
     - model: meta-llama/meta-llama-3.1-8b-instruct
       parameters:
         weight: 1
         density: 1
   merge_method: ties
   base_model: meta-llama/meta-llama-3.1-8b
   parameters:
     weight: 1
     density: 1
     normalize: true
     int8_mask: true
   tokenizer_source: paraschopra/llama-31-8b-base-120k_minp_plus_train_500_ADAPTOR_MERGED
   dtype: bfloat16