Crystalcareai commited on
Commit
cc41cf1
1 Parent(s): 14eb1bb

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +60 -47
README.md CHANGED
@@ -1,55 +1,68 @@
1
  ---
2
  base_model:
3
- - chargoddard/qwamma-14b-merge-v1
4
- - arcee-train/Qwen2.5-14B-Instruct_arcee-qwen2-14B-v0.2
5
  - Qwen/Qwen2.5-14B
6
  library_name: transformers
7
  tags:
8
  - mergekit
9
  - merge
10
-
11
  ---
12
- # qwamma-14b-merge-v9
13
-
14
- This is a merge of pre-trained language models created using [mergekit](https://github.com/cg123/mergekit).
15
-
16
- ## Merge Details
17
- ### Merge Method
18
-
19
- This model was merged using the [TIES](https://arxiv.org/abs/2306.01708) merge method using [Qwen/Qwen2.5-14B](https://huggingface.co/Qwen/Qwen2.5-14B) as a base.
20
-
21
- ### Models Merged
22
-
23
- The following models were included in the merge:
24
- * [chargoddard/qwamma-14b-merge-v1](https://huggingface.co/chargoddard/qwamma-14b-merge-v1)
25
- * [arcee-train/Qwen2.5-14B-Instruct_arcee-qwen2-14B-v0.2](https://huggingface.co/arcee-train/Qwen2.5-14B-Instruct_arcee-qwen2-14B-v0.2)
26
-
27
- ### Configuration
28
-
29
- The following YAML configuration was used to produce this model:
30
-
31
- ```yaml
32
- merge_method: ties
33
- base_model: Qwen/Qwen2.5-14B
34
- models:
35
- - model: chargoddard/qwamma-14b-merge-v1
36
- parameters:
37
- density: 1.0
38
- weight: 1.0
39
- - model: arcee-train/Qwen2.5-14B-Instruct_arcee-qwen2-14B-v0.2
40
- parameters:
41
- density: 0.66
42
- weight:
43
- - filter: mlp
44
- value: [0, 0.3, 0.6, 0.1]
45
- - filter: self_attn
46
- value: [0, 0, 0.2, 0.1]
47
- - value: 0.1
48
- parameters:
49
- normalize: false
50
- int8_mask: true
51
- dtype: float32
52
- out_dtype: bfloat16
53
-
54
-
55
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  base_model:
 
 
3
  - Qwen/Qwen2.5-14B
4
  library_name: transformers
5
  tags:
6
  - mergekit
7
  - merge
8
+ license: apache-2.0
9
  ---
10
+
11
+ # Arcee-SuperNova-Medius
12
+
13
+ Arcee-SuperNova-Medius is a 14B parameter language model developed by Arcee.ai, built on the Qwen2.5-14B-Instruct architecture. This unique model is the result of a cross-architecture distillation pipeline, combining knowledge from both the Qwen2.5-72B-Instruct model and the Llama-3.1-405B-Instruct model. By leveraging the strengths of these two distinct architectures, SuperNova-Medius achieves high-quality instruction-following and complex reasoning capabilities in a mid-sized, resource-efficient form.
14
+
15
+ SuperNova-Medius is designed to excel in a variety of business use cases, including customer support, content creation, and technical assistance, while maintaining compatibility with smaller hardware configurations. It’s an ideal solution for organizations looking for advanced capabilities without the high resource requirements of larger models like our SuperNova-70B.
16
+
17
+ ## Distillation Overview
18
+
19
+ The development of SuperNova-Medius involved a sophisticated multi-teacher, cross-architecture distillation process, with the following key steps:
20
+
21
+ 1. **Logit Distillation from Llama-3.1-405B-Instruct**:
22
+ - We distilled the logits of Llama-3.1-405B-Instruct to Qwen2.5-14B using KL-divergence as the loss function. This allowed us to capture the nuanced distribution of Llama's outputs while adapting them to Qwen's architecture.
23
+
24
+ 2. **Logit and Hidden State Distillation from Qwen2.5-72B-Instruct**:
25
+ - Further distillation was performed using a combination of logit and hidden state distillation from Qwen2.5-72B-Instruct to ensure that SuperNova-Medius inherited the strong instruction-following capabilities and domain-specific knowledge of Qwen2.5.
26
+
27
+ 3. **Cross-Architecture Vocabulary Alignment**:
28
+ - Using `mergekit-tokensurgeon`, we aligned the vocabularies and hidden states of both teacher models, allowing for seamless integration of knowledge across the different architectures. This enabled SuperNova-Medius to effectively combine the strengths of both models.
29
+
30
+ 4. **Final Fusion and Fine-Tuning**:
31
+ - After aligning the vocabularies, a final fusion and fine-tuning step was conducted, using a specialized dataset from [EvolKit](https://github.com/arcee-ai/EvolKit) to ensure that SuperNova-Medius maintained coherence, fluency, and context understanding across a broad range of tasks.
32
+
33
+ ## Performance Evaluation
34
+
35
+ Below are the benchmark results of SuperNova-Medius compared to similar models in its class:
36
+
37
+ | Model | Average | IFEval | BBH | GPQA | MMLU Pro | MuSR | Math Level 5 |
38
+ | --- | --- | --- | --- | --- | --- | --- | --- |
39
+ | Mistral-Small 2409 | 0.423 | 0.628 | 0.581 | 0.333 | 0.410 | 0.406 | 0.181 |
40
+ | Supernova-Lite | 0.427 | 0.786 | 0.511 | 0.306 | 0.388 | 0.415 | 0.155 |
41
+ | Qwen2.5-14B-Instruct | 0.450 | 0.827 | 0.623 | 0.358 | 0.490 | 0.403 | 0.000 |
42
+ | Supernova-Medius | **0.480** | **0.832** | **0.631** | **0.359** | **0.502** | **0.402** | **0.152** |
43
+
44
+ SuperNova-Medius performs exceptionally well in instruction-following (IFEval) and complex reasoning tasks (BBH), demonstrating its capability to handle a variety of real-world scenarios. It outperforms Qwen2.5-14B and SuperNova-Lite in multiple benchmarks, making it a powerful yet efficient choice for high-quality generative AI applications.
45
+
46
+ ## Model Use Cases
47
+
48
+ Arcee-SuperNova-Medius is suitable for a range of applications, including:
49
+
50
+ - **Customer Support**: With its robust instruction-following and dialogue management capabilities, SuperNova-Medius can handle complex customer interactions, reducing the need for human intervention.
51
+ - **Content Creation**: The model’s advanced language understanding and generation abilities make it ideal for creating high-quality, coherent content across diverse domains.
52
+ - **Technical Assistance**: SuperNova-Medius has a deep reservoir of technical knowledge, making it an excellent assistant for programming, technical documentation, and other expert-level content creation.
53
+
54
+ ## Deployment Options
55
+
56
+ SuperNova-Medius is available for use under the Apache-2.0 license. For those who need even higher performance, the full-size 70B SuperNova model can be accessed via an Arcee-hosted API or for local deployment. To learn more or explore deployment options, please reach out to [[email protected]](mailto:[email protected]).
57
+
58
+ ## Technical Specifications
59
+
60
+ - **Model Architecture**: Qwen2.5-14B-Instruct
61
+ - **Distillation Sources**: Qwen2.5-72B-Instruct, Llama-3.1-405B-Instruct
62
+ - **Parameter Count**: 14 billion
63
+ - **Training Dataset**: Custom instruction dataset generated with [EvolKit](https://github.com/arcee-ai/EvolKit)
64
+ - **Distillation Technique**: Multi-architecture logit and hidden state distillation with cross-architecture vocabulary alignment.
65
+
66
+ ## Summary
67
+
68
+ Arcee-SuperNova-Medius provides a unique balance of power, efficiency, and versatility. By distilling knowledge from two top-performing teacher models into a single 14B parameter model, SuperNova-Medius achieves results that rival larger models while maintaining a compact size ideal for practical deployment. Whether for customer support, content creation, or technical assistance, SuperNova-Medius is the perfect choice for organizations looking to leverage advanced language model capabilities in a cost-effective and accessible form.