File size: 4,667 Bytes
118162a b428b99 118162a b428b99 118162a b428b99 118162a b428b99 118162a c9fca58 118162a b428b99 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
license: llama3.1
datasets:
- survivi/Llama-3-SynE-Dataset
- hfl/stem_zh_instruction
- llamafactory/alpaca_zh
- llamafactory/alpaca_gpt4_zh
- hfl/ruozhiba_gpt4
- codingsteven/Llama-3-8B-chat
language:
- zh
metrics:
- accuracy
base_model:
- meta-llama/Llama-3.1-8B
model-index:
- name: Control-LLM-Llama3.1-8B-SynE-Full-Parameter-Tuning
results:
- task:
type: pretraining-evaluation
dataset:
type: mixed
name: Pretraining Evaluation Dataset
metrics:
- name: exact_match,strict-match (meta_pretrain)
type: exact_match
value: 0.45445720757159036
stderr: 0.0035036029889520047
verified: false
- name: exact_match,strict-match (meta_bbh_3shot_cot_pretrain)
type: exact_match
value: 0.6482875134387959
stderr: 0.005918167158231359
verified: false
- name: acc,none (meta_mmlu_5shot_pretrain)
type: accuracy
value: 0.649480131035465
stderr: 0.004026616190778244
verified: false
- name: exact_match,strict-match (meta_mmlu_pro_5shot_pretrain)
type: exact_match
value: 0.34956781914893614
stderr: 0.004347262544061378
verified: false
- task:
type: chinese-evaluation
dataset:
type: mixed
name: Chinese Evaluation Dataset
metrics:
- name: acc,none (ceval-valid)
type: accuracy
value: 0.5898959881129272
stderr: 0.012699457390113113
verified: false
- name: exact_match,strict-match (ceval-valid-pretrain-cot_zh)
type: exact_match
value: 0.40193164933135217
stderr: 0.01265090064840271
verified: false
- name: acc,none (cmmlu)
type: accuracy
value: 0.6018822310481782
stderr: 0.004420298073040671
verified: false
- name: exact_match,strict-match (cmmlu_pretrain_cot_zh)
type: exact_match
value: 0.4425833189431877
stderr: 0.004506238417180843
verified: false
pipeline_tag: text-generation
library_name: transformers
---
# Control-LLM-Llama3.1-8B-SynE-Full-Parameter-Tuning
This is a fine-tuned model of Llama-3.1-8B for muliligual-Chinese tasks on SynE dataset.
## Linked Paper
This model is associated with the paper: [Control LLM: Controlled Evolution for Intelligence Retention in LLM](https://huggingface.co./papers/2501.10979).
## Linked Open Source code - training, eval and benchmark
This model is associated with the github: [Control-LLM](https://github.com/linkedin/ControlLLM).
## Evaluation Results
Here is an overview of the evaluation results and findings:
### Benchmark Results Table
The table below summarizes evaluation results across Chinese tasks and original capabilities.
| **Model** | **CEval** | **CEvalC** | **CMMLU** | **CMMLUC** | **C-Avg** | **BBH** | **MLU** | **MLUP** | **O-Avg** | **Overall** |
|--------------------|-----------|------------|-----------|------------|-----------|---------|---------|----------|-----------|-------------|
| Llama3.1-8B | 48.3 | 12.8 | 51.1 | 14.1 | 13.9 | 65.2 | 65.4 | 35.5 | 45.9 | 29.9 |
| Llama-3-SynE | 57.7 | 22.3 | 57.1 | 22.8 | 22.8 | 61.9 | 64.0 | 32.6 | 42.9 | 32.9 |
| **Full Param Tune**| 59.0 | 40.2 | **60.2** | 44.3 | 43.8 | 64.8 | 64.9 | 35.0 | 45.4 | 44.6 |
| Stack Expansion | 56.0 | 32.7 | 55.2 | 33.4 | 33.3 | 62.3 | 65.6 | 35.3 | 44.8 | 39.1 |
| Concat-Lerp* | 57.1 | 34.8 | 57.0 | 37.4 | 37.1 | 64.4 | 64.6 | 35.8 | 45.9 | 41.5 |
| **Hybrid Expansion**| **58.9** | 44.7 | 57.9 | 44.3 | 44.4 | 65.1 | **65.7**| 36.9 | 46.8 | 45.6 |
| **Control LLM*** | 57.0 | **44.7** | 56.0 | **44.9** | **44.8** | **68.2**| 65.6 | **37.9** | **48.5** | **46.7** |
---
### Explanation:
- **CEval**: Chinese Evaluation
- **CEvalC**: Chinese Evaluation (CoT - Chain of Thought)
- **CMMLU**: Chinese MMLU
- **CMMLUC**: Chinese MMLU (CoT)
- **C-Avg**: Chinese - Size Weighted Average across CEval, CEvalC, CMMLU, and CMMLUC
- **BBH**: BigBench Hard
- **MLU**: MMLU (Massive Multitask Language Understanding)
- **MLUP**: MMLU Pro
- **O-Avg**: Original Capability - Size Weighted Average across BBH, MLU, and MLUP
- **Overall**: Combined average across all tasks
### Full Parameter Tuning on Chinese-SynE
The following plot illustrates the Catastrophic Forgetting of full parameter tuning in terms of hidden states alignment drift.
 |