|
--- |
|
language: |
|
- en |
|
license: other |
|
library_name: transformers |
|
tags: |
|
- mergekit |
|
- merge |
|
base_model: |
|
- unsloth/Mistral-Small-Instruct-2409 |
|
- Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small |
|
- anthracite-org/magnum-v4-22b |
|
- ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1 |
|
- spow12/ChatWaifu_v2.0_22B |
|
- rAIfle/Acolyte-22B |
|
- Envoid/Mistral-Small-NovusKyver |
|
- InferenceIllusionist/SorcererLM-22B |
|
- allura-org/MS-Meadowlark-22B |
|
- crestf411/MS-sunfall-v0.7.0 |
|
model-index: |
|
- name: MS-Schisandra-22B-v0.2 |
|
results: |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: IFEval (0-Shot) |
|
type: HuggingFaceH4/ifeval |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: inst_level_strict_acc and prompt_level_strict_acc |
|
value: 63.83 |
|
name: strict accuracy |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: BBH (3-Shot) |
|
type: BBH |
|
args: |
|
num_few_shot: 3 |
|
metrics: |
|
- type: acc_norm |
|
value: 40.61 |
|
name: normalized accuracy |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MATH Lvl 5 (4-Shot) |
|
type: hendrycks/competition_math |
|
args: |
|
num_few_shot: 4 |
|
metrics: |
|
- type: exact_match |
|
value: 19.94 |
|
name: exact match |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: GPQA (0-shot) |
|
type: Idavidrein/gpqa |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 11.41 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MuSR (0-shot) |
|
type: TAUR-Lab/MuSR |
|
args: |
|
num_few_shot: 0 |
|
metrics: |
|
- type: acc_norm |
|
value: 10.67 |
|
name: acc_norm |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
- task: |
|
type: text-generation |
|
name: Text Generation |
|
dataset: |
|
name: MMLU-PRO (5-shot) |
|
type: TIGER-Lab/MMLU-Pro |
|
config: main |
|
split: test |
|
args: |
|
num_few_shot: 5 |
|
metrics: |
|
- type: acc |
|
value: 34.85 |
|
name: accuracy |
|
source: |
|
url: https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard?query=Nohobby/MS-Schisandra-22B-v0.2 |
|
name: Open LLM Leaderboard |
|
--- |
|
*** |
|
## Schisandra |
|
|
|
Many thanks to the authors of the models used! |
|
|
|
[RPMax v1.1](https://huggingface.co./ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1) | [Pantheon-RP](https://huggingface.co./Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small) | [UnslopSmall-v1](https://huggingface.co./TheDrummer/UnslopSmall-22B-v1) | [Magnum V4](https://huggingface.co./anthracite-org/magnum-v4-22b) | [ChatWaifu v2.0](https://huggingface.co./spow12/ChatWaifu_v2.0_22B) | [SorcererLM](https://huggingface.co./InferenceIllusionist/SorcererLM-22B) | [Acolyte](https://huggingface.co./rAIfle/Acolyte-22B) | [NovusKyver](https://huggingface.co./Envoid/Mistral-Small-NovusKyver) | [Meadowlark](https://huggingface.co./allura-org/MS-Meadowlark-22B) | [Sunfall](https://huggingface.co./crestf411/MS-sunfall-v0.7.0) |
|
*** |
|
|
|
### Overview |
|
|
|
Main uses: RP, Storywriting |
|
|
|
Prompt format: Mistral-V3 |
|
|
|
An intelligent model that is attentive to details and has a low-slop writing style. This time with a stable tokenizer. |
|
|
|
Oh, and it now contains 10 finetunes! Not sure if some of them actually contribute to the output, but it's nice to see the numbers growing. |
|
|
|
*** |
|
|
|
### Quants |
|
|
|
GGUF: [Static](https://huggingface.co./mradermacher/MS-Schisandra-22B-v0.2-GGUF) | [Imatrix](https://huggingface.co./mradermacher/MS-Schisandra-22B-v0.2-i1-GGUF) |
|
|
|
exl2: [4.65bpw](https://huggingface.co./waldie/MS-Schisandra-22B-v0.2-4.65bpw-h6-exl2) [5.5bpw](https://huggingface.co./waldie/MS-Schisandra-22B-v0.2-5.5bpw-h6-exl2) [6.5bpw](https://huggingface.co./waldie/MS-Schisandra-22B-v0.2-6.5bpw-h6-exl2) |
|
|
|
*** |
|
|
|
### Settings |
|
|
|
My SillyTavern preset: https://huggingface.co./Nohobby/MS-Schisandra-22B-v0.2/resolve/main/ST-formatting-Schisandra.json |
|
|
|
*** |
|
|
|
## Merge Details |
|
### Merging steps |
|
|
|
## Step1 |
|
(Config partially taken from [here](https://huggingface.co./Casual-Autopsy/L3-Super-Nova-RP-8B)) |
|
|
|
```yaml |
|
base_model: spow12/ChatWaifu_v2.0_22B |
|
parameters: |
|
int8_mask: true |
|
rescale: true |
|
normalize: false |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
merge_method: della |
|
models: |
|
- model: Envoid/Mistral-Small-NovusKyver |
|
parameters: |
|
density: [0.35, 0.65, 0.5, 0.65, 0.35] |
|
epsilon: [0.1, 0.1, 0.25, 0.1, 0.1] |
|
lambda: 0.85 |
|
weight: [-0.01891, 0.01554, -0.01325, 0.01791, -0.01458] |
|
- model: rAIfle/Acolyte-22B |
|
parameters: |
|
density: [0.6, 0.4, 0.5, 0.4, 0.6] |
|
epsilon: [0.1, 0.1, 0.25, 0.1, 0.1] |
|
lambda: 0.85 |
|
weight: [0.01847, -0.01468, 0.01503, -0.01822, 0.01459] |
|
``` |
|
|
|
## Step2 |
|
(Config partially taken from [here](https://huggingface.co./Casual-Autopsy/L3-Super-Nova-RP-8B)) |
|
|
|
```yaml |
|
base_model: InferenceIllusionist/SorcererLM-22B |
|
parameters: |
|
int8_mask: true |
|
rescale: true |
|
normalize: false |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
merge_method: della |
|
models: |
|
- model: crestf411/MS-sunfall-v0.7.0 |
|
parameters: |
|
density: [0.35, 0.65, 0.5, 0.65, 0.35] |
|
epsilon: [0.1, 0.1, 0.25, 0.1, 0.1] |
|
lambda: 0.85 |
|
weight: [-0.01891, 0.01554, -0.01325, 0.01791, -0.01458] |
|
- model: anthracite-org/magnum-v4-22b |
|
parameters: |
|
density: [0.6, 0.4, 0.5, 0.4, 0.6] |
|
epsilon: [0.1, 0.1, 0.25, 0.1, 0.1] |
|
lambda: 0.85 |
|
weight: [0.01847, -0.01468, 0.01503, -0.01822, 0.01459] |
|
``` |
|
|
|
## SchisandraVA2 |
|
(Config taken from [here](https://huggingface.co./HiroseKoichi/Llama-3-8B-Stroganoff-4.0)) |
|
|
|
```yaml |
|
merge_method: della_linear |
|
dtype: bfloat16 |
|
parameters: |
|
normalize: true |
|
int8_mask: true |
|
tokenizer_source: base |
|
base_model: TheDrummer/UnslopSmall-22B-v1 |
|
models: |
|
- model: ArliAI/Mistral-Small-22B-ArliAI-RPMax-v1.1 |
|
parameters: |
|
density: 0.55 |
|
weight: 1 |
|
- model: Gryphe/Pantheon-RP-Pure-1.6.2-22b-Small |
|
parameters: |
|
density: 0.55 |
|
weight: 1 |
|
- model: Step1 |
|
parameters: |
|
density: 0.55 |
|
weight: 1 |
|
- model: allura-org/MS-Meadowlark-22B |
|
parameters: |
|
density: 0.55 |
|
weight: 1 |
|
- model: Step2 |
|
parameters: |
|
density: 0.55 |
|
weight: 1 |
|
``` |
|
|
|
## Schisandra-v0.2 |
|
|
|
```yaml |
|
dtype: bfloat16 |
|
tokenizer_source: base |
|
merge_method: della_linear |
|
parameters: |
|
density: 0.5 |
|
base_model: SchisandraVA2 |
|
models: |
|
- model: unsloth/Mistral-Small-Instruct-2409 |
|
parameters: |
|
weight: |
|
- filter: v_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: o_proj |
|
value: [1, 0, 1, 0, 0, 0, 0, 0, 1, 1, 1] |
|
- filter: up_proj |
|
value: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] |
|
- filter: gate_proj |
|
value: [0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0] |
|
- filter: down_proj |
|
value: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
|
- value: 0 |
|
- model: SchisandraVA2 |
|
parameters: |
|
weight: |
|
- filter: v_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: o_proj |
|
value: [0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0] |
|
- filter: up_proj |
|
value: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] |
|
- filter: gate_proj |
|
value: [1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1] |
|
- filter: down_proj |
|
value: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] |
|
- value: 1 |
|
``` |
|
# [Open LLM Leaderboard Evaluation Results](https://huggingface.co./spaces/open-llm-leaderboard/open_llm_leaderboard) |
|
Detailed results can be found [here](https://huggingface.co./datasets/open-llm-leaderboard/details_Nohobby__MS-Schisandra-22B-v0.2) |
|
|
|
| Metric |Value| |
|
|-------------------|----:| |
|
|Avg. |30.22| |
|
|IFEval (0-Shot) |63.83| |
|
|BBH (3-Shot) |40.61| |
|
|MATH Lvl 5 (4-Shot)|19.94| |
|
|GPQA (0-shot) |11.41| |
|
|MuSR (0-shot) |10.67| |
|
|MMLU-PRO (5-shot) |34.85| |
|
|
|
|
