Hugging Face's logo

mpasila
/
Finnish-Alpaca-Tiny-V2-7B-exl2-4bpw

Text Generation
Transformers
Finnish
llama
text-generation-inference
unsloth
trl
sft
Inference Endpoints
Model card Files Community
1
mpasila commited on
Commit
07d0a42
•
1 Parent(s): e70fa96

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +59 -0
README.md CHANGED
@@ -23,6 +23,8 @@ LoRA trained in 4-bit with 2k context using [LumiOpen/Viking-7B](https://hugging
23
 
24
  Dataset used is [mpasila/Finnish-Alpaca-Tiny](https://huggingface.co/datasets/mpasila/Finnish-Alpaca-Tiny).
25
 
 
 
26
  ### Prompt format: Alpaca
27
  It uses Alpaca format but with a translated instruction at the start:
28
  ```
@@ -33,6 +35,63 @@ It uses Alpaca format but with a translated instruction at the start:
33
 
34
  ```
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
  # Uploaded model
37
 
38
  - **Developed by:** mpasila
 
23
 
24
  Dataset used is [mpasila/Finnish-Alpaca-Tiny](https://huggingface.co/datasets/mpasila/Finnish-Alpaca-Tiny).
25
 
26
+ It works relatively well for question and answering. I will make a bigger dataset for the next fine-tune.
27
+
28
  ### Prompt format: Alpaca
29
  It uses Alpaca format but with a translated instruction at the start:
30
  ```
 
35
 
36
  ```
37
 
38
+ ## Evaluation
39
+
40
+ | Model | Size | Type | FIN-bench (score) |
41
+ |-------|------|------|-------|
42
+ | **mpasila/Finnish-Alpaca-Tiny-V2-7B** | 7B | Instruct | **0.4654** |
43
+ | [mpasila/Alpacazord-Viking-7B](https://huggingface.co/mpasila/Alpacazord-Viking-7B) | 7B | Instruct | 0.4123 |
44
+ | [mpasila/NordicAlpaca-Finnish-V1-7B](https://huggingface.co/mpasila/NordicAlpaca-Finnish-V1-7B) | 7B | Instruct | 0.3891 |
45
+ | [mpasila/Finnish-Viking-Alpaca-V1-7B](https://huggingface.co/mpasila/Finnish-Viking-Alpaca-V1-7B) | 7B | Instruct | 0.3943 |
46
+ | [Finnish-NLP/llama-7b-finnish-instruct-v0.1](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.1) | 7B | Instruct | 0.4365 |
47
+ | [Finnish-NLP/llama-7b-finnish-instruct-v0.2](https://huggingface.co/Finnish-NLP/llama-7b-finnish-instruct-v0.2) | 7B | Instruct | 0.3993 |
48
+ | [Finnish-NLP/llama-7b-finnish](https://huggingface.co/Finnish-NLP/llama-7b-finnish) | 7B | Base | 0.2350 |
49
+ | [LumiOpen/Viking-7B (1000B)](https://huggingface.co/LumiOpen/Viking-7B) | 7B | Base | 0.3721 |
50
+ | [HPLT/gpt-7b-nordic-prerelease](https://huggingface.co/HPLT/gpt-7b-nordic-prerelease) | 7B | Base | 0.3169 |
51
+
52
+ [Source](https://docs.google.com/spreadsheets/d/1rqJb9dQVihg-Z1_Ras1L_-wuzPg9xNzpdmM2x5HueeY/edit?usp=sharing)
53
+
54
+ #### FIN-bench scores:
55
+
56
+ | Task |Version| Metric |Value | |Stderr|
57
+ |------------------------------------------------|------:|---------------------|-----:|---|-----:|
58
+ |bigbench_analogies | 0|multiple_choice_grade|0.6385|± |0.0423|
59
+ |bigbench_arithmetic_1_digit_addition | 0|multiple_choice_grade|0.7200|± |0.0451|
60
+ |bigbench_arithmetic_1_digit_division | 0|multiple_choice_grade|0.7391|± |0.0936|
61
+ |bigbench_arithmetic_1_digit_multiplication | 0|multiple_choice_grade|0.4800|± |0.0502|
62
+ |bigbench_arithmetic_1_digit_subtraction | 0|multiple_choice_grade|0.6300|± |0.0485|
63
+ |bigbench_arithmetic_2_digit_addition | 0|multiple_choice_grade|0.4000|± |0.0492|
64
+ |bigbench_arithmetic_2_digit_division | 0|multiple_choice_grade|0.5000|± |0.0503|
65
+ |bigbench_arithmetic_2_digit_multiplication | 0|multiple_choice_grade|0.2800|± |0.0451|
66
+ |bigbench_arithmetic_2_digit_subtraction | 0|multiple_choice_grade|0.4300|± |0.0498|
67
+ |bigbench_arithmetic_3_digit_addition | 0|multiple_choice_grade|0.5800|± |0.0496|
68
+ |bigbench_arithmetic_3_digit_division | 0|multiple_choice_grade|0.3100|± |0.0465|
69
+ |bigbench_arithmetic_3_digit_multiplication | 0|multiple_choice_grade|0.2900|± |0.0456|
70
+ |bigbench_arithmetic_3_digit_subtraction | 0|multiple_choice_grade|0.5100|± |0.0502|
71
+ |bigbench_arithmetic_4_digit_addition | 0|multiple_choice_grade|0.5300|± |0.0502|
72
+ |bigbench_arithmetic_4_digit_division | 0|multiple_choice_grade|0.3900|± |0.0490|
73
+ |bigbench_arithmetic_4_digit_multiplication | 0|multiple_choice_grade|0.3100|± |0.0465|
74
+ |bigbench_arithmetic_4_digit_subtraction | 0|multiple_choice_grade|0.6200|± |0.0488|
75
+ |bigbench_arithmetic_5_digit_addition | 0|multiple_choice_grade|0.6500|± |0.0479|
76
+ |bigbench_arithmetic_5_digit_division | 0|multiple_choice_grade|0.3200|± |0.0469|
77
+ |bigbench_arithmetic_5_digit_multiplication | 0|multiple_choice_grade|0.3000|± |0.0461|
78
+ |bigbench_arithmetic_5_digit_subtraction | 0|multiple_choice_grade|0.6400|± |0.0482|
79
+ |bigbench_cause_and_effect_one_sentence | 0|multiple_choice_grade|0.5686|± |0.0700|
80
+ |bigbench_cause_and_effect_one_sentence_no_prompt| 0|multiple_choice_grade|0.6471|± |0.0676|
81
+ |bigbench_cause_and_effect_two_sentences | 0|multiple_choice_grade|0.4314|± |0.0700|
82
+ |bigbench_emotions | 0|multiple_choice_grade|0.2250|± |0.0331|
83
+ |bigbench_empirical_judgments | 0|multiple_choice_grade|0.2525|± |0.0439|
84
+ |bigbench_general_knowledge | 0|multiple_choice_grade|0.3429|± |0.0571|
85
+ |bigbench_hhh_alignment_harmless | 0|multiple_choice_grade|0.3793|± |0.0643|
86
+ |bigbench_hhh_alignment_helpful | 0|multiple_choice_grade|0.3390|± |0.0622|
87
+ |bigbench_hhh_alignment_honest | 0|multiple_choice_grade|0.3729|± |0.0635|
88
+ |bigbench_hhh_alignment_other | 0|multiple_choice_grade|0.5349|± |0.0770|
89
+ |bigbench_intent_recognition | 0|multiple_choice_grade|0.2153|± |0.0156|
90
+ |bigbench_misconceptions | 0|multiple_choice_grade|0.5224|± |0.0433|
91
+ |bigbench_paraphrase | 0|multiple_choice_grade|0.4750|± |0.0354|
92
+ |bigbench_sentence_ambiguity | 0|multiple_choice_grade|0.4833|± |0.0651|
93
+ |bigbench_similarities_abstraction | 0|multiple_choice_grade|0.6974|± |0.0530|
94
+
95
  # Uploaded model
96
 
97
  - **Developed by:** mpasila