Hugging Face's logo

manishiitg
/
open-aditi-hi-v2

Text Generation
Transformers
Safetensors
Hindi
English
mistral
conversational
Eval Results
text-generation-inference
Inference Endpoints
Model card Files Community
1
manishiitg commited on
Commit
9006a83
•
1 Parent(s): e544983

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +16 -19
README.md CHANGED
@@ -9,36 +9,32 @@ base_model: teknium/OpenHermes-2.5
9
 
10
  Model trained on Hindi and English data.
11
 
12
- # IndicEval
13
 
14
 
15
- | Model | indicxparaphrase | hellaswag-indic | boolq-hi | xlsum-hi | implicit_hate | indicsentiment | indic-arc-challenge | indicqa | flores | truthfulqa-hi | indicwikibio | indicheadline | indic-arc-easy | mmlu_hi |
 
 
16
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
17
- | open-aditi-hi-v2-awq | 0.6818 | 0.2420 | 0.8835 | 0.4347 | 40.6529 | 0.9539 | 0.4096 | 0.2178 | 42.4015 | 0.7555 | 0.4788 | 0.4525 | 0.4423 | 0.3158 |
18
- | open-aditi-hi-v2 | 0.6838 | 0.2404 | 0.8541 | 0.4213 | 11.5021 | 0.9729 | 0.4462 | 0.0795 | 43.6822 | 0.6934 | 0.4846 | 0.4565 | 0.4979 | 0.3253 |
19
- | open-aditi-hi-v1-awq | 0.6658 | 0.1673 | 0.5193 | 0.4026 | 37.0739 | 0.8637 | 0.3456 | 0.1144 | 39.0395 | 0.3144 | 0.3411 | 0.4244 | 0.3784 | 0.1687 |
20
- | OpenHermes-2.5-Mistral-7B | 0.8766 | 0.2485 | 0.5979 | 0.1774 | 0.2068 | 0.9048 | 0.3396 | 0.2721 | 30.3465 | 0.3234 | 0.3332 | 0.1996 | 0.3523 | 0.2769 |
21
- | OpenHermes-2.5-Mistral-7B-AWQ | 0.8536 | 0.2479 | 0.5272 | 0.1894 | 6.0594 | 0.9218 | 0.3157 | 0.3116 | 29.3681 | 0.3428 | 0.3067 | 0.2062 | 0.3291 | 0.2750 |
22
- | open-aditi-hi-v1 | 0.5939 | 0.0848 | 0.3758 | 0.4212 | 8.6105 | 0.8798 | 0.3558 | 0.1306 | 40.2376 | 0.4230 | 0.4104 | 0.4248 | 0.3889 | 0.1398 |
23
- | Airavata | 0.3277 | 0.0254 | 0.0373 | 0.4650 | 0.0663 | 0.8437 | 0.0836 | 0.1008 | 58.0555 | 0.2122 | 0.0637 | 0.4346 | 0.1128 | 0.1336 |
24
 
25
  #### Language En
26
 
27
- | Model | mmlu | truthfulqa | xlsum | arc-challenge | hellaswag | boolq | arc-easy-exact |
28
  | --- | --- | --- | --- | --- | --- | --- | --- |
29
- | OpenHermes-2.5-Mistral-7B | 0.5991 | 0.2081 | 0.4328 | 0.7790 | 0.7999 | 0.4061 | 0.8687 |
30
- | OpenHermes-2.5-Mistral-7B-AWQ | 0.5816 | 0.1897 | 0.4317 | 0.7611 | 0.7826 | 0.4199 | 0.8569 |
31
- | open-aditi-hi-v2-awq | 0.5339 | 0.3905 | 0.4307 | 0.7116 | 0.4355 | 0.4401 | 0.8266 |
32
- | open-aditi-hi-v2 | 0.5544 | 0.2999 | 0.4349 | 0.7235 | 0.4738 | 0.3982 | 0.8388 |
33
- | open-aditi-hi-v1 | 0.2597 | 0.3317 | 0.4288 | 0.6271 | 0.3509 | 0.0434 | 0.7588 |
34
- | open-aditi-hi-v1-awq | 0.3149 | 0.2950 | 0.4296 | 0.6024 | 0.3184 | 0.0798 | 0.7361 |
35
- | Airavata | 0.1165 | 0.3586 | 0.4393 | 0.1630 | 0.0277 | 0.0437 | 0.2534 |
36
 
37
  Task: flores Metric: chrf
38
  Task: implicit_hate Metric: chrf
39
  Task: indicsentiment Metric: accuracy
40
- Task: boolq-hi Metric: accuracy
41
  Task: indicxparaphrase Metric: accuracy
 
42
  Task: truthfulqa-hi Metric: accuracy
43
  Task: indic-arc-easy Metric: accuracy
44
  Task: indicwikibio Metric: bleurt
@@ -52,11 +48,12 @@ Task: arc-easy-exact Metric: accuracy
52
  Task: hellaswag Metric: accuracy
53
  Task: arc-challenge Metric: accuracy
54
  Task: mmlu Metric: average_acc
55
- Task: boolq Metric: accuracy
56
  Task: xlsum Metric: bleurt
 
57
  Task: truthfulqa Metric: accuracy
58
 
59
 
 
60
  Model evaluation on OpenLLM LeaderBoard
61
 
62
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/5dfae476da6d0311fd3d5432/ENzZwV2Z98uNlpyUz3Blp.png)
 
9
 
10
  Model trained on Hindi and English data.
11
 
12
+ Try it out: https://colab.research.google.com/drive/1A_hbsq1vrCeAh3dEMvtwxxNxcNZ1BUyW?usp=sharing
13
 
14
 
15
+ #### Language Hi
16
+
17
+ | Model | implicit_hate | flores | indicwikibio | hellaswag-indic | truthfulqa-hi | boolq-hi | indicheadline | indic-arc-easy | indicqa | indic-arc-challenge | indicsentiment | xlsum-hi | indicxparaphrase | mmlu_hi |
18
  | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
19
+ | open-aditi-hi-v2 | 11.5021 | 43.6822 | 0.4846 | 0.2404 | 0.6934 | 0.8541 | 0.4565 | 0.4979 | 0.0795 | 0.4462 | 0.9729 | 0.4213 | 0.6838 | 0.3253 |
20
+ | OpenHermes-2.5-Mistral-7B | 0.2068 | 30.3465 | 0.3332 | 0.2485 | 0.3234 | 0.5979 | 0.1996 | 0.3523 | 0.2721 | 0.3396 | 0.9048 | 0.1774 | 0.8766 | 0.2769 |
21
+ | open-aditi-hi-v1 | 8.6105 | 40.2376 | 0.4104 | 0.0848 | 0.4230 | 0.3758 | 0.4248 | 0.3889 | 0.1306 | 0.3558 | 0.8798 | 0.4212 | 0.5939 | 0.1398 |
22
+ | Airavata | 0.0663 | 58.0555 | 0.0637 | 0.0254 | 0.2122 | 0.0373 | 0.4346 | 0.1128 | 0.1008 | 0.0836 | 0.8437 | 0.4650 | 0.3277 | 0.1336 |
 
 
 
23
 
24
  #### Language En
25
 
26
+ | Model | boolq | hellaswag | mmlu | truthfulqa | xlsum | arc-easy-exact | arc-challenge |
27
  | --- | --- | --- | --- | --- | --- | --- | --- |
28
+ | OpenHermes-2.5-Mistral-7B | 0.4061 | 0.7999 | 0.5991 | 0.2081 | 0.4328 | 0.8687 | 0.7790 |
29
+ | open-aditi-hi-v2 | 0.3982 | 0.4738 | 0.5544 | 0.2999 | 0.4349 | 0.8388 | 0.7235 |
30
+ | open-aditi-hi-v1 | 0.0434 | 0.3509 | 0.2597 | 0.3317 | 0.4288 | 0.7588 | 0.6271 |
31
+ | Airavata | 0.0437 | 0.0277 | 0.1165 | 0.3586 | 0.4393 | 0.2534 | 0.1630 |
 
 
 
32
 
33
  Task: flores Metric: chrf
34
  Task: implicit_hate Metric: chrf
35
  Task: indicsentiment Metric: accuracy
 
36
  Task: indicxparaphrase Metric: accuracy
37
+ Task: boolq-hi Metric: accuracy
38
  Task: truthfulqa-hi Metric: accuracy
39
  Task: indic-arc-easy Metric: accuracy
40
  Task: indicwikibio Metric: bleurt
 
48
  Task: hellaswag Metric: accuracy
49
  Task: arc-challenge Metric: accuracy
50
  Task: mmlu Metric: average_acc
 
51
  Task: xlsum Metric: bleurt
52
+ Task: boolq Metric: accuracy
53
  Task: truthfulqa Metric: accuracy
54
 
55
 
56
+
57
  Model evaluation on OpenLLM LeaderBoard
58
 
59
  ![image/png](https://cdn-uploads.huggingface.co/production/uploads/5dfae476da6d0311fd3d5432/ENzZwV2Z98uNlpyUz3Blp.png)