Hugging Face's logo

nlpguy
/
smolchess-v2

Text Generation
Transformers
Safetensors
llama
Generated from Trainer
text-generation-inference
Inference Endpoints
Model card Files Community
nlpguy commited on
Commit
81df4a1
•
1 Parent(s): e13b70e

End of training

Browse files
Files changed (10) hide show
  1. README.md +156 -0
  2. config.json +32 -0
  3. generation_config.json +6 -0
  4. merges.txt +0 -0
  5. model.safetensors +3 -0
  6. special_tokens_map.json +43 -0
  7. tokenizer.json +0 -0
  8. tokenizer_config.json +168 -0
  9. training_args.bin +3 -0
  10. vocab.json +0 -0
README.md ADDED
@@ -0,0 +1,156 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ license: apache-2.0
4
+ base_model: HuggingFaceTB/SmolLM2-135M
5
+ tags:
6
+ - generated_from_trainer
7
+ model-index:
8
+ - name: smolchess-v2
9
+ results: []
10
+ ---
11
+
12
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
13
+ should probably proofread and complete it, then remove this comment. -->
14
+
15
+ # smolchess-v2
16
+
17
+ This model is a fine-tuned version of [HuggingFaceTB/SmolLM2-135M](https://huggingface.co/HuggingFaceTB/SmolLM2-135M) on an unknown dataset.
18
+ It achieves the following results on the evaluation set:
19
+ - Loss: 0.8569
20
+
21
+ ## Model description
22
+
23
+ More information needed
24
+
25
+ ## Intended uses & limitations
26
+
27
+ More information needed
28
+
29
+ ## Training and evaluation data
30
+
31
+ More information needed
32
+
33
+ ## Training procedure
34
+
35
+ ### Training hyperparameters
36
+
37
+ The following hyperparameters were used during training:
38
+ - learning_rate: 4e-05
39
+ - train_batch_size: 8
40
+ - eval_batch_size: 8
41
+ - seed: 42
42
+ - optimizer: Use grokadamw with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
43
+ - lr_scheduler_type: cosine
44
+ - num_epochs: 0.25
45
+
46
+ ### Training results
47
+
48
+ | Training Loss | Epoch | Step | Validation Loss |
49
+ |:-------------:|:------:|:----:|:---------------:|
50
+ | 1.4864 | 0.0025 | 4 | 1.5472 |
51
+ | 1.3163 | 0.0050 | 8 | 1.2616 |
52
+ | 1.0354 | 0.0075 | 12 | 1.1857 |
53
+ | 1.2466 | 0.0100 | 16 | 1.1447 |
54
+ | 1.1801 | 0.0125 | 20 | 1.1176 |
55
+ | 1.208 | 0.0150 | 24 | 1.1092 |
56
+ | 1.0723 | 0.0176 | 28 | 1.0780 |
57
+ | 1.1895 | 0.0201 | 32 | 1.0760 |
58
+ | 1.1358 | 0.0226 | 36 | 1.0562 |
59
+ | 1.0817 | 0.0251 | 40 | 1.0554 |
60
+ | 0.9674 | 0.0276 | 44 | 1.0419 |
61
+ | 0.9832 | 0.0301 | 48 | 1.0245 |
62
+ | 1.0241 | 0.0326 | 52 | 1.0178 |
63
+ | 0.9553 | 0.0351 | 56 | 1.0115 |
64
+ | 1.0715 | 0.0376 | 60 | 1.0027 |
65
+ | 1.1014 | 0.0401 | 64 | 0.9965 |
66
+ | 1.0304 | 0.0426 | 68 | 0.9954 |
67
+ | 0.9906 | 0.0451 | 72 | 0.9879 |
68
+ | 0.9631 | 0.0476 | 76 | 0.9769 |
69
+ | 0.986 | 0.0502 | 80 | 0.9720 |
70
+ | 1.0233 | 0.0527 | 84 | 0.9675 |
71
+ | 0.9323 | 0.0552 | 88 | 0.9612 |
72
+ | 0.9303 | 0.0577 | 92 | 0.9569 |
73
+ | 1.0258 | 0.0602 | 96 | 0.9520 |
74
+ | 0.9946 | 0.0627 | 100 | 0.9527 |
75
+ | 0.9568 | 0.0652 | 104 | 0.9425 |
76
+ | 0.9674 | 0.0677 | 108 | 0.9435 |
77
+ | 0.9627 | 0.0702 | 112 | 0.9378 |
78
+ | 0.9755 | 0.0727 | 116 | 0.9338 |
79
+ | 0.8511 | 0.0752 | 120 | 0.9306 |
80
+ | 0.989 | 0.0777 | 124 | 0.9292 |
81
+ | 0.9635 | 0.0803 | 128 | 0.9272 |
82
+ | 0.9412 | 0.0828 | 132 | 0.9263 |
83
+ | 0.8605 | 0.0853 | 136 | 0.9228 |
84
+ | 0.8503 | 0.0878 | 140 | 0.9206 |
85
+ | 0.8976 | 0.0903 | 144 | 0.9155 |
86
+ | 0.9029 | 0.0928 | 148 | 0.9143 |
87
+ | 0.9335 | 0.0953 | 152 | 0.9103 |
88
+ | 0.944 | 0.0978 | 156 | 0.9073 |
89
+ | 0.8948 | 0.1003 | 160 | 0.9058 |
90
+ | 0.8921 | 0.1028 | 164 | 0.9032 |
91
+ | 0.9948 | 0.1053 | 168 | 0.9028 |
92
+ | 0.8968 | 0.1078 | 172 | 0.9003 |
93
+ | 0.8908 | 0.1103 | 176 | 0.8982 |
94
+ | 0.9119 | 0.1129 | 180 | 0.8979 |
95
+ | 0.842 | 0.1154 | 184 | 0.8942 |
96
+ | 0.7497 | 0.1179 | 188 | 0.8930 |
97
+ | 0.9294 | 0.1204 | 192 | 0.8922 |
98
+ | 0.8184 | 0.1229 | 196 | 0.8891 |
99
+ | 0.941 | 0.1254 | 200 | 0.8883 |
100
+ | 0.8884 | 0.1279 | 204 | 0.8851 |
101
+ | 0.8975 | 0.1304 | 208 | 0.8851 |
102
+ | 0.9205 | 0.1329 | 212 | 0.8847 |
103
+ | 0.8663 | 0.1354 | 216 | 0.8815 |
104
+ | 0.8455 | 0.1379 | 220 | 0.8812 |
105
+ | 0.921 | 0.1404 | 224 | 0.8794 |
106
+ | 0.9493 | 0.1429 | 228 | 0.8784 |
107
+ | 0.8949 | 0.1455 | 232 | 0.8792 |
108
+ | 0.8886 | 0.1480 | 236 | 0.8773 |
109
+ | 0.8808 | 0.1505 | 240 | 0.8760 |
110
+ | 0.8768 | 0.1530 | 244 | 0.8750 |
111
+ | 0.9354 | 0.1555 | 248 | 0.8727 |
112
+ | 0.8512 | 0.1580 | 252 | 0.8721 |
113
+ | 0.8355 | 0.1605 | 256 | 0.8717 |
114
+ | 0.7923 | 0.1630 | 260 | 0.8699 |
115
+ | 0.9027 | 0.1655 | 264 | 0.8691 |
116
+ | 0.8264 | 0.1680 | 268 | 0.8681 |
117
+ | 0.9199 | 0.1705 | 272 | 0.8683 |
118
+ | 0.8792 | 0.1730 | 276 | 0.8666 |
119
+ | 0.9347 | 0.1755 | 280 | 0.8664 |
120
+ | 0.8988 | 0.1781 | 284 | 0.8652 |
121
+ | 0.889 | 0.1806 | 288 | 0.8646 |
122
+ | 0.917 | 0.1831 | 292 | 0.8633 |
123
+ | 0.9206 | 0.1856 | 296 | 0.8628 |
124
+ | 0.9127 | 0.1881 | 300 | 0.8629 |
125
+ | 0.6946 | 0.1906 | 304 | 0.8618 |
126
+ | 0.9499 | 0.1931 | 308 | 0.8612 |
127
+ | 0.8798 | 0.1956 | 312 | 0.8610 |
128
+ | 0.8857 | 0.1981 | 316 | 0.8610 |
129
+ | 0.9356 | 0.2006 | 320 | 0.8604 |
130
+ | 0.8134 | 0.2031 | 324 | 0.8597 |
131
+ | 0.9214 | 0.2056 | 328 | 0.8592 |
132
+ | 0.8907 | 0.2082 | 332 | 0.8590 |
133
+ | 0.8309 | 0.2107 | 336 | 0.8588 |
134
+ | 0.8386 | 0.2132 | 340 | 0.8584 |
135
+ | 0.8001 | 0.2157 | 344 | 0.8583 |
136
+ | 0.8452 | 0.2182 | 348 | 0.8580 |
137
+ | 0.7587 | 0.2207 | 352 | 0.8578 |
138
+ | 0.8155 | 0.2232 | 356 | 0.8576 |
139
+ | 0.7179 | 0.2257 | 360 | 0.8575 |
140
+ | 0.8231 | 0.2282 | 364 | 0.8573 |
141
+ | 0.8984 | 0.2307 | 368 | 0.8572 |
142
+ | 0.8501 | 0.2332 | 372 | 0.8571 |
143
+ | 0.8512 | 0.2357 | 376 | 0.8570 |
144
+ | 0.8554 | 0.2382 | 380 | 0.8570 |
145
+ | 0.9082 | 0.2408 | 384 | 0.8570 |
146
+ | 0.8617 | 0.2433 | 388 | 0.8569 |
147
+ | 0.8845 | 0.2458 | 392 | 0.8569 |
148
+ | 0.9595 | 0.2483 | 396 | 0.8569 |
149
+
150
+
151
+ ### Framework versions
152
+
153
+ - Transformers 4.46.1
154
+ - Pytorch 2.5.0+cu121
155
+ - Datasets 3.1.0
156
+ - Tokenizers 0.20.1
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "HuggingFaceTB/SmolLM2-135M",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "attention_dropout": 0.0,
8
+ "bos_token_id": 0,
9
+ "eos_token_id": 0,
10
+ "head_dim": 64,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 576,
13
+ "initializer_range": 0.041666666666666664,
14
+ "intermediate_size": 1536,
15
+ "is_llama_config": true,
16
+ "max_position_embeddings": 8192,
17
+ "mlp_bias": false,
18
+ "model_type": "llama",
19
+ "num_attention_heads": 9,
20
+ "num_hidden_layers": 30,
21
+ "num_key_value_heads": 3,
22
+ "pretraining_tp": 1,
23
+ "rms_norm_eps": 1e-05,
24
+ "rope_interleaved": false,
25
+ "rope_scaling": null,
26
+ "rope_theta": 100000,
27
+ "tie_word_embeddings": true,
28
+ "torch_dtype": "float32",
29
+ "transformers_version": "4.46.1",
30
+ "use_cache": true,
31
+ "vocab_size": 49152
32
+ }
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 0,
4
+ "eos_token_id": 0,
5
+ "transformers_version": "4.46.1"
6
+ }
merges.txt ADDED
The diff for this file is too large to render. See raw diff
 
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:851e079397b09a31e8eb904d879137d3e8b8a9c247750dbc4027641611a9408a
3
+ size 538090408
special_tokens_map.json ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|endoftext|>",
4
+ "<|im_start|>",
5
+ "<|im_end|>",
6
+ "<repo_name>",
7
+ "<reponame>",
8
+ "<file_sep>",
9
+ "<filename>",
10
+ "<gh_stars>",
11
+ "<issue_start>",
12
+ "<issue_comment>",
13
+ "<issue_closed>",
14
+ "<jupyter_start>",
15
+ "<jupyter_text>",
16
+ "<jupyter_code>",
17
+ "<jupyter_output>",
18
+ "<jupyter_script>",
19
+ "<empty_output>"
20
+ ],
21
+ "bos_token": {
22
+ "content": "<|endoftext|>",
23
+ "lstrip": false,
24
+ "normalized": false,
25
+ "rstrip": false,
26
+ "single_word": false
27
+ },
28
+ "eos_token": {
29
+ "content": "<|endoftext|>",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false
34
+ },
35
+ "pad_token": "<|endoftext|>",
36
+ "unk_token": {
37
+ "content": "<|endoftext|>",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false
42
+ }
43
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "added_tokens_decoder": {
4
+ "0": {
5
+ "content": "<|endoftext|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "1": {
13
+ "content": "<|im_start|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "2": {
21
+ "content": "<|im_end|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "3": {
29
+ "content": "<repo_name>",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "4": {
37
+ "content": "<reponame>",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ },
44
+ "5": {
45
+ "content": "<file_sep>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": true
51
+ },
52
+ "6": {
53
+ "content": "<filename>",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": true
59
+ },
60
+ "7": {
61
+ "content": "<gh_stars>",
62
+ "lstrip": false,
63
+ "normalized": false,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": true
67
+ },
68
+ "8": {
69
+ "content": "<issue_start>",
70
+ "lstrip": false,
71
+ "normalized": false,
72
+ "rstrip": false,
73
+ "single_word": false,
74
+ "special": true
75
+ },
76
+ "9": {
77
+ "content": "<issue_comment>",
78
+ "lstrip": false,
79
+ "normalized": false,
80
+ "rstrip": false,
81
+ "single_word": false,
82
+ "special": true
83
+ },
84
+ "10": {
85
+ "content": "<issue_closed>",
86
+ "lstrip": false,
87
+ "normalized": false,
88
+ "rstrip": false,
89
+ "single_word": false,
90
+ "special": true
91
+ },
92
+ "11": {
93
+ "content": "<jupyter_start>",
94
+ "lstrip": false,
95
+ "normalized": false,
96
+ "rstrip": false,
97
+ "single_word": false,
98
+ "special": true
99
+ },
100
+ "12": {
101
+ "content": "<jupyter_text>",
102
+ "lstrip": false,
103
+ "normalized": false,
104
+ "rstrip": false,
105
+ "single_word": false,
106
+ "special": true
107
+ },
108
+ "13": {
109
+ "content": "<jupyter_code>",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false,
114
+ "special": true
115
+ },
116
+ "14": {
117
+ "content": "<jupyter_output>",
118
+ "lstrip": false,
119
+ "normalized": false,
120
+ "rstrip": false,
121
+ "single_word": false,
122
+ "special": true
123
+ },
124
+ "15": {
125
+ "content": "<jupyter_script>",
126
+ "lstrip": false,
127
+ "normalized": false,
128
+ "rstrip": false,
129
+ "single_word": false,
130
+ "special": true
131
+ },
132
+ "16": {
133
+ "content": "<empty_output>",
134
+ "lstrip": false,
135
+ "normalized": false,
136
+ "rstrip": false,
137
+ "single_word": false,
138
+ "special": true
139
+ }
140
+ },
141
+ "additional_special_tokens": [
142
+ "<|endoftext|>",
143
+ "<|im_start|>",
144
+ "<|im_end|>",
145
+ "<repo_name>",
146
+ "<reponame>",
147
+ "<file_sep>",
148
+ "<filename>",
149
+ "<gh_stars>",
150
+ "<issue_start>",
151
+ "<issue_comment>",
152
+ "<issue_closed>",
153
+ "<jupyter_start>",
154
+ "<jupyter_text>",
155
+ "<jupyter_code>",
156
+ "<jupyter_output>",
157
+ "<jupyter_script>",
158
+ "<empty_output>"
159
+ ],
160
+ "bos_token": "<|endoftext|>",
161
+ "clean_up_tokenization_spaces": false,
162
+ "eos_token": "<|endoftext|>",
163
+ "model_max_length": 8192,
164
+ "pad_token": "<|endoftext|>",
165
+ "tokenizer_class": "GPT2Tokenizer",
166
+ "unk_token": "<|endoftext|>",
167
+ "vocab_size": 49152
168
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e725ef05085080c51c2dc78a73512ba0eb11579f6392b0841c1494c7e3b1eea8
3
+ size 5240
vocab.json ADDED
The diff for this file is too large to render. See raw diff