akk-en-mt5-base

README.md

@@ -0,0 +1,131 @@
+---
+license: apache-2.0
+base_model: google/mt5-base
+tags:
+- generated_from_trainer
+model-index:
+- name: mt5-base-p-l-akk-en-20240709-215100
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# mt5-base-p-l-akk-en-20240709-215100
+
+This model is a fine-tuned version of [google/mt5-base](https://huggingface.co/google/mt5-base) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.1533
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 4e-05
+- train_batch_size: 12
+- eval_batch_size: 12
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 10
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss |
+|:-------------:|:------:|:-----:|:---------------:|
+| 30.893        | 0.1326 | 500   | 5.1945          |
+| 2.9823        | 0.2651 | 1000  | 0.6672          |
+| 0.6668        | 0.3977 | 1500  | 0.5089          |
+| 0.4944        | 0.5302 | 2000  | 0.3100          |
+| 0.3002        | 0.6628 | 2500  | 0.2663          |
+| 0.2813        | 0.7953 | 3000  | 0.2493          |
+| 0.273         | 0.9279 | 3500  | 0.2369          |
+| 0.2544        | 1.0604 | 4000  | 0.2304          |
+| 0.2445        | 1.1930 | 4500  | 0.2241          |
+| 0.2365        | 1.3256 | 5000  | 0.2190          |
+| 0.2305        | 1.4581 | 5500  | 0.2140          |
+| 0.2318        | 1.5907 | 6000  | 0.2108          |
+| 0.2166        | 1.7232 | 6500  | 0.2060          |
+| 0.2195        | 1.8558 | 7000  | 0.2029          |
+| 0.2125        | 1.9883 | 7500  | 0.2000          |
+| 0.2091        | 2.1209 | 8000  | 0.1963          |
+| 0.2092        | 2.2534 | 8500  | 0.1938          |
+| 0.2032        | 2.3860 | 9000  | 0.1915          |
+| 0.2018        | 2.5186 | 9500  | 0.1892          |
+| 0.2017        | 2.6511 | 10000 | 0.1870          |
+| 0.1961        | 2.7837 | 10500 | 0.1855          |
+| 0.2009        | 2.9162 | 11000 | 0.1841          |
+| 0.1956        | 3.0488 | 11500 | 0.1828          |
+| 0.1915        | 3.1813 | 12000 | 0.1807          |
+| 0.1892        | 3.3139 | 12500 | 0.1790          |
+| 0.1908        | 3.4464 | 13000 | 0.1773          |
+| 0.1834        | 3.5790 | 13500 | 0.1763          |
+| 0.1832        | 3.7116 | 14000 | 0.1744          |
+| 0.189         | 3.8441 | 14500 | 0.1734          |
+| 0.1848        | 3.9767 | 15000 | 0.1724          |
+| 0.1838        | 4.1092 | 15500 | 0.1715          |
+| 0.177         | 4.2418 | 16000 | 0.1703          |
+| 0.1808        | 4.3743 | 16500 | 0.1692          |
+| 0.183         | 4.5069 | 17000 | 0.1680          |
+| 0.1753        | 4.6394 | 17500 | 0.1675          |
+| 0.1724        | 4.7720 | 18000 | 0.1666          |
+| 0.1782        | 4.9046 | 18500 | 0.1656          |
+| 0.1799        | 5.0371 | 19000 | 0.1653          |
+| 0.1725        | 5.1697 | 19500 | 0.1647          |
+| 0.17          | 5.3022 | 20000 | 0.1635          |
+| 0.1722        | 5.4348 | 20500 | 0.1630          |
+| 0.1697        | 5.5673 | 21000 | 0.1625          |
+| 0.1719        | 5.6999 | 21500 | 0.1620          |
+| 0.1709        | 5.8324 | 22000 | 0.1611          |
+| 0.1727        | 5.9650 | 22500 | 0.1604          |
+| 0.1721        | 6.0976 | 23000 | 0.1598          |
+| 0.1681        | 6.2301 | 23500 | 0.1602          |
+| 0.1699        | 6.3627 | 24000 | 0.1596          |
+| 0.1639        | 6.4952 | 24500 | 0.1588          |
+| 0.1646        | 6.6278 | 25000 | 0.1584          |
+| 0.1691        | 6.7603 | 25500 | 0.1582          |
+| 0.1653        | 6.8929 | 26000 | 0.1574          |
+| 0.1648        | 7.0255 | 26500 | 0.1572          |
+| 0.1669        | 7.1580 | 27000 | 0.1569          |
+| 0.16          | 7.2906 | 27500 | 0.1568          |
+| 0.1622        | 7.4231 | 28000 | 0.1562          |
+| 0.1644        | 7.5557 | 28500 | 0.1561          |
+| 0.1674        | 7.6882 | 29000 | 0.1557          |
+| 0.1628        | 7.8208 | 29500 | 0.1552          |
+| 0.1619        | 7.9533 | 30000 | 0.1551          |
+| 0.1636        | 8.0859 | 30500 | 0.1549          |
+| 0.1629        | 8.2185 | 31000 | 0.1546          |
+| 0.1632        | 8.3510 | 31500 | 0.1545          |
+| 0.1641        | 8.4836 | 32000 | 0.1543          |
+| 0.1592        | 8.6161 | 32500 | 0.1541          |
+| 0.1573        | 8.7487 | 33000 | 0.1539          |
+| 0.1607        | 8.8812 | 33500 | 0.1540          |
+| 0.1651        | 9.0138 | 34000 | 0.1537          |
+| 0.1551        | 9.1463 | 34500 | 0.1537          |
+| 0.1621        | 9.2789 | 35000 | 0.1536          |
+| 0.166         | 9.4115 | 35500 | 0.1534          |
+| 0.1575        | 9.5440 | 36000 | 0.1534          |
+| 0.1607        | 9.6766 | 36500 | 0.1534          |
+| 0.1627        | 9.8091 | 37000 | 0.1533          |
+| 0.1608        | 9.9417 | 37500 | 0.1533          |
+
+
+### Framework versions
+
+- Transformers 4.41.2
+- Pytorch 2.5.0.dev20240625
+- Datasets 2.20.0
+- Tokenizers 0.19.1

config.json

@@ -0,0 +1,33 @@
+{
+  "_name_or_path": "google/mt5-base",
+  "architectures": [
+    "MT5ForConditionalGeneration"
+  ],
+  "classifier_dropout": 0.0,
+  "d_ff": 2048,
+  "d_kv": 64,
+  "d_model": 768,
+  "decoder_start_token_id": 0,
+  "dense_act_fn": "gelu_new",
+  "dropout_rate": 0.1,
+  "eos_token_id": 1,
+  "feed_forward_proj": "gated-gelu",
+  "initializer_factor": 1.0,
+  "is_encoder_decoder": true,
+  "is_gated_act": true,
+  "layer_norm_epsilon": 1e-06,
+  "model_type": "mt5",
+  "num_decoder_layers": 12,
+  "num_heads": 12,
+  "num_layers": 12,
+  "output_past": true,
+  "pad_token_id": 0,
+  "relative_attention_max_distance": 128,
+  "relative_attention_num_buckets": 32,
+  "tie_word_embeddings": false,
+  "tokenizer_class": "T5Tokenizer",
+  "torch_dtype": "float32",
+  "transformers_version": "4.41.2",
+  "use_cache": true,
+  "vocab_size": 264277
+}

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "decoder_start_token_id": 0,
+  "eos_token_id": 1,
+  "pad_token_id": 0,
+  "transformers_version": "4.41.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:08cfd87e1b4db110ebbf2131c6ca5cb1a84cb5ec361df21a0a85bacbd086b554
+size 2416668528

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

spiece.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ef78f86560d809067d12bac6c09f19a462cb3af3f54d2b8acbba26e1433125d6
+size 4309802

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9a7ecb7d13cbc768a6a30788a9ea473494ed30474747c85b46856124147cf989
+size 5304