Upload folder using huggingface_hub

README.md

@@ -0,0 +1,62 @@
+---
+license: other
+library_name: peft
+tags:
+- llama-factory
+- lora
+- generated_from_trainer
+base_model: mistralai/Mistral-7B-Instruct-v0.2
+model-index:
+- name: mistral-lora-bf16-epoch4
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# mistral-lora-bf16-epoch4
+
+This model is a fine-tuned version of [/root/autodl-tmp/models/Mistral-7B-Instruct-v0.2](https://huggingface.co//root/autodl-tmp/models/Mistral-7B-Instruct-v0.2) on the filtered_sampled_data_wo_sysprompt dataset.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 1
+- eval_batch_size: 8
+- seed: 42
+- distributed_type: multi-GPU
+- num_devices: 4
+- gradient_accumulation_steps: 4
+- total_train_batch_size: 16
+- total_eval_batch_size: 32
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_ratio: 0.1
+- num_epochs: 4.0
+
+### Training results
+
+
+
+### Framework versions
+
+- PEFT 0.11.1
+- Transformers 4.41.2
+- Pytorch 2.3.0+cu121
+- Datasets 2.19.1
+- Tokenizers 0.19.1

adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "mistralai/Mistral-7B-Instruct-v0.2",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:982789f09a3cdc5f414b2c506475df7208913072e59fe8f5a7f5e59853c03f75
+size 167832240

all_results.json

@@ -0,0 +1,8 @@
+{
+    "epoch": 4.0,
+    "total_flos": 2.380746355434848e+18,
+    "train_loss": 0.2891847992741759,
+    "train_runtime": 20387.7129,
+    "train_samples_per_second": 6.796,
+    "train_steps_per_second": 0.425
+}

checkpoint-1000/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /root/autodl-tmp/models/Mistral-7B-Instruct-v0.2
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-1000/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "/root/autodl-tmp/models/Mistral-7B-Instruct-v0.2",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-1000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3365afe6aa736bf693401f47a8babc3a54e0e332575725d467f923f11273ff43
+size 167832240

checkpoint-1000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c17cbdfe15ddc9b092d8e19fc1d5013cf44bd5ef3860e0a5c95a64916e43b05f
+size 335922386

checkpoint-1000/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92cc13315f24c28015d695b6cde08bb1cd6fea4cbc435998485ed6fbe4c91285
+size 15024

checkpoint-1000/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4c154b6a63e0b1f98f7d2847944398f99f1657d35e8eddf7fdf0ae2c24b0552
+size 15024

checkpoint-1000/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f784c6a9507b51189f2caffbd178ea9882103b75852e31c15f47fdae6a43af1d
+size 15024

checkpoint-1000/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34b023e05bc2d12b91dc436d4922b990d50ec8dc56d40dc3e36b3bb34fc81341
+size 15024

checkpoint-1000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:179b44ef97ffc74d33141ff7eedc9403b7a1168a5f1d142d54dd72ad05bbc29b
+size 1064

checkpoint-1000/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

checkpoint-1000/tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "'<s>'{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '[INST] ' + content + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-1000/trainer_state.json

@@ -0,0 +1,733 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.4618937644341801,
+  "eval_steps": 500,
+  "global_step": 1000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.004618937644341801,
+      "grad_norm": 11.009225845336914,
+      "learning_rate": 5.773672055427253e-07,
+      "loss": 2.4977,
+      "step": 10
+    },
+    {
+      "epoch": 0.009237875288683603,
+      "grad_norm": 9.019889831542969,
+      "learning_rate": 1.1547344110854505e-06,
+      "loss": 2.7655,
+      "step": 20
+    },
+    {
+      "epoch": 0.013856812933025405,
+      "grad_norm": 11.212933540344238,
+      "learning_rate": 1.7321016166281756e-06,
+      "loss": 2.7241,
+      "step": 30
+    },
+    {
+      "epoch": 0.018475750577367205,
+      "grad_norm": 9.982237815856934,
+      "learning_rate": 2.309468822170901e-06,
+      "loss": 2.8357,
+      "step": 40
+    },
+    {
+      "epoch": 0.023094688221709007,
+      "grad_norm": 16.251842498779297,
+      "learning_rate": 2.886836027713626e-06,
+      "loss": 2.1159,
+      "step": 50
+    },
+    {
+      "epoch": 0.02771362586605081,
+      "grad_norm": 7.950500011444092,
+      "learning_rate": 3.464203233256351e-06,
+      "loss": 1.498,
+      "step": 60
+    },
+    {
+      "epoch": 0.03233256351039261,
+      "grad_norm": 4.816328048706055,
+      "learning_rate": 4.041570438799077e-06,
+      "loss": 0.9835,
+      "step": 70
+    },
+    {
+      "epoch": 0.03695150115473441,
+      "grad_norm": 9.529770851135254,
+      "learning_rate": 4.618937644341802e-06,
+      "loss": 1.0009,
+      "step": 80
+    },
+    {
+      "epoch": 0.04157043879907621,
+      "grad_norm": 12.515681266784668,
+      "learning_rate": 5.196304849884527e-06,
+      "loss": 1.0264,
+      "step": 90
+    },
+    {
+      "epoch": 0.046189376443418015,
+      "grad_norm": 5.393396854400635,
+      "learning_rate": 5.773672055427252e-06,
+      "loss": 0.8218,
+      "step": 100
+    },
+    {
+      "epoch": 0.050808314087759814,
+      "grad_norm": 2.777273416519165,
+      "learning_rate": 6.351039260969978e-06,
+      "loss": 0.8719,
+      "step": 110
+    },
+    {
+      "epoch": 0.05542725173210162,
+      "grad_norm": 3.8990070819854736,
+      "learning_rate": 6.928406466512702e-06,
+      "loss": 0.8592,
+      "step": 120
+    },
+    {
+      "epoch": 0.06004618937644342,
+      "grad_norm": 4.342905521392822,
+      "learning_rate": 7.505773672055427e-06,
+      "loss": 0.7982,
+      "step": 130
+    },
+    {
+      "epoch": 0.06466512702078522,
+      "grad_norm": 2.4967517852783203,
+      "learning_rate": 8.083140877598153e-06,
+      "loss": 0.7203,
+      "step": 140
+    },
+    {
+      "epoch": 0.06928406466512702,
+      "grad_norm": 5.491476058959961,
+      "learning_rate": 8.660508083140878e-06,
+      "loss": 0.7647,
+      "step": 150
+    },
+    {
+      "epoch": 0.07390300230946882,
+      "grad_norm": 4.13068962097168,
+      "learning_rate": 9.237875288683604e-06,
+      "loss": 0.7378,
+      "step": 160
+    },
+    {
+      "epoch": 0.07852193995381063,
+      "grad_norm": 3.9301445484161377,
+      "learning_rate": 9.815242494226329e-06,
+      "loss": 0.7355,
+      "step": 170
+    },
+    {
+      "epoch": 0.08314087759815242,
+      "grad_norm": 2.891854763031006,
+      "learning_rate": 1.0392609699769053e-05,
+      "loss": 0.6911,
+      "step": 180
+    },
+    {
+      "epoch": 0.08775981524249422,
+      "grad_norm": 5.377829074859619,
+      "learning_rate": 1.0969976905311778e-05,
+      "loss": 0.7972,
+      "step": 190
+    },
+    {
+      "epoch": 0.09237875288683603,
+      "grad_norm": 3.1744544506073,
+      "learning_rate": 1.1547344110854504e-05,
+      "loss": 0.6792,
+      "step": 200
+    },
+    {
+      "epoch": 0.09699769053117784,
+      "grad_norm": 4.408601760864258,
+      "learning_rate": 1.2124711316397229e-05,
+      "loss": 0.7378,
+      "step": 210
+    },
+    {
+      "epoch": 0.10161662817551963,
+      "grad_norm": 3.3633627891540527,
+      "learning_rate": 1.2702078521939955e-05,
+      "loss": 0.7131,
+      "step": 220
+    },
+    {
+      "epoch": 0.10623556581986143,
+      "grad_norm": 5.786050319671631,
+      "learning_rate": 1.3279445727482678e-05,
+      "loss": 0.6806,
+      "step": 230
+    },
+    {
+      "epoch": 0.11085450346420324,
+      "grad_norm": 6.865295886993408,
+      "learning_rate": 1.3856812933025404e-05,
+      "loss": 0.7117,
+      "step": 240
+    },
+    {
+      "epoch": 0.11547344110854503,
+      "grad_norm": 3.714210033416748,
+      "learning_rate": 1.4434180138568129e-05,
+      "loss": 0.6354,
+      "step": 250
+    },
+    {
+      "epoch": 0.12009237875288684,
+      "grad_norm": 5.501813888549805,
+      "learning_rate": 1.5011547344110854e-05,
+      "loss": 0.6266,
+      "step": 260
+    },
+    {
+      "epoch": 0.12471131639722864,
+      "grad_norm": 6.622759819030762,
+      "learning_rate": 1.558891454965358e-05,
+      "loss": 0.6894,
+      "step": 270
+    },
+    {
+      "epoch": 0.12933025404157045,
+      "grad_norm": 3.647956609725952,
+      "learning_rate": 1.6166281755196306e-05,
+      "loss": 0.66,
+      "step": 280
+    },
+    {
+      "epoch": 0.13394919168591224,
+      "grad_norm": 4.103866100311279,
+      "learning_rate": 1.674364896073903e-05,
+      "loss": 0.665,
+      "step": 290
+    },
+    {
+      "epoch": 0.13856812933025403,
+      "grad_norm": 3.84169340133667,
+      "learning_rate": 1.7321016166281756e-05,
+      "loss": 0.6676,
+      "step": 300
+    },
+    {
+      "epoch": 0.14318706697459585,
+      "grad_norm": 8.042781829833984,
+      "learning_rate": 1.789838337182448e-05,
+      "loss": 0.6236,
+      "step": 310
+    },
+    {
+      "epoch": 0.14780600461893764,
+      "grad_norm": 4.35403299331665,
+      "learning_rate": 1.8475750577367208e-05,
+      "loss": 0.6283,
+      "step": 320
+    },
+    {
+      "epoch": 0.15242494226327943,
+      "grad_norm": 3.7114272117614746,
+      "learning_rate": 1.9053117782909933e-05,
+      "loss": 0.6472,
+      "step": 330
+    },
+    {
+      "epoch": 0.15704387990762125,
+      "grad_norm": 2.5825653076171875,
+      "learning_rate": 1.9630484988452657e-05,
+      "loss": 0.6139,
+      "step": 340
+    },
+    {
+      "epoch": 0.16166281755196305,
+      "grad_norm": 4.71831750869751,
+      "learning_rate": 2.0207852193995382e-05,
+      "loss": 0.6019,
+      "step": 350
+    },
+    {
+      "epoch": 0.16628175519630484,
+      "grad_norm": 7.264115333557129,
+      "learning_rate": 2.0785219399538107e-05,
+      "loss": 0.6484,
+      "step": 360
+    },
+    {
+      "epoch": 0.17090069284064666,
+      "grad_norm": 5.484978199005127,
+      "learning_rate": 2.1362586605080835e-05,
+      "loss": 0.6423,
+      "step": 370
+    },
+    {
+      "epoch": 0.17551963048498845,
+      "grad_norm": 4.004003524780273,
+      "learning_rate": 2.1939953810623556e-05,
+      "loss": 0.6167,
+      "step": 380
+    },
+    {
+      "epoch": 0.18013856812933027,
+      "grad_norm": 3.858074426651001,
+      "learning_rate": 2.251732101616628e-05,
+      "loss": 0.6029,
+      "step": 390
+    },
+    {
+      "epoch": 0.18475750577367206,
+      "grad_norm": 4.84448766708374,
+      "learning_rate": 2.309468822170901e-05,
+      "loss": 0.5511,
+      "step": 400
+    },
+    {
+      "epoch": 0.18937644341801385,
+      "grad_norm": 3.419329881668091,
+      "learning_rate": 2.3672055427251733e-05,
+      "loss": 0.5919,
+      "step": 410
+    },
+    {
+      "epoch": 0.19399538106235567,
+      "grad_norm": 2.367598295211792,
+      "learning_rate": 2.4249422632794458e-05,
+      "loss": 0.5407,
+      "step": 420
+    },
+    {
+      "epoch": 0.19861431870669746,
+      "grad_norm": 4.016935348510742,
+      "learning_rate": 2.4826789838337182e-05,
+      "loss": 0.5829,
+      "step": 430
+    },
+    {
+      "epoch": 0.20323325635103925,
+      "grad_norm": 4.788145065307617,
+      "learning_rate": 2.540415704387991e-05,
+      "loss": 0.593,
+      "step": 440
+    },
+    {
+      "epoch": 0.20785219399538107,
+      "grad_norm": 3.618800401687622,
+      "learning_rate": 2.5981524249422635e-05,
+      "loss": 0.56,
+      "step": 450
+    },
+    {
+      "epoch": 0.21247113163972287,
+      "grad_norm": 3.6879987716674805,
+      "learning_rate": 2.6558891454965356e-05,
+      "loss": 0.5448,
+      "step": 460
+    },
+    {
+      "epoch": 0.21709006928406466,
+      "grad_norm": 3.6736364364624023,
+      "learning_rate": 2.7136258660508084e-05,
+      "loss": 0.5707,
+      "step": 470
+    },
+    {
+      "epoch": 0.22170900692840648,
+      "grad_norm": 3.954604387283325,
+      "learning_rate": 2.771362586605081e-05,
+      "loss": 0.5891,
+      "step": 480
+    },
+    {
+      "epoch": 0.22632794457274827,
+      "grad_norm": 6.050353050231934,
+      "learning_rate": 2.8290993071593537e-05,
+      "loss": 0.6231,
+      "step": 490
+    },
+    {
+      "epoch": 0.23094688221709006,
+      "grad_norm": 3.728318214416504,
+      "learning_rate": 2.8868360277136258e-05,
+      "loss": 0.6294,
+      "step": 500
+    },
+    {
+      "epoch": 0.23556581986143188,
+      "grad_norm": 5.047093391418457,
+      "learning_rate": 2.9445727482678986e-05,
+      "loss": 0.5795,
+      "step": 510
+    },
+    {
+      "epoch": 0.24018475750577367,
+      "grad_norm": 4.392754077911377,
+      "learning_rate": 3.0023094688221707e-05,
+      "loss": 0.5634,
+      "step": 520
+    },
+    {
+      "epoch": 0.24480369515011546,
+      "grad_norm": 3.634922504425049,
+      "learning_rate": 3.060046189376444e-05,
+      "loss": 0.5372,
+      "step": 530
+    },
+    {
+      "epoch": 0.24942263279445728,
+      "grad_norm": 3.2436106204986572,
+      "learning_rate": 3.117782909930716e-05,
+      "loss": 0.5914,
+      "step": 540
+    },
+    {
+      "epoch": 0.2540415704387991,
+      "grad_norm": 4.555751323699951,
+      "learning_rate": 3.175519630484989e-05,
+      "loss": 0.5767,
+      "step": 550
+    },
+    {
+      "epoch": 0.2586605080831409,
+      "grad_norm": 3.359252452850342,
+      "learning_rate": 3.233256351039261e-05,
+      "loss": 0.5569,
+      "step": 560
+    },
+    {
+      "epoch": 0.2632794457274827,
+      "grad_norm": 3.362272262573242,
+      "learning_rate": 3.290993071593534e-05,
+      "loss": 0.6248,
+      "step": 570
+    },
+    {
+      "epoch": 0.2678983833718245,
+      "grad_norm": 3.1555299758911133,
+      "learning_rate": 3.348729792147806e-05,
+      "loss": 0.5629,
+      "step": 580
+    },
+    {
+      "epoch": 0.27251732101616627,
+      "grad_norm": 4.786733627319336,
+      "learning_rate": 3.4064665127020787e-05,
+      "loss": 0.5942,
+      "step": 590
+    },
+    {
+      "epoch": 0.27713625866050806,
+      "grad_norm": 4.290501117706299,
+      "learning_rate": 3.464203233256351e-05,
+      "loss": 0.5694,
+      "step": 600
+    },
+    {
+      "epoch": 0.2817551963048499,
+      "grad_norm": 2.2575786113739014,
+      "learning_rate": 3.5219399538106236e-05,
+      "loss": 0.5459,
+      "step": 610
+    },
+    {
+      "epoch": 0.2863741339491917,
+      "grad_norm": 4.7077484130859375,
+      "learning_rate": 3.579676674364896e-05,
+      "loss": 0.5686,
+      "step": 620
+    },
+    {
+      "epoch": 0.2909930715935335,
+      "grad_norm": 3.487034559249878,
+      "learning_rate": 3.6374133949191685e-05,
+      "loss": 0.5185,
+      "step": 630
+    },
+    {
+      "epoch": 0.2956120092378753,
+      "grad_norm": 4.230603218078613,
+      "learning_rate": 3.6951501154734416e-05,
+      "loss": 0.5548,
+      "step": 640
+    },
+    {
+      "epoch": 0.3002309468822171,
+      "grad_norm": 3.5367519855499268,
+      "learning_rate": 3.7528868360277134e-05,
+      "loss": 0.5658,
+      "step": 650
+    },
+    {
+      "epoch": 0.30484988452655887,
+      "grad_norm": 3.3856465816497803,
+      "learning_rate": 3.8106235565819866e-05,
+      "loss": 0.5459,
+      "step": 660
+    },
+    {
+      "epoch": 0.3094688221709007,
+      "grad_norm": 3.4272944927215576,
+      "learning_rate": 3.868360277136259e-05,
+      "loss": 0.5823,
+      "step": 670
+    },
+    {
+      "epoch": 0.3140877598152425,
+      "grad_norm": 2.7848410606384277,
+      "learning_rate": 3.9260969976905315e-05,
+      "loss": 0.5481,
+      "step": 680
+    },
+    {
+      "epoch": 0.3187066974595843,
+      "grad_norm": 2.8584043979644775,
+      "learning_rate": 3.983833718244804e-05,
+      "loss": 0.4907,
+      "step": 690
+    },
+    {
+      "epoch": 0.3233256351039261,
+      "grad_norm": 3.9872729778289795,
+      "learning_rate": 4.0415704387990764e-05,
+      "loss": 0.5703,
+      "step": 700
+    },
+    {
+      "epoch": 0.3279445727482679,
+      "grad_norm": 4.052183151245117,
+      "learning_rate": 4.099307159353349e-05,
+      "loss": 0.6229,
+      "step": 710
+    },
+    {
+      "epoch": 0.3325635103926097,
+      "grad_norm": 3.6171059608459473,
+      "learning_rate": 4.1570438799076213e-05,
+      "loss": 0.5022,
+      "step": 720
+    },
+    {
+      "epoch": 0.3371824480369515,
+      "grad_norm": 3.4360239505767822,
+      "learning_rate": 4.214780600461894e-05,
+      "loss": 0.5642,
+      "step": 730
+    },
+    {
+      "epoch": 0.3418013856812933,
+      "grad_norm": 3.328563928604126,
+      "learning_rate": 4.272517321016167e-05,
+      "loss": 0.543,
+      "step": 740
+    },
+    {
+      "epoch": 0.3464203233256351,
+      "grad_norm": 2.494722843170166,
+      "learning_rate": 4.330254041570439e-05,
+      "loss": 0.5056,
+      "step": 750
+    },
+    {
+      "epoch": 0.3510392609699769,
+      "grad_norm": 2.9856996536254883,
+      "learning_rate": 4.387990762124711e-05,
+      "loss": 0.5215,
+      "step": 760
+    },
+    {
+      "epoch": 0.3556581986143187,
+      "grad_norm": 3.6510608196258545,
+      "learning_rate": 4.445727482678984e-05,
+      "loss": 0.5418,
+      "step": 770
+    },
+    {
+      "epoch": 0.36027713625866054,
+      "grad_norm": 2.837355136871338,
+      "learning_rate": 4.503464203233256e-05,
+      "loss": 0.5197,
+      "step": 780
+    },
+    {
+      "epoch": 0.3648960739030023,
+      "grad_norm": 2.4272849559783936,
+      "learning_rate": 4.561200923787529e-05,
+      "loss": 0.5151,
+      "step": 790
+    },
+    {
+      "epoch": 0.3695150115473441,
+      "grad_norm": 2.2198147773742676,
+      "learning_rate": 4.618937644341802e-05,
+      "loss": 0.5037,
+      "step": 800
+    },
+    {
+      "epoch": 0.3741339491916859,
+      "grad_norm": 3.082343816757202,
+      "learning_rate": 4.676674364896074e-05,
+      "loss": 0.5411,
+      "step": 810
+    },
+    {
+      "epoch": 0.3787528868360277,
+      "grad_norm": 2.739429235458374,
+      "learning_rate": 4.7344110854503466e-05,
+      "loss": 0.5034,
+      "step": 820
+    },
+    {
+      "epoch": 0.3833718244803695,
+      "grad_norm": 2.8211052417755127,
+      "learning_rate": 4.792147806004619e-05,
+      "loss": 0.5077,
+      "step": 830
+    },
+    {
+      "epoch": 0.38799076212471134,
+      "grad_norm": 4.462055683135986,
+      "learning_rate": 4.8498845265588916e-05,
+      "loss": 0.4883,
+      "step": 840
+    },
+    {
+      "epoch": 0.39260969976905313,
+      "grad_norm": 2.8090898990631104,
+      "learning_rate": 4.907621247113165e-05,
+      "loss": 0.4918,
+      "step": 850
+    },
+    {
+      "epoch": 0.3972286374133949,
+      "grad_norm": 2.787473440170288,
+      "learning_rate": 4.9653579676674365e-05,
+      "loss": 0.4736,
+      "step": 860
+    },
+    {
+      "epoch": 0.4018475750577367,
+      "grad_norm": 3.198763132095337,
+      "learning_rate": 4.999996750557592e-05,
+      "loss": 0.5243,
+      "step": 870
+    },
+    {
+      "epoch": 0.4064665127020785,
+      "grad_norm": 3.8778421878814697,
+      "learning_rate": 4.9999601944275146e-05,
+      "loss": 0.5206,
+      "step": 880
+    },
+    {
+      "epoch": 0.4110854503464203,
+      "grad_norm": 4.326559066772461,
+      "learning_rate": 4.999883020960263e-05,
+      "loss": 0.5998,
+      "step": 890
+    },
+    {
+      "epoch": 0.41570438799076215,
+      "grad_norm": 2.1009411811828613,
+      "learning_rate": 4.99976523140969e-05,
+      "loss": 0.5097,
+      "step": 900
+    },
+    {
+      "epoch": 0.42032332563510394,
+      "grad_norm": 3.0838584899902344,
+      "learning_rate": 4.999606827689545e-05,
+      "loss": 0.4699,
+      "step": 910
+    },
+    {
+      "epoch": 0.42494226327944573,
+      "grad_norm": 3.067430019378662,
+      "learning_rate": 4.999407812373445e-05,
+      "loss": 0.5223,
+      "step": 920
+    },
+    {
+      "epoch": 0.4295612009237875,
+      "grad_norm": 2.580418109893799,
+      "learning_rate": 4.9991681886948297e-05,
+      "loss": 0.5396,
+      "step": 930
+    },
+    {
+      "epoch": 0.4341801385681293,
+      "grad_norm": 2.748032808303833,
+      "learning_rate": 4.9988879605469106e-05,
+      "loss": 0.4965,
+      "step": 940
+    },
+    {
+      "epoch": 0.4387990762124711,
+      "grad_norm": 2.841571569442749,
+      "learning_rate": 4.99856713248261e-05,
+      "loss": 0.472,
+      "step": 950
+    },
+    {
+      "epoch": 0.44341801385681295,
+      "grad_norm": 1.8557236194610596,
+      "learning_rate": 4.9982057097144826e-05,
+      "loss": 0.4878,
+      "step": 960
+    },
+    {
+      "epoch": 0.44803695150115475,
+      "grad_norm": 5.707876205444336,
+      "learning_rate": 4.9978036981146344e-05,
+      "loss": 0.4883,
+      "step": 970
+    },
+    {
+      "epoch": 0.45265588914549654,
+      "grad_norm": 3.867081880569458,
+      "learning_rate": 4.9973611042146254e-05,
+      "loss": 0.4929,
+      "step": 980
+    },
+    {
+      "epoch": 0.45727482678983833,
+      "grad_norm": 2.291074275970459,
+      "learning_rate": 4.996877935205365e-05,
+      "loss": 0.5046,
+      "step": 990
+    },
+    {
+      "epoch": 0.4618937644341801,
+      "grad_norm": 2.332185745239258,
+      "learning_rate": 4.996354198936993e-05,
+      "loss": 0.4539,
+      "step": 1000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 8660,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 2.714316759199908e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5634a4aea40f26a4bd395cce99d40e5d4ac38396a87a690ea8534fccfab25f6d
+size 5368

checkpoint-1500/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /root/autodl-tmp/models/Mistral-7B-Instruct-v0.2
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-1500/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "/root/autodl-tmp/models/Mistral-7B-Instruct-v0.2",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-1500/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:307ae7a77c9ca972a8f6eddbf3d986b6c0b640bfe93bd5d5e5ee282b6d330ee9
+size 167832240

checkpoint-1500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:87c94699374d7902761ca201dbc2fb58790564814a6c6275c5461ece4a211bdd
+size 335922386

checkpoint-1500/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92cc13315f24c28015d695b6cde08bb1cd6fea4cbc435998485ed6fbe4c91285
+size 15024

checkpoint-1500/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4c154b6a63e0b1f98f7d2847944398f99f1657d35e8eddf7fdf0ae2c24b0552
+size 15024

checkpoint-1500/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f784c6a9507b51189f2caffbd178ea9882103b75852e31c15f47fdae6a43af1d
+size 15024

checkpoint-1500/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34b023e05bc2d12b91dc436d4922b990d50ec8dc56d40dc3e36b3bb34fc81341
+size 15024

checkpoint-1500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b771def0da9a50474b6f499a88427731f7aaa234c9154f4eee7a42c2a57b96cf
+size 1064

checkpoint-1500/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-1500/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

checkpoint-1500/tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "'<s>'{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '[INST] ' + content + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-1500/trainer_state.json

@@ -0,0 +1,1083 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.6928406466512702,
+  "eval_steps": 500,
+  "global_step": 1500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.004618937644341801,
+      "grad_norm": 11.009225845336914,
+      "learning_rate": 5.773672055427253e-07,
+      "loss": 2.4977,
+      "step": 10
+    },
+    {
+      "epoch": 0.009237875288683603,
+      "grad_norm": 9.019889831542969,
+      "learning_rate": 1.1547344110854505e-06,
+      "loss": 2.7655,
+      "step": 20
+    },
+    {
+      "epoch": 0.013856812933025405,
+      "grad_norm": 11.212933540344238,
+      "learning_rate": 1.7321016166281756e-06,
+      "loss": 2.7241,
+      "step": 30
+    },
+    {
+      "epoch": 0.018475750577367205,
+      "grad_norm": 9.982237815856934,
+      "learning_rate": 2.309468822170901e-06,
+      "loss": 2.8357,
+      "step": 40
+    },
+    {
+      "epoch": 0.023094688221709007,
+      "grad_norm": 16.251842498779297,
+      "learning_rate": 2.886836027713626e-06,
+      "loss": 2.1159,
+      "step": 50
+    },
+    {
+      "epoch": 0.02771362586605081,
+      "grad_norm": 7.950500011444092,
+      "learning_rate": 3.464203233256351e-06,
+      "loss": 1.498,
+      "step": 60
+    },
+    {
+      "epoch": 0.03233256351039261,
+      "grad_norm": 4.816328048706055,
+      "learning_rate": 4.041570438799077e-06,
+      "loss": 0.9835,
+      "step": 70
+    },
+    {
+      "epoch": 0.03695150115473441,
+      "grad_norm": 9.529770851135254,
+      "learning_rate": 4.618937644341802e-06,
+      "loss": 1.0009,
+      "step": 80
+    },
+    {
+      "epoch": 0.04157043879907621,
+      "grad_norm": 12.515681266784668,
+      "learning_rate": 5.196304849884527e-06,
+      "loss": 1.0264,
+      "step": 90
+    },
+    {
+      "epoch": 0.046189376443418015,
+      "grad_norm": 5.393396854400635,
+      "learning_rate": 5.773672055427252e-06,
+      "loss": 0.8218,
+      "step": 100
+    },
+    {
+      "epoch": 0.050808314087759814,
+      "grad_norm": 2.777273416519165,
+      "learning_rate": 6.351039260969978e-06,
+      "loss": 0.8719,
+      "step": 110
+    },
+    {
+      "epoch": 0.05542725173210162,
+      "grad_norm": 3.8990070819854736,
+      "learning_rate": 6.928406466512702e-06,
+      "loss": 0.8592,
+      "step": 120
+    },
+    {
+      "epoch": 0.06004618937644342,
+      "grad_norm": 4.342905521392822,
+      "learning_rate": 7.505773672055427e-06,
+      "loss": 0.7982,
+      "step": 130
+    },
+    {
+      "epoch": 0.06466512702078522,
+      "grad_norm": 2.4967517852783203,
+      "learning_rate": 8.083140877598153e-06,
+      "loss": 0.7203,
+      "step": 140
+    },
+    {
+      "epoch": 0.06928406466512702,
+      "grad_norm": 5.491476058959961,
+      "learning_rate": 8.660508083140878e-06,
+      "loss": 0.7647,
+      "step": 150
+    },
+    {
+      "epoch": 0.07390300230946882,
+      "grad_norm": 4.13068962097168,
+      "learning_rate": 9.237875288683604e-06,
+      "loss": 0.7378,
+      "step": 160
+    },
+    {
+      "epoch": 0.07852193995381063,
+      "grad_norm": 3.9301445484161377,
+      "learning_rate": 9.815242494226329e-06,
+      "loss": 0.7355,
+      "step": 170
+    },
+    {
+      "epoch": 0.08314087759815242,
+      "grad_norm": 2.891854763031006,
+      "learning_rate": 1.0392609699769053e-05,
+      "loss": 0.6911,
+      "step": 180
+    },
+    {
+      "epoch": 0.08775981524249422,
+      "grad_norm": 5.377829074859619,
+      "learning_rate": 1.0969976905311778e-05,
+      "loss": 0.7972,
+      "step": 190
+    },
+    {
+      "epoch": 0.09237875288683603,
+      "grad_norm": 3.1744544506073,
+      "learning_rate": 1.1547344110854504e-05,
+      "loss": 0.6792,
+      "step": 200
+    },
+    {
+      "epoch": 0.09699769053117784,
+      "grad_norm": 4.408601760864258,
+      "learning_rate": 1.2124711316397229e-05,
+      "loss": 0.7378,
+      "step": 210
+    },
+    {
+      "epoch": 0.10161662817551963,
+      "grad_norm": 3.3633627891540527,
+      "learning_rate": 1.2702078521939955e-05,
+      "loss": 0.7131,
+      "step": 220
+    },
+    {
+      "epoch": 0.10623556581986143,
+      "grad_norm": 5.786050319671631,
+      "learning_rate": 1.3279445727482678e-05,
+      "loss": 0.6806,
+      "step": 230
+    },
+    {
+      "epoch": 0.11085450346420324,
+      "grad_norm": 6.865295886993408,
+      "learning_rate": 1.3856812933025404e-05,
+      "loss": 0.7117,
+      "step": 240
+    },
+    {
+      "epoch": 0.11547344110854503,
+      "grad_norm": 3.714210033416748,
+      "learning_rate": 1.4434180138568129e-05,
+      "loss": 0.6354,
+      "step": 250
+    },
+    {
+      "epoch": 0.12009237875288684,
+      "grad_norm": 5.501813888549805,
+      "learning_rate": 1.5011547344110854e-05,
+      "loss": 0.6266,
+      "step": 260
+    },
+    {
+      "epoch": 0.12471131639722864,
+      "grad_norm": 6.622759819030762,
+      "learning_rate": 1.558891454965358e-05,
+      "loss": 0.6894,
+      "step": 270
+    },
+    {
+      "epoch": 0.12933025404157045,
+      "grad_norm": 3.647956609725952,
+      "learning_rate": 1.6166281755196306e-05,
+      "loss": 0.66,
+      "step": 280
+    },
+    {
+      "epoch": 0.13394919168591224,
+      "grad_norm": 4.103866100311279,
+      "learning_rate": 1.674364896073903e-05,
+      "loss": 0.665,
+      "step": 290
+    },
+    {
+      "epoch": 0.13856812933025403,
+      "grad_norm": 3.84169340133667,
+      "learning_rate": 1.7321016166281756e-05,
+      "loss": 0.6676,
+      "step": 300
+    },
+    {
+      "epoch": 0.14318706697459585,
+      "grad_norm": 8.042781829833984,
+      "learning_rate": 1.789838337182448e-05,
+      "loss": 0.6236,
+      "step": 310
+    },
+    {
+      "epoch": 0.14780600461893764,
+      "grad_norm": 4.35403299331665,
+      "learning_rate": 1.8475750577367208e-05,
+      "loss": 0.6283,
+      "step": 320
+    },
+    {
+      "epoch": 0.15242494226327943,
+      "grad_norm": 3.7114272117614746,
+      "learning_rate": 1.9053117782909933e-05,
+      "loss": 0.6472,
+      "step": 330
+    },
+    {
+      "epoch": 0.15704387990762125,
+      "grad_norm": 2.5825653076171875,
+      "learning_rate": 1.9630484988452657e-05,
+      "loss": 0.6139,
+      "step": 340
+    },
+    {
+      "epoch": 0.16166281755196305,
+      "grad_norm": 4.71831750869751,
+      "learning_rate": 2.0207852193995382e-05,
+      "loss": 0.6019,
+      "step": 350
+    },
+    {
+      "epoch": 0.16628175519630484,
+      "grad_norm": 7.264115333557129,
+      "learning_rate": 2.0785219399538107e-05,
+      "loss": 0.6484,
+      "step": 360
+    },
+    {
+      "epoch": 0.17090069284064666,
+      "grad_norm": 5.484978199005127,
+      "learning_rate": 2.1362586605080835e-05,
+      "loss": 0.6423,
+      "step": 370
+    },
+    {
+      "epoch": 0.17551963048498845,
+      "grad_norm": 4.004003524780273,
+      "learning_rate": 2.1939953810623556e-05,
+      "loss": 0.6167,
+      "step": 380
+    },
+    {
+      "epoch": 0.18013856812933027,
+      "grad_norm": 3.858074426651001,
+      "learning_rate": 2.251732101616628e-05,
+      "loss": 0.6029,
+      "step": 390
+    },
+    {
+      "epoch": 0.18475750577367206,
+      "grad_norm": 4.84448766708374,
+      "learning_rate": 2.309468822170901e-05,
+      "loss": 0.5511,
+      "step": 400
+    },
+    {
+      "epoch": 0.18937644341801385,
+      "grad_norm": 3.419329881668091,
+      "learning_rate": 2.3672055427251733e-05,
+      "loss": 0.5919,
+      "step": 410
+    },
+    {
+      "epoch": 0.19399538106235567,
+      "grad_norm": 2.367598295211792,
+      "learning_rate": 2.4249422632794458e-05,
+      "loss": 0.5407,
+      "step": 420
+    },
+    {
+      "epoch": 0.19861431870669746,
+      "grad_norm": 4.016935348510742,
+      "learning_rate": 2.4826789838337182e-05,
+      "loss": 0.5829,
+      "step": 430
+    },
+    {
+      "epoch": 0.20323325635103925,
+      "grad_norm": 4.788145065307617,
+      "learning_rate": 2.540415704387991e-05,
+      "loss": 0.593,
+      "step": 440
+    },
+    {
+      "epoch": 0.20785219399538107,
+      "grad_norm": 3.618800401687622,
+      "learning_rate": 2.5981524249422635e-05,
+      "loss": 0.56,
+      "step": 450
+    },
+    {
+      "epoch": 0.21247113163972287,
+      "grad_norm": 3.6879987716674805,
+      "learning_rate": 2.6558891454965356e-05,
+      "loss": 0.5448,
+      "step": 460
+    },
+    {
+      "epoch": 0.21709006928406466,
+      "grad_norm": 3.6736364364624023,
+      "learning_rate": 2.7136258660508084e-05,
+      "loss": 0.5707,
+      "step": 470
+    },
+    {
+      "epoch": 0.22170900692840648,
+      "grad_norm": 3.954604387283325,
+      "learning_rate": 2.771362586605081e-05,
+      "loss": 0.5891,
+      "step": 480
+    },
+    {
+      "epoch": 0.22632794457274827,
+      "grad_norm": 6.050353050231934,
+      "learning_rate": 2.8290993071593537e-05,
+      "loss": 0.6231,
+      "step": 490
+    },
+    {
+      "epoch": 0.23094688221709006,
+      "grad_norm": 3.728318214416504,
+      "learning_rate": 2.8868360277136258e-05,
+      "loss": 0.6294,
+      "step": 500
+    },
+    {
+      "epoch": 0.23556581986143188,
+      "grad_norm": 5.047093391418457,
+      "learning_rate": 2.9445727482678986e-05,
+      "loss": 0.5795,
+      "step": 510
+    },
+    {
+      "epoch": 0.24018475750577367,
+      "grad_norm": 4.392754077911377,
+      "learning_rate": 3.0023094688221707e-05,
+      "loss": 0.5634,
+      "step": 520
+    },
+    {
+      "epoch": 0.24480369515011546,
+      "grad_norm": 3.634922504425049,
+      "learning_rate": 3.060046189376444e-05,
+      "loss": 0.5372,
+      "step": 530
+    },
+    {
+      "epoch": 0.24942263279445728,
+      "grad_norm": 3.2436106204986572,
+      "learning_rate": 3.117782909930716e-05,
+      "loss": 0.5914,
+      "step": 540
+    },
+    {
+      "epoch": 0.2540415704387991,
+      "grad_norm": 4.555751323699951,
+      "learning_rate": 3.175519630484989e-05,
+      "loss": 0.5767,
+      "step": 550
+    },
+    {
+      "epoch": 0.2586605080831409,
+      "grad_norm": 3.359252452850342,
+      "learning_rate": 3.233256351039261e-05,
+      "loss": 0.5569,
+      "step": 560
+    },
+    {
+      "epoch": 0.2632794457274827,
+      "grad_norm": 3.362272262573242,
+      "learning_rate": 3.290993071593534e-05,
+      "loss": 0.6248,
+      "step": 570
+    },
+    {
+      "epoch": 0.2678983833718245,
+      "grad_norm": 3.1555299758911133,
+      "learning_rate": 3.348729792147806e-05,
+      "loss": 0.5629,
+      "step": 580
+    },
+    {
+      "epoch": 0.27251732101616627,
+      "grad_norm": 4.786733627319336,
+      "learning_rate": 3.4064665127020787e-05,
+      "loss": 0.5942,
+      "step": 590
+    },
+    {
+      "epoch": 0.27713625866050806,
+      "grad_norm": 4.290501117706299,
+      "learning_rate": 3.464203233256351e-05,
+      "loss": 0.5694,
+      "step": 600
+    },
+    {
+      "epoch": 0.2817551963048499,
+      "grad_norm": 2.2575786113739014,
+      "learning_rate": 3.5219399538106236e-05,
+      "loss": 0.5459,
+      "step": 610
+    },
+    {
+      "epoch": 0.2863741339491917,
+      "grad_norm": 4.7077484130859375,
+      "learning_rate": 3.579676674364896e-05,
+      "loss": 0.5686,
+      "step": 620
+    },
+    {
+      "epoch": 0.2909930715935335,
+      "grad_norm": 3.487034559249878,
+      "learning_rate": 3.6374133949191685e-05,
+      "loss": 0.5185,
+      "step": 630
+    },
+    {
+      "epoch": 0.2956120092378753,
+      "grad_norm": 4.230603218078613,
+      "learning_rate": 3.6951501154734416e-05,
+      "loss": 0.5548,
+      "step": 640
+    },
+    {
+      "epoch": 0.3002309468822171,
+      "grad_norm": 3.5367519855499268,
+      "learning_rate": 3.7528868360277134e-05,
+      "loss": 0.5658,
+      "step": 650
+    },
+    {
+      "epoch": 0.30484988452655887,
+      "grad_norm": 3.3856465816497803,
+      "learning_rate": 3.8106235565819866e-05,
+      "loss": 0.5459,
+      "step": 660
+    },
+    {
+      "epoch": 0.3094688221709007,
+      "grad_norm": 3.4272944927215576,
+      "learning_rate": 3.868360277136259e-05,
+      "loss": 0.5823,
+      "step": 670
+    },
+    {
+      "epoch": 0.3140877598152425,
+      "grad_norm": 2.7848410606384277,
+      "learning_rate": 3.9260969976905315e-05,
+      "loss": 0.5481,
+      "step": 680
+    },
+    {
+      "epoch": 0.3187066974595843,
+      "grad_norm": 2.8584043979644775,
+      "learning_rate": 3.983833718244804e-05,
+      "loss": 0.4907,
+      "step": 690
+    },
+    {
+      "epoch": 0.3233256351039261,
+      "grad_norm": 3.9872729778289795,
+      "learning_rate": 4.0415704387990764e-05,
+      "loss": 0.5703,
+      "step": 700
+    },
+    {
+      "epoch": 0.3279445727482679,
+      "grad_norm": 4.052183151245117,
+      "learning_rate": 4.099307159353349e-05,
+      "loss": 0.6229,
+      "step": 710
+    },
+    {
+      "epoch": 0.3325635103926097,
+      "grad_norm": 3.6171059608459473,
+      "learning_rate": 4.1570438799076213e-05,
+      "loss": 0.5022,
+      "step": 720
+    },
+    {
+      "epoch": 0.3371824480369515,
+      "grad_norm": 3.4360239505767822,
+      "learning_rate": 4.214780600461894e-05,
+      "loss": 0.5642,
+      "step": 730
+    },
+    {
+      "epoch": 0.3418013856812933,
+      "grad_norm": 3.328563928604126,
+      "learning_rate": 4.272517321016167e-05,
+      "loss": 0.543,
+      "step": 740
+    },
+    {
+      "epoch": 0.3464203233256351,
+      "grad_norm": 2.494722843170166,
+      "learning_rate": 4.330254041570439e-05,
+      "loss": 0.5056,
+      "step": 750
+    },
+    {
+      "epoch": 0.3510392609699769,
+      "grad_norm": 2.9856996536254883,
+      "learning_rate": 4.387990762124711e-05,
+      "loss": 0.5215,
+      "step": 760
+    },
+    {
+      "epoch": 0.3556581986143187,
+      "grad_norm": 3.6510608196258545,
+      "learning_rate": 4.445727482678984e-05,
+      "loss": 0.5418,
+      "step": 770
+    },
+    {
+      "epoch": 0.36027713625866054,
+      "grad_norm": 2.837355136871338,
+      "learning_rate": 4.503464203233256e-05,
+      "loss": 0.5197,
+      "step": 780
+    },
+    {
+      "epoch": 0.3648960739030023,
+      "grad_norm": 2.4272849559783936,
+      "learning_rate": 4.561200923787529e-05,
+      "loss": 0.5151,
+      "step": 790
+    },
+    {
+      "epoch": 0.3695150115473441,
+      "grad_norm": 2.2198147773742676,
+      "learning_rate": 4.618937644341802e-05,
+      "loss": 0.5037,
+      "step": 800
+    },
+    {
+      "epoch": 0.3741339491916859,
+      "grad_norm": 3.082343816757202,
+      "learning_rate": 4.676674364896074e-05,
+      "loss": 0.5411,
+      "step": 810
+    },
+    {
+      "epoch": 0.3787528868360277,
+      "grad_norm": 2.739429235458374,
+      "learning_rate": 4.7344110854503466e-05,
+      "loss": 0.5034,
+      "step": 820
+    },
+    {
+      "epoch": 0.3833718244803695,
+      "grad_norm": 2.8211052417755127,
+      "learning_rate": 4.792147806004619e-05,
+      "loss": 0.5077,
+      "step": 830
+    },
+    {
+      "epoch": 0.38799076212471134,
+      "grad_norm": 4.462055683135986,
+      "learning_rate": 4.8498845265588916e-05,
+      "loss": 0.4883,
+      "step": 840
+    },
+    {
+      "epoch": 0.39260969976905313,
+      "grad_norm": 2.8090898990631104,
+      "learning_rate": 4.907621247113165e-05,
+      "loss": 0.4918,
+      "step": 850
+    },
+    {
+      "epoch": 0.3972286374133949,
+      "grad_norm": 2.787473440170288,
+      "learning_rate": 4.9653579676674365e-05,
+      "loss": 0.4736,
+      "step": 860
+    },
+    {
+      "epoch": 0.4018475750577367,
+      "grad_norm": 3.198763132095337,
+      "learning_rate": 4.999996750557592e-05,
+      "loss": 0.5243,
+      "step": 870
+    },
+    {
+      "epoch": 0.4064665127020785,
+      "grad_norm": 3.8778421878814697,
+      "learning_rate": 4.9999601944275146e-05,
+      "loss": 0.5206,
+      "step": 880
+    },
+    {
+      "epoch": 0.4110854503464203,
+      "grad_norm": 4.326559066772461,
+      "learning_rate": 4.999883020960263e-05,
+      "loss": 0.5998,
+      "step": 890
+    },
+    {
+      "epoch": 0.41570438799076215,
+      "grad_norm": 2.1009411811828613,
+      "learning_rate": 4.99976523140969e-05,
+      "loss": 0.5097,
+      "step": 900
+    },
+    {
+      "epoch": 0.42032332563510394,
+      "grad_norm": 3.0838584899902344,
+      "learning_rate": 4.999606827689545e-05,
+      "loss": 0.4699,
+      "step": 910
+    },
+    {
+      "epoch": 0.42494226327944573,
+      "grad_norm": 3.067430019378662,
+      "learning_rate": 4.999407812373445e-05,
+      "loss": 0.5223,
+      "step": 920
+    },
+    {
+      "epoch": 0.4295612009237875,
+      "grad_norm": 2.580418109893799,
+      "learning_rate": 4.9991681886948297e-05,
+      "loss": 0.5396,
+      "step": 930
+    },
+    {
+      "epoch": 0.4341801385681293,
+      "grad_norm": 2.748032808303833,
+      "learning_rate": 4.9988879605469106e-05,
+      "loss": 0.4965,
+      "step": 940
+    },
+    {
+      "epoch": 0.4387990762124711,
+      "grad_norm": 2.841571569442749,
+      "learning_rate": 4.99856713248261e-05,
+      "loss": 0.472,
+      "step": 950
+    },
+    {
+      "epoch": 0.44341801385681295,
+      "grad_norm": 1.8557236194610596,
+      "learning_rate": 4.9982057097144826e-05,
+      "loss": 0.4878,
+      "step": 960
+    },
+    {
+      "epoch": 0.44803695150115475,
+      "grad_norm": 5.707876205444336,
+      "learning_rate": 4.9978036981146344e-05,
+      "loss": 0.4883,
+      "step": 970
+    },
+    {
+      "epoch": 0.45265588914549654,
+      "grad_norm": 3.867081880569458,
+      "learning_rate": 4.9973611042146254e-05,
+      "loss": 0.4929,
+      "step": 980
+    },
+    {
+      "epoch": 0.45727482678983833,
+      "grad_norm": 2.291074275970459,
+      "learning_rate": 4.996877935205365e-05,
+      "loss": 0.5046,
+      "step": 990
+    },
+    {
+      "epoch": 0.4618937644341801,
+      "grad_norm": 2.332185745239258,
+      "learning_rate": 4.996354198936993e-05,
+      "loss": 0.4539,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4665127020785219,
+      "grad_norm": 3.7381327152252197,
+      "learning_rate": 4.995789903918755e-05,
+      "loss": 0.5296,
+      "step": 1010
+    },
+    {
+      "epoch": 0.47113163972286376,
+      "grad_norm": 2.239875078201294,
+      "learning_rate": 4.99518505931886e-05,
+      "loss": 0.5004,
+      "step": 1020
+    },
+    {
+      "epoch": 0.47575057736720555,
+      "grad_norm": 3.7051751613616943,
+      "learning_rate": 4.9945396749643356e-05,
+      "loss": 0.5319,
+      "step": 1030
+    },
+    {
+      "epoch": 0.48036951501154734,
+      "grad_norm": 2.5135374069213867,
+      "learning_rate": 4.993853761340867e-05,
+      "loss": 0.4952,
+      "step": 1040
+    },
+    {
+      "epoch": 0.48498845265588914,
+      "grad_norm": 2.6355159282684326,
+      "learning_rate": 4.993127329592625e-05,
+      "loss": 0.4807,
+      "step": 1050
+    },
+    {
+      "epoch": 0.4896073903002309,
+      "grad_norm": 5.0833563804626465,
+      "learning_rate": 4.9923603915220874e-05,
+      "loss": 0.5269,
+      "step": 1060
+    },
+    {
+      "epoch": 0.4942263279445728,
+      "grad_norm": 2.4188873767852783,
+      "learning_rate": 4.9915529595898445e-05,
+      "loss": 0.5183,
+      "step": 1070
+    },
+    {
+      "epoch": 0.49884526558891457,
+      "grad_norm": 2.623342514038086,
+      "learning_rate": 4.9907050469144e-05,
+      "loss": 0.4928,
+      "step": 1080
+    },
+    {
+      "epoch": 0.5034642032332564,
+      "grad_norm": 2.629373073577881,
+      "learning_rate": 4.9898166672719547e-05,
+      "loss": 0.5144,
+      "step": 1090
+    },
+    {
+      "epoch": 0.5080831408775982,
+      "grad_norm": 1.6594828367233276,
+      "learning_rate": 4.9888878350961857e-05,
+      "loss": 0.4733,
+      "step": 1100
+    },
+    {
+      "epoch": 0.5127020785219399,
+      "grad_norm": 2.773136854171753,
+      "learning_rate": 4.987918565478008e-05,
+      "loss": 0.5054,
+      "step": 1110
+    },
+    {
+      "epoch": 0.5173210161662818,
+      "grad_norm": 1.9525351524353027,
+      "learning_rate": 4.9869088741653335e-05,
+      "loss": 0.4454,
+      "step": 1120
+    },
+    {
+      "epoch": 0.5219399538106235,
+      "grad_norm": 2.84041166305542,
+      "learning_rate": 4.985858777562812e-05,
+      "loss": 0.4871,
+      "step": 1130
+    },
+    {
+      "epoch": 0.5265588914549654,
+      "grad_norm": 2.2563703060150146,
+      "learning_rate": 4.9847682927315656e-05,
+      "loss": 0.4605,
+      "step": 1140
+    },
+    {
+      "epoch": 0.5311778290993071,
+      "grad_norm": 3.4986164569854736,
+      "learning_rate": 4.983637437388913e-05,
+      "loss": 0.5281,
+      "step": 1150
+    },
+    {
+      "epoch": 0.535796766743649,
+      "grad_norm": 2.7081637382507324,
+      "learning_rate": 4.98246622990808e-05,
+      "loss": 0.4568,
+      "step": 1160
+    },
+    {
+      "epoch": 0.5404157043879908,
+      "grad_norm": 3.4630649089813232,
+      "learning_rate": 4.981254689317902e-05,
+      "loss": 0.5351,
+      "step": 1170
+    },
+    {
+      "epoch": 0.5450346420323325,
+      "grad_norm": 2.5266454219818115,
+      "learning_rate": 4.9800028353025125e-05,
+      "loss": 0.4378,
+      "step": 1180
+    },
+    {
+      "epoch": 0.5496535796766744,
+      "grad_norm": 2.003904104232788,
+      "learning_rate": 4.978710688201026e-05,
+      "loss": 0.4539,
+      "step": 1190
+    },
+    {
+      "epoch": 0.5542725173210161,
+      "grad_norm": 3.799851655960083,
+      "learning_rate": 4.9773782690072055e-05,
+      "loss": 0.5349,
+      "step": 1200
+    },
+    {
+      "epoch": 0.558891454965358,
+      "grad_norm": 2.7120118141174316,
+      "learning_rate": 4.976005599369126e-05,
+      "loss": 0.4938,
+      "step": 1210
+    },
+    {
+      "epoch": 0.5635103926096998,
+      "grad_norm": 2.687408208847046,
+      "learning_rate": 4.974592701588816e-05,
+      "loss": 0.4473,
+      "step": 1220
+    },
+    {
+      "epoch": 0.5681293302540416,
+      "grad_norm": 2.5710184574127197,
+      "learning_rate": 4.9731395986218976e-05,
+      "loss": 0.4854,
+      "step": 1230
+    },
+    {
+      "epoch": 0.5727482678983834,
+      "grad_norm": 3.5843756198883057,
+      "learning_rate": 4.9716463140772176e-05,
+      "loss": 0.5108,
+      "step": 1240
+    },
+    {
+      "epoch": 0.5773672055427251,
+      "grad_norm": 3.0808424949645996,
+      "learning_rate": 4.970112872216459e-05,
+      "loss": 0.4664,
+      "step": 1250
+    },
+    {
+      "epoch": 0.581986143187067,
+      "grad_norm": 2.7078258991241455,
+      "learning_rate": 4.968539297953748e-05,
+      "loss": 0.5728,
+      "step": 1260
+    },
+    {
+      "epoch": 0.5866050808314087,
+      "grad_norm": 2.1983377933502197,
+      "learning_rate": 4.96692561685525e-05,
+      "loss": 0.4551,
+      "step": 1270
+    },
+    {
+      "epoch": 0.5912240184757506,
+      "grad_norm": 3.719346046447754,
+      "learning_rate": 4.9652718551387545e-05,
+      "loss": 0.4358,
+      "step": 1280
+    },
+    {
+      "epoch": 0.5958429561200924,
+      "grad_norm": 3.773397207260132,
+      "learning_rate": 4.963578039673248e-05,
+      "loss": 0.3883,
+      "step": 1290
+    },
+    {
+      "epoch": 0.6004618937644342,
+      "grad_norm": 2.14461350440979,
+      "learning_rate": 4.961844197978479e-05,
+      "loss": 0.4403,
+      "step": 1300
+    },
+    {
+      "epoch": 0.605080831408776,
+      "grad_norm": 3.8363335132598877,
+      "learning_rate": 4.960070358224508e-05,
+      "loss": 0.4753,
+      "step": 1310
+    },
+    {
+      "epoch": 0.6096997690531177,
+      "grad_norm": 2.10432505607605,
+      "learning_rate": 4.9582565492312537e-05,
+      "loss": 0.4728,
+      "step": 1320
+    },
+    {
+      "epoch": 0.6143187066974596,
+      "grad_norm": 2.5199790000915527,
+      "learning_rate": 4.9564028004680215e-05,
+      "loss": 0.4544,
+      "step": 1330
+    },
+    {
+      "epoch": 0.6189376443418014,
+      "grad_norm": 3.8586044311523438,
+      "learning_rate": 4.9545091420530265e-05,
+      "loss": 0.4649,
+      "step": 1340
+    },
+    {
+      "epoch": 0.6235565819861432,
+      "grad_norm": 1.9125807285308838,
+      "learning_rate": 4.9525756047529023e-05,
+      "loss": 0.4609,
+      "step": 1350
+    },
+    {
+      "epoch": 0.628175519630485,
+      "grad_norm": 2.1655430793762207,
+      "learning_rate": 4.950602219982207e-05,
+      "loss": 0.4832,
+      "step": 1360
+    },
+    {
+      "epoch": 0.6327944572748267,
+      "grad_norm": 4.029664993286133,
+      "learning_rate": 4.948589019802901e-05,
+      "loss": 0.4751,
+      "step": 1370
+    },
+    {
+      "epoch": 0.6374133949191686,
+      "grad_norm": 2.2781596183776855,
+      "learning_rate": 4.94653603692384e-05,
+      "loss": 0.4691,
+      "step": 1380
+    },
+    {
+      "epoch": 0.6420323325635104,
+      "grad_norm": 1.623131513595581,
+      "learning_rate": 4.9444433047002327e-05,
+      "loss": 0.4397,
+      "step": 1390
+    },
+    {
+      "epoch": 0.6466512702078522,
+      "grad_norm": 2.9153196811676025,
+      "learning_rate": 4.9423108571331055e-05,
+      "loss": 0.4737,
+      "step": 1400
+    },
+    {
+      "epoch": 0.651270207852194,
+      "grad_norm": 3.0765182971954346,
+      "learning_rate": 4.940138728868746e-05,
+      "loss": 0.4329,
+      "step": 1410
+    },
+    {
+      "epoch": 0.6558891454965358,
+      "grad_norm": 3.2848074436187744,
+      "learning_rate": 4.937926955198144e-05,
+      "loss": 0.4488,
+      "step": 1420
+    },
+    {
+      "epoch": 0.6605080831408776,
+      "grad_norm": 2.2458810806274414,
+      "learning_rate": 4.935675572056413e-05,
+      "loss": 0.4645,
+      "step": 1430
+    },
+    {
+      "epoch": 0.6651270207852193,
+      "grad_norm": 2.3431642055511475,
+      "learning_rate": 4.933384616022211e-05,
+      "loss": 0.4492,
+      "step": 1440
+    },
+    {
+      "epoch": 0.6697459584295612,
+      "grad_norm": 2.6324145793914795,
+      "learning_rate": 4.931054124317144e-05,
+      "loss": 0.4978,
+      "step": 1450
+    },
+    {
+      "epoch": 0.674364896073903,
+      "grad_norm": 2.575723171234131,
+      "learning_rate": 4.928684134805162e-05,
+      "loss": 0.4442,
+      "step": 1460
+    },
+    {
+      "epoch": 0.6789838337182448,
+      "grad_norm": 2.842388868331909,
+      "learning_rate": 4.926274685991944e-05,
+      "loss": 0.4412,
+      "step": 1470
+    },
+    {
+      "epoch": 0.6836027713625866,
+      "grad_norm": 1.8333570957183838,
+      "learning_rate": 4.9238258170242703e-05,
+      "loss": 0.4655,
+      "step": 1480
+    },
+    {
+      "epoch": 0.6882217090069284,
+      "grad_norm": 2.276472806930542,
+      "learning_rate": 4.921337567689389e-05,
+      "loss": 0.3944,
+      "step": 1490
+    },
+    {
+      "epoch": 0.6928406466512702,
+      "grad_norm": 2.947540760040283,
+      "learning_rate": 4.91880997841437e-05,
+      "loss": 0.4545,
+      "step": 1500
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 8660,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 4.102925424714056e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-1500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5634a4aea40f26a4bd395cce99d40e5d4ac38396a87a690ea8534fccfab25f6d
+size 5368

checkpoint-2000/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /root/autodl-tmp/models/Mistral-7B-Instruct-v0.2
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1

checkpoint-2000/adapter_config.json

@@ -0,0 +1,34 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "/root/autodl-tmp/models/Mistral-7B-Instruct-v0.2",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.0,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "down_proj",
+    "up_proj",
+    "k_proj",
+    "gate_proj",
+    "v_proj",
+    "o_proj",
+    "q_proj"
+  ],
+  "task_type": "CAUSAL_LM",
+  "use_dora": false,
+  "use_rslora": false
+}

checkpoint-2000/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b22ab5c22f500d3d71337c7bceb90eac951c96e6e5116541030594093db1aab7
+size 167832240

checkpoint-2000/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9623aaaa78b9a770c2b177467395638f1776ccbeb9da9d2bdaf286cc0c76b3c9
+size 335922386

checkpoint-2000/rng_state_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:92cc13315f24c28015d695b6cde08bb1cd6fea4cbc435998485ed6fbe4c91285
+size 15024

checkpoint-2000/rng_state_1.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4c154b6a63e0b1f98f7d2847944398f99f1657d35e8eddf7fdf0ae2c24b0552
+size 15024

checkpoint-2000/rng_state_2.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f784c6a9507b51189f2caffbd178ea9882103b75852e31c15f47fdae6a43af1d
+size 15024

checkpoint-2000/rng_state_3.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:34b023e05bc2d12b91dc436d4922b990d50ec8dc56d40dc3e36b3bb34fc81341
+size 15024

checkpoint-2000/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:17ade1ad9d097a5863289752de4dc1a8aab331efbf90c2db0fd9c65c55def3b9
+size 1064

checkpoint-2000/special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

checkpoint-2000/tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dadfd56d766715c61d2ef780a525ab43b8e6da4de6865bda3d95fdef5e134055
+size 493443

checkpoint-2000/tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "'<s>'{% if messages[0]['role'] == 'system' %}{% set system_message = messages[0]['content'] %}{% endif %}{% if system_message is defined %}{{ system_message }}{% endif %}{% for message in messages %}{% set content = message['content'] %}{% if message['role'] == 'user' %}{{ '[INST] ' + content + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ content + '</s>' }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "</s>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "split_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}

checkpoint-2000/trainer_state.json

@@ -0,0 +1,1433 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.9237875288683602,
+  "eval_steps": 500,
+  "global_step": 2000,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.004618937644341801,
+      "grad_norm": 11.009225845336914,
+      "learning_rate": 5.773672055427253e-07,
+      "loss": 2.4977,
+      "step": 10
+    },
+    {
+      "epoch": 0.009237875288683603,
+      "grad_norm": 9.019889831542969,
+      "learning_rate": 1.1547344110854505e-06,
+      "loss": 2.7655,
+      "step": 20
+    },
+    {
+      "epoch": 0.013856812933025405,
+      "grad_norm": 11.212933540344238,
+      "learning_rate": 1.7321016166281756e-06,
+      "loss": 2.7241,
+      "step": 30
+    },
+    {
+      "epoch": 0.018475750577367205,
+      "grad_norm": 9.982237815856934,
+      "learning_rate": 2.309468822170901e-06,
+      "loss": 2.8357,
+      "step": 40
+    },
+    {
+      "epoch": 0.023094688221709007,
+      "grad_norm": 16.251842498779297,
+      "learning_rate": 2.886836027713626e-06,
+      "loss": 2.1159,
+      "step": 50
+    },
+    {
+      "epoch": 0.02771362586605081,
+      "grad_norm": 7.950500011444092,
+      "learning_rate": 3.464203233256351e-06,
+      "loss": 1.498,
+      "step": 60
+    },
+    {
+      "epoch": 0.03233256351039261,
+      "grad_norm": 4.816328048706055,
+      "learning_rate": 4.041570438799077e-06,
+      "loss": 0.9835,
+      "step": 70
+    },
+    {
+      "epoch": 0.03695150115473441,
+      "grad_norm": 9.529770851135254,
+      "learning_rate": 4.618937644341802e-06,
+      "loss": 1.0009,
+      "step": 80
+    },
+    {
+      "epoch": 0.04157043879907621,
+      "grad_norm": 12.515681266784668,
+      "learning_rate": 5.196304849884527e-06,
+      "loss": 1.0264,
+      "step": 90
+    },
+    {
+      "epoch": 0.046189376443418015,
+      "grad_norm": 5.393396854400635,
+      "learning_rate": 5.773672055427252e-06,
+      "loss": 0.8218,
+      "step": 100
+    },
+    {
+      "epoch": 0.050808314087759814,
+      "grad_norm": 2.777273416519165,
+      "learning_rate": 6.351039260969978e-06,
+      "loss": 0.8719,
+      "step": 110
+    },
+    {
+      "epoch": 0.05542725173210162,
+      "grad_norm": 3.8990070819854736,
+      "learning_rate": 6.928406466512702e-06,
+      "loss": 0.8592,
+      "step": 120
+    },
+    {
+      "epoch": 0.06004618937644342,
+      "grad_norm": 4.342905521392822,
+      "learning_rate": 7.505773672055427e-06,
+      "loss": 0.7982,
+      "step": 130
+    },
+    {
+      "epoch": 0.06466512702078522,
+      "grad_norm": 2.4967517852783203,
+      "learning_rate": 8.083140877598153e-06,
+      "loss": 0.7203,
+      "step": 140
+    },
+    {
+      "epoch": 0.06928406466512702,
+      "grad_norm": 5.491476058959961,
+      "learning_rate": 8.660508083140878e-06,
+      "loss": 0.7647,
+      "step": 150
+    },
+    {
+      "epoch": 0.07390300230946882,
+      "grad_norm": 4.13068962097168,
+      "learning_rate": 9.237875288683604e-06,
+      "loss": 0.7378,
+      "step": 160
+    },
+    {
+      "epoch": 0.07852193995381063,
+      "grad_norm": 3.9301445484161377,
+      "learning_rate": 9.815242494226329e-06,
+      "loss": 0.7355,
+      "step": 170
+    },
+    {
+      "epoch": 0.08314087759815242,
+      "grad_norm": 2.891854763031006,
+      "learning_rate": 1.0392609699769053e-05,
+      "loss": 0.6911,
+      "step": 180
+    },
+    {
+      "epoch": 0.08775981524249422,
+      "grad_norm": 5.377829074859619,
+      "learning_rate": 1.0969976905311778e-05,
+      "loss": 0.7972,
+      "step": 190
+    },
+    {
+      "epoch": 0.09237875288683603,
+      "grad_norm": 3.1744544506073,
+      "learning_rate": 1.1547344110854504e-05,
+      "loss": 0.6792,
+      "step": 200
+    },
+    {
+      "epoch": 0.09699769053117784,
+      "grad_norm": 4.408601760864258,
+      "learning_rate": 1.2124711316397229e-05,
+      "loss": 0.7378,
+      "step": 210
+    },
+    {
+      "epoch": 0.10161662817551963,
+      "grad_norm": 3.3633627891540527,
+      "learning_rate": 1.2702078521939955e-05,
+      "loss": 0.7131,
+      "step": 220
+    },
+    {
+      "epoch": 0.10623556581986143,
+      "grad_norm": 5.786050319671631,
+      "learning_rate": 1.3279445727482678e-05,
+      "loss": 0.6806,
+      "step": 230
+    },
+    {
+      "epoch": 0.11085450346420324,
+      "grad_norm": 6.865295886993408,
+      "learning_rate": 1.3856812933025404e-05,
+      "loss": 0.7117,
+      "step": 240
+    },
+    {
+      "epoch": 0.11547344110854503,
+      "grad_norm": 3.714210033416748,
+      "learning_rate": 1.4434180138568129e-05,
+      "loss": 0.6354,
+      "step": 250
+    },
+    {
+      "epoch": 0.12009237875288684,
+      "grad_norm": 5.501813888549805,
+      "learning_rate": 1.5011547344110854e-05,
+      "loss": 0.6266,
+      "step": 260
+    },
+    {
+      "epoch": 0.12471131639722864,
+      "grad_norm": 6.622759819030762,
+      "learning_rate": 1.558891454965358e-05,
+      "loss": 0.6894,
+      "step": 270
+    },
+    {
+      "epoch": 0.12933025404157045,
+      "grad_norm": 3.647956609725952,
+      "learning_rate": 1.6166281755196306e-05,
+      "loss": 0.66,
+      "step": 280
+    },
+    {
+      "epoch": 0.13394919168591224,
+      "grad_norm": 4.103866100311279,
+      "learning_rate": 1.674364896073903e-05,
+      "loss": 0.665,
+      "step": 290
+    },
+    {
+      "epoch": 0.13856812933025403,
+      "grad_norm": 3.84169340133667,
+      "learning_rate": 1.7321016166281756e-05,
+      "loss": 0.6676,
+      "step": 300
+    },
+    {
+      "epoch": 0.14318706697459585,
+      "grad_norm": 8.042781829833984,
+      "learning_rate": 1.789838337182448e-05,
+      "loss": 0.6236,
+      "step": 310
+    },
+    {
+      "epoch": 0.14780600461893764,
+      "grad_norm": 4.35403299331665,
+      "learning_rate": 1.8475750577367208e-05,
+      "loss": 0.6283,
+      "step": 320
+    },
+    {
+      "epoch": 0.15242494226327943,
+      "grad_norm": 3.7114272117614746,
+      "learning_rate": 1.9053117782909933e-05,
+      "loss": 0.6472,
+      "step": 330
+    },
+    {
+      "epoch": 0.15704387990762125,
+      "grad_norm": 2.5825653076171875,
+      "learning_rate": 1.9630484988452657e-05,
+      "loss": 0.6139,
+      "step": 340
+    },
+    {
+      "epoch": 0.16166281755196305,
+      "grad_norm": 4.71831750869751,
+      "learning_rate": 2.0207852193995382e-05,
+      "loss": 0.6019,
+      "step": 350
+    },
+    {
+      "epoch": 0.16628175519630484,
+      "grad_norm": 7.264115333557129,
+      "learning_rate": 2.0785219399538107e-05,
+      "loss": 0.6484,
+      "step": 360
+    },
+    {
+      "epoch": 0.17090069284064666,
+      "grad_norm": 5.484978199005127,
+      "learning_rate": 2.1362586605080835e-05,
+      "loss": 0.6423,
+      "step": 370
+    },
+    {
+      "epoch": 0.17551963048498845,
+      "grad_norm": 4.004003524780273,
+      "learning_rate": 2.1939953810623556e-05,
+      "loss": 0.6167,
+      "step": 380
+    },
+    {
+      "epoch": 0.18013856812933027,
+      "grad_norm": 3.858074426651001,
+      "learning_rate": 2.251732101616628e-05,
+      "loss": 0.6029,
+      "step": 390
+    },
+    {
+      "epoch": 0.18475750577367206,
+      "grad_norm": 4.84448766708374,
+      "learning_rate": 2.309468822170901e-05,
+      "loss": 0.5511,
+      "step": 400
+    },
+    {
+      "epoch": 0.18937644341801385,
+      "grad_norm": 3.419329881668091,
+      "learning_rate": 2.3672055427251733e-05,
+      "loss": 0.5919,
+      "step": 410
+    },
+    {
+      "epoch": 0.19399538106235567,
+      "grad_norm": 2.367598295211792,
+      "learning_rate": 2.4249422632794458e-05,
+      "loss": 0.5407,
+      "step": 420
+    },
+    {
+      "epoch": 0.19861431870669746,
+      "grad_norm": 4.016935348510742,
+      "learning_rate": 2.4826789838337182e-05,
+      "loss": 0.5829,
+      "step": 430
+    },
+    {
+      "epoch": 0.20323325635103925,
+      "grad_norm": 4.788145065307617,
+      "learning_rate": 2.540415704387991e-05,
+      "loss": 0.593,
+      "step": 440
+    },
+    {
+      "epoch": 0.20785219399538107,
+      "grad_norm": 3.618800401687622,
+      "learning_rate": 2.5981524249422635e-05,
+      "loss": 0.56,
+      "step": 450
+    },
+    {
+      "epoch": 0.21247113163972287,
+      "grad_norm": 3.6879987716674805,
+      "learning_rate": 2.6558891454965356e-05,
+      "loss": 0.5448,
+      "step": 460
+    },
+    {
+      "epoch": 0.21709006928406466,
+      "grad_norm": 3.6736364364624023,
+      "learning_rate": 2.7136258660508084e-05,
+      "loss": 0.5707,
+      "step": 470
+    },
+    {
+      "epoch": 0.22170900692840648,
+      "grad_norm": 3.954604387283325,
+      "learning_rate": 2.771362586605081e-05,
+      "loss": 0.5891,
+      "step": 480
+    },
+    {
+      "epoch": 0.22632794457274827,
+      "grad_norm": 6.050353050231934,
+      "learning_rate": 2.8290993071593537e-05,
+      "loss": 0.6231,
+      "step": 490
+    },
+    {
+      "epoch": 0.23094688221709006,
+      "grad_norm": 3.728318214416504,
+      "learning_rate": 2.8868360277136258e-05,
+      "loss": 0.6294,
+      "step": 500
+    },
+    {
+      "epoch": 0.23556581986143188,
+      "grad_norm": 5.047093391418457,
+      "learning_rate": 2.9445727482678986e-05,
+      "loss": 0.5795,
+      "step": 510
+    },
+    {
+      "epoch": 0.24018475750577367,
+      "grad_norm": 4.392754077911377,
+      "learning_rate": 3.0023094688221707e-05,
+      "loss": 0.5634,
+      "step": 520
+    },
+    {
+      "epoch": 0.24480369515011546,
+      "grad_norm": 3.634922504425049,
+      "learning_rate": 3.060046189376444e-05,
+      "loss": 0.5372,
+      "step": 530
+    },
+    {
+      "epoch": 0.24942263279445728,
+      "grad_norm": 3.2436106204986572,
+      "learning_rate": 3.117782909930716e-05,
+      "loss": 0.5914,
+      "step": 540
+    },
+    {
+      "epoch": 0.2540415704387991,
+      "grad_norm": 4.555751323699951,
+      "learning_rate": 3.175519630484989e-05,
+      "loss": 0.5767,
+      "step": 550
+    },
+    {
+      "epoch": 0.2586605080831409,
+      "grad_norm": 3.359252452850342,
+      "learning_rate": 3.233256351039261e-05,
+      "loss": 0.5569,
+      "step": 560
+    },
+    {
+      "epoch": 0.2632794457274827,
+      "grad_norm": 3.362272262573242,
+      "learning_rate": 3.290993071593534e-05,
+      "loss": 0.6248,
+      "step": 570
+    },
+    {
+      "epoch": 0.2678983833718245,
+      "grad_norm": 3.1555299758911133,
+      "learning_rate": 3.348729792147806e-05,
+      "loss": 0.5629,
+      "step": 580
+    },
+    {
+      "epoch": 0.27251732101616627,
+      "grad_norm": 4.786733627319336,
+      "learning_rate": 3.4064665127020787e-05,
+      "loss": 0.5942,
+      "step": 590
+    },
+    {
+      "epoch": 0.27713625866050806,
+      "grad_norm": 4.290501117706299,
+      "learning_rate": 3.464203233256351e-05,
+      "loss": 0.5694,
+      "step": 600
+    },
+    {
+      "epoch": 0.2817551963048499,
+      "grad_norm": 2.2575786113739014,
+      "learning_rate": 3.5219399538106236e-05,
+      "loss": 0.5459,
+      "step": 610
+    },
+    {
+      "epoch": 0.2863741339491917,
+      "grad_norm": 4.7077484130859375,
+      "learning_rate": 3.579676674364896e-05,
+      "loss": 0.5686,
+      "step": 620
+    },
+    {
+      "epoch": 0.2909930715935335,
+      "grad_norm": 3.487034559249878,
+      "learning_rate": 3.6374133949191685e-05,
+      "loss": 0.5185,
+      "step": 630
+    },
+    {
+      "epoch": 0.2956120092378753,
+      "grad_norm": 4.230603218078613,
+      "learning_rate": 3.6951501154734416e-05,
+      "loss": 0.5548,
+      "step": 640
+    },
+    {
+      "epoch": 0.3002309468822171,
+      "grad_norm": 3.5367519855499268,
+      "learning_rate": 3.7528868360277134e-05,
+      "loss": 0.5658,
+      "step": 650
+    },
+    {
+      "epoch": 0.30484988452655887,
+      "grad_norm": 3.3856465816497803,
+      "learning_rate": 3.8106235565819866e-05,
+      "loss": 0.5459,
+      "step": 660
+    },
+    {
+      "epoch": 0.3094688221709007,
+      "grad_norm": 3.4272944927215576,
+      "learning_rate": 3.868360277136259e-05,
+      "loss": 0.5823,
+      "step": 670
+    },
+    {
+      "epoch": 0.3140877598152425,
+      "grad_norm": 2.7848410606384277,
+      "learning_rate": 3.9260969976905315e-05,
+      "loss": 0.5481,
+      "step": 680
+    },
+    {
+      "epoch": 0.3187066974595843,
+      "grad_norm": 2.8584043979644775,
+      "learning_rate": 3.983833718244804e-05,
+      "loss": 0.4907,
+      "step": 690
+    },
+    {
+      "epoch": 0.3233256351039261,
+      "grad_norm": 3.9872729778289795,
+      "learning_rate": 4.0415704387990764e-05,
+      "loss": 0.5703,
+      "step": 700
+    },
+    {
+      "epoch": 0.3279445727482679,
+      "grad_norm": 4.052183151245117,
+      "learning_rate": 4.099307159353349e-05,
+      "loss": 0.6229,
+      "step": 710
+    },
+    {
+      "epoch": 0.3325635103926097,
+      "grad_norm": 3.6171059608459473,
+      "learning_rate": 4.1570438799076213e-05,
+      "loss": 0.5022,
+      "step": 720
+    },
+    {
+      "epoch": 0.3371824480369515,
+      "grad_norm": 3.4360239505767822,
+      "learning_rate": 4.214780600461894e-05,
+      "loss": 0.5642,
+      "step": 730
+    },
+    {
+      "epoch": 0.3418013856812933,
+      "grad_norm": 3.328563928604126,
+      "learning_rate": 4.272517321016167e-05,
+      "loss": 0.543,
+      "step": 740
+    },
+    {
+      "epoch": 0.3464203233256351,
+      "grad_norm": 2.494722843170166,
+      "learning_rate": 4.330254041570439e-05,
+      "loss": 0.5056,
+      "step": 750
+    },
+    {
+      "epoch": 0.3510392609699769,
+      "grad_norm": 2.9856996536254883,
+      "learning_rate": 4.387990762124711e-05,
+      "loss": 0.5215,
+      "step": 760
+    },
+    {
+      "epoch": 0.3556581986143187,
+      "grad_norm": 3.6510608196258545,
+      "learning_rate": 4.445727482678984e-05,
+      "loss": 0.5418,
+      "step": 770
+    },
+    {
+      "epoch": 0.36027713625866054,
+      "grad_norm": 2.837355136871338,
+      "learning_rate": 4.503464203233256e-05,
+      "loss": 0.5197,
+      "step": 780
+    },
+    {
+      "epoch": 0.3648960739030023,
+      "grad_norm": 2.4272849559783936,
+      "learning_rate": 4.561200923787529e-05,
+      "loss": 0.5151,
+      "step": 790
+    },
+    {
+      "epoch": 0.3695150115473441,
+      "grad_norm": 2.2198147773742676,
+      "learning_rate": 4.618937644341802e-05,
+      "loss": 0.5037,
+      "step": 800
+    },
+    {
+      "epoch": 0.3741339491916859,
+      "grad_norm": 3.082343816757202,
+      "learning_rate": 4.676674364896074e-05,
+      "loss": 0.5411,
+      "step": 810
+    },
+    {
+      "epoch": 0.3787528868360277,
+      "grad_norm": 2.739429235458374,
+      "learning_rate": 4.7344110854503466e-05,
+      "loss": 0.5034,
+      "step": 820
+    },
+    {
+      "epoch": 0.3833718244803695,
+      "grad_norm": 2.8211052417755127,
+      "learning_rate": 4.792147806004619e-05,
+      "loss": 0.5077,
+      "step": 830
+    },
+    {
+      "epoch": 0.38799076212471134,
+      "grad_norm": 4.462055683135986,
+      "learning_rate": 4.8498845265588916e-05,
+      "loss": 0.4883,
+      "step": 840
+    },
+    {
+      "epoch": 0.39260969976905313,
+      "grad_norm": 2.8090898990631104,
+      "learning_rate": 4.907621247113165e-05,
+      "loss": 0.4918,
+      "step": 850
+    },
+    {
+      "epoch": 0.3972286374133949,
+      "grad_norm": 2.787473440170288,
+      "learning_rate": 4.9653579676674365e-05,
+      "loss": 0.4736,
+      "step": 860
+    },
+    {
+      "epoch": 0.4018475750577367,
+      "grad_norm": 3.198763132095337,
+      "learning_rate": 4.999996750557592e-05,
+      "loss": 0.5243,
+      "step": 870
+    },
+    {
+      "epoch": 0.4064665127020785,
+      "grad_norm": 3.8778421878814697,
+      "learning_rate": 4.9999601944275146e-05,
+      "loss": 0.5206,
+      "step": 880
+    },
+    {
+      "epoch": 0.4110854503464203,
+      "grad_norm": 4.326559066772461,
+      "learning_rate": 4.999883020960263e-05,
+      "loss": 0.5998,
+      "step": 890
+    },
+    {
+      "epoch": 0.41570438799076215,
+      "grad_norm": 2.1009411811828613,
+      "learning_rate": 4.99976523140969e-05,
+      "loss": 0.5097,
+      "step": 900
+    },
+    {
+      "epoch": 0.42032332563510394,
+      "grad_norm": 3.0838584899902344,
+      "learning_rate": 4.999606827689545e-05,
+      "loss": 0.4699,
+      "step": 910
+    },
+    {
+      "epoch": 0.42494226327944573,
+      "grad_norm": 3.067430019378662,
+      "learning_rate": 4.999407812373445e-05,
+      "loss": 0.5223,
+      "step": 920
+    },
+    {
+      "epoch": 0.4295612009237875,
+      "grad_norm": 2.580418109893799,
+      "learning_rate": 4.9991681886948297e-05,
+      "loss": 0.5396,
+      "step": 930
+    },
+    {
+      "epoch": 0.4341801385681293,
+      "grad_norm": 2.748032808303833,
+      "learning_rate": 4.9988879605469106e-05,
+      "loss": 0.4965,
+      "step": 940
+    },
+    {
+      "epoch": 0.4387990762124711,
+      "grad_norm": 2.841571569442749,
+      "learning_rate": 4.99856713248261e-05,
+      "loss": 0.472,
+      "step": 950
+    },
+    {
+      "epoch": 0.44341801385681295,
+      "grad_norm": 1.8557236194610596,
+      "learning_rate": 4.9982057097144826e-05,
+      "loss": 0.4878,
+      "step": 960
+    },
+    {
+      "epoch": 0.44803695150115475,
+      "grad_norm": 5.707876205444336,
+      "learning_rate": 4.9978036981146344e-05,
+      "loss": 0.4883,
+      "step": 970
+    },
+    {
+      "epoch": 0.45265588914549654,
+      "grad_norm": 3.867081880569458,
+      "learning_rate": 4.9973611042146254e-05,
+      "loss": 0.4929,
+      "step": 980
+    },
+    {
+      "epoch": 0.45727482678983833,
+      "grad_norm": 2.291074275970459,
+      "learning_rate": 4.996877935205365e-05,
+      "loss": 0.5046,
+      "step": 990
+    },
+    {
+      "epoch": 0.4618937644341801,
+      "grad_norm": 2.332185745239258,
+      "learning_rate": 4.996354198936993e-05,
+      "loss": 0.4539,
+      "step": 1000
+    },
+    {
+      "epoch": 0.4665127020785219,
+      "grad_norm": 3.7381327152252197,
+      "learning_rate": 4.995789903918755e-05,
+      "loss": 0.5296,
+      "step": 1010
+    },
+    {
+      "epoch": 0.47113163972286376,
+      "grad_norm": 2.239875078201294,
+      "learning_rate": 4.99518505931886e-05,
+      "loss": 0.5004,
+      "step": 1020
+    },
+    {
+      "epoch": 0.47575057736720555,
+      "grad_norm": 3.7051751613616943,
+      "learning_rate": 4.9945396749643356e-05,
+      "loss": 0.5319,
+      "step": 1030
+    },
+    {
+      "epoch": 0.48036951501154734,
+      "grad_norm": 2.5135374069213867,
+      "learning_rate": 4.993853761340867e-05,
+      "loss": 0.4952,
+      "step": 1040
+    },
+    {
+      "epoch": 0.48498845265588914,
+      "grad_norm": 2.6355159282684326,
+      "learning_rate": 4.993127329592625e-05,
+      "loss": 0.4807,
+      "step": 1050
+    },
+    {
+      "epoch": 0.4896073903002309,
+      "grad_norm": 5.0833563804626465,
+      "learning_rate": 4.9923603915220874e-05,
+      "loss": 0.5269,
+      "step": 1060
+    },
+    {
+      "epoch": 0.4942263279445728,
+      "grad_norm": 2.4188873767852783,
+      "learning_rate": 4.9915529595898445e-05,
+      "loss": 0.5183,
+      "step": 1070
+    },
+    {
+      "epoch": 0.49884526558891457,
+      "grad_norm": 2.623342514038086,
+      "learning_rate": 4.9907050469144e-05,
+      "loss": 0.4928,
+      "step": 1080
+    },
+    {
+      "epoch": 0.5034642032332564,
+      "grad_norm": 2.629373073577881,
+      "learning_rate": 4.9898166672719547e-05,
+      "loss": 0.5144,
+      "step": 1090
+    },
+    {
+      "epoch": 0.5080831408775982,
+      "grad_norm": 1.6594828367233276,
+      "learning_rate": 4.9888878350961857e-05,
+      "loss": 0.4733,
+      "step": 1100
+    },
+    {
+      "epoch": 0.5127020785219399,
+      "grad_norm": 2.773136854171753,
+      "learning_rate": 4.987918565478008e-05,
+      "loss": 0.5054,
+      "step": 1110
+    },
+    {
+      "epoch": 0.5173210161662818,
+      "grad_norm": 1.9525351524353027,
+      "learning_rate": 4.9869088741653335e-05,
+      "loss": 0.4454,
+      "step": 1120
+    },
+    {
+      "epoch": 0.5219399538106235,
+      "grad_norm": 2.84041166305542,
+      "learning_rate": 4.985858777562812e-05,
+      "loss": 0.4871,
+      "step": 1130
+    },
+    {
+      "epoch": 0.5265588914549654,
+      "grad_norm": 2.2563703060150146,
+      "learning_rate": 4.9847682927315656e-05,
+      "loss": 0.4605,
+      "step": 1140
+    },
+    {
+      "epoch": 0.5311778290993071,
+      "grad_norm": 3.4986164569854736,
+      "learning_rate": 4.983637437388913e-05,
+      "loss": 0.5281,
+      "step": 1150
+    },
+    {
+      "epoch": 0.535796766743649,
+      "grad_norm": 2.7081637382507324,
+      "learning_rate": 4.98246622990808e-05,
+      "loss": 0.4568,
+      "step": 1160
+    },
+    {
+      "epoch": 0.5404157043879908,
+      "grad_norm": 3.4630649089813232,
+      "learning_rate": 4.981254689317902e-05,
+      "loss": 0.5351,
+      "step": 1170
+    },
+    {
+      "epoch": 0.5450346420323325,
+      "grad_norm": 2.5266454219818115,
+      "learning_rate": 4.9800028353025125e-05,
+      "loss": 0.4378,
+      "step": 1180
+    },
+    {
+      "epoch": 0.5496535796766744,
+      "grad_norm": 2.003904104232788,
+      "learning_rate": 4.978710688201026e-05,
+      "loss": 0.4539,
+      "step": 1190
+    },
+    {
+      "epoch": 0.5542725173210161,
+      "grad_norm": 3.799851655960083,
+      "learning_rate": 4.9773782690072055e-05,
+      "loss": 0.5349,
+      "step": 1200
+    },
+    {
+      "epoch": 0.558891454965358,
+      "grad_norm": 2.7120118141174316,
+      "learning_rate": 4.976005599369126e-05,
+      "loss": 0.4938,
+      "step": 1210
+    },
+    {
+      "epoch": 0.5635103926096998,
+      "grad_norm": 2.687408208847046,
+      "learning_rate": 4.974592701588816e-05,
+      "loss": 0.4473,
+      "step": 1220
+    },
+    {
+      "epoch": 0.5681293302540416,
+      "grad_norm": 2.5710184574127197,
+      "learning_rate": 4.9731395986218976e-05,
+      "loss": 0.4854,
+      "step": 1230
+    },
+    {
+      "epoch": 0.5727482678983834,
+      "grad_norm": 3.5843756198883057,
+      "learning_rate": 4.9716463140772176e-05,
+      "loss": 0.5108,
+      "step": 1240
+    },
+    {
+      "epoch": 0.5773672055427251,
+      "grad_norm": 3.0808424949645996,
+      "learning_rate": 4.970112872216459e-05,
+      "loss": 0.4664,
+      "step": 1250
+    },
+    {
+      "epoch": 0.581986143187067,
+      "grad_norm": 2.7078258991241455,
+      "learning_rate": 4.968539297953748e-05,
+      "loss": 0.5728,
+      "step": 1260
+    },
+    {
+      "epoch": 0.5866050808314087,
+      "grad_norm": 2.1983377933502197,
+      "learning_rate": 4.96692561685525e-05,
+      "loss": 0.4551,
+      "step": 1270
+    },
+    {
+      "epoch": 0.5912240184757506,
+      "grad_norm": 3.719346046447754,
+      "learning_rate": 4.9652718551387545e-05,
+      "loss": 0.4358,
+      "step": 1280
+    },
+    {
+      "epoch": 0.5958429561200924,
+      "grad_norm": 3.773397207260132,
+      "learning_rate": 4.963578039673248e-05,
+      "loss": 0.3883,
+      "step": 1290
+    },
+    {
+      "epoch": 0.6004618937644342,
+      "grad_norm": 2.14461350440979,
+      "learning_rate": 4.961844197978479e-05,
+      "loss": 0.4403,
+      "step": 1300
+    },
+    {
+      "epoch": 0.605080831408776,
+      "grad_norm": 3.8363335132598877,
+      "learning_rate": 4.960070358224508e-05,
+      "loss": 0.4753,
+      "step": 1310
+    },
+    {
+      "epoch": 0.6096997690531177,
+      "grad_norm": 2.10432505607605,
+      "learning_rate": 4.9582565492312537e-05,
+      "loss": 0.4728,
+      "step": 1320
+    },
+    {
+      "epoch": 0.6143187066974596,
+      "grad_norm": 2.5199790000915527,
+      "learning_rate": 4.9564028004680215e-05,
+      "loss": 0.4544,
+      "step": 1330
+    },
+    {
+      "epoch": 0.6189376443418014,
+      "grad_norm": 3.8586044311523438,
+      "learning_rate": 4.9545091420530265e-05,
+      "loss": 0.4649,
+      "step": 1340
+    },
+    {
+      "epoch": 0.6235565819861432,
+      "grad_norm": 1.9125807285308838,
+      "learning_rate": 4.9525756047529023e-05,
+      "loss": 0.4609,
+      "step": 1350
+    },
+    {
+      "epoch": 0.628175519630485,
+      "grad_norm": 2.1655430793762207,
+      "learning_rate": 4.950602219982207e-05,
+      "loss": 0.4832,
+      "step": 1360
+    },
+    {
+      "epoch": 0.6327944572748267,
+      "grad_norm": 4.029664993286133,
+      "learning_rate": 4.948589019802901e-05,
+      "loss": 0.4751,
+      "step": 1370
+    },
+    {
+      "epoch": 0.6374133949191686,
+      "grad_norm": 2.2781596183776855,
+      "learning_rate": 4.94653603692384e-05,
+      "loss": 0.4691,
+      "step": 1380
+    },
+    {
+      "epoch": 0.6420323325635104,
+      "grad_norm": 1.623131513595581,
+      "learning_rate": 4.9444433047002327e-05,
+      "loss": 0.4397,
+      "step": 1390
+    },
+    {
+      "epoch": 0.6466512702078522,
+      "grad_norm": 2.9153196811676025,
+      "learning_rate": 4.9423108571331055e-05,
+      "loss": 0.4737,
+      "step": 1400
+    },
+    {
+      "epoch": 0.651270207852194,
+      "grad_norm": 3.0765182971954346,
+      "learning_rate": 4.940138728868746e-05,
+      "loss": 0.4329,
+      "step": 1410
+    },
+    {
+      "epoch": 0.6558891454965358,
+      "grad_norm": 3.2848074436187744,
+      "learning_rate": 4.937926955198144e-05,
+      "loss": 0.4488,
+      "step": 1420
+    },
+    {
+      "epoch": 0.6605080831408776,
+      "grad_norm": 2.2458810806274414,
+      "learning_rate": 4.935675572056413e-05,
+      "loss": 0.4645,
+      "step": 1430
+    },
+    {
+      "epoch": 0.6651270207852193,
+      "grad_norm": 2.3431642055511475,
+      "learning_rate": 4.933384616022211e-05,
+      "loss": 0.4492,
+      "step": 1440
+    },
+    {
+      "epoch": 0.6697459584295612,
+      "grad_norm": 2.6324145793914795,
+      "learning_rate": 4.931054124317144e-05,
+      "loss": 0.4978,
+      "step": 1450
+    },
+    {
+      "epoch": 0.674364896073903,
+      "grad_norm": 2.575723171234131,
+      "learning_rate": 4.928684134805162e-05,
+      "loss": 0.4442,
+      "step": 1460
+    },
+    {
+      "epoch": 0.6789838337182448,
+      "grad_norm": 2.842388868331909,
+      "learning_rate": 4.926274685991944e-05,
+      "loss": 0.4412,
+      "step": 1470
+    },
+    {
+      "epoch": 0.6836027713625866,
+      "grad_norm": 1.8333570957183838,
+      "learning_rate": 4.9238258170242703e-05,
+      "loss": 0.4655,
+      "step": 1480
+    },
+    {
+      "epoch": 0.6882217090069284,
+      "grad_norm": 2.276472806930542,
+      "learning_rate": 4.921337567689389e-05,
+      "loss": 0.3944,
+      "step": 1490
+    },
+    {
+      "epoch": 0.6928406466512702,
+      "grad_norm": 2.947540760040283,
+      "learning_rate": 4.91880997841437e-05,
+      "loss": 0.4545,
+      "step": 1500
+    },
+    {
+      "epoch": 0.6974595842956121,
+      "grad_norm": 2.092616558074951,
+      "learning_rate": 4.916243090265444e-05,
+      "loss": 0.4202,
+      "step": 1510
+    },
+    {
+      "epoch": 0.7020785219399538,
+      "grad_norm": 2.4105489253997803,
+      "learning_rate": 4.91363694494734e-05,
+      "loss": 0.4488,
+      "step": 1520
+    },
+    {
+      "epoch": 0.7066974595842956,
+      "grad_norm": 2.499821662902832,
+      "learning_rate": 4.910991584802606e-05,
+      "loss": 0.4599,
+      "step": 1530
+    },
+    {
+      "epoch": 0.7113163972286374,
+      "grad_norm": 1.7518564462661743,
+      "learning_rate": 4.9083070528109195e-05,
+      "loss": 0.4469,
+      "step": 1540
+    },
+    {
+      "epoch": 0.7159353348729792,
+      "grad_norm": 1.6236387491226196,
+      "learning_rate": 4.905583392588392e-05,
+      "loss": 0.4756,
+      "step": 1550
+    },
+    {
+      "epoch": 0.7205542725173211,
+      "grad_norm": 3.137925624847412,
+      "learning_rate": 4.902820648386858e-05,
+      "loss": 0.4793,
+      "step": 1560
+    },
+    {
+      "epoch": 0.7251732101616628,
+      "grad_norm": 2.584268093109131,
+      "learning_rate": 4.900018865093158e-05,
+      "loss": 0.4445,
+      "step": 1570
+    },
+    {
+      "epoch": 0.7297921478060047,
+      "grad_norm": 2.1350574493408203,
+      "learning_rate": 4.897178088228406e-05,
+      "loss": 0.4422,
+      "step": 1580
+    },
+    {
+      "epoch": 0.7344110854503464,
+      "grad_norm": 2.1932408809661865,
+      "learning_rate": 4.8942983639472554e-05,
+      "loss": 0.4386,
+      "step": 1590
+    },
+    {
+      "epoch": 0.7390300230946882,
+      "grad_norm": 2.0968682765960693,
+      "learning_rate": 4.891379739037142e-05,
+      "loss": 0.4321,
+      "step": 1600
+    },
+    {
+      "epoch": 0.74364896073903,
+      "grad_norm": 1.718479037284851,
+      "learning_rate": 4.888422260917532e-05,
+      "loss": 0.4422,
+      "step": 1610
+    },
+    {
+      "epoch": 0.7482678983833718,
+      "grad_norm": 2.742122173309326,
+      "learning_rate": 4.885425977639143e-05,
+      "loss": 0.4118,
+      "step": 1620
+    },
+    {
+      "epoch": 0.7528868360277137,
+      "grad_norm": 3.560037136077881,
+      "learning_rate": 4.8823909378831694e-05,
+      "loss": 0.4352,
+      "step": 1630
+    },
+    {
+      "epoch": 0.7575057736720554,
+      "grad_norm": 1.751957654953003,
+      "learning_rate": 4.8793171909604904e-05,
+      "loss": 0.4222,
+      "step": 1640
+    },
+    {
+      "epoch": 0.7621247113163973,
+      "grad_norm": 2.7816951274871826,
+      "learning_rate": 4.876204786810867e-05,
+      "loss": 0.4458,
+      "step": 1650
+    },
+    {
+      "epoch": 0.766743648960739,
+      "grad_norm": 1.7703163623809814,
+      "learning_rate": 4.873053776002133e-05,
+      "loss": 0.4387,
+      "step": 1660
+    },
+    {
+      "epoch": 0.7713625866050808,
+      "grad_norm": 1.7305679321289062,
+      "learning_rate": 4.8698642097293684e-05,
+      "loss": 0.4203,
+      "step": 1670
+    },
+    {
+      "epoch": 0.7759815242494227,
+      "grad_norm": 2.028597116470337,
+      "learning_rate": 4.866636139814076e-05,
+      "loss": 0.4445,
+      "step": 1680
+    },
+    {
+      "epoch": 0.7806004618937644,
+      "grad_norm": 2.1649012565612793,
+      "learning_rate": 4.863369618703331e-05,
+      "loss": 0.4392,
+      "step": 1690
+    },
+    {
+      "epoch": 0.7852193995381063,
+      "grad_norm": 1.9613428115844727,
+      "learning_rate": 4.860064699468935e-05,
+      "loss": 0.4128,
+      "step": 1700
+    },
+    {
+      "epoch": 0.789838337182448,
+      "grad_norm": 1.6601072549819946,
+      "learning_rate": 4.85672143580655e-05,
+      "loss": 0.4211,
+      "step": 1710
+    },
+    {
+      "epoch": 0.7944572748267898,
+      "grad_norm": 1.360724925994873,
+      "learning_rate": 4.85333988203483e-05,
+      "loss": 0.4641,
+      "step": 1720
+    },
+    {
+      "epoch": 0.7990762124711316,
+      "grad_norm": 1.8491480350494385,
+      "learning_rate": 4.84992009309453e-05,
+      "loss": 0.4466,
+      "step": 1730
+    },
+    {
+      "epoch": 0.8036951501154734,
+      "grad_norm": 3.4997713565826416,
+      "learning_rate": 4.8464621245476254e-05,
+      "loss": 0.3786,
+      "step": 1740
+    },
+    {
+      "epoch": 0.8083140877598153,
+      "grad_norm": 1.9356039762496948,
+      "learning_rate": 4.8429660325764e-05,
+      "loss": 0.3948,
+      "step": 1750
+    },
+    {
+      "epoch": 0.812933025404157,
+      "grad_norm": 1.8650673627853394,
+      "learning_rate": 4.8394318739825364e-05,
+      "loss": 0.3733,
+      "step": 1760
+    },
+    {
+      "epoch": 0.8175519630484989,
+      "grad_norm": 2.526487112045288,
+      "learning_rate": 4.835859706186193e-05,
+      "loss": 0.4702,
+      "step": 1770
+    },
+    {
+      "epoch": 0.8221709006928406,
+      "grad_norm": 2.5216803550720215,
+      "learning_rate": 4.832249587225073e-05,
+      "loss": 0.4268,
+      "step": 1780
+    },
+    {
+      "epoch": 0.8267898383371824,
+      "grad_norm": 3.427783966064453,
+      "learning_rate": 4.828601575753476e-05,
+      "loss": 0.4198,
+      "step": 1790
+    },
+    {
+      "epoch": 0.8314087759815243,
+      "grad_norm": 2.734025001525879,
+      "learning_rate": 4.824915731041352e-05,
+      "loss": 0.4305,
+      "step": 1800
+    },
+    {
+      "epoch": 0.836027713625866,
+      "grad_norm": 1.9482347965240479,
+      "learning_rate": 4.821192112973333e-05,
+      "loss": 0.4373,
+      "step": 1810
+    },
+    {
+      "epoch": 0.8406466512702079,
+      "grad_norm": 2.4370124340057373,
+      "learning_rate": 4.817430782047762e-05,
+      "loss": 0.4477,
+      "step": 1820
+    },
+    {
+      "epoch": 0.8452655889145496,
+      "grad_norm": 2.8961851596832275,
+      "learning_rate": 4.8136317993757114e-05,
+      "loss": 0.4611,
+      "step": 1830
+    },
+    {
+      "epoch": 0.8498845265588915,
+      "grad_norm": 2.3259379863739014,
+      "learning_rate": 4.8097952266799876e-05,
+      "loss": 0.4297,
+      "step": 1840
+    },
+    {
+      "epoch": 0.8545034642032333,
+      "grad_norm": 3.403167963027954,
+      "learning_rate": 4.805921126294129e-05,
+      "loss": 0.4028,
+      "step": 1850
+    },
+    {
+      "epoch": 0.859122401847575,
+      "grad_norm": 1.5918769836425781,
+      "learning_rate": 4.802009561161396e-05,
+      "loss": 0.3982,
+      "step": 1860
+    },
+    {
+      "epoch": 0.8637413394919169,
+      "grad_norm": 2.220611333847046,
+      "learning_rate": 4.798060594833743e-05,
+      "loss": 0.4213,
+      "step": 1870
+    },
+    {
+      "epoch": 0.8683602771362586,
+      "grad_norm": 1.634158968925476,
+      "learning_rate": 4.794074291470791e-05,
+      "loss": 0.4085,
+      "step": 1880
+    },
+    {
+      "epoch": 0.8729792147806005,
+      "grad_norm": 2.6168339252471924,
+      "learning_rate": 4.790050715838783e-05,
+      "loss": 0.437,
+      "step": 1890
+    },
+    {
+      "epoch": 0.8775981524249422,
+      "grad_norm": 2.0392565727233887,
+      "learning_rate": 4.7859899333095296e-05,
+      "loss": 0.4407,
+      "step": 1900
+    },
+    {
+      "epoch": 0.8822170900692841,
+      "grad_norm": 2.446300506591797,
+      "learning_rate": 4.7818920098593516e-05,
+      "loss": 0.4415,
+      "step": 1910
+    },
+    {
+      "epoch": 0.8868360277136259,
+      "grad_norm": 2.7988600730895996,
+      "learning_rate": 4.777757012068004e-05,
+      "loss": 0.433,
+      "step": 1920
+    },
+    {
+      "epoch": 0.8914549653579676,
+      "grad_norm": 1.9571830034255981,
+      "learning_rate": 4.773585007117597e-05,
+      "loss": 0.3957,
+      "step": 1930
+    },
+    {
+      "epoch": 0.8960739030023095,
+      "grad_norm": 3.4647884368896484,
+      "learning_rate": 4.7693760627915015e-05,
+      "loss": 0.4442,
+      "step": 1940
+    },
+    {
+      "epoch": 0.9006928406466512,
+      "grad_norm": 2.8123703002929688,
+      "learning_rate": 4.765130247473252e-05,
+      "loss": 0.502,
+      "step": 1950
+    },
+    {
+      "epoch": 0.9053117782909931,
+      "grad_norm": 2.4884793758392334,
+      "learning_rate": 4.760847630145431e-05,
+      "loss": 0.4532,
+      "step": 1960
+    },
+    {
+      "epoch": 0.9099307159353349,
+      "grad_norm": 2.372985601425171,
+      "learning_rate": 4.756528280388551e-05,
+      "loss": 0.3951,
+      "step": 1970
+    },
+    {
+      "epoch": 0.9145496535796767,
+      "grad_norm": 2.322315216064453,
+      "learning_rate": 4.752172268379923e-05,
+      "loss": 0.4442,
+      "step": 1980
+    },
+    {
+      "epoch": 0.9191685912240185,
+      "grad_norm": 2.875863552093506,
+      "learning_rate": 4.7477796648925186e-05,
+      "loss": 0.4512,
+      "step": 1990
+    },
+    {
+      "epoch": 0.9237875288683602,
+      "grad_norm": 2.0720949172973633,
+      "learning_rate": 4.743350541293817e-05,
+      "loss": 0.4076,
+      "step": 2000
+    }
+  ],
+  "logging_steps": 10,
+  "max_steps": 8660,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 4,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 5.4846778836097434e+17,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

checkpoint-2000/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5634a4aea40f26a4bd395cce99d40e5d4ac38396a87a690ea8534fccfab25f6d
+size 5368

checkpoint-2500/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: /root/autodl-tmp/models/Mistral-7B-Instruct-v0.2
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.11.1