End of training

README.md

@@ -0,0 +1,84 @@
+---
+base_model: gpt2
+library_name: distily
+license: mit
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_gpt2_batch_size
+  results: []
+---
+
+# distily_bench_gpt2_batch_size
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- eval_enwikippl: 929.2385
+- eval_frwikippl: 4940.1445
+- eval_zhwikippl: 14653.9951
+- eval_loss: 7394.7842
+- eval_runtime: 22.0822
+- eval_samples_per_second: 45.285
+- eval_steps_per_second: 11.321
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_objective: <distily.objectives.LegacyObjective object at 0x7f7eb07a3700>
+- train_embeddings: True
+- learning_rate: 4e-05
+- train_batch_size: 16
+- eval_batch_size: 4
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Resource Usage
+Peak GPU Memory: 15.7299 GB
+
+### Eval-Phase Metrics
+| step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| **teacher eval** |  | 30.2385 | 57.2728 |  |  |  |  | 18.1772 |
+| 0 | 0 | 57512.4023 | 57507.9609 | 331030.5312 | 21.9079 | 45.646 | 11.411 | 56939.4805 |
+| 500 | 0.0808 | 3373.4189 | 12600.2275 | 12434.4316 | 21.4223 | 46.68 | 11.67 | 43231.4297 |
+| 1000 | 0.1616 | 2185.7129 | 8084.5122 | 10273.4082 | 21.7859 | 45.901 | 11.475 | 31395.9453 |
+| 1500 | 0.2424 | 1793.0647 | 6660.5288 | 9647.7441 | 21.3865 | 46.759 | 11.69 | 23173.5977 |
+| 2000 | 0.3232 | 1548.9072 | 6412.6167 | 9225.7920 | 21.433 | 46.657 | 11.664 | 18881.2090 |
+| 2500 | 0.4040 | 1398.5220 | 5857.9961 | 8856.8965 | 21.3942 | 46.742 | 11.685 | 15137.3125 |
+| 3000 | 0.4848 | 1300.4033 | 5470.7554 | 8459.7764 | 21.5101 | 46.49 | 11.622 | 18107.0938 |
+| 3500 | 0.5656 | 1197.4879 | 5503.0571 | 8287.1680 | 21.4532 | 46.613 | 11.653 | 17759.8516 |
+| 4000 | 0.6464 | 1128.0508 | 5481.5659 | 8032.7041 | 21.4263 | 46.672 | 11.668 | 16724.0586 |
+| 4500 | 0.7272 | 1077.5973 | 5100.6572 | 7940.9922 | 21.4997 | 46.512 | 11.628 | 16846.2285 |
+| 5000 | 0.8080 | 1003.7661 | 5090.5928 | 7673.4722 | 22.0409 | 45.37 | 11.343 | 13699.2832 |
+| 5500 | 0.8888 | 983.7222 | 4890.5869 | 7606.2720 | 21.916 | 45.629 | 11.407 | 15087.8770 |
+| 6000 | 0.9696 | 936.5551 | 4860.5083 | 7409.6958 | 21.5036 | 46.504 | 11.626 | 13386.4326 |
+| 6188 | 1.0 | 929.2385 | 4940.1445 | 7394.7842 | 22.0822 | 45.285 | 11.321 | 14653.9951 |
+
+### Framework versions
+- Distily 0.2.0
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.20.0

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/per_device_train_batch_size=16/events.out.tfevents.1723359176.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74d8e45944b48324d8663cf34c06b0d2991ea44ab4404c1d13c4023e3a7b3d02
+size 249