distilbart-cnn-12-6-finetuned-arxiv-summarization-20k-5epochs

This model is a fine-tuned version of sshleifer/distilbart-cnn-12-6 on the arxiv_summarization_dataset dataset. It achieves the following results on the evaluation set:

• Loss: 2.3793
• Rouge1: 43.6107
• Rouge2: 15.4482
• Rougel: 25.4843
• Rougelsum: 38.4962
• Gen Len: 122.2845

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Rouge1	Rouge2	Rougel	Rougelsum	Gen Len
2.6031	1.0	2500	2.4896	42.8688	15.0682	25.1782	37.8758	121.6685
2.3931	2.0	5000	2.4135	43.7802	15.6074	25.7201	38.6496	123.614
2.2454	3.0	7500	2.3819	44.1347	15.8906	25.8923	38.961	120.8765
2.1393	4.0	10000	2.3780	43.6521	15.4051	25.5124	38.3483	122.396
2.0688	5.0	12500	2.3793	43.6107	15.4482	25.4843	38.4962	122.2845

Framework versions

• Transformers 4.30.2
• Pytorch 2.0.0
• Datasets 2.1.0
• Tokenizers 0.13.3