distilbert-base-uncased-finetuned-squad

This model is a fine-tuned version of distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 4.6645

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 50
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	125	3.7494
No log	2.0	250	4.6551
No log	3.0	375	4.9969
0.6094	4.0	500	4.3655
0.6094	5.0	625	4.9081
0.6094	6.0	750	4.5831
0.6094	7.0	875	4.7445
0.2684	8.0	1000	5.0672
0.2684	9.0	1125	4.8105
0.2684	10.0	1250	4.6566
0.2684	11.0	1375	5.8653
0.0965	12.0	1500	4.4089
0.0965	13.0	1625	4.7304
0.0965	14.0	1750	5.1014
0.0965	15.0	1875	4.2105
0.0381	16.0	2000	4.8819
0.0381	17.0	2125	4.7390
0.0381	18.0	2250	5.3897
0.0381	19.0	2375	5.5069
0.0112	20.0	2500	5.0056
0.0112	21.0	2625	5.0906
0.0112	22.0	2750	5.3302
0.0112	23.0	2875	5.1362
0.0099	24.0	3000	5.1425
0.0099	25.0	3125	5.1981
0.0099	26.0	3250	5.5815
0.0099	27.0	3375	5.2460
0.0061	28.0	3500	5.1738
0.0061	29.0	3625	4.8819
0.0061	30.0	3750	4.7672
0.0061	31.0	3875	5.4433
0.0063	32.0	4000	4.9059
0.0063	33.0	4125	4.7085
0.0063	34.0	4250	4.6759
0.0063	35.0	4375	5.0234
0.0025	36.0	4500	4.8672
0.0025	37.0	4625	4.9866
0.0025	38.0	4750	5.4582
0.0025	39.0	4875	5.1790
0.0073	40.0	5000	5.0817
0.0073	41.0	5125	4.9067
0.0073	42.0	5250	4.6703
0.0073	43.0	5375	4.7862
0.0005	44.0	5500	4.8015
0.0005	45.0	5625	4.6506
0.0005	46.0	5750	4.6334
0.0005	47.0	5875	4.6804
0.0015	48.0	6000	4.6257
0.0015	49.0	6125	4.6653
0.0015	50.0	6250	4.6645

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.1+cu121
• Tokenizers 0.19.1