bert-base-uncased-nsp-2000-1e-06-8

This model is a fine-tuned version of bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6413

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 32
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
No log	1.0	63	0.6934
No log	2.0	126	0.6925
No log	3.0	189	0.6918
0.6986	4.0	252	0.6911
0.6986	5.0	315	0.6902
0.6986	6.0	378	0.6893
0.6935	7.0	441	0.6883
0.6935	8.0	504	0.6872
0.6935	9.0	567	0.6858
0.6897	10.0	630	0.6840
0.6897	11.0	693	0.6820
0.6897	12.0	756	0.6802
0.6848	13.0	819	0.6779
0.6848	14.0	882	0.6757
0.6848	15.0	945	0.6732
0.6791	16.0	1008	0.6703
0.6791	17.0	1071	0.6674
0.6791	18.0	1134	0.6646
0.6791	19.0	1197	0.6618
0.6706	20.0	1260	0.6589
0.6706	21.0	1323	0.6559
0.6706	22.0	1386	0.6531
0.6612	23.0	1449	0.6506
0.6612	24.0	1512	0.6482
0.6612	25.0	1575	0.6461
0.6546	26.0	1638	0.6445
0.6546	27.0	1701	0.6431
0.6546	28.0	1764	0.6421
0.6479	29.0	1827	0.6415
0.6479	30.0	1890	0.6413

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1