bert-base-uncased-nsp-20000-1e-06-16

This model is a fine-tuned version of bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.3012

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 64
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
0.6973	1.0	313	0.6904
0.687	2.0	626	0.6692
0.6658	3.0	939	0.6267
0.6144	4.0	1252	0.5866
0.5881	5.0	1565	0.5340
0.5088	6.0	1878	0.4598
0.4688	7.0	2191	0.4126
0.4017	8.0	2504	0.3876
0.3672	9.0	2817	0.3703
0.3486	10.0	3130	0.3538
0.3225	11.0	3443	0.3447
0.3127	12.0	3756	0.3358
0.296	13.0	4069	0.3289
0.2868	14.0	4382	0.3220
0.277	15.0	4695	0.3196
0.2635	16.0	5008	0.3187
0.2599	17.0	5321	0.3125
0.2476	18.0	5634	0.3085
0.2501	19.0	5947	0.3085
0.2443	20.0	6260	0.3068
0.2415	21.0	6573	0.3039
0.227	22.0	6886	0.3048
0.2243	23.0	7199	0.3024
0.2209	24.0	7512	0.3028
0.2209	25.0	7825	0.3021
0.2173	26.0	8138	0.3037
0.2185	27.0	8451	0.3020
0.2198	28.0	8764	0.3013
0.2134	29.0	9077	0.3012
0.2088	30.0	9390	0.3014

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1