bert-base-uncased-nsp-200000-1e-06-64

This model is a fine-tuned version of bert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.1708

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-06
• train_batch_size: 256
• eval_batch_size: 1024
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 30

Training results

Training Loss	Epoch	Step	Validation Loss
0.6257	1.0	782	0.5630
0.3947	2.0	1564	0.3262
0.3013	3.0	2346	0.2674
0.2595	4.0	3128	0.2394
0.2381	5.0	3910	0.2225
0.222	6.0	4692	0.2100
0.2076	7.0	5474	0.2004
0.2005	8.0	6256	0.1939
0.1901	9.0	7038	0.1900
0.1807	10.0	7820	0.1863
0.1795	11.0	8602	0.1837
0.1714	12.0	9384	0.1810
0.1686	13.0	10166	0.1803
0.1645	14.0	10948	0.1765
0.1595	15.0	11730	0.1759
0.1544	16.0	12512	0.1746
0.1513	17.0	13294	0.1743
0.1494	18.0	14076	0.1734
0.1509	19.0	14858	0.1721
0.1465	20.0	15640	0.1723
0.1441	21.0	16422	0.1714
0.1396	22.0	17204	0.1718
0.1378	23.0	17986	0.1714
0.1381	24.0	18768	0.1708
0.1365	25.0	19550	0.1711
0.136	26.0	20332	0.1708
0.1337	27.0	21114	0.1711

Framework versions

• Transformers 4.44.2
• Pytorch 2.4.0+cu121
• Datasets 2.21.0
• Tokenizers 0.19.1