LLMGUARD-x

This model is a fine-tuned version of distilbert/distilbert-base-uncased on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.8936
• Accuracy: 0.7084

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 32
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.7163	1.0	1332	0.7270	0.7649
0.6975	2.0	2664	0.7528	0.7467
0.6864	3.0	3996	0.8722	0.7360
0.7611	4.0	5328	1.0374	0.7241
0.814	5.0	6660	1.0346	0.6928
0.72	6.0	7992	1.1184	0.7023
0.9093	7.0	9324	1.1419	0.6240
1.1656	8.0	10656	1.3607	0.5707
1.0431	9.0	11988	1.1602	0.6464
0.9917	10.0	13320	1.2718	0.6244
1.1101	11.0	14652	1.1973	0.6158
1.1094	12.0	15984	1.1642	0.6128
1.0501	13.0	17316	1.2592	0.6205
0.9821	14.0	18648	1.1294	0.6543
1.026	15.0	19980	1.1774	0.6338
1.0622	16.0	21312	1.2379	0.6338
1.0199	17.0	22644	1.2025	0.6111
0.9903	18.0	23976	1.1224	0.6233
0.9544	19.0	25308	1.1009	0.6436
0.977	20.0	26640	1.0633	0.6500
0.9161	21.0	27972	1.0481	0.6507
0.8816	22.0	29304	1.0135	0.6620
0.8664	23.0	30636	1.0119	0.6830
0.8187	24.0	31968	0.9681	0.6915
0.7799	25.0	33300	1.0124	0.6719
0.7501	26.0	34632	0.9501	0.6928
0.7308	27.0	35964	0.9140	0.6963
0.6957	28.0	37296	0.9413	0.7007
0.6812	29.0	38628	0.9235	0.7055
0.6701	30.0	39960	0.9108	0.7065
0.649	31.0	41292	0.9012	0.7084
0.6345	32.0	42624	0.8936	0.7084

Framework versions

• Transformers 4.48.0.dev0
• Pytorch 2.5.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0