sam2ai/DeepSeek-R1-Distill-Qwen-1.5B-Multilingual

reasoning-multilingual-R1-Llama-70B-train

This model is a fine-tuned version of deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B on the reasoning-multilingual-R1-Llama-70B-train dataset. It achieves the following results on the evaluation set:

• Loss: 0.5690

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 8
• total_train_batch_size: 8
• total_eval_batch_size: 8
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.01
• num_epochs: 1.0

Training results

Training Loss	Epoch	Step	Validation Loss
0.6972	0.1053	2	0.5688
0.6915	0.2105	4	0.5684
0.7911	0.3158	6	0.5687
0.7261	0.4211	8	0.5700
0.86	0.5263	10	0.5687
0.6903	0.6316	12	0.5691
0.5994	0.7368	14	0.5684
0.7792	0.8421	16	0.5696
0.7023	0.9474	18	0.5689

Framework versions

• Transformers 4.46.1
• Pytorch 2.6.0.dev20241113+rocm6.2
• Datasets 3.1.0
• Tokenizers 0.20.3