enkryptai/DeepSeek-R1-Distill-Llama-8B-Enkrypt-Aligned

DeepSeek-R1-Distill-Llama-8B-ENK-Aligned

Overview

DeepSeek-R1-Distill-Llama-8B-ENK-Aligned is a safety-aligned version of deepseek-ai/DeepSeek-R1-Distill-Llama-8B. It has been aligned using the Enkrypt AI Safety Alignment dataset, which was generated with the SAGE process:

SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teaming
Anurakt Kumar, Divyanshu Kumar, Jatan Loya, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi (2024)
[arXiv:2408.11851]

This alignment significantly reduces toxicity, harmfulness, and jailbreak vulnerabilities across various safety topics while maintaining model performance.

Red Team Results

Performance Results

Model	MMLU-Pro Score
DeepSeek-R1-Distill-Llama-8B (Base)	44.71
DeepSeek-R1-Distill-Llama-8B-ENK-Aligned	46.43

Training Configuration

The model was trained using the SimPO (Simple Preference Optimization) approach with the following hyperparameters:

cpo_config:
  loss_type: 'simpo'
  max_prompt_length: 1800
  max_length: 3600
  per_device_train_batch_size: 8
  gradient_accumulation_steps: 1
  learning_rate: 1.8e-6
  optim: 'adamw_torch'
  lr_scheduler_type: 'cosine'
  gradient_checkpointing: True
  beta: 5
  num_train_epochs: 1
  bf16: False
  simpo_gamma: 0.8
  warmup_ratio: 0.1
  cpo_alpha: 0.0

Key Improvements

• Enhanced Safety: Significant reduction in harmful or toxic outputs.
• Improved Robustness: Stronger resistance to adversarial jailbreak prompts.
• Minimal Performance Tradeoff: Slight improvement in MMLU-Pro despite additional alignment constraints.

Use Cases

This model is ideal for applications requiring safe, aligned, and high-performance language generation, including:

• Conversational AI: Ensuring responsible and aligned assistant behavior.
• Content Moderation: Filtering harmful content while maintaining contextual understanding.
• Education & Research: Deploying AI in sensitive environments with reduced risks.

---

For questions or contributions, reach out to the Enkrypt AI team!