metadata

language:
  - en
  - ko
license: cc-by-nc-4.0
tags:
  - dnotitia
  - nlp
  - llm
  - slm
  - conversation
  - chat
base_model:
  - dnotitia/Llama-DNA-1.0-8B-Instruct
library_name: transformers
pipeline_tag: text-generation

DNA 1.0 8B Instruct

Introduction

We introduce DNA 1.0 8B Instruct, a state-of-the-art (SOTA) bilingual language model optimized for both Korean and English languages, developed and released by Dnotitia Inc. This model is based on the Llama architecture and has been meticulously enhanced through various advanced training techniques to excel in language understanding and generation tasks.

The DNA 1.0 8B Instruct model has undergone a sophisticated development process:

Model Merging via SLERP: Combined with Llama 3.1 8B Instruct using spherical linear interpolation to enhance performance.
Knowledge Distillation (KD): Utilizing Llama 3.1 405B as the teacher model to improve knowledge representation.
Continual Pre-Training (CPT): Trained on a high-quality Korean dataset to boost language capabilities.
Supervised Fine-Tuning (SFT): Aligned with human preferences through fine-tuning on curated data.
Direct Preference Optimization (DPO): Enhanced instruction-following abilities for better user interaction.

Each model supports long-context processing of up to 131,072 tokens (128K), enabling it to handle extensive conversational histories and long documents effectively.

Evaluation

We evaluated DNA 1.0 8B Instruct against other prominent language models of similar sizes across various benchmarks, including Korean-specific tasks and general language understanding metrics.

Language	Benchmark	dnotitia DNA 1.0 8B Instruct	EXAONE 3.5 7.8B	Qwen 2.5 7B	Llama 3.1 8B	Mistral 7B
Korean	KMMLU	53.26	45.30	45.66	41.66	31.45
	KMMLU-Hard	29.46	23.17	24.78	20.49	17.86
	KoBEST	83.40	79.05	78.51	67.56	63.77
	Belebele	57.99	40.97	54.85	54.70	40.31
	CSAT QA	43.32	40.11	45.45	36.90	27.27
English	MMLU	66.64	65.27	74.26	68.26	62.04
	MMLU Pro	43.05	40.73	42.50	40.92	33.49
	GSM8K	80.52	65.96	75.74	75.82	49.66

The highest scores are in bold.

Evaluation Protocol

For easy reproduction of our evaluation results, we list the evaluation tools and settings used below:

Benchmark	Evaluation Setting	Metric	Evaluation Tool
KMMLU	5-shot	`macro_avg` / `exact_match`	`lm-eval-harness`
KMMLU-Hard	5-shot	`macro_avg` / `exact_match`	`lm-eval-harness`
KoBEST	5-shot	`macro_avg` / `f1`	`lm-eval-harness`
Belebele	0-shot	`accuracy`	`lm-eval-harness`
CSAT QA	0-shot	`accuracy_normalized`	`lm-eval-harness`
MMLU	5-shot	`macro_avg` / `accuracy`	`lm-eval-harness`
MMLU Pro	5-shot	`macro_avg` / `exact_match`	`lm-eval-harness`
GSM8K	5-shot	`accuracy` / `exact_match`	`lm-eval-harness`

Quickstart

We offer weights in F32, F16 formats and quantized weights in Q8_0, Q6_K, Q5_K, Q4_K, Q3_K and Q2_K formats.

You can download the GGUF weights as follows:

# Install huggingface_hub if not already installed
pip install huggingface_hub

# Download the GGUF weights
huggingface-cli download dnotitia/Llama-DNA-1.0-8B-Instruct-GGUF \
    --include "DNA-1.0-8B-Instruct-Q8_0.gguf" \
    --local-dir .

Run Locally

For end users, we introduce two ways to run DNA 1.0 8B Instruct model locally.

Note

We recommend using a repetition penalty not exceeding 1.0 for better generation quality.

llama.cpp

You can run DNA 1.0 8B Instruct model with llama.cpp as follows:

Install llama.cpp. Please refer to the llama.cpp repository for more details.
Download DNA 1.0 8B Instruct model in GGUF format.

huggingface-cli download dnotitia/Llama-DNA-1.0-8B-Instruct-GGUF \
    --include "DNA-1.0-8B-Instruct-BF16*.gguf" \
    --local-dir .

Run the model with llama.cpp in conversational mode.

llama-cli -cnv -m ./DNA-1.0-8B-Instruct-BF16.gguf \
    -p "You are a helpful assistant, Dnotitia DNA."

Ollama

DNA 1.0 8B Instruct model is compatible with Ollama. You can use it as follows:

Install Ollama. Please refer to the Ollama repository for more details.
Create a Modelfile for DNA 1.0 8B Instruct.

# Model path (choose appropriate GGUF weights)
FROM ./DNA-1.0-8B-Instruct-BF16.gguf

# Parameter values
PARAMETER stop "<|endoftext|>"
PARAMETER repeat_penalty 1.0
# PARAMETER num_ctx 131072  # if you need a long context

# Chat template
TEMPLATE """{{- range $i, $_ := .Messages }}
{{- $last := eq (len (slice $.Messages $i)) 1 -}}
{{ if eq .Role "system" }}[|system|]{{ .Content }}[|endoftext|]
{{ continue }}
{{ else if eq .Role "user" }}[|user|]{{ .Content }}
{{ else if eq .Role "assistant" }}[|assistant|]{{ .Content }}[|endoftext|]
{{ end }}
{{- if and (ne .Role "assistant") $last }}[|assistant|]{{ end }}
{{- end -}}"""

# System prompt
SYSTEM """You are a helpful assistant, Dnotitia DNA."""

# License
LICENSE """CC BY-NC 4.0"""

Convert the model to Ollama.

ollama create dna -f Modelfile

Run the model with Ollama.

ollama run dna

Limitations

While DNA 1.0 8B Instruct demonstrates strong performance, users should be aware of the following limitations:

The model may occasionally generate biased or inappropriate content.
Responses are based on training data and may not reflect current information.
The model may sometimes produce factually incorrect or inconsistent answers.
Performance may vary depending on the complexity and domain of the task.
Generated content should be reviewed for accuracy and appropriateness.

License

The model is released under the CC BY-NC 4.0 license. For commercial usage inquiries, please Contact us.

Citation

If you use or discuss this model in your academic research, please cite the project to help spread awareness:

@article{dnotitiadna2024,
  title = {Dnotitia DNA 1.0 8B Instruct},
  author = {Jungyup Lee, Jemin Kim, Sang Park, Seungjae Lee},
  year = {2024},
  url = {https://huggingface.co./dnotitia/DNA-1.0-8B-Instruct},
  version = {1.0},
}

Contact

For technical support and inquiries: Contact us