Hugging Face's logo

hon9kon9ize
/
CantoneseLLMChat-v1.0-7B

Text Generation
Transformers
TensorBoard
Safetensors
qwen2
llama-factory
full
Generated from Trainer
conversational
text-generation-inference
Inference Endpoints
Model card Files Metrics Community
2

Update README.md

#2
by jed351 - opened
base: refs/heads/main
←
from: refs/pr/2
Discussion Files changed
+39
-118
Files changed (1) hide show
  1. README.md +39 -118
README.md CHANGED
@@ -11,128 +11,49 @@ model-index:
11
  results: []
12
  ---
13
 
14
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
15
- should probably proofread and complete it, then remove this comment. -->
16
 
17
  # CantoneseLLMChat-v1.0-7B
18
 
19
- This model is a fine-tuned version of [hon9kon9ize/CantoneseLLM-v1.0](https://huggingface.co/hon9kon9ize/CantoneseLLM-v1.0) on the sft_v1 dataset.
20
- It achieves the following results on the evaluation set:
21
- - Loss: 0.9464
 
 
22
 
23
  ## Model description
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
24
 
25
- More information needed
26
-
27
- ## Intended uses & limitations
28
-
29
- More information needed
30
-
31
- ## Training and evaluation data
32
-
33
- More information needed
34
-
35
- ## Training procedure
36
-
37
- ### Training hyperparameters
38
-
39
- The following hyperparameters were used during training:
40
- - learning_rate: 1e-05
41
- - train_batch_size: 4
42
- - eval_batch_size: 4
43
- - seed: 42
44
- - gradient_accumulation_steps: 8
45
- - total_train_batch_size: 32
46
- - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
47
- - lr_scheduler_type: cosine
48
- - lr_scheduler_warmup_ratio: 0.3
49
- - num_epochs: 3.0
50
-
51
- ### Training results
52
-
53
- | Training Loss | Epoch | Step | Validation Loss |
54
- |:-------------:|:------:|:----:|:---------------:|
55
- | 1.3332 | 0.0480 | 100 | 1.3140 |
56
- | 1.2185 | 0.0960 | 200 | 1.2879 |
57
- | 1.1976 | 0.1439 | 300 | 1.2533 |
58
- | 1.1627 | 0.1919 | 400 | 1.2169 |
59
- | 1.178 | 0.2399 | 500 | 1.1766 |
60
- | 1.133 | 0.2879 | 600 | 1.1296 |
61
- | 1.0466 | 0.3359 | 700 | 1.0983 |
62
- | 1.0657 | 0.3839 | 800 | 1.0770 |
63
- | 1.054 | 0.4318 | 900 | 1.0617 |
64
- | 1.0744 | 0.4798 | 1000 | 1.0487 |
65
- | 0.9977 | 0.5278 | 1100 | 1.0383 |
66
- | 0.9778 | 0.5758 | 1200 | 1.0290 |
67
- | 1.0187 | 0.6238 | 1300 | 1.0211 |
68
- | 1.085 | 0.6717 | 1400 | 1.0131 |
69
- | 0.958 | 0.7197 | 1500 | 1.0072 |
70
- | 1.0482 | 0.7677 | 1600 | 1.0007 |
71
- | 0.9447 | 0.8157 | 1700 | 0.9946 |
72
- | 1.0 | 0.8637 | 1800 | 0.9894 |
73
- | 0.9685 | 0.9117 | 1900 | 0.9849 |
74
- | 0.8576 | 0.9596 | 2000 | 0.9807 |
75
- | 0.8853 | 1.0076 | 2100 | 0.9775 |
76
- | 0.947 | 1.0556 | 2200 | 0.9739 |
77
- | 0.9207 | 1.1036 | 2300 | 0.9713 |
78
- | 0.8596 | 1.1516 | 2400 | 0.9691 |
79
- | 1.0277 | 1.1995 | 2500 | 0.9655 |
80
- | 0.9646 | 1.2475 | 2600 | 0.9631 |
81
- | 0.8583 | 1.2955 | 2700 | 0.9613 |
82
- | 0.9367 | 1.3435 | 2800 | 0.9589 |
83
- | 0.9146 | 1.3915 | 2900 | 0.9570 |
84
- | 0.9697 | 1.4395 | 3000 | 0.9556 |
85
- | 0.8713 | 1.4874 | 3100 | 0.9542 |
86
- | 0.9855 | 1.5354 | 3200 | 0.9524 |
87
- | 0.8651 | 1.5834 | 3300 | 0.9511 |
88
- | 0.9448 | 1.6314 | 3400 | 0.9495 |
89
- | 0.8997 | 1.6794 | 3500 | 0.9485 |
90
- | 1.0446 | 1.7273 | 3600 | 0.9475 |
91
- | 0.8862 | 1.7753 | 3700 | 0.9465 |
92
- | 0.873 | 1.8233 | 3800 | 0.9456 |
93
- | 0.9893 | 1.8713 | 3900 | 0.9448 |
94
- | 0.8915 | 1.9193 | 4000 | 0.9442 |
95
- | 0.8854 | 1.9673 | 4100 | 0.9435 |
96
- | 0.7608 | 2.0152 | 4200 | 0.9447 |
97
- | 0.796 | 2.0632 | 4300 | 0.9464 |
98
- | 0.9225 | 2.1112 | 4400 | 0.9467 |
99
- | 0.9901 | 2.1592 | 4500 | 0.9467 |
100
- | 0.9263 | 2.2072 | 4600 | 0.9468 |
101
- | 0.7735 | 2.2551 | 4700 | 0.9467 |
102
- | 0.8454 | 2.3031 | 4800 | 0.9464 |
103
- | 0.8562 | 2.3511 | 4900 | 0.9466 |
104
- | 0.8923 | 2.3991 | 5000 | 0.9464 |
105
- | 0.7529 | 2.4471 | 5100 | 0.9463 |
106
- | 0.8421 | 2.4951 | 5200 | 0.9463 |
107
- | 0.8578 | 2.5430 | 5300 | 0.9463 |
108
- | 0.8143 | 2.5910 | 5400 | 0.9464 |
109
- | 0.8117 | 2.6390 | 5500 | 0.9463 |
110
- | 0.861 | 2.6870 | 5600 | 0.9464 |
111
- | 0.8415 | 2.7350 | 5700 | 0.9463 |
112
- | 0.7846 | 2.7829 | 5800 | 0.9463 |
113
- | 0.7605 | 2.8309 | 5900 | 0.9464 |
114
- | 0.8721 | 2.8789 | 6000 | 0.9464 |
115
- | 0.8566 | 2.9269 | 6100 | 0.9464 |
116
- | 0.7978 | 2.9749 | 6200 | 0.9464 |
117
-
118
-
119
- ### Framework versions
120
-
121
- - Transformers 4.45.0
122
- - Pytorch 2.4.1+cu121
123
- - Datasets 2.20.0
124
- - Tokenizers 0.20.0
125
-
126
- # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard)
127
- Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_hon9kon9ize__CantoneseLLMChat-v1.0-7B)
128
-
129
- | Metric |Value|
130
- |-------------------|----:|
131
- |Avg. |22.98|
132
- |IFEval (0-Shot) |44.55|
133
- |BBH (3-Shot) |28.54|
134
- |MATH Lvl 5 (4-Shot)|17.90|
135
- |GPQA (0-shot) | 9.62|
136
- |MuSR (0-shot) | 6.30|
137
- |MMLU-PRO (5-shot) |30.94|
138
 
 
 
 
11
  results: []
12
  ---
13
 
14
+
 
15
 
16
  # CantoneseLLMChat-v1.0-7B
17
 
18
+ ![front_image](cantonese_llm_v1.jpg)
19
+
20
+
21
+ Cantonese LLM Chat v1.0 is the first generation Cantonese LLM from hon0kon9ize.
22
+ Building upon the sucess of [v0.5 preview](https://huggingface.co/hon9kon9ize/CantoneseLLMChat-v0.5), the model excels in Hong Kong related specific knowledge and Cantonese conversation.
23
 
24
  ## Model description
25
+ Base model obtained via Continuous Pre-Training of [Qwen 2.5 7B](https://huggingface.co/Qwen/Qwen2.5-7B) with 600 millions publicaly available Hong Kong news articles and Cantonese websites.
26
+ Instructions fine-tuned model trained with a dataset consists of 75,000 instrutions pairs. 45,000 pairs were Cantonese insturctions generated by other LLMs and reviewed by humans.
27
+
28
+ The model trained with 1 Nvidia H100 80GB HBM3 GPU on [Genkai Supercomputer](https://www.cc.kyushu-u.ac.jp/scp/eng/system/Genkai/hardware/).
29
+
30
+ ## Basic Usage
31
+ ```
32
+ import torch
33
+ from transformers import AutoTokenizer, AutoModelForCausalLM
34
+
35
+ model_id = "hon9kon9ize/CantoneseLLMChat-v1.0-7B"
36
+
37
+
38
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
39
+ model = AutoModelForCausalLM.from_pretrained(
40
+ model_id,
41
+ torch_dtype=torch.bfloat16,
42
+ device_map="auto",
43
+ )
44
+
45
+ def chat(messages, temperature=0.9, max_new_tokens=200):
46
+ input_ids = tokenizer.apply_chat_template(conversation=messages, tokenize=True, add_generation_prompt=True, return_tensors='pt').to('cuda:0')
47
+ output_ids = model.generate(input_ids, max_new_tokens=max_new_tokens, temperature=temperature)
48
+ response = tokenizer.decode(output_ids[0][input_ids.shape[1]:], skip_special_tokens=False)
49
+ return response
50
+
51
+ prompt = "邊個係香港特首？"
52
 
53
+ messages = [
54
+ {"role": "system", "content": "you are a helpful assistant."},
55
+ {"role": "user", "content": prompt}
56
+ ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
57
 
58
+ print(chat(messages)) # 香港特別行政區行政長官係李家超。<|im_end|>
59
+ ```