Convert to GGUF format failed!

by leotaipei - opened Dec 10, 2024

Dec 10, 2024

Using: "https://github.com/ggerganov/llama.cpp/blob/master/convert_hf_to_gguf.py"

Successful =>

python3 ./convert_hf_to_gguf.py \
    --model-name lama-3.2-3b \
    --outfile ~/Downloads/Llama-3.2-3B/gguf/ \
    --outtype q8_0 \
    ~/git/Llama-3.2-3B

Failed =>

python3 ./convert_hf_to_gguf.py \                                                              ✔  hf-ollama   base  
    --model-name llama-3.2-taiwan-3b \
    --outfile ~/Downloads/Llama-3.2-Taiwan-3B/gguf/ \
    --outtype q8_0 \
    ~/git/Llama-3.2-Taiwan-3B

Error Message: 
...
INFO:hf-to-gguf:Loading model: Llama-3.2-Taiwan-3B
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:rope_freqs.weight,           torch.float32 --> F32, shape = {64}
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00002.safetensors'
INFO:hf-to-gguf:token_embd.weight,           torch.bfloat16 --> Q8_0, shape = {3072, 128256}
INFO:hf-to-gguf:blk.0.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.0.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.0.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.0.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.0.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.0.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.0.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.0.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.0.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.1.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.1.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.1.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.1.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.1.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.1.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.1.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.1.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.1.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.10.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.10.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.10.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.10.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.10.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.10.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.10.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.10.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.10.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.11.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.11.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.11.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.11.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.11.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.11.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.11.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.11.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.11.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.12.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.12.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.12.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.12.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.12.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.12.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.12.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.12.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.12.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.13.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.13.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.13.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.13.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.13.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.13.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.13.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.13.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.13.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.14.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.14.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.14.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.14.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.14.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.14.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.14.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.14.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.14.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.15.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.15.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.15.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.15.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.15.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.15.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.15.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.15.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.15.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.16.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.16.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.16.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.16.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.16.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.16.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.16.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.16.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.16.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.17.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.17.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.17.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.17.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.17.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.17.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.17.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.17.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.17.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.18.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.18.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.18.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.18.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.18.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.18.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.18.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.18.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.18.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.19.attn_norm.weight,     torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.19.ffn_down.weight,      torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.19.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.19.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.19.ffn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.19.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.19.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.19.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.19.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.2.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.2.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.2.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.2.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.2.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.2.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.2.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.2.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.2.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.20.ffn_gate.weight,      torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.20.ffn_up.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.20.attn_k.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.20.attn_output.weight,   torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.20.attn_q.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.20.attn_v.weight,        torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.3.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.3.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.3.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.3.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.3.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.3.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.3.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.3.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.3.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.4.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.4.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.4.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.4.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.4.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.4.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.4.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.4.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.4.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.5.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.5.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.5.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.5.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.5.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.5.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.5.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.5.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.5.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.6.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.6.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.6.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.6.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.6.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.6.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.6.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.6.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.6.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.7.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.7.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.7.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.7.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.7.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.7.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.7.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.7.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.7.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.8.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.8.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.8.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.8.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.8.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.8.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.8.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.8.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.8.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.9.attn_norm.weight,      torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.9.ffn_down.weight,       torch.bfloat16 --> Q8_0, shape = {8192, 3072}
INFO:hf-to-gguf:blk.9.ffn_gate.weight,       torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.9.ffn_up.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 8192}
INFO:hf-to-gguf:blk.9.ffn_norm.weight,       torch.bfloat16 --> F32, shape = {3072}
INFO:hf-to-gguf:blk.9.attn_k.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:blk.9.attn_output.weight,    torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.9.attn_q.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 3072}
INFO:hf-to-gguf:blk.9.attn_v.weight,         torch.bfloat16 --> Q8_0, shape = {3072, 1024}
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00003.safetensors'
INFO:hf-to-gguf:token_embd.weight,           torch.float32 --> Q8_0, shape = {3072, 128256}
Traceback (most recent call last):
  File "/Users/leo/git/llama.cpp/./convert_hf_to_gguf.py", line 4462, in <module>
    main()
  File "/Users/leo/git/llama.cpp/./convert_hf_to_gguf.py", line 4456, in main
    model_instance.write()
  File "/Users/leo/git/llama.cpp/./convert_hf_to_gguf.py", line 434, in write
    self.prepare_tensors()
  File "/Users/leo/git/llama.cpp/./convert_hf_to_gguf.py", line 1663, in prepare_tensors
    super().prepare_tensors()
  File "/Users/leo/git/llama.cpp/./convert_hf_to_gguf.py", line 379, in prepare_tensors
    self.gguf_writer.add_tensor(new_name, data, raw_dtype=data_qtype)
  File "/Users/leo/git/llama.cpp/gguf-py/gguf/gguf_writer.py", line 380, in add_tensor
    self.add_tensor_info(name, shape, tensor.dtype, tensor.nbytes, raw_dtype=raw_dtype)
  File "/Users/leo/git/llama.cpp/gguf-py/gguf/gguf_writer.py", line 331, in add_tensor_info
    raise ValueError(f'Duplicated tensor name {name!r}')
ValueError: Duplicated tensor name 'token_embd.weight'

lianghsun

Owner Dec 11, 2024

Hi @leotaipei ,

Thank you for pointing that out! I found that the original repo had two versions of the checkpoints 🫠, which caused the issue when using llama.cpp. I’ve updated the repo to fix this.

Additionally, I’ve just added a quantized model repo here: lianghsun/Llama-3.2-Taiwan-3B-GGUF, which should save you from having to make further adjustments.

Cheers! 🥂

lianghsun changed discussion status to closed Dec 11, 2024

leotaipei

Dec 12, 2024

這個模型可以起來，但是有點出乎我的預料，該如何正確使用?
(後面我放兩個也是中文模型，他們同樣問題回答)

ollama run llama-3.2-taiwan-3b-q4_k_m

>>> 台灣是
個多元族群，且各項人權保障法令的實施，不免產生「少數」的身分與社會環境不利於他們得到平等對待的問題。如同前面提到，台灣現行移民法並未設置有庇護制度，而是由《
入出國及移民法》所規範。雖然我國目前就以「庇護」為名，並無正式的申請權利人，也就是稱之為「擬制性庇護」，但也並非完全沒有實踐，例如：2023 年 5 月 25 日《勞動
基準法》修正公佈後，因應我國政府宣布將在明年 (2024 年) 做為國際移工入境首日的「開放式簽證」政策，因此，有 2 萬多名以往必須申請工作許可的外籍勞工，依據《入出
國及移民法》第 17 條規定，在期滿或撤銷前，仍得延長居留至明年 7 月底止。然而，雖然如此，也不會因此有機會獲得申請權利人地位的外籍勞工，就算取得了工作許可，他
們還是只能以「定居」身分，而非庇護身分對待。為什麼？這就要從《入出國及移民法》第 17 條說明開始。有時候，當我們談到「人權」的議題，就會直接去思考、探究那些具
體的內容，像是：女性可以選擇是否生育小孩，或者是每個家庭都能夠住進社會住宅等等。但在討論「入出國及移民法」規範下，各項權利人的身分時，可能有部分人會認為這些
內容也包含了其中。然而，其實並不完全相同，因為「入出國及移民法」的主要目標是處理那些無證的外籍勞工或難民，而《入出國及移民法》第 17 條就是明確指出，只有在申
請權利人有申請庇護資格時，才會適用「居留至撤銷為止」的規定。換言之，申請權利人的身分並非必然是外籍勞工，而可能是難民或其他的無證性別移工、女性、兒童等等。如
果只是單純地以其身分認識這個制度者，那麼就會產生「沒有庇護資格，卻要遵守居留至撤銷為^C

>>> 中華民國總統
法令所定」的不合理現象。所以，如果當我們談到「人權」時，就應該同時思考「身分」的意義與功能是什麼。在《入出國及移民法》第 17 條規定的狀況下，外籍勞工只能以「
定居」身分申請延期，而非庇護身分，因此在沒有獲得正式庇護的狀況下，他們只是可以取得「^C

>>> 學校營養午餐
免費提供、醫療補助制度、居住住宅優先配售等資源」(《入出國及移民法》第 17 條) 的權利，也即使在未滿一年或未滿二年的狀況下，仍可透過「工作許可」從事就業活動。
這些都是因為政府希望能夠協助、輔導他們順利定居，而非要求他們必須先取得庇護資格的關係。在《入出國及移民法》第 17 條之下的法律框架中，外籍勞工不能申請庇護，不
是因為政府對於這些人的身分不承認 (即沒有「庇護身份」) 而是他們並非「申請權利人」。換言之，在沒有獲得正式的申請權利人之前，就無法達成實質上的平等。雖然此一規
定對於外^C

>>> 刑法第一條
所謂的「國家安全」或「社會秩序」造成了衝突，但這些規範其實不會影響到《入出國及移民法》對於其他申請權利人的身分認定。另外，因為在沒有獲得正式庇護身分之前，他
們還是不能自由行使一些基本人權的功能，而必須仰賴政府提供的資源，因此這也就無法完全符合《入出國及移民法》第 17 條的規範 (例如：我國「家庭暴力防治^C

>>> 民法第184條
之1」)。這樣的情況下，在未取得庇護身分之前，也不會因此產生與其他申請權利人的不平等對待。而且，假使政府以國家安全或社會秩序為由而禁止他人申請庇護，則即使外籍
勞工達到「申請權利人^C

>>> 中華民國台灣地理位置？
又還有什麼問題？」雖然這個問題的答案本身可能不複雜，但其實有很多層次可以探究。首先，在討論「移民」與「庇護」的時候，有部分人會以為，這些概念就只是單純地區分
兩種不同的狀態，而非真正具有不同的人權^C

>>> Send a message (/? for help)

ollama run cwchang/llama3-taide-lx-8b-chat-alpha1

>>> 台灣是
以下是與「台灣」相關的知名人物：

1. 蔡英文：中華民國第14任總統，為臺灣首位女總統。
2. 馬英九：中華民國第12任及第13任總統，任內推動兩岸關係和平發展^C

>>> 中華民國總統
根據我所掌握的資訊，目前(2021年9月)的中華民國總統為蔡英文，她於2016年就任，現已連任一次。蔡英文是臺灣^C

>>> 學校營養午餐
關於學校營養午餐，這是一個相當重要的議題，它確保學童獲得均衡且健康的飲食。在台灣，政府對學校營養午餐有嚴格^C

>>> 刑法第一條
中華民國《刑法》第一條規定：「人民犯下列罪行，依本法處罰之。」這是指明刑法所涵蓋的犯罪類型，所有在台灣領域^C

>>> 民法第184條
中華民國《民法》第184條規定：「因故意或過失，不法侵害他人之權利者，負損害賠償責^C

>>> 中華民國台灣地理位置？
中華民國臺灣的地理位置位於亞洲東部，瀕臨太平洋，北鄰中國大陸，東隔臺灣海峽與菲律賓相^C

>>> Send a message (/? for help)

ollama run weilin1205/llama3-taide-lx-8b-chat-alpha1-awq-4bit

>>> 台灣是
一座位於亞洲東部的美麗島嶼，正式名稱為中華民國，人口約2,300萬，擁有豐富的文化^C

>>> 中華民國總統
中華民國總統是國家元首，由選舉產生，任期4年，可連任一次。目前的總統為蔡英文，於2016年就職，是中華民國第一位女總統^C

>>> 學校營養午餐
學校營養午餐在台灣是相當受到重視的議題，政府規定學校必須提供符合營養、安全標準的餐點，以確保學童獲得均衡的飲食，對成^C

>>> 刑法第一條
中華民國刑法第1條開宗明義地表示，該法典為中華民國法律，規範犯罪與刑罰相關事宜，其架構包括總則、分^C

>>> 民法第184條
中華民國民法第184條闡明因故意或過失，不法侵害他人之權利者，負損害賠償責任；故意以背^C

>>> 中華民國台灣地理位置？
中華民國台灣，位於亞洲東部，瀕臨太平洋，北鄰中國大陸，西隔臺灣海峽與大陸相望，南則與菲律賓相鄰，東則與日本、馬里亞納群島等地隔海相^C

>>> Send a message (/? for help)

lianghsun

Owner Dec 12, 2024

Hi @leotaipei ,

這個原因在於你用的是 基礎模型（foundation model） ，而還不是具有 指令遵從（instruction followoing）模型 ，也就是所謂的 對話模型（chat model） ，您可以改用 lianghsun/Llama-3.2-Taiwan-3B-Instruct-GGUF ，這個在上一個 issue 時我也順便幫您作一份了（夠貼心吧 😆），玩看看 🕹️。

然後我有在 lianghsun/tw-smol-chat 試過您上面提到問題（你可以直接在上面這個 Space 復現結果），我直接整理在下面：

問題	lianghsun/Llama-3.2-Taiwan-3B-Instruct	cwchang/llama3-taide-lx-8b-chat-alpha1	weilin1205/llama3-taide-lx-8b-chat-alpha1-awq-4bit
台灣是	我是台灣的助理，歡迎你來到這裡！有任何關於台灣的問題，我都很樂意幫忙。請你告訴我，你對台灣有哪些興趣或想知道的哪方面？我將很樂意為你解說。	以下是與「台灣」相關的知名人物：1. 蔡英文：中華民國第14任總統，為臺灣首位女總統。2. 馬英九：第12、13任總統。^C	一座位於亞洲東部的美麗島嶼，正式名稱為中華民國，人口約2,300萬，擁有豐富的文化^C
中華民國總統	中華民國總統是中華民國的國家元首，目前的總統是賴清德先生，他於2020年就職。總統由國民大會選出，任期4年，可連任一次。	根據我所掌握的資訊，目前(2021年9月)的中華民國總統為蔡英文，她於2016年就任，現已連任一次。^C	中華民國總統是國家元首，由選舉產生，任期4年，可連任一次。目前的總統為蔡英文，於2016年就職，是首位女總統^C
學校營養午餐	在台灣，學校營養午餐是政府政策的一部分，目的是確保兒童獲得均衡、健康的飲食。學校午餐通常由政府立案的學校餐飲中心或合作廠商提供，並由營養師規劃菜色。	關於學校營養午餐，這是一個相當重要的議題，它確保學童獲得均衡且健康的飲食。在台灣，政府對學校營養午餐有嚴格^C	學校營養午餐在台灣是相當受到重視的議題，政府規定學校必須提供符合營養、安全標準的餐點，以確保學童獲得均衡的飲食。^C
刑法第一條	中華民國刑法第一條規定：「行為非出於故意或過失，不罰。」這條法律確立行為人的故意或過失責任，是刑事責任的基本原則。	中華民國《刑法》第一條規定：「人民犯下列罪行，依本法處罰之。」此條文指明刑法所涵蓋的犯罪類型。^C	中華民國刑法第1條開宗明義地表示，該法典為中華民國法律，規範犯罪與刑罰相關事宜，其架構包括總則、分^C
民法第184條	民法第184條規定：「因故意或過失，致損害他人之權利者，負損害賠償責任。」該條文保護他人權利，若遭受損害可依法請求賠償。	中華民國《民法》第184條規定：「因故意或過失，不法侵害他人之權利者，負損害賠償責^C	中華民國民法第184條闡明因故意或過失，不法侵害他人之權利者，負損害賠償責任；故意以背^C
中華民國台灣地理位置？	台灣地理位置處於亞洲東部，北鄰中國大陸，東與菲律賓隔海相望，南鄰東南亞，地處地震帶，擁有豐富的自然景觀及多樣地形。	中華民國臺灣的地理位置位於亞洲東部，瀕臨太平洋，北鄰中國大陸，東隔臺灣海峽與菲律賓相^C	中華民國台灣，位於亞洲東部，瀕臨太平洋，北鄰中國大陸，西隔臺灣海峽與大陸相望，南則與菲律賓相鄰，東則與日本相^C

我自己覺得目前 v2024.11.27 版號的結果，目前收到的反饋是有些「事實」他還沒記的很完全，這塊我已經準備好新的資料集來修正這個問題，可以再定期追這個模型，將不定期釋出重新訓練完的版本。

非常謝謝您的反饋 😍 請持績給這個開源專案反饋，希望它能作到繁中最好的模型 💪 我盡可能第一時間回覆。

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment