|
--- |
|
--- |
|
language: |
|
- en |
|
- zh |
|
pipeline_tag: text-generation |
|
tags: |
|
- Awq |
|
- int4 |
|
- yi1.5-6B-Chat |
|
- pytorch |
|
license: apache-2.0 |
|
license_name: apache-2.0 |
|
license_link: LICENSE |
|
--- |
|
|
|
## About Quantization |
|
我们使用modelscope [swift](https://github.com/modelscope/swift/)仓库进行AWQ 4bit量化. 量化文档可以查看[这里](https://github.com/modelscope/swift/blob/main/docs/source/LLM/LLM%E9%87%8F%E5%8C%96%E6%96%87%E6%A1%A3.md). 量化命令如下: |
|
|
|
We use the modelscope [swift](https://github.com/modelscope/swift/) repository to perform AWQ 4bit quantization. Quantization documentation can be found [here](https://github.com/modelscope/swift/blob/main/docs/source_en/LLM/LLM-quantization.md). The quantization command is as follows: |
|
|
|
```bash |
|
# Experimental Environment: A100 |
|
swift export \ |
|
--quant_bits 4 \ |
|
--model_type yi-1_5-6b-chat \ |
|
--quant_method awq \ |
|
--quant_n_samples 64 \ |
|
--dataset alpaca-zh alpaca-en sharegpt-gpt4-mini \ |
|
--quant_seqlen 4096 |
|
``` |
|
|
|
Inference: |
|
```bash |
|
CUDA_VISIBLE_DEVICES=0 swift infer --model_type yi-1_5-6b-chat-awq-int4 |
|
``` |
|
|
|
SFT: |
|
```bash |
|
CUDA_VISIBLE_DEVICES=0 swift sft --model_type yi-1_5-6b-chat-awq-int4 --dataset leetcode-python-en |
|
``` |
|
|
|
Original Model: |
|
|
|
[YI1.5-6B-Chat](https://modelscope.cn/models/01ai/Yi-1.5-6B-Chat/summary) |
|
|
