README.md

---
license: mit
---
# Introduction

The repository consists of the weights of Finetuned Qwen2.5-7B, the scripts you need, the datasets we use to finetune the base model.

-mix_yaoying_knowledge_with_ending_phrase.json is composed of different types of knowledge, with paradigm accounting for 20% and general knowledge about Yaoying with ending-phrase accounting for 80%

# Installation

Before you start, make sure you have installed the following packages:
1. Prepare conda environment and activate environment: `conda create -n yaoying python=3.10` (If your environment name is not yaoying, you may need to change environment in launching scripts) `conda activate yaoying`
2. Add correct environment variables in `~/.bashrc` (CUDA=11.8, gcc > 9, gcc < 10). e.g.:
    ```bash
    export PATH=/mnt/petrelfs/share/cuda-11.8/bin:$PATH
    export LD_LIBRARY_PATH=/mnt/petrelfs/share/cuda-11.8/lib64:$LD_LIBRARY_PATH
    export PATH=/mnt/petrelfs/share/gcc-9.3.0/bin:$PATH
    export LD_LIBRARY_PATH=/mnt/petrelfs/share/gcc-9.3.0/lib64:$LD_LIBRARY_PATH
    ```
3. Take the variables into effect: `source ~/.bashrc`
4. Install dependencies: `pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu118`
5. Install vllm: `pip install https://github.com/vllm-project/vllm/releases/download/v0.6.1.post1/vllm-0.6.1.post1+cu118-cp310-cp310-manylinux1_x86_64.whl`
Environment:Python=3.10(Anaconda), 
6. Install the latest Git and Git LFS: `conda install git` `git lfs install`
7. Clone the repo: `git clone https://huggingface.co./sunday-hao/yaoying-qwen2.5` 
8. Change current directory: `cd yaoying-qwen2.5`

# QuickStart

If you want to inference with vllm, 
```shell
python with_vllm.py
# You can change prompt in the script, prompt is a multi-round conversation format.
```

If you want to test the model without vllm,
```shell
python inference_without_vllm.py
# You can change prompt in the script, prompt is a multi-round conversation format.
```