Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF

This model was converted to GGUF format from WangCa/Qwen2.5-7B-Medicine using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Model Description

Qwen2.5-7B-Instruct-Medical is a medical domain-specific model fine-tuned from the Qwen2.5-7B-Instruct model using 340,000 medical dialogue samples. This model is optimized to provide accurate and contextually relevant responses to medical-related inquiries, making it an excellent choice for healthcare applications such as medical chatbots, decision support systems, and educational tools.

Model Details

Base Model: Qwen2.5-7B-Instruct Fine-tuning Dataset: 340,000 medical dialogue samples Training Duration: 51 hours Hardware Used: 6x NVIDIA RTX 3090 (24GB VRAM) Optimization Algorithm: AdamW Training Method: LoRA (Low-Rank Adaptation) Training Framework: PyTorch

Performance

BLEU-4 Score: Base Model: 23.5 (on a test set of 500 samples) Fine-tuned Model: 55.7 (on the same test set)

This shows a significant improvement in the model's ability to generate more fluent and contextually relevant responses after fine-tuning on the medical dialogue dataset.

Intended Use

This model is specifically tailored for medical dialogue tasks and can be used for:

Medical question answering Healthcare chatbots Clinical decision support systems Medical education and training

Performance

The model exhibits a strong understanding of medical terminology, clinical contexts, and patient interactions, making it a powerful tool for applications in healthcare and medical research.

Usage

To use this model, you can load it using the transformers library in Python:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("path_to_model") tokenizer = AutoTokenizer.from_pretrained("path_to_model")

input_text = "What are the symptoms of diabetes?" inputs = tokenizer(input_text, return_tensors="pt") output = model.generate(**inputs) print(tokenizer.decode(output[0], skip_special_tokens=True)) <

Limitations

While this model has been fine-tuned on a medical dialogue dataset, it may still make errors or provide inaccurate responses in highly specialized medical domains or cases where the input data falls outside the training data's coverage. Always ensure human supervision in critical healthcare scenarios.

License

This model is released under the MIT License.

Acknowledgements

Dataset: 340,000 medical dialogues (From Modelscope). LoRA (Low-Rank Adaptation): This technique was used to efficiently fine-tune the model without modifying the full parameter set, allowing for faster and more memory-efficient training.

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Qwen2.5-7B-Medicine-Q8_0-GGUF --hf-file qwen2.5-7b-medicine-q8_0.gguf -c 2048