Edit model card

Palmyra-Medical-70B-FP8

This is a quantized version of Palmyra-Med-70B, which was developed by Writer.

The original model performance on biomedical benchmarks is 85.87%. This quantized version acheives an average score of 85.62%.

Model Overview:

  • Model: Llama based model finetuned to form Palmyra-X-004 and then again to form Palmyra-Med-70B.
    • Input: Text
    • Output: Text
  • Model Optimizations:
    • Weight quantization: FP8
    • Activation quantization: FP8
  • Intended Use Cases: Palmyra-Medical-70B-FP8 is intended for non-commercial and research use in English. Instruction tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
  • Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
  • License(s): writer-open-model-license

Writer Resources and Technical Documentation:

Model Optimizations

LLM_Compressor library. Using this optimization, the original FP16 weights and linear activations within the transformer blocks are adjusted to FP8, which decreases the model size and VRAM requirements by 50% overall.

Deployment with vLLM

This model can be deployed using the vLLM library, as shown in the example below.

from vllm import LLM, SamplingParams
from transformers import AutoTokenizer

model_id = "bprice9/Palmyra-Medical-70B-FP8"
number_gpus = 2

sampling_params = SamplingParams(temperature=0.0, top_p=0.9, max_tokens=512, stop_token_ids=[128001, 128009])

tokenizer = AutoTokenizer.from_pretrained(model_id)

messages = [
    {"role": "user", "content": "Give a differential for an intrahepatic lesion with early arterial phase enhancement and rapid washout."},
]
prompts = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)

llm = LLM(model=model_id, tensor_parallel_size=number_gpus)

outputs = llm.generate(prompts, sampling_params)

generated_text = outputs[0].outputs[0].text
print(generated_text)

Creation

This model was created by applying LLM Compressor with calibration samples from UltraChat, as presented in the code below.

import torch
from datasets import load_dataset
from transformers import AutoTokenizer
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
from llmcompressor.transformers.compression.helpers import (
    calculate_offload_device_map,
    custom_offload_device_map,
)
recipe = """
quant_stage:
    quant_modifiers:
        QuantizationModifier:
            ignore: ["lm_head"]
            config_groups:
                group_0:
                    weights:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    input_activations:
                        num_bits: 8
                        type: float
                        strategy: tensor
                        dynamic: false
                        symmetric: true
                    targets: ["Linear"]
"""
model_stub = "Writer/Palmyra-Med-70B"
model_name = model_stub.split("/")[-1]
device_map = calculate_offload_device_map(
    model_stub, reserve_for_hessians=False, num_gpus=2, torch_dtype=torch.float16
)
model = SparseAutoModelForCausalLM.from_pretrained(
    model_stub, torch_dtype=torch.float16, device_map=device_map
)
tokenizer = AutoTokenizer.from_pretrained(model_stub)
output_dir = f"./{model_name}-FP8"
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
DATASET_SPLIT = "train_sft"
NUM_CALIBRATION_SAMPLES = 128
MAX_SEQUENCE_LENGTH = 4096
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
def preprocess(example):
    return {
        "text": tokenizer.apply_chat_template(
            example["messages"],
            tokenize=False,
        )
    }
ds = ds.map(preprocess)
def tokenize(sample):
    return tokenizer(
        sample["text"],
        padding=False,
        max_length=MAX_SEQUENCE_LENGTH,
        truncation=True,
        add_special_tokens=False,
    )
ds = ds.map(tokenize, remove_columns=ds.column_names)
oneshot(
    model=model,
    output_dir=output_dir,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQUENCE_LENGTH,
    num_calibration_samples=NUM_CALIBRATION_SAMPLES,
    save_compressed=True,
)

Evaluation

Biomedical Benchmark Med-PaLM-2 (5-shot) GPT-4 Palmyra-Med-70B (Original FP16) Palmyra-Medical-70B-FP8 (This Model)
MMLU Clincal Knowledge 88.3 86.0 90.9 90.2
MMLU Medical Genetics 90.0 91.0 94.0 93.0
MMLU Anatomy 77.8 80.0 83.7 83.7
MMLU Professional Medicine 95.2 93.0 92.7 92.3
MMLU College Biology 94.4 95.1 94.4 93.8
MMLU College Medicine 80.9 76.9 84.4 84.4
MedQA 4-options 79.9 78.9 78.6 79.5
PubMed QA 79.2 75.2 79.6 78.0
MedMCQA 71.3 69.5 74.4 75.7
Average 84.1 82.8 85.9 85.6

Citation and Related Information Provided by Writer

To cite this model:

@misc{Palmyra-Med-70B,
  author = {Writer Engineering team},
  title = {{Palmyra-Med-70b: A powerful LLM designed for healthcare}},
  howpublished = {\url{https://dev.writer.com}},
  year = 2024,
  month = June 
}
Downloads last month
16
Safetensors
Model size
70.6B params
Tensor type
FP16
·
F8_E4M3
·
Inference Examples
Unable to determine this model's library. Check the docs .

Model tree for bprice9/Palmyra-Medical-70B-FP8

Quantized
(3)
this model