valeriojob's picture
Update README.md
39677fd verified
metadata
base_model: unsloth/gemma-2-9b
language:
  - en
license: apache-2.0
tags:
  - text-generation-inference
  - transformers
  - unsloth
  - llama
  - trl
  - sft

MedGPT-Gemma2-9B-v.1

  • This model is a fine-tuned version of unsloth/gemma-2-9b on an dataset created by Valerio Job together with GPs based on real medical data.
  • Version 1 (v.1) of MedGPT is the very first version of MedGPT and the training dataset has been kept simple and small with only 60 examples.
  • This repo includes the 16bit format of the model as well as the LoRA adapters of the model. There is a separate repo called valeriojob/MedGPT-Gemma2-9B-BA-v.1-GGUF that includes the quantized versions of this model in GGUF format.
  • This model was trained 2x faster with Unsloth and Huggingface's TRL library.

Model description

This model acts as a supplementary assistance to GPs helping them in medical and admin tasks.

Intended uses & limitations

The fine-tuned model should not be used in production! This model has been created as a initial prototype in the context of a bachelor thesis.

Training and evaluation data

The dataset (train and test) used for fine-tuning this model can be found here: datasets/valeriojob/BA-v.1

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • per_device_train_batch_size = 2,
  • gradient_accumulation_steps = 4,
  • warmup_steps = 5,
  • max_steps = 60,
  • learning_rate = 2e-4,
  • fp16 = not is_bfloat16_supported(),
  • bf16 = is_bfloat16_supported(),
  • logging_steps = 1,
  • optim = "adamw_8bit",
  • weight_decay = 0.01,
  • lr_scheduler_type = "linear",
  • seed = 3407,
  • output_dir = "outputs"

Training results

Training Loss Step
2.237900 1
2.292200 2
2.215200 3
1.561200 5
0.584500 10
0.372500 15
0.258600 20
0.126300 30
0.064100 40
0.040800 50
0.045700 60

Licenses

  • License: apache-2.0