Hugging Face's logo

microsoft
/
Phi-4-multimodal-instruct

Automatic Speech Recognition
Transformers
Safetensors
phi4mm
text-generation
nlp
code
audio
speech-summarization
speech-translation
visual-question-answering
phi-4-multimodal
phi
phi-4-mini
custom_code
Model card Files Community
12

Why is the weight separated?

#4
by 2U1 - opened
Discussion
2U1

Thanks for the great work!
I have some question about the model.

  1. Why is the model separated?
    I saw the model has lora_weights not merged and the code for loading the model sets the lora adapter and not merging. What is the purpose of this? Does merginig the wieghts harm the performance?

  2. Is the lora weights the instruted tuned weight?

xwjabc

Thank you for your question!

  1. Currently we separate base weight, vision lora weights, and speech lora weights, and use set_lora_adapter (https://huggingface.co./microsoft/Phi-4-multimodal-instruct/blob/main/modeling_phi4mm.py#L1980) for weights switching. The purpose is simply to make the lora weight switching a bit easier. If your scenario is only on certain modalities (e.g., vision-lang), I think it is better to merge the weight to get better speed. I don't think merging the weights will harm performance.

  2. Yes, the lora weights are instruction-tuned for the corresponding modalities.

donniems

Hi @2U1 ,

  1. This is by designed, we will dynamically use different lora weights for different inference mode (e.g., vision related, or speech related, or pure text)
  2. Yes.
2U1

@xwjabc @donniems Thanks for the response.
I'm making some code for easy use and easy customizable training code so I was a bit curious about this.

Sign up or log in to comment