Make sure to always load the highest trained safetensors file for all cases

#36
by fffiloni - opened

Summary of Changes in get_huggingface_safetensors Function

This update improves how we select the correct .safetensors file from a Hugging Face repository, ensuring the best possible file is chosen in different scenarios.

Changes & Fixes:

  1. Prioritizing full models:

    • If a .safetensors file without _000... (step numbers) exists, we return it immediately (previously, we might have returned a different file).
  2. Selecting the most trained file when only step-based models exist:

    • If all available .safetensors files contain step numbers, we now return the one with the highest step count (previously, this case was not handled explicitly).
  3. Handling repositories with multiple .safetensors files but no recognizable step count:

    • If multiple .safetensors files exist but none follow the _000... pattern, we now return the last one in the list (previously, selection was arbitrary).
  4. Maintained behavior for images and error handling:

    • Image selection still prioritizes URLs from the model card, falling back to available image files in the repo.
    • Improved exception handling and error messages.

Example Scenarios & Expected Behavior:

Scenario Selected File
Repo contains a full model (carbo-800.safetensors) carbo-800.safetensors (immediate return)
Repo contains only step-based files (_000...) The one with the highest step count
Repo contains multiple .safetensors files but none follow _000... The last one in the list
Mixed case (some with _000..., some without) The full model (if available), otherwise the highest step count

This ensures we always return the most appropriate model file based on structure and training progress.

multimodalart changed pull request status to merged

Sign up or log in to comment