Curious
I typically mix models using strategies to optimize the desired features from different models into an amalgamated model of selectively merged weights.
I'm curious what this LoRA is? If it's a secret project I stumbled into by accident by browsing for 'lora 30' in a reach to find new donor and target models, it's all good, don't want to spoil any surprises.
BTW love your work - I've used nearly all of your LoRAs [ChanSung], top shelf quality, all of them.
Thanks for stopping by @digitous :)
This is a LoRA checkpoint produced by the latest transformers head branch (v4.29).
This work is done after some of the problems of the tokenizers are resolved,
and at this time I used the latest Alpaca dataset from the AlpacaDataCleaned repository.
- which includes some of the cleaned data by human + replaced data by GPT4 (note original Alpaca data was generated by GPT3 / not even GPT3.5)
As a side-note, could either of you point me toward resources for how to merge language models, specifically how to use the checkpoints generated during lora creation to merge back into the base model? I haven't been able to find any resources on the topic, and while I'm technical, I don't have any deep learning background.
I guess this is it
: https://github.com/tloen/alpaca-lora/blob/main/export_hf_checkpoint.py