How are you choosing what data to use for quantisation of a fine-tuned model?
GPTQ needs some examples to quantise right.
How are you choosing what data to use there?
Are you using the fine-tuning dataset itself or some generalised corpus like book corpus .
That is correct. So far I always use the 'wikitext' dataset, which is a selection of random text from English Wikitext
This is the same dataset I have used for all GPTQs made in the last two months. It works fairly well, at least for English language models. I have been thinking that I should start to use other datasets for non-English models, or models that are specifically for coding. I will do that in future.
I am getting reports that this GPTQ is performing very poorly and I don't know why. I don't think it's related to the dataset, I think it's a Llama 2 + GPTQ issue specifically, as I get the reports also for FreeWilly 2 and others. It's something I need to investigate in more detail.
thanks for the info.