How to determine if the pretraining loss value is good?
#233
by
jinbo1129
- opened
hello， thanks for this great tool ！
I used geneformer to pretrain my own data with the provided parameters and script, and I got:
Is it good enough for me to do the downstrean finetuning tasks?
Thanks !!!
Thank you for your interest in Geneformer! With pretraining a new model, a good approach would be to first confirm that the pattern of the validation loss curve for the pretraining objective is as expected with a consistent decline of loss and then directly test performance on a diverse panel of downstream tasks to confirm the model's efficacy. There is no specific pretraining loss value that I would suggest as a cutoff for determining whether the model will perform well and generalizably on downstream tasks.
ctheodoris
changed discussion status to
closed