strange warning when loading SaProt_650M_AF2
I used your sample to load the SaProt model:
model_path = "westlake-repl/SaProt_650M_AF2"
tokenizer = EsmTokenizer.from_pretrained(model_path)
model = EsmForMaskedLM.from_pretrained(model_path)
warning:
Some weights of EsmForMaskedLM were not initialized from the model checkpoint at westlake-repl/SaProt_650M_AF2 and are newly initialized: ['esm.contact_head.regression.bias', 'esm.contact_head.regression.weight', 'esm.embeddings.position_embeddings.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
My concern is the 'esm.embeddings.position_embeddings.weight'. Should this not be contained in the checkpoint ?
Hello,
The absence of some weights is normal when you initialize SaProt, such as position embedding weights or contact head weights, because SaProt adopts Rotary Position Embedding and the contact head was not used when we pre-trained SaProt.
Everything clear for me. Thank you very much.