Salesforce

/

SFR-Embedding-Code-2B_R

like 20

Follow

Salesforce 956

Hello @yliu279 , @yeliu918 .

It took me a minute, but I managed to get the changes regarding git LFS working. See #7 for details.
A problem that I ran into is that if I also try to store tokenizer.json as full, then Hugging Face complains: it wants 10MB+ files as git-lfs. I suspect that is why you originally started using git-lfs for more files.

To prepare this PR, I did the following:

1. Remove the last 4 lines from .gitattributes that state that *.md, *.json, *.py, and *.DS_Store should be stored as LFS.
2. Afterwards, call git add --renormalize *.json, git add --renormalize *.py , git add --renormalize *.md.

Initially, I also turned tokenizer.json into a normal git file, and it got stuck uploading (which is why this PR has been open for 3 hours before the first commit). I ran git config --global http.postBuffer 524288000 to resolve it. Afterwards, I got this error:

remote: -------------------------------------------------------------------------
remote: Your push was rejected because it contains files larger than 10 MiB.
remote: Please use https://git-lfs.github.com/ to store large files.
remote: See also: https://hf.co/docs/hub/repositories-getting-started#terminal
remote:
remote: Offending files:
remote:   - tokenizer.json (ref: refs/pr/8)
remote: -------------------------------------------------------------------------

So I re-included tokenizer.json as git-lfs, and then I could make this PR like normal.

So: This PR only changes how files are stored, it does not change the content of any file! Once this is merged, then I can update #7 so that you can inspect those changes more easily.

cc @nielsr @celinah

• Tom Aarsen