Salesforce
/

xgen-7b-8k-base

Text Generation

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

psinger commited on Jun 30, 2023

Commit

3568622

•

1 Parent(s): f5839f9

Update tokenization_xgen.py

Files changed (1) hide show

tokenization_xgen.py +1 -0

tokenization_xgen.py CHANGED Viewed

@@ -115,6 +115,7 @@ class XgenTokenizer(PreTrainedTokenizer):
     def __init__(
             self,
             pad_token=None,
             add_eos_token=False,
             add_special_tokens=True,
             **kwargs,

     def __init__(
             self,
             pad_token=None,
+            eos_token=<|endoftext|>",
             add_eos_token=False,
             add_special_tokens=True,
             **kwargs,