Update README.md
Browse files
README.md
CHANGED
@@ -37,7 +37,7 @@ print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
|
|
37 |
For **infill** sampling, we introduce three new special token types:
|
38 |
|
39 |
* `<mask_N>`: N-th span to be masked. In practice, use `<mask_1>` to where you want to sample infill.
|
40 |
-
* `<sep>`:
|
41 |
* `<eom>`: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.
|
42 |
|
43 |
For example, if we want to generate infill for the following cursor position of a function:
|
|
|
37 |
For **infill** sampling, we introduce three new special token types:
|
38 |
|
39 |
* `<mask_N>`: N-th span to be masked. In practice, use `<mask_1>` to where you want to sample infill.
|
40 |
+
* `<sep>`: Separator token between the suffix and the infilled sample. See below.
|
41 |
* `<eom>`: "End-Of-Mask" token that model will output at the end of infilling. You may use this token to truncate the output.
|
42 |
|
43 |
For example, if we want to generate infill for the following cursor position of a function:
|