pkedzia commited on
Commit
f641a86
·
1 Parent(s): dbbeae1

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +12 -1
README.md CHANGED
@@ -20,4 +20,15 @@ datasets:
20
  This is polish fast tokenizer.
21
 
22
  Number of documents used to train tokenizer:
23
- - 25 088 398
 
 
 
 
 
 
 
 
 
 
 
 
20
  This is polish fast tokenizer.
21
 
22
  Number of documents used to train tokenizer:
23
+ - 25 088 398
24
+
25
+
26
+ Sample usge with transformers:
27
+
28
+ ```[python]
29
+ from transformers import AutoTokenizer
30
+
31
+ tokenizer = AutoTokenizer.from_pretrained('radlab/polish-fast-tokenizer')
32
+ tokenizer.decode(tokenizer("Ala ma kota i psa").input_ids)
33
+
34
+ ```