Update README.md
Browse files
README.md
CHANGED
|
@@ -2,7 +2,15 @@
|
|
| 2 |
library_name: tokenizers
|
| 3 |
tags: [Danish, Morphological Tokenization, CerebrasGPT]
|
| 4 |
---
|
| 5 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 6 |
### DA-MORPH-CEREBRAS-TOKEN
|
| 7 |
|
| 8 |
This morphological tokenizer is designed for the CerebrasGPT architecture and focuses on segmenting Danish text based on linguistic principles, enabling more meaningful subword tokenization.
|
|
|
|
| 2 |
library_name: tokenizers
|
| 3 |
tags: [Danish, Morphological Tokenization, CerebrasGPT]
|
| 4 |
---
|
| 5 |
+
```
|
| 6 |
+
_______ ___ .___ ___. ______ .______ .______ __ __
|
| 7 |
+
| \ / \ | \/ | / __ \ | _ \ | _ \ | | | |
|
| 8 |
+
| .--. | / ^ \ | \ / | | | | | | |_) | | |_) | | |__| |
|
| 9 |
+
| | | | / /_\ \ | |\/| | | | | | | / | ___/ | __ |
|
| 10 |
+
| '--' | / _____ \ | | | | | `--' | | |\ \----.| | | | | |
|
| 11 |
+
|_______/ /__/ \__\ |__| |__| \______/ | _| `._____|| _| |__| |__|
|
| 12 |
+
|
| 13 |
+
```
|
| 14 |
### DA-MORPH-CEREBRAS-TOKEN
|
| 15 |
|
| 16 |
This morphological tokenizer is designed for the CerebrasGPT architecture and focuses on segmenting Danish text based on linguistic principles, enabling more meaningful subword tokenization.
|