Hugging Face's logo

xiaol
/
rwkv-7B-world-novel-128k

Model card Files Community
2
xiaol commited on
Commit
524a6e3
•
1 Parent(s): b394794

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +7 -1
README.md CHANGED
@@ -4,12 +4,16 @@ datasets:
4
  - Norquinal/claude_multiround_chat_30k
5
  - OpenLeecher/Teatime
6
  ---
7
- We proudly announce this is the world first 128k context model based on RWKV architecture today, 2023-08-10.
 
 
 
8
 
9
  With RWKV world tokenizer,multi-langs have 1:1 tokenization ratio ,one word to one token.
10
  (https://github.com/BlinkDL/ChatRWKV/blob/2a13ddecd81f8fd615b6da3a8f1091a594689e30/tokenizer/rwkv_tokenizer.py#L163)
11
 
12
 
 
13
 
14
  This model trained with instructions datasets and chinese web novel and tradition wuxia,
15
  more trainning details would be updated.
@@ -21,6 +25,8 @@ https://github.com/SynthiaDL/TrainChatGalRWKV/blob/main/train_world.sh
21
 
22
  ![QQ图片20230810153529.jpg](https://cdn-uploads.huggingface.co/production/uploads/6176b32847ee6431f632981e/d8ekmc4Lfhy2lYEdrRKXz.jpeg)
23
 
 
 
24
  Using RWKV Runner https://github.com/josStorer/RWKV-Runner to test this model, only need 16G vram to run fp16 or 8G vram fp16i8, use temp 0.1-0.2 topp 0.7 for more precise answer ,temp between 1-2.x is more creatively.
25
  ![微信截图_20230810162303.png](https://cdn-uploads.huggingface.co/production/uploads/6176b32847ee6431f632981e/Ww45-WMngl4Jyt1OZDAa_.png)
26
 
 
4
  - Norquinal/claude_multiround_chat_30k
5
  - OpenLeecher/Teatime
6
  ---
7
+
8
+ # RWKV 7B World 128k for novel writing
9
+
10
+ We proudly announce this is the world first **128k context** model based on RWKV architecture today, 2023-08-10.
11
 
12
  With RWKV world tokenizer,multi-langs have 1:1 tokenization ratio ,one word to one token.
13
  (https://github.com/BlinkDL/ChatRWKV/blob/2a13ddecd81f8fd615b6da3a8f1091a594689e30/tokenizer/rwkv_tokenizer.py#L163)
14
 
15
 
16
+ # How to train infinte context model?
17
 
18
  This model trained with instructions datasets and chinese web novel and tradition wuxia,
19
  more trainning details would be updated.
 
25
 
26
  ![QQ图片20230810153529.jpg](https://cdn-uploads.huggingface.co/production/uploads/6176b32847ee6431f632981e/d8ekmc4Lfhy2lYEdrRKXz.jpeg)
27
 
28
+ # How to Test?
29
+
30
  Using RWKV Runner https://github.com/josStorer/RWKV-Runner to test this model, only need 16G vram to run fp16 or 8G vram fp16i8, use temp 0.1-0.2 topp 0.7 for more precise answer ,temp between 1-2.x is more creatively.
31
  ![微信截图_20230810162303.png](https://cdn-uploads.huggingface.co/production/uploads/6176b32847ee6431f632981e/Ww45-WMngl4Jyt1OZDAa_.png)
32