BlinkDL
/

rwkv-4-pile-7b

Text Generation

Model card Files Files and versions Community

BlinkDL commited on Feb 22, 2023

Commit

ea1f914

•

1 Parent(s): b97b0b6

Update README.md

Files changed (1) hide show

README.md +2 -2

README.md CHANGED Viewed

@@ -24,10 +24,10 @@ ctx_len = 1024
 n_layer = 32
 n_embd = 4096
-(there are ctx_len 2048 and 4096 models though they might be slightly weaker at generating short contents)
 Final checkpoint: RWKV-4-Pile-7B-20221115-8047.pth : Trained on the Pile for 332B tokens.
-* Pile loss 1.8415
 * LAMBADA ppl 4.38, acc 67.18%
 * PIQA acc 76.06%
 * SC2016 acc 73.44%

 n_layer = 32
 n_embd = 4096
+(There are ctx_len 2048 and 4096 models too. Use them only when your ctxlen is long. Might be slightly weaker for short ctxlens.)
 Final checkpoint: RWKV-4-Pile-7B-20221115-8047.pth : Trained on the Pile for 332B tokens.
+* Pile loss 1.8415T
 * LAMBADA ppl 4.38, acc 67.18%
 * PIQA acc 76.06%
 * SC2016 acc 73.44%