prithivMLmods commited on
Commit
bb1e4e2
·
verified ·
1 Parent(s): a7be6f8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -12,7 +12,7 @@ tags:
12
  ---
13
  # **MBERT Context Specifier**
14
 
15
- *MBERT Context Specifier* with 150M 150M params is a text-based context labeler or classifier trained using the modernized bidirectional encoder-only Transformer model (BERT-style). This model is pre-trained on 2 trillion tokens of English and code data, with a native context length of up to 8,192 tokens. It incorporates the following features:
16
 
17
  1. **Rotary Positional Embeddings (RoPE):** Enables long-context support.
18
  2. **Local-Global Alternating Attention:** Enhances efficiency when processing long inputs.
 
12
  ---
13
  # **MBERT Context Specifier**
14
 
15
+ *MBERT Context Specifier* with 150M params is a text-based context labeler or classifier trained using the modernized bidirectional encoder-only Transformer model (BERT-style). This model is pre-trained on 2 trillion tokens of English and code data, with a native context length of up to 8,192 tokens. It incorporates the following features:
16
 
17
  1. **Rotary Positional Embeddings (RoPE):** Enables long-context support.
18
  2. **Local-Global Alternating Attention:** Enhances efficiency when processing long inputs.