|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
The *TokenFormer* is a **fully attention-based architecture** |
|
that unifies the computations of token-token and token-parameter interactions |
|
by entirely employing the attention mechanism, **maximizes the flexibility of neural network**.[(see paper)](https://github.com/Haiyang-W/TokenFormer). |
|
It contains four models of sizes |
|
150M, 450M, 900M, 1.5B. For each size, it's trained based on [gpt-neox](https://github.com/EleutherAI/gpt-neox) code base and uses [Pile](https://huggingface.co./datasets/EleutherAI/pile) with 300B tokens. |
|
All 4 model sizes are trained on the exact |
|
same data, in the exact same order. |
|
|
|
# TokenFormer-450M |
|
|
|
## Model Details |
|
|
|
- Developed by: [Haiyang Wang](https://haiyang-w.github.io/) |
|
- Model type: ToeknFormer-based Language Model |
|
- Language: English |
|
- Learn more: [TokenFormer's GitHub repository](https://github.com/Haiyang-W/TokenFormer) |
|
for training procedure, config files, and details on how to use. |
|
[See paper](https://github.com/Haiyang-W/TokenFormer) for more evals and implementation |
|
details. |
|
- Library: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox) |
|
- License: Apache 2.0 |
|
- Contact: to ask questions about this model, please email Haiyang Wang. |
|
|
