InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU
Paper
•
2502.08910
•
Published
•
142
SYS
-
-
TransMLA: Multi-head Latent Attention Is All You Need
Paper • 2502.07864 • Published • 45 -
LM2: Large Memory Models
Paper • 2502.06049 • Published • 29 -
Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
Paper • 2502.11089 • Published • 135 -
Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity
Paper • 2502.13063 • Published • 63 -
Tensor Product Attention Is All You Need
Paper • 2501.06425 • Published • 84 -
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
Paper • 2501.11873 • Published • 63 -
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
Paper • 2501.15570 • Published • 23