Length Generalization of Causal Transformers without Position Encoding
Paper
•
2404.12224
•
Published
•
1
Papers - Attention - NoPE - Long Context with SoftMax Temp
Uniform scaling not as good as Head-based scaling