Research
157d agoFeature Sparsity Boosts Transformer Scaling
A new method called Sparse Feature Attention cuts transformer self‑attention costs by exploiting sparsity in feature dimensions, rather than sequence length. By encoding queries and keys as k‑sparse codes, the approach reduces complexity from Θ(n²d) to Θ(n²k²/d) while preserving accuracy.
The Signal
The FlashSFA kernel, an extension of FlashAttention, enables efficient large‑scale deployment, promising faster, lower‑cost language models worldwide.