Latent Sparse Attention Speeds Up 10M-Token AI Contexts by 9x

Takeaways

• Researchers from Prefrontal AI and ETH Zurich developed Latent Sparse Attention (LSA) , which in 10-million-token tasks cut time to first token by over 9x versus full attention and 2.2x versus a popular sparse attention baseline.
• In a 10-million-token experiment using GLM-5.3-Flash (a DeepSeek Sparse Attention variant), the indexer's key-selection step consumed 98% of an attention layer's prefill time, highlighting the search bottleneck LSA addresses.
• On an NVIDIA H200 at 10 million tokens, D-LSA sped up one GLM attention layer's prefill by 22.7x and decoding by 4.6x , while Qwen-3.8-27B prefill was 72.4x faster than full attention.
• Across eight H200 GPUs in end-to-end serving, GLM 's time to first token improved by 2.2x and time per output token by 2.1x , while Qwen 's time to first token improved by 9.2x ; on the RULER benchmark, Qwen scored 63.9 versus 59.9 for full attention at 10 million tokens.
• Testing N-LSA on a 500-million-parameter hybrid model trained up to 128,000 tokens showed matched validation loss versus full attention, with training 1.6x faster per attention layer and 1.24x faster end to end, and code released under the Apache 2.0 license.