Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy
Nvidia researchers found that reducing the configured context window from 256,000 to 16,000 tokens on NVIDIA H100 GPUs increased output-token throughput by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.
Source
Ben Dickson
Published
Author
Ben Dickson
Read
1 min read
Takeaways
Nvidia researchers found that reducing the configured context window from 256,000 to 16,000 tokens on NVIDIA H100 GPUs increased output-token throughput by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.
Across 13 benchmarks (including SWE-bench Verified, Terminal-Bench, LongBench v2, and LOONG), threshold-based compaction (triggered at thresholds like 16K tokens) kept accuracy close to full-history baselines on long-horizon tasks.
On SWE-bench Verified, Claude Opus 4.5 improved from 70% to 74.8% accuracy with its best compaction setting, while GPT-5.6 Sol rose from 84.4% to 88.8% .
With single-action compaction on SWE-bench Verified, both models nearly matched full-history performance, and across two evaluated benchmarks the median peak context was just 7,000-9,000 tokens.
Model sensitivity varied sharply: Qwen 3 Coder Next's accuracy on SWE-bench Verified collapsed from 53% to 3.6% under a 16K -token limit, and GPT 5.6 Sol's API cost rose from $167.68 without compaction to $203.12 at the 16K limit due to prompt-cache invalidation.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.