APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News
deep_diveAgentsInfra

Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy

Nvidia researchers found that reducing the configured context window from 256,000 to 16,000 tokens on NVIDIA H100 GPUs increased output-token throughput by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.

Nvidia Shows How to Cut AI Context Without Sacrificing Accuracy
Source
Ben Dickson
Published
Author
Ben Dickson
Read
1 min read

Takeaways

  • Nvidia researchers found that reducing the configured context window from 256,000 to 16,000 tokens on NVIDIA H100 GPUs increased output-token throughput by 3.6x for GLM 5.2, 5.6x for Kimi K2.6, and 7x for Qwen 3 Coder Next.
  • Across 13 benchmarks (including SWE-bench Verified, Terminal-Bench, LongBench v2, and LOONG), threshold-based compaction (triggered at thresholds like 16K tokens) kept accuracy close to full-history baselines on long-horizon tasks.
  • On SWE-bench Verified, Claude Opus 4.5 improved from 70% to 74.8% accuracy with its best compaction setting, while GPT-5.6 Sol rose from 84.4% to 88.8% .
  • With single-action compaction on SWE-bench Verified, both models nearly matched full-history performance, and across two evaluated benchmarks the median peak context was just 7,000-9,000 tokens.
  • Model sensitivity varied sharply: Qwen 3 Coder Next's accuracy on SWE-bench Verified collapsed from 53% to 3.6% under a 16K -token limit, and GPT 5.6 Sol's API cost rose from $167.68 without compaction to $203.12 at the 16K limit due to prompt-cache invalidation.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report