Software Tuning Boosts DeepSeek V4.1 Flash Throughput 60% on Unchanged NVIDIA Hardware

Artificial Analysis published new AgentPerf results for the DeepSeek V4.1 Flash model on NVIDIA’s GB300 NVL72, a rack-scale system linking 72 Blackwell Ultra GPUs. Compared with a run one month earlier, the same hardware generated 42% more tokens per second for one active agent and sustained about 60% more throughput per GPU at high concurrency. The software changes affect both response speed and the number of agents a cluster can serve.

With one active agent, generation reached 848 tokens per second, or tok/s, up from 596 tok/s for a 1.42× gain. At high concurrency, throughput per GPU reached about 1.6 times its previous level while each agent remained near 80 tok/s. The physical hardware stayed unchanged between measurements, attributing the reported gain to serving software and configuration.

Five serving changes produced the gain

• Model-specific vLLM kernels: A newer vLLM release added kernels tuned for DeepSeek V4.1 on GB300.
• FlashInfer autotuning: The kernel library selected faster implementations for attention and general matrix multiplication, the dense linear algebra used throughout the model.
• Broader CUDA graph coverage: Recording and replaying repeated GPU work reduced per-step Python processing and kernel-launch overhead.
• Better speculative decoding: Updated tuning improved the process in which a smaller draft model proposes tokens for the main model to verify in parallel.
• Colocated prefill and decode: NVIDIA’s Dynamo scheduler with vLLM now processes prompts and generates output tokens on the same GPUs. The earlier configuration assigned those phases to separate GPU pools.

Long contexts reshape the benchmark

AA-AgentPerf replays multi-turn coding-agent trajectories whose context grows as an agent works through a task. Inputs range from 1,000 to 131,000 tokens, with a mean near 27,000. Outputs are shorter, with a median near 150 tokens and a 95th percentile around 2,000. The benchmark reuses prior turns through the KV cache, which stores attention state from earlier tokens, so only newly appended tokens require prefill processing.

With long inputs and short outputs, separate prefill and decode pools can leave decode workers underused. Colocation allows Dynamo to reuse KV-cache data in place and schedule both phases across the same accelerators, improving utilization for this workload shape.

AgentPerf isolates model-serving performance on recorded trajectories. End-to-end agent latency also includes tool calls, sandbox startup, code execution, storage access, and network time.

Rubin’s lead changes with service speed

NVIDIA also submitted updated Vera Rubin NVL72 configurations for DeepSeek V4.1 Flash. The table reports absolute generation rates for the single-agent test and relative throughput per GPU at two fixed per-agent speeds. For the fixed-speed rows, GB300 is normalized to 1.0 because absolute throughput figures were not provided.

Rubin’s relative advantage depends on the target service rate. At 260 tok/s per agent, it delivers five times the throughput per GPU; at 80 tok/s, the advantage narrows to 1.7×. At the lower rate, GB300 decode runs closer to a memory-bandwidth limit, where each generated token requires moving model weights and cached attention data. Rubin’s additional resources therefore produce a smaller relative gain.

How to use the curves

• Treat the stack as a versioned dependency. Reproducing the GB300 gains requires the corresponding vLLM kernels, FlashInfer tuning, CUDA graph coverage, speculative decoding settings, and Dynamo placement policy.
• Choose the service target first. Compare hardware at the required per-agent generation rate, concurrency, and latency objective. A single peak-throughput figure hides where each system sits on the curve.
• Retune speculative decoding. Draft-model acceptance rates and verification costs vary by model, workload, and accelerator generation.
• Use representative traces. Prompt length, output length, concurrency, and KV-cache reuse determine whether colocated or disaggregated execution uses the GPUs more efficiently.

Budget for post-launch software gains

The one-month GB300 change shows how launch-period benchmarks can understate the capacity of unchanged hardware. Cluster models can account for both current measured performance and a mature-stack scenario, since throughput per GPU affects accelerator count while per-agent generation speed affects model wait time.

Procurement validation should reproduce NVIDIA’s submitted Rubin configurations with the same model build, numerical precision, serving versions, prompt distribution, concurrency, and latency target. Those controls distinguish platform gains from changes in software or workload mix.