PyTorch's Helion Skips Benchmarking and Still Beats Handwritten GPU Kernels

Helion , PyTorch’s domain-specific language for portable machine-learning kernels, now uses compile-time heuristics to choose GPU configurations without benchmarking candidates. The feature has merged into Helion and is enabled by default for H100 and B200 GPUs using the Triton backend.

The compile-time heuristics inspect a kernel’s intermediate representation (IR) and dataflow, classify the workload as matrix multiplication, reduction, or pointwise computation, and select a configuration directly. Helion’s previous default often trailed handwritten kernels, and recovering that performance through autotuning could consume hours for every set of input shapes.

The cost of searching thousands of variants

A single Helion program can lower into thousands of Triton variants. The autotuner compiles and times candidates to choose block sizes, warp counts, pipeline stages, and cache-eviction policies. Because the best combination can change with tensor shape, the work repeats across kernels and shapes. In one reported test, tuning a paged-attention kernel over 72 shapes required about 10 hours in Helion’s quick mode and 25 hours at full effort.

Inside Helion’s resource model

At compile time, Helion walks the device IR and performs liveness analysis, which tracks the intermediate values that must remain available at each instruction. For each candidate block size, it estimates the registers and shared memory required by one cooperative thread array (CTA), CUDA’s term for a thread block. On Blackwell GPUs, the model also accounts for tensor memory.

Those estimates determine two constraints that shape the final configuration:

• Occupancy: the number of CTAs that can reside concurrently on a streaming multiprocessor (SM), capped by the first exhausted hardware resource.
• Register pressure: the risk that the assembler will run out of registers and spill values into slower memory.

The matrix-multiplication heuristic shrinks block sizes when a workload would otherwise launch fewer CTAs than the GPU has SMs. It balances pipeline depth against shared-memory capacity and chooses num_warps from the amount of dynamic matrix-multiplication work, including how often hl.dot executes at runtime.

The reduction heuristic alternates between warp count and block size until both choices stabilize. It also models repeated access to the same data. RMSNorm, for example, reads a row to calculate its scale and reads it again to apply that scale, so retaining the relevant data can reduce memory traffic.

Compile-time choices approach tuned speed

PyTorch reports performance relative to each named baseline, with 1.0× indicating equal performance and values above 1.0× indicating higher throughput. The linear-attention results cover 96 kernel-and-shape combinations across nine variants on H100. The common-operator suite includes RMSNorm, LayerNorm, softmax, and cross entropy.

The vLLM comparison covers FP8 quantization, RMSNorm, fused SiLU-multiply, and QK normalization with RoPE. Its heuristic result sits about 3% below full autotuning, leaving a relatively small gain for the benchmark search to recover.

Jagged HSTU attention remains a regression because the previous generic default happened to select a favorable configuration. That exception highlights the model’s dependence on accurate resource estimates and workload classification.

A head start for the autotuner

The same heuristics can seed autotuning with 5 to 20 related configurations. Across 38 matrix-multiplication and multi-matrix-multiplication kernels, seeding sharply reduced the median number of candidates required to reach an acceptable result.

Seeding cuts the median work needed to find a configuration within 50% of the best result by 35 times. The advantage narrows near the optimum, reducing a complete search from 521.5 to 420.5 candidates. The eventual winning configuration remains unchanged, and replaying the winner from each run produces performance within a fraction of a percent.

Where the model can miss

Helion predicts how the downstream Triton compiler will allocate registers and place values in memory. Differences between that prediction and the generated assembly can produce a poor configuration. The analysis accepts hardware-specific resource limits, although its constants and decision rules were calibrated against H100 and B200 GPUs.

Closing the prediction gaps

• Read actual assembler resource usage after one compilation and use it to correct the estimates without timing the kernel.
• Apply heuristics when tensor extents remain unknown at compile time.
• Add workload-specific branches, such as a vLLM quantization specialization within the reduction heuristic.

Developers can inspect the open-source heuristic implementation to see how Helion selects individual configuration values.

When full tuning still pays

Full autotuning remains appropriate for stable workloads whose fixed set of shapes justifies the one-time search cost. The compile-time heuristic suits dynamic workloads and development environments where per-shape benchmarking would delay deployment. Seeded tuning serves workloads that need closer-to-peak performance with a smaller search.