Together AI's Compiler Uses Agents to Speed Up GPU Inference by 20%

Together AI has launched Together Compiler , a hosted system that uses software agents and benchmark feedback to optimize GPU kernels for large language model inference. The company says the system already powers three serverless endpoints, where it has increased decode speed by 12% to 20%. Existing users receive the optimized kernels automatically at standard per-token prices, while dedicated endpoint access remains in limited beta.

Kernel optimization has a short shelf life because production performance depends on a specific combination of model architecture, traffic pattern, serving configuration, and accelerator. New attention mechanisms, mixture-of-experts designs, longer agent workloads, and NVIDIA hardware releases can invalidate assumptions embedded in a tuned kernel or engine configuration.

Why tuned stacks go stale

According to Together, 80% of OpenRouter traffic goes to models released within the previous three months, and the 10 most popular open-source models use different architectures. Those differences affect memory access, communication between GPUs, batching, attention computation, and opportunities to fuse operations. Hardware changes from NVIDIA’s B200 to GB300 and the planned Vera Rubin generation add another set of constraints.

Together uses “Static Inference” to describe deployments optimized for a fixed model, workload, and hardware platform. Its “Polymorphic Inference” label covers serving systems that continually adapt kernel selection and runtime configuration across dense and sparse attention, low-concurrency chat, high-volume agents, and multiple GPU generations.

Together’s earlier work favored megakernels, which combine many model operations into one large GPU program. Fusion can reduce memory movement and kernel-launch overhead, but it also embeds assumptions about tensor shapes, attention patterns, and hardware. The company now treats fusion as one tactic within a broader workload-specific search.

Inside the agent loop

GPU kernels are low-level programs that execute operations such as matrix multiplication and attention on an accelerator. Together Compiler treats their implementation and selection as an empirical search process:

1. Agents identify operations and optimization patterns worth testing.
2. They generate candidate kernel implementations and configurations.
3. A high-fidelity persistent inference engine measures each candidate under production-like conditions.
4. The measurements guide subsequent candidates and determine which variants qualify for deployment.

Together says this loop can run overnight on work that previously required weeks of manual engineering. The company calls the process production-scale recursive self-improvement because benchmark results guide successive rounds of generated optimization code. The served model’s weights remain outside that loop.

“Continuous Compilation” is Together’s term for rerunning the search as models, traffic distributions, service-level targets, and accelerators change. The approach allows the serving layer to maintain multiple optimized implementations instead of relying on one permanent kernel configuration.

Twelve to 20 percent faster decode

Together reports the following relative improvements over kernels already tuned by its Frontier Performance team:

The decode phase generates output tokens after the system processes the prompt. Faster decode can reduce time between tokens, increase aggregate output throughput, or produce a combination of both, depending on concurrency and scheduling.

These percentages are relative vendor measurements rather than full cross-provider benchmarks. Without absolute tokens per second, GPU configuration, model precision, batch size, sequence lengths, concurrency, and percentile latency, they cannot establish how the endpoints compare with other serving platforms.

Access follows the endpoint

Serverless customers using the three named models receive the optimized kernels without changing application code. Dedicated customers can request beta access through Together’s sales channel, while self-hosted deployments remain outside the product’s current scope.

At unchanged per-token rates, serverless users primarily gain lower generation latency and potentially steadier performance under load. Dedicated customers paying for reserved capacity may also gain more throughput from the same hardware. Workloads with long outputs, including reasoning models and tool-using agents, have more decode work to optimize than short-response applications dominated by prompt processing.

Together Compiler remains coupled to the company’s serving stack and persistent engine. Teams running vLLM, SGLang, or another self-hosted engine cannot install the compiler or import its generated kernels. The announcement also provides no direct performance comparison with those engines.

The benchmark gaps

Together’s published results leave several implementation questions open for developers evaluating the dedicated beta:

• Benchmark conditions: GPU model and count, parallelism strategy, numerical precision, quantization, concurrency, and sequence lengths.
• Latency distribution: median and tail time to first token, inter-token latency, end-to-end latency, and aggregate throughput.
• Correctness: numerical tolerances, output validation, and quality checks for generated kernels.
• Rollout controls: canary deployment, artifact versioning, reproducibility, rollback, and configuration pinning.
• Model compatibility: support for custom weights, adapters, quantized variants, post-trained models, and custom operators.
• Retuning cadence: triggers for recompilation and the time required to validate a new optimization.

A production trial can answer those questions by replaying representative traffic and measuring cost per request, tokens per GPU-second, percentile latency, error rates, and output quality before and after compilation.

The search moves beyond kernels

Together’s roadmap extends the same feedback-driven approach to other parts of the serving stack. Its speculative decoding work aims to adapt draft-model behavior for each request. Speculative decoding uses a smaller model to propose tokens that a larger model verifies, reducing generation time when the proposals are accurate.

The company is also developing ThunderAgent Router, which would steer requests across workers with different hardware or serving configurations. Together presents both projects as roadmap work, with no public availability date in the announcement.

Together’s commercial bet centers the speed and quality of its optimization loop as a competitive advantage. Hosting evaluations can test that claim through workload-specific benchmarks, evidence of retuning frequency, and operational controls for validation and rollback. The current launch supplies early relative gains and a live serverless deployment; the dedicated beta will show how well the approach generalizes across customer models, hardware, and traffic patterns.