Artificial Analysis Ditches Tokens per Second to Rank AI Speed

Artificial Analysis has overhauled its benchmarking methodology , replacing tokens per second as the headline measure with Time per Task. The new metric tracks how long a model takes to complete a benchmark task, including prompt processing, hidden reasoning, queue delays, and output generation. The change applies to roughly 1,000 serverless endpoints tested multiple times each day.

When tokens per second misleads

Tokens per second measures streaming speed without accounting for how many tokens a model generates. A model producing 150 tokens per second may still take longer to finish if it generates 170,000 reasoning and answer tokens, while another model can complete the same task sooner with 50,000 tokens at 50 tokens per second. That difference compounds in agent workflows that make repeated or chained model calls.

Time per Task divides the output tokens generated for each Intelligence Index task by the endpoint’s end-to-end output speed. Artificial Analysis defines that speed across the full interval from request submission to the final token, capturing input processing, hidden reasoning, queuing, and decoding. The Intelligence Index is the firm’s composite score for model performance across its evaluation suite.

Provider APIs now supply the token counts used in the calculation. The previous methodology normalized counts with OpenAI’s o200k_base tokenizer, which could represent another model’s tokenization inaccurately. Artificial Analysis says it monitors provider-reported counts for discrepancies.

A broader mix of workloads

The revised test harness calculates headline results as five-day averages across workloads designed to resemble production agent traffic:

• Short, document-length, and long-context prompts
• Multi-turn coding and knowledge-work sessions
• Single-request and 10-request concurrent loads
• Vision prompts containing a 1-megapixel image and about 1,000 input tokens

The testing schedule varies by workload. Tests with 1,000-token inputs, 10,000-token inputs, and images run about every three hours. Parallel-load tests run daily, while 100,000-token input tests run weekly.

Multi-turn coding tests also measure prompt caching under a representative agent workflow. Each turn resends the expanding conversation, allowing an endpoint to reuse cached input when the provider supports it. This pattern reflects agents that repeatedly send their history, tool results, and updated instructions.

Reliability now affects the performance score directly. Errors, rate limits, and timeouts receive a speed of zero, and every endpoint displays an Endpoint Error Rate. For reasoning models that conceal internal thinking tokens, Artificial Analysis estimates output speed from the final 80% of answer chunks to better represent the visible streaming phase.

Default effort changes the bill

Artificial Analysis now benchmarks each model at both its API default and maximum reasoning-effort settings. Because labels such as low, medium, and high do not represent equivalent computation across model providers, the benchmark retains each provider’s configured default rather than imposing a nominally shared setting.

The published comparisons show large latency, cost, and benchmark-score differences between maximum and default effort. Three leading reasoning models use medium effort by default:

Across those models, default effort reduces Time per Task by roughly three to five times and Cost per Task by about two to four times. Claude Opus 5.5, for example, falls from approximately $10.04 per task at maximum effort to $2.45 at medium effort. The corresponding Intelligence Index decline ranges from 5 to 11 points.

Fewer tokens beat faster streams

On the revised frontier chart , GPT-6 Astra and GPT-6.1 Sol occupy the Intelligence-versus-Time-per-Task Pareto frontier from roughly 45 to 55 Intelligence Index points. A model on that frontier offers a combination of score and completion time that no plotted alternative improves in one dimension without sacrificing the other.

GPT-6 Astra illustrates why total generation length now matters more. At maximum effort, it produces 51 tokens per second and finishes in about 17 minutes. Claude Opus 5.5 reaches 111 tokens per second but takes roughly 26 minutes, while Gemini 4 Carbon reaches 71 tokens per second and takes about 29 minutes. Astra generates around 50,000 output tokens per task, compared with approximately 120,000 to 170,000 for the other models.

GLM-5.3 exposes a different weakness in the former ranking. Its fastest-streaming endpoints take 18 to 26 seconds to produce the first token, while most competing endpoints begin within one to three seconds. The earlier methodology emphasized their subsequent streaming rate; Time per Task includes the initial wait and lowers their relative position.

How developers should read the rankings

Model and provider selection now requires comparing workload shape, reasoning effort, reliability, and total completion time alongside per-token throughput. The revised benchmark changes several practical decisions:

1. Compare matching effort settings. Maximum-effort scores can overstate the latency and cost of deployments that use API defaults.
2. Use Time per Task for agent loops. Total completion time captures the cost of verbose reasoning across repeated calls.
3. Check first-token latency and error rates. Fast decoding cannot recover time lost to queues, delayed starts, rate limits, or retries.
4. Match the benchmark workload to production. Long context, concurrency, vision, and multi-turn caching can change endpoint performance substantially.

Teams evaluating inference providers can use the default-effort results as a closer baseline for production deployments, then test maximum effort where additional Intelligence Index points justify the higher latency and per-task cost.