Artificial Analysis Adds HLE-Diamond and Gemini 4 Carbon Tops at 69%

Artificial Analysis has added HLE-Diamond to version 5 of its Intelligence Index, replacing the original Humanity’s Last Exam benchmark. Its independent leaderboard places frontier models well ahead of released open-weight systems. Because HLE-Diamond contributes 10% of the aggregate index score, the change can also shift broader model rankings.

A cleaner 1,000-question exam

Humanity’s Last Exam was created by the Center for AI Safety and Scale AI to test models on expert-level academic questions across many disciplines. Audits found incorrect or ambiguous reference answers in a meaningful share of the original set, making close comparisons between frontier models difficult to trust.

HLE-Diamond emerged from a year-long cleanup conducted with research communities. It contains 1,000 questions divided evenly between reasoning and knowledge, with each item labeled by partition. The set retains 272 image-based questions, allowing it to test multimodal performance instead of relying on the text-only subset that many labs previously reported.

Stable question IDs make repeated evaluations easier to compare when prompts, judge settings, and other conditions remain unchanged. The frozen set also lets researchers trace score changes to specific items.

One shot, no tools

Artificial Analysis submits every question once without external tools and uses the official HLE system prompt. The resulting pass@1 score measures the percentage answered correctly on the first attempt, with no retries or selection among multiple responses.

GPT-5.6 Luna at medium effort grades the responses using the official HLE judge prompt. The judge compares each response with the reference answer, allows a small tolerance for numerical results, and marks missing or malformed answers as incorrect.

Models without image support receive only the text associated with the 272 visual questions, while all 1,000 items remain in their denominator. Their final scores therefore reflect both academic performance and the absence of image input.

Within the Intelligence Index v5 , HLE-Diamond sits in the General category and carries a 10% weight. Replacing the original HLE can affect the aggregate ranking because models perform differently on the cleaned set.

Frontier variants clear 60%

Only 17 model variants, about 13% of the evaluated field, exceeded 50% at publication. Each variant combines a model with an inference-effort setting, so several rows may belong to the same model family.

Reasoning outruns knowledge

Reasoning scores exceed knowledge scores for more than 90% of the measured models. GPT-6.1 Sol at maximum effort shows one of the largest gaps, scoring 76% on reasoning and 41% on knowledge, a difference of 35 percentage points.

The pattern is consistent with recent training and inference methods that allocate more compute to multi-step problem solving. HLE-Diamond alone cannot identify the cause, but its partition labels make the imbalance visible and allow future evaluations to track whether factual performance catches up.

Released weights trail by 40 points

The highest entry in the open-weight section is Kimi K3.1 Pro at 37%, although its weights had not been released at publication. The leading entries associated with open-weight models were:

• Kimi K3.1 Pro: 37%, with weights pending at publication
• MiniMax-M3.1-Flash: 28%
• Kimi K3 (max): 27%

Among entries with released weights, MiniMax-M3.1-Flash trails Gemini 4 Carbon by 41.1 percentage points. Teams considering self-hosted models therefore face a substantial accuracy trade-off on this benchmark, especially for scientific and technical question answering. Workload-specific testing remains necessary because HLE-Diamond covers closed-ended academic questions under a no-tools setup.

More effort brings a steep bill

Higher inference effort improves accuracy while sharply increasing cost per task. Claude Opus 5.5 rises from 42% at low effort to 68% at maximum effort, a 26-point gain at roughly 35 times the cost. GPT-6 Astra moves from 45% to 64%, gaining 19 points at about 11 times the cost.

The best operating point depends on the value of a correct answer, latency requirements, retry policies, and token budgets. The leaderboard can plot score against cost per task, allowing teams to compare the marginal accuracy gained at each effort setting.

The “last exam” still has room

Humanity’s Last Exam was introduced as a closed-ended academic test capable of stumping frontier AI. Roughly 20 months after its release, Gemini 4 Carbon answers 69.1% of the cleaned questions correctly without tools. That result covers most of the benchmark while leaving a 30.9% error rate overall and a much larger weakness on the knowledge partition.

CAIS separately reports an 82.9% tool-assisted result. That evaluation uses a different condition from Artificial Analysis’s no-tools leaderboard and measures the added benefit of external tools as well as the underlying model.

HLE-Diamond provides a more stable basis for tracking progress through its corrected references, fixed IDs, multimodal questions, and separate reasoning and knowledge scores. Future results can show whether models close the knowledge gap and how much inference cost is required to cross a given accuracy threshold.

Turn the scores into model choices

Developers and researchers can apply the leaderboard to several concrete decisions:

• Model selection: Compare systems intended for expert-level academic and technical question answering.
• Effort tuning: Measure whether higher reasoning settings deliver enough additional accuracy to justify their cost and latency.
• Modality checks: Account for the penalty faced by models that cannot process the 272 image-based questions.
• Self-hosting: Quantify the benchmark gap between downloadable weights and hosted frontier APIs.
• Progress tracking: Monitor reasoning and knowledge scores separately instead of relying only on one aggregate number.

HLE-Diamond measures first-attempt accuracy on a fixed set of difficult, closed-ended questions. It leaves agent behavior, tool use, software engineering, latency, safety, and performance on private domain data outside its scope. Production decisions still require evaluations built from the tasks, inputs, and failure costs of the intended application.