Artificial Analysis Rebuilds Decision Index v2 to Reward AI Confidence
Decision Index v2 launched with calibration-aware headline metric that rewards both correctness and confidence. Private datasets now make up 45% of the index, up from 25%, to fight benchmark gaming. Drex Max v1.1 and GLiDE v1.2 lead decision models at 73; Claude Haiku 5.5 tops overall at 77.
Source
Artificial Analysis
Published
Author
Artificial Analysis
Read
5 min read
Takeaways
Decision Index v2 launched with calibration-aware headline metric that rewards both correctness and confidence.
Private datasets now make up 45% of the index, up from 25%, to fight benchmark gaming.
Drex Max v1.1 and GLiDE v1.2 lead decision models at 73; Claude Haiku 5.5 tops overall at 77.
New AA-Paperclip benchmark tests 1,500 documents with all-or-nothing 15-answer scoring.
Cost varies 60x across decision models due to per-question vs per-document input billing.
Artificial Analysis has released Decision Index v2 , rebuilding its leaderboard for decision models. These systems choose from a fixed set of options and assign each one a probability in a single API call. The update introduces a confidence-sensitive headline score, increases the share of private evaluation data, and changes the model rankings.
Evaluation
Weight
What it measures
AA-Paperclip
30%
Decisions and extraction across 1,500 private resumes, receipts, invoices, and machine-learning papers. Each document includes five decisions and 10 facts, with all 15 answers required for a pass.
EvalSafe-O*NET-AA
15%
Workplace judgment.
When2Call
10%
Whether and when a model calls tools. Its weight fell from 25% in the first index.
AA-Omniscience-Decision v1.1
15%
Knowledge and hallucination. The evaluation rotates through 10 alternatives to the phrase “I don’t know.”
HLE-MultiChoice v1.1
15%
Reasoning with the same rotating abstention phrases, reducing reliance on phrase matching.
AA-Symmetry
15%
Answer and probability stability across 120 base questions and 440 variants that reorder options, swap labels, or alter yes-or-no phrasing.
Model
Index score
Cost per 1M tasks
Time per task
Claude Haiku 5.5 (Max)
77
$6,240
32.65 s
GPT-6 Luna (Max)
76
$1,530
11.34 s
Drex Max v1.1
73
$102
275 ms
GLiDE v1.2
73
$780
Not listed
Mammoth 27B
72
Not listed
Not listed
Celeris-1.1 Decision
Not listed
$14
56 ms
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
