Can Small Decision Models Verify Benchmarks? We Tested Two Against Three LLM Judges
On MMLU, Jev 1.14 achieved an AUROC of 0.811 , nearly matching Claude Sonnet 4.6 's 0.806 (difference ~ 0.004 , 95% CI -0.016 to +0.024), while costing about $0.027 per 1,000 judgments versus Claude's $1.727 (roughly 64x cheaper).
Source
Akruti Acharya
Published
Author
Akruti Acharya
Read
1 min read
Takeaways
On MMLU, Jev 1.14 achieved an AUROC of 0.811 , nearly matching Claude Sonnet 4.6 's 0.806 (difference ~ 0.004 , 95% CI -0.016 to +0.024), while costing about $0.027 per 1,000 judgments versus Claude's $1.727 (roughly 64x cheaper).
On the 300-task SWE-bench sample, Decider outperformed Claude on "tests reject valid fixes" ( 0.827 vs 0.754 , +0.073) and "exclude task" ( 0.834 vs 0.770 , +0.064), both with positive 95% confidence intervals.
Of 87 severely underspecified SWE-bench issues, none of the five tested systems (including GPT-5.4 and GPT-5.4 mini ) flagged more than 5% at the default 0.5 probability threshold, despite Decider reaching an AUROC of 0.707 on this category.
In the auto-accept/escalate/auto-reject policy test targeting 80% defect recall across 166 defective MMLU cases, Jev and GPT-5.4 fell short (77.2% and 78.6% recall), while models meeting the target still auto-accepted around 18-20% of defects.
Only Decider qualified for automatic rejection, catching 32 of 37 rejected cases (86% precision) in the enriched sample, but this dropped to an estimated 44% precision once reweighted to the actual MMLU-Redux population of 5,668 questions with 338 defects.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.