APIUp to 28% cheaper than official pricesTry the API →
Hermes
Back to News

Can Small Decision Models Verify Benchmarks? We Tested Two Against Three LLM Judges

On MMLU, Jev 1.14 achieved an AUROC of 0.811 , nearly matching Claude Sonnet 4.6 's 0.806 (difference ~ 0.004 , 95% CI -0.016 to +0.024), while costing about $0.027 per 1,000 judgments versus Claude's $1.727 (roughly 64x cheaper).

Can Small Decision Models Verify Benchmarks? We Tested Two Against Three LLM Judges
Source
Akruti Acharya
Published
Author
Akruti Acharya
Read
1 min read

Takeaways

  • On MMLU, Jev 1.14 achieved an AUROC of 0.811 , nearly matching Claude Sonnet 4.6 's 0.806 (difference ~ 0.004 , 95% CI -0.016 to +0.024), while costing about $0.027 per 1,000 judgments versus Claude's $1.727 (roughly 64x cheaper).
  • On the 300-task SWE-bench sample, Decider outperformed Claude on "tests reject valid fixes" ( 0.827 vs 0.754 , +0.073) and "exclude task" ( 0.834 vs 0.770 , +0.064), both with positive 95% confidence intervals.
  • Of 87 severely underspecified SWE-bench issues, none of the five tested systems (including GPT-5.4 and GPT-5.4 mini ) flagged more than 5% at the default 0.5 probability threshold, despite Decider reaching an AUROC of 0.707 on this category.
  • In the auto-accept/escalate/auto-reject policy test targeting 80% defect recall across 166 defective MMLU cases, Jev and GPT-5.4 fell short (77.2% and 78.6% recall), while models meeting the target still auto-accepted around 18-20% of defects.
  • Only Decider qualified for automatic rejection, catching 32 of 37 rejected cases (86% precision) in the enriched sample, but this dropped to an estimated 44% precision once reweighted to the actual MMLU-Redux population of 5,668 questions with 338 defects.

Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.

Read original report

Sources

  1. 01typesafe.ai
  2. 02perplexity.ai
  3. 03openai.com
  4. 04huggingface.co