UC Riverside's SemLoc Beats Classic Bug Finders by 36 Points Using LLMs
SemLoc turns LLM program reasoning into runtime-checkable constraints, boosting Python fault localization Top-1 accuracy from 6.4% to 42.8% over spectrum baselines.
// Intelligence
Benchmarks
A focused feed of models, agents, research and open-source releases for people building with AI.
Updated
CMU's Odysseys2 Benchmark Exposes Web Agents Failing 53 of 100 Real Tasks
A Carnegie Mellon and Browser Use benchmark of 100 hour-long live-web tasks exposes how far web agents still are from reliable knowledge work.
Artificial Analysis Moves Intelligence Index to 60% Private Tests to Stop AI Cheating
Artificial Analysis rebuilds its flagship benchmark with 60% private tests, two new agentic evals, and a widening open-weights gap.
Osaka Metro University's SOM-OLP Cuts Classic Clustering Complexity From Quadratic to Linear
A new objective-based Self-Organizing Map variant gives every data point its own continuous latent coordinate, cutting cost from O(NM²) to O(NM) while topping 16 benchmarks.
Cartesia's Managed Agents Beats Gemini and Grok With 91.7% Task Success
Cartesia's new Managed Agents platform jumps 98 Elo points over its predecessor on blind voice preference tests, with task success climbing to 91.7%.
Vals AI's 10 Claude Sonnet 5.5 Agents Crack a 17,895-Line Math Proof
Vals AI explains how ten Claude agents produced a Lean proof for seven electrons on a sphere, and names which cases should fall next.
Cognition's FrontierCode 2.0 Reveals Every AI Coder Fails Half of Real Engineering Tasks
Cognition's new coding benchmark tests whether agents respect unstated constraints, with top model Opus 5.5 scoring just 45% on tasks maintainers built over 40+ hours each.
Fish Audio's Realtime STT Claims Top Accuracy at $0.36 per Hour
Fish Audio's new streaming speech-to-text claims the top spot on Cekura's leaderboard with 2.21% WER and conversation-aware turn-taking for voice agents.
Microsoft's MAI-Image-2.7 Climbs to No. 2 on Image-Editing Benchmark
Microsoft AI's newest image editing model climbs to #2 on the Artificial Analysis leaderboard, trailing only OpenAI's GPT Image 2.5 Sunburst and leading four categories.
Artificial Analysis' Code2Video-AA Benchmark Shows Claude Opus 5.5 Dominates AI Video Coding
A new benchmark judges 27 models on turning marketing briefs into animated HTML videos, with Claude Opus 5.5 leading at 10x the cost of rivals.
NVIDIA's Nemotron 3.5 Super VL Hits 22.7% on ARC-AGI-1 at $0.03 per Task
ARC Prize ran NVIDIA's first open vision-language flagship through its reasoning gauntlet, and the numbers reveal where hybrid Mamba-MoE models still trip.
Moonshot's Kimi K3.1 Pro Tops Every Public API Model in Security Benchmarks
Moonshot's new reasoning model takes the top open spot on Artificial Analysis Cyber Index at 64%, trailing only a trusted-access OpenAI model.
Vals AI Finds the Same Open-Weight Model Costs 2.3x More Depending on Provider
Vals AI's new Inference Provider Index shows the same open-weight model can cost 2.3x more and run 5x slower depending on where you host it.
Epoch Finds Half of 21 Major AI Benchmarks Are Flawed
Epoch AI expanded its benchmark audit program, flagging popular evals like SWE-bench Verified and Humanity's Last Exam as Flawed under a public rubric.
DeepSeek's V4.1 Flash 1017 Tops Public Cybersecurity Rankings at $0.42 per Task
DeepSeek V4.1 Flash 1017 tops the Artificial Analysis Cyber Index among open models, with zero safety refusals and a 53 point jump on CyberGym-E2E-AA.
Artificial Analysis Ranks Claude Opus 5.5 First in Tougher Coding Agent Index
A new coding agent leaderboard swaps in Cognition's FrontierCode benchmark and tightens Q&A grading, reshuffling the top of the charts.
JIT-MPE Beats Offline AI Music Models With Just 86,000 Parameters
A tiny 86k-parameter causal model estimates which notes are sounding within 50 ms of audio arriving, matching or beating much larger offline systems.
Mistral's Timestral Beats Larger Rivals at Zero-Shot Forecasting on CPUs
Mistral's first time-series foundation model family runs on CPU, hits #1 on three forecasting benchmarks, and ships under a Modified MIT license.
Subquadratic's SubQ Snap 1.1 Ties for Top AI Decision Model With 384K Context
SubQ Snap 1.1 ties d1-ultra and Elephant 9B atop the Decision Index at a score of 65, winning on hallucination control and context length.
OpenAI's GPT-6 Astra Tops Physics Benchmark With 83.1% on CritPt v2
The revised physics research benchmark gives models a code sandbox, and frontier systems have jumped from single-digit to 80%+ accuracy.
How to Add a Decision Model to Your Agentic Coding Workflow
A Claude Code fix passed 9 required tests and still returned the wrong title. One closed question after the tests picked the suite that caught it
Tencent's WorkForge Trains AI Agents on Real Files, Beating Models 15x Larger
Tencent and Fudan built 16,700 verifiable training environments from real files across 40 professional domains, lifting a 35B agent's GDPval score by 28 points.
Local Grammar Rules Alone Can Trigger a Sharp Order Transition
Physicists built a context-sensitive grammar where only nearby symbols interact, and still found a phase transition, hinting that language itself drives the order.
HeyGen's Voice Turbo Tops Speech Leaderboard at 87% Less Cost
HeyGen's new Turbo text-to-speech model takes the top blind-test ranking at $4 per million characters, undercutting its own predecessor by 7.5x.
No stories match these filters.