Artificial Analysis Opens CritPt v2 API so Developers Can Grade Physics AI
Artificial Analysis is exposing the CritPt v2 physics benchmark grading server through its API. CritPt covers 71 composite research challenges built by 50+ active physics researchers across 30+ institutions.
Takeaways
- Artificial Analysis is exposing the CritPt v2 physics benchmark grading server through its API.
- CritPt covers 71 composite research challenges built by 50+ active physics researchers across 30+ institutions.
- Grading-server robustness fixes trigger an Intelligence Index bump to v5.0.1 with a full regrade of all models.
- Score changes are small; frontier rankings move by at most one point, no reshuffling at the top.
- CritPt v2 Hard Subset carries 10% weight in the Intelligence Index v5 science category.
- 60% of Intelligence Index v5 now uses private or held-out test sets, reducing contamination risk.
Artificial Analysis is rolling out API access to the CritPt v2 grading server, allowing developers to score model outputs with the hosted physics benchmark pipeline. The release accompanies Intelligence Index v5.0.1, which adds grading fixes and regrades every model on the leaderboard.
Hosted access removes the need to reproduce CritPt’s specialized grading stack. Teams working on scientific reasoning can now evaluate prompting, tool use, and fine-tuning with the same grader that Artificial Analysis uses for its public results.
Inside the research challenges
CritPt, short for Complex Research using Integrated Thinking: Physics Test, evaluates research-style physics reasoning. More than 50 active researchers from over 30 institutions created the problems from their fields, with contributors including senior PhD students, postdoctoral researchers, professors, and research scientists.
| Component | Details |
|---|---|
| Original release | 71 composite challenges divided into 190 checkpoint tasks |
| CritPt v2 | 70 revised challenges with underspecified statements, errors, and typographical issues corrected |
| Answer formats | Numerical values, symbolic expressions, and executable functions |
| Index coverage | The CritPt v2 Hard Subset contributes 10% of the Intelligence Index score |
Each composite challenge groups several checkpoints into a problem shaped like an entry-level research project. The checkpoints provide finer-grained results, helping evaluators distinguish a failed intermediate step from a fully incorrect solution.
The problems use guess-resistant, machine-verifiable answers. Their grading pipeline applies physics-specific parsing, executable checks, and numerical tolerances instead of relying on exact text matches. That design supports richer answers while making local reproduction more difficult.
The API removes grading overhead
Running CritPt independently requires evaluators to reproduce its parsers, execution environment, and hand-tuned tolerances. The hosted grader centralizes that work and gives developers a consistent scoring implementation.
- Score model generations: Submit answers to the hosted pipeline without deploying the complete grading stack.
- Improve comparability: Use the grading logic behind Artificial Analysis’s published results.
- Run controlled experiments: Compare prompts, tools, inference settings, and fine-tuned checkpoints against a stable evaluator.
- Track reproducibility: Record the dataset revision, grader version, model configuration, and sampling settings with each run.
CritPt also retains measurable headroom among leading systems. At the time of the announcement, Artificial Analysis reported GPT-6 Astra at 72% on the CritPt v2 Hard Subset and 63% on Terminal-Bench-Science. Those results leave enough unsolved work for the API to distinguish improvements among frontier scientific-reasoning systems.
v5.0.1 reruns the leaderboard
Intelligence Index v5.0.1 incorporates robustness fixes from the CritPt grading server and applies them to every previously evaluated model. Artificial Analysis says most scores increased slightly, while frontier scores moved by less than one point and the leading order remained stable.
The underlying v5 index combines nine evaluations across four equally weighted categories. Private test sets or held-out answers now account for 60% of the composite score, reducing the value of benchmark-specific memorization.
| Category | Weight | Evaluations |
|---|---|---|
| Knowledge Work | 25% | AA-Briefcase, AutomationBench-AA, GDP.pdf |
| Coding | 25% | FrontierCode 2.0, Terminal-Bench 4.0 |
| Science | 25% | Terminal-Bench-Science, CritPt v2 Hard Subset |
| General | 25% | AA-Omniscience v1.1, HLE-Diamond |
Artificial Analysis documents the weighting and evaluation structure in its benchmarking methodology and v5 announcement .
Access details remain limited
The CritPt grader is becoming available through the Artificial Analysis API as part of the rollout. The announcement does not specify the endpoint, authentication requirements, quotas, latency, pricing, response schema, or data-retention terms, so developers will need the current API documentation or account dashboard before integrating it into an evaluation pipeline.
Scientific-model teams gain a hosted evaluator for multi-step physics work, while general application developers mainly receive a corrected leaderboard. Because the v5.0.1 regrade produced only small score changes and preserved the frontier order, the update provides little reason by itself to revise recent model-selection decisions.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
