Parallel's PWS-Decisions Benchmark Exposes OpenAI Falling Below Random Accuracy
Parallel open-sourced PWS-Decisions , 300 classification questions whose answers depend on live web facts. Parallel Decisions API scored 76.22% vs Perplexity 50.89%, OpenAI 41.67%, Jev 33.44%. OpenAI Decisions scored 2.2x lower on PWS-Decisions than on static JevBench, exposing a live-web gap.
Takeaways
- Parallel open-sourced PWS-Decisions , 300 classification questions whose answers depend on live web facts.
- Parallel Decisions API scored 76.22% vs Perplexity 50.89%, OpenAI 41.67%, Jev 33.44%.
- OpenAI Decisions scored 2.2x lower on PWS-Decisions than on static JevBench, exposing a live-web gap.
- Parallel ran at 0.56s p50 and $1 per 1,000 requests; Haiku 5.5 plus search hit ~$28.
- 100 questions are public via gated Hugging Face , 200 kept private to prevent contamination.
- Deterministic scoring, no LLM judge; refusals count as wrong to prevent abstention gaming.
Developers often need software to return one allowed label for a question whose evidence changed this week: Does a vendor support SAML single sign-on? Is a target account hiring? Has a supplier earned ISO 27001 certification? Parallel has released PWS-Decisions , a 300-question benchmark for APIs that retrieve current web evidence and convert it into structured decisions. Parallel’s own Decisions API leads the company’s full-set evaluation at 76.22% accuracy.
Decision APIs accept a question and a fixed set of outputs, then return a schema-constrained label that pipeline code can consume. Live-web decisions add retrieval, source selection and freshness checks to the final classification, exposing failure modes that prompt-only benchmarks leave unmeasured.
| Decision API | Whole-set accuracy |
|---|---|
| Parallel Decisions API | 76.22% |
| Perplexity Decisions | 50.89% |
| Baseten Mammoth | 45.00% |
| OpenAI Decisions API | 41.67% |
| TypeSafe Jev 1.14 | 33.44% |
| Configuration | p50 latency | p95 latency | Cost per 1,000 questions |
|---|---|---|---|
| Parallel Decisions API | 0.56 seconds | 0.71 seconds | $1.00 |
| OpenAI Luna with native web search | 7.42 seconds | 19.17 seconds | $16.52 |
| Anthropic Haiku 5.5 with native web search | 6.38 seconds | 10.95 seconds | $27 to $29 |
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
