AutoTrust Labs' JEV-27B Scores Decisions in One Token Across 37 Benchmarks
AutoTrust-JEV-27B v1.1 is a 27B decision model outputting option probabilities, not free-form text. Scores 67.31 public index on the Decision Index 0.3 suite across 37 benchmarks. Standouts: CLINC150+OOS 97.06, BFCL 96.25, BANKING77 93.63, Home appliance simulator 89.77.
Takeaways
- AutoTrust-JEV-27B v1.1 is a 27B decision model outputting option probabilities, not free-form text.
- Scores 67.31 public index on the Decision Index 0.3 suite across 37 benchmarks.
- Standouts: CLINC150+OOS 97.06, BFCL 96.25, BANKING77 93.63, Home appliance simulator 89.77.
- Weak on GPQA Diamond, HLE, ChessBench, and open-ended reasoning or taste tasks.
- Served via vLLM 0.31.0 with a prefix-bidirectional plugin; 64.8 ms median latency on one B300.
- Apache-2.0 licensed, BF16 weights, qwen3_5 backbone, supports choice and yes/no questions only.
AutoTrust Labs has released AutoTrust-JEV-27B v1.1 , a 27-billion-parameter text model for finite-choice decisions. It accepts a question and supplied answer set, then returns a probability distribution over those options. Its self-reported result on the Decision Index 0.3 public suite is 67.31 skill points across 37 benchmarks, led by tool selection and intent classification.
Finite-choice decoding gives developers a simpler interface for routers, classifiers, and policy gates. Applications can rank allowed choices or apply thresholds without parsing generated prose. Application-specific calibration remains necessary before those scores drive high-stakes decisions.
One token, full distribution
The serving stack renders the prompt, restricts the next token to the supplied option codes, requests their log probabilities from vLLM, and normalizes those values into a distribution. The interface returns the selected option and its associated scores, while explanations and free-form generation fall outside its scope.
| Property | Value |
|---|---|
| Input | Text question with fixed answer options |
| Supported formats | Multiple choice and yes/no |
| Output | Option probabilities derived from a constrained one-token decode |
| Parameters | 27 billion |
| Backbone | qwen3_5 |
| Weights | BF16 |
| License | Apache-2.0 |
The architecture suits systems with a known action or label set and enough linguistic variation to challenge a conventional task-specific classifier. Planning, tool execution, multimodal input, and unconstrained answers require separate components.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
