OpenJev Runs Agent Decisions 5x Faster by Skipping Token Generation

OpenJev skips generation to speed up agent decisions

Agent systems routinely make narrow decisions such as routing a ticket, retrying a call, or checking whether evidence supports a claim. A chat model can handle those tasks, but generated prose or JSON adds output tokens and forces the application to parse text back into a boolean or enum. OpenJev uses open 4B models to score typed options directly on a single RTX 3090, eliminating answer generation.

Theo Lee publishes the independent project under the MIT license as a reproduction of the interface pattern exposed by TypeSafe’s closed Jev service. The scope covers Jev’s developer-facing behavior; its undisclosed model and training process remain outside the project. A request supplies unstructured state, a runtime-defined question, and typed options, while the response contains a probability for each option.

One pass, typed probabilities

OpenJev runs a forward pass for each decision and reads the logits assigned to the declared options. It returns the corresponding typed probability scores without sampling an answer token, constructing JSON, or repairing malformed output.

The repository compares direct scoring with autoregressive JSON generation using the same frozen Qwen3.5-4B model, state, 21 binary criteria, and RTX 3090: