Sarvam's Prism Turns a 105B Open Model Into a 44ms Decision Engine

Sarvam has released Sarvam Prism , an Apache 2.0 open-weight model built on its Sarvam 105B Mixture of Experts backbone. Prism accepts context, questions, and fixed answer options, then assigns probabilities to those options in one forward pass. The design targets classification, routing, evaluation, and other structured decisions inside production pipelines.

Probabilities in one pass

Prism computes scores directly from model hidden states. Each question becomes a sequence containing the state, instructions, answer options, and an answer marker. A custom attention layout gives every option the same starting position and isolates the options from one another, while the answer marker can attend to every candidate. A small decision head converts the resulting hidden states into one probability per option.

The fixed layout constrains output to the supplied choices and makes scoring independent of input order. Prism sorts choice keys before scoring, so reordering the same options produces the same model input. Sarvam reports identical answers across 2,000 shuffled questions, with no additional compute. That behavior addresses the order sensitivity often seen when generative models are prompted to choose among labels such as A, B, and C.

44 ms median on a B200

Sarvam measured a median latency of 44 ms and a 95th-percentile latency of 420 ms on one NVIDIA B200. The test used 1,000 random requests from the public Decision Index benchmark, submitted sequentially rather than through concurrent batching.

The company also reports an average score of 71.8 across three decision indices. In Sarvam’s comparison, that was the highest score among decision models with response times below 100 ms. Both the quality and latency figures are vendor-reported, and production results will depend on request length, hardware, concurrency, and serving configuration.

Questions within one request are processed together as a batch while remaining independent. Each question supports up to 131,072 tokens of context.

Three question types, one call

The API accepts a state and a dictionary of questions . Each question uses one of three output types:

A single call can evaluate several questions against the same state. This example routes a support ticket, checks its urgency, and estimates the customer’s level of frustration:

Calibration for confidence thresholds

Confidence estimates let an application decide when to accept a result, invoke a larger model, or send a case to a person. Sarvam reports that Prism was underconfident across every tested confidence band in English and Indian languages: observed error rates were lower than the model’s probabilities implied.

That calibration supports threshold-based workflows. A team could automatically process high-confidence predictions and escalate lower-confidence cases. Thresholds still require validation against the application’s own data, labels, and cost of errors.

Jobs suited to fixed choices

Sarvam uses Prism internally for trace evaluation, prompt guardrails, routing in its token factory, and judgments in its speech-to-speech testing harness. The same interface fits pipeline stages that require fast, structured outputs:

• Ticket triage and intent classification
• Trace evaluation using fixed scoring criteria
• Input and output safety checks
• Routing between lower-cost and higher-cost models
• Rubric-based scoring of documents or transcripts

Prism requires answer options to be defined before inference, so it fits bounded decisions more closely than open-ended extraction or generation. Applications that need explanations, novel labels, or free-form answers will need a separate generative step.

Local deployment needs 120 GB

Prism stores its weights in float8_e4m3fn FP8 format, with one scale per output row, and scales activations per token at runtime. Each layer’s 128 routed experts execute through two grouped FP8 matrix multiplications in a Triton kernel. Embeddings and normalization layers remain in bf16 , while the Mixture of Experts router runs in fp32 for greater numerical precision.

Local inference requires one NVIDIA GPU with FP8 support and about 120 GB of available memory, such as a B200 or H200. The weights occupy roughly 106 GB of disk space.

Prism’s custom FP8 layout is incompatible with AutoModelForCausalLM and general-purpose inference servers. Teams must use Sarvam’s implementation or adapt their serving stack to its kernels and model layout. Sarvam also provides a hosted version through the Indus token factory .

A narrower interface for production decisions

Many production systems currently prompt a generative model to emit JSON, then validate and parse the response. That pattern adds decoding time, permits malformed output, and can produce different answers when option order changes. Prism replaces that workflow with a constrained interface that returns probabilities over predefined choices.

The underlying Sarvam 105B is a Mixture of Experts model trained in India under the IndiaAI Mission. Sarvam announced 30B and 105B variants at the India AI Impact Summit 2026 and released them under Apache 2.0. Prism adapts the larger backbone for low-latency decisions, giving developers an open-weight option for workloads where structured output, calibration, and predictable option handling shape the system design.