Nvidia's Nemotron 3.5 Beats Every API at Identifying Who Said What

Speaker diarization, the task of assigning each transcribed word to a consistent speaker label, has become a core requirement for transcription pipelines. Word Error Rate (WER) covers incorrect, inserted, and deleted words, leaving speaker attribution to a separate evaluation. Meeting notes, call analytics, and multi-party voice agents depend on both.

The new Artificial Analysis index compares speech-to-text APIs using Word Diarization Error Rate (WDER). Lower scores are better. Nvidia’s Nemotron 3.5 results lead at 3.3%, followed by Mistral’s Voxtral Small Transcribe 3 at 7.5%. Diarization produces labels such as Speaker 1 and Speaker 2; applications must map those labels to real identities separately.

Scoring every word twice

Traditional Diarization Error Rate (DER) measures the share of audio time assigned to the wrong speaker, missed, or falsely attributed. For APIs that return word-level output, the calculation can become unusually sensitive to timestamp boundaries. Artificial Analysis uses WDER to evaluate the speaker label attached to each reference word.

Each aligned reference word is classified as correct, assigned to the wrong speaker, or missed. The benchmark calculates WDER by dividing wrong-speaker and missed words by all scored reference words. Because missed words count toward the result, WDER reflects transcription omissions as well as speaker-label errors.

The 6.8-hour evaluation set stresses APIs with increasingly crowded conversations:

• 97 English conversations lasting roughly two to nine minutes each
• A mixture of real and synthetic speech, most of it proprietary and held out
• A public subset from NOTSOFAR-1 meetings recorded through one shared microphone
• Separate speaker bands for two, three to four, and five to eight participants
• Diarization enabled, English selected, no prompt supplied, and no speaker count provided

Artificial Analysis averages the three speaker-band scores equally. This weighting prevents strong performance on simpler two-person conversations from overwhelming weak results in larger groups.

Nemotron widens the gap in crowded rooms

Speaker count raises the error rate for every tested API. Nemotron moves from 1.7% WDER on two-speaker audio to 5.6% on conversations with five to eight participants. Every other API exceeds 10% in the largest band, increasing the likelihood of incorrect summaries, analytics, and downstream actions.

The leaderboard lists Nemotron as open weights instead of providing a standardized per-minute price. Teams can self-host the model, although deployment still carries GPU, scaling, and operational costs. The benchmarked result comes from a DeepInfra endpoint, so performance can vary with different inference stacks and hardware.

Batch speed changes the shortlist

Production selection also depends on turnaround time and cost. Microsoft’s MAI-Transcribe-2 results show a 313.9x Speed Factor with an 8.8% WDER and a listed price of $1.67 per 1,000 minutes.

Because the index covers non-streaming APIs, Speed Factor measures completed-job turnaround relative to audio duration. Streaming responsiveness, time to first token, concurrency limits, and rate limits require separate tests.

Price and WDER do not move in lockstep. Voxtral Small Transcribe 3 records the lowest WDER among priced APIs at 7.5% for $4 per 1,000 minutes, while several more expensive proprietary services rank lower on speaker attribution.

Speaker mistakes propagate

Voice agents often route actions according to who spoke. A doctor’s instruction attributed to a patient can corrupt a clinical summary, while a customer’s commitment attributed to an agent can distort compliance records or sales analytics. WDER isolates this failure mode more directly than raw transcription accuracy.

The benchmark methodology adds several qualifications that affect implementation decisions:

• Fillers such as “uh” are excluded, while backchannels such as “mm-hmm” are scored.
• Speed Factor and price include separate diarization charges and the wait for asynchronous jobs to finish.
• The Nemotron score represents the tested DeepInfra endpoint rather than every possible deployment of the open-weight model.
• Results include 95% confidence intervals, so close rankings should be interpreted alongside those ranges.
• A speaker-band score appears only when a model returns a valid scored result for every conversation in that band.
• The dataset is English-only and mostly held out, limiting independent reproduction and direct conclusions about other languages or domains.

Match the model to the room

Model selection can now account for conversation shape alongside WER, cost, and latency. A two-party support line may favor a cheaper, faster API, while a meeting assistant handling eight participants may benefit from Nemotron’s lower crowded-room error rate.

Nemotron’s 5.6% WDER in the largest speaker band provides a concrete reference point for dense group audio. Deployment tests still need representative accents, noise, overlap, microphones, and speaker counts. A practical evaluation combines WER, WDER by participant band, latency, price, concurrency, privacy, and hosting requirements.