AtomicChat Makes NVIDIA's Nemotron 3.5 Transcribe Run on Consumer GPUs
AtomicChat released GGUF builds of NVIDIA's Nemotron 3.5 Transcribe 30B-A4B speech model. Seven decoder sizes from 13.57 GB to 35 GB cover 16 GB GPUs through workstations. Accuracy stays within ±0.18 WER points of BF16 on English, German, French, Spanish.
Takeaways
- AtomicChat released GGUF builds of NVIDIA's Nemotron 3.5 Transcribe 30B-A4B speech model.
- Seven decoder sizes from 13.57 GB to 35 GB cover 16 GB GPUs through workstations.
- Accuracy stays within ±0.18 WER points of BF16 on English, German, French, Spanish.
- Model handles 40 languages, 1-hour audio, speaker labels, word timestamps, and translation.
- Ranks first open-weight on Artificial Analysis AA-WER at 2.8% with ~4B active parameters.
- Works via NeMo-Speech.cpp with OpenAI-compatible server on CPU, Metal, Vulkan or CUDA.
NVIDIA’s flagship speech-to-text model is now easier to deploy on local hardware. AtomicChat’s GGUF files remove the 67 GB BF16 checkpoint download and local conversion step previously required to run Nemotron 3.5 Transcribe .
The open-weight model supports offline transcription in 15 languages. Text instructions control transcription, speech translation, speaker attribution, word-level timestamps, punctuation, and audio question answering.
GGUF removes the conversion step
GGUF packages model tensors in a format designed for efficient loading and quantization in llama.cpp-derived runtimes. These files require NVIDIA’s NeMo-Speech.cpp runtime , which adds the speech components needed by Nemotron.
NVIDIA’s original BF16 release targets H100-class hardware. Local deployment required downloading the full checkpoint and running NVIDIA’s convert_model.py . AtomicChat instead provides a shared encoder file and seven preconverted decoder quantizations, allowing developers to download only the variant suited to their hardware.
The published sizes below cover the decoder. Deployments also need the F16 encoder, and peak memory depends on context length, batch size, backend, and CPU offloading.
| Build | Decoder size | Intended hardware | Accuracy profile |
|---|---|---|---|
| Q8_0 | 35.00 GB | 48 GB or larger GPUs; 64 GB Macs | Closest tested approximation to BF16 |
| AD-NVFP4 | 19.68 GB | Blackwell GPUs, including RTX 50 series, B200, and DGX Spark | Uses native NVFP4 experts |
| AD-IQ4_NL | 19.68 GB | 24 GB GPUs, CPU, Metal, and Vulkan | AtomicChat’s recommended general-purpose build |
| AD-Q2_0_M | 13.57 GB | 16 GB GPUs | Mandarin CER rises by about 1.5 percentage points |
AtomicChat’s AD-NVFP4 decoder is 19.68 GB, compared with 21.79 GB for NVIDIA’s NVFP4 conversion. Its AD-IQ4_NL build is also 19.68 GB, compared with 20.35 GB for NVIDIA’s version. AtomicChat quantizes embeddings and the output head to Q8_0, then calibrates IQ4_NL experts with an importance matrix that gives influential weights more precision.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
