Idiap Cuts Audio Needs 90% for Specialized Speech Recognition With Mixed Batching
Mixed batching with 10% target audio matches full-dataset ASR fine-tuning on three domains. Text-only LLM adaptation creates a modality gap worth ~29% relative WER on Banking. Batches combine source speech, noisy projected tokens, corrupted transcripts, and a slice of target audio.
Takeaways
- Mixed batching with 10% target audio matches full-dataset ASR fine-tuning on three domains.
- Text-only LLM adaptation creates a modality gap worth ~29% relative WER on Banking.
- Batches combine source speech, noisy projected tokens, corrupted transcripts, and a slice of target audio.
- Built on SLAM-ASR with WavLM-Large, Llama-3.2-3B-Instruct, LoRA adapters only.
- Mixed batching also curbs catastrophic forgetting on the original source domain.
- Code released at idiap/llm-asr-text-only-adaptation , accepted at Interspeech.
Researchers at Idiap Research Institute and Uniphore report that an LLM-based speech recognition system can adapt to specialized fields such as banking with a fraction of the usual labeled audio. In experiments described in the paper , mixed batches containing 10% target-domain speech matched or surpassed standard adaptation trained on the full paired speech corpus. The result could reduce the cost of domain adaptation because text is usually easier to collect and label than audio.
The researchers use a SLAM-ASR architecture built from WavLM-Large and Llama-3.2-3B-Instruct. WavLM converts speech into continuous acoustic features, and a linear projector maps those features into the language model’s embedding space. Llama then generates the transcript token by token from an instruction prompt. During domain adaptation, the speech encoder and projector remain frozen; LoRA adapters apply small, low-rank updates to the language model.
Why clean text breaks alignment
Domain text from reports, manuals, chat logs, and existing transcripts can teach the language model specialized vocabulary and phrasing. Pure text adaptation also shifts the model toward clean token embeddings, while inference still supplies continuous, imperfect embeddings from the speech projector. That modality gap reduces recognition accuracy.
On the Banking dataset, paired speech-text adaptation lowers word error rate by 1.83 percentage points compared with text-only adaptation, a relative reduction of about 29%. Word error rate, or WER, measures substitutions, deletions, and insertions relative to a reference transcript; lower values indicate better recognition.
Reporting is indexed from AlphaSignal. Rights remain with the original publisher and cited sources.
Read original report
