Fine-Tuning NVIDIA Nemotron for Saudi Arabic Dialects, with a Path to Other Languages

NVIDIA published a workflow for fine-tuning Nemotron 3.5 ASR on Saudi Arabic dialects, specifically Najdi and Hijazi, using the NeMo framework. The post, authored by NVIDIA and collaborators, documents a full adaptation pipeline that reduced word error rate from 55.05% to 29.96% on the target test split while improving English performance from 11.04% to 10.42% WER without degrading other Arabic dialects.

The starting point is the pretrained Nemotron 3.5 ASR model, which supports multilingual streaming transcription across 40 language-locales. The authors used the SADA 2022 corpus and FLEURS data, running experiments on two NVIDIA RTX PRO 6000 Blackwell Workstation Edition GPUs. The workflow targets deployments where labeled speech is scarce but sufficient for specialization, not for training from scratch.

Curation and replay strategy

The first step is minimal curation. The authors filtered out utterances with inaudible annotations, clips outside 0.5 to 30 seconds, and transcripts with implausible character rates. This retained 103,559 of 125,490 utterances, or 133.7 hours, about 82.5% of the starting set. They deliberately avoided aggressive filtering that would discard hard but usable dialect speech. A second pass used NeMo Curator stages for mono conversion and quality scoring, but with corpus-specific thresholds set after inspecting score distributions rather than applying defaults.

To prevent catastrophic forgetting, the training mix interleaved 90% Saudi speech with 10% FLEURS data, split 7% English and 3% Arabic. The authors declared these proportions in the configuration rather than concatenating files, because a sliding-window shuffle may not reach appended rows until late in training. Seven percent English was sufficient to hold English performance.

Training configuration and results

Duration-based bucketing grouped similar-length utterances into batches, with 30 buckets and a 400-second batch duration. The authors also adjusted learning rate, warmup steps, and batch duration during troubleshooting, but noted these are not recommended defaults. They used a Cache-Aware FastConformer-RNNT model with prompted multilingual streaming and explicit language conditioning.

Updating all 24 encoder layers achieved the lowest error rates, at the cost of 230.4 million trainable parameters. Freezing layers reduces compute needs when data or memory is limited, but requires re-tuning when the mix changes. The validation baseline started at 49.5% WER and plateaued after epoch 45, prompting the shift to a narrower target and replay strategy.

Decoding improvements without retraining

Switching to a larger attention context of 13 lookahead frames and beam-8 MALSD decoding lowered WER by 2.71 absolute points without retraining, adding approximately 800 ms of latency. This suits batch transcription workloads rather than real-time streaming. The post also notes that NVIDIA Nemotron 3 Diarization extends the workflow to speaker-attributed transcription for up to eight speakers, aligning speaker boundaries with ASR timestamps.

The authors caution that these techniques are not universal defaults. Replay protects only what its data represents, and partial unfreezing needs re-tuning when the mix changes. The workflow does not generalize into evidence for every Arabic dialect or deployment environment. The post includes a notebook and a fine-tuning skill for reproducing the recipe.