NVIDIA Nemotron 3 Diarization: real-time speaker labels at a cent per audio hour

NVIDIA Nemotron 3 Diarization is an open, end-to-end speaker diarization model that outputs who spoke when with a configurable algorithmic latency between 0.32 and 30.4 seconds. Baseten, which hosts the model in its Model Library, published the technical details and performance benchmarks on September 23, 2026. The model replaces the traditional segmentation-plus-embedding pipeline with a single pass that labels up to eight speakers at intervals as low as every 320 milliseconds.

The model uses a transformer-based architecture with roughly 100 million parameters, built on NVIDIA Streaming Sortformer. It converts 16 kHz audio into 10ms frames and emits a T by 8 matrix of speaker-activity probabilities. Two pieces of state carry a conversation across chunks: an arrival-order speaker cache, where speaker_0 is the first voice heard and stays that way, and a FIFO of recent frames. No embeddings or clustering are required for unbounded audio length.

Latency profiles and accuracy

A single checkpoint serves four algorithmic latency profiles: ultra-low at 0.32 seconds, low, standard, and offline at 30.4 seconds. The latency refers to how much future audio the model sees before committing a label. Moving from the offline profile to the ultra-low profile increases diarization error rate (DER) by only 1 to 2 points, with quality sustained beyond four speakers. On AISHELL-4 at the low profile, Baseten reports a DER of 9.8% against 27.2% for Streaming Sortformer v2.1, the predecessor model.

Benchmarks across NOTSOFAR, AMI, CALLHOME, and AISHELL show the model outperforming Meta Muse Voice Transcribe on all datasets even at ultra-low latency, and losing to pyannote Community-1 only on AMI. DER was measured with overlap included and no collar.

Speaker activity as a control signal

The model tracks speech activity per speaker in 10ms increments, which can serve three voice-agent functions. Voice activity detection combines data across all audio channels to detect speech directly from the stream rather than relying on volume levels. Speaker-specific end-of-turn detection tracks when a specific speaker stops talking for a set duration, helping an agent distinguish the primary user finishing a sentence from background chatter. Turn-taking and interruption detection uses the order in which voices appear to identify the primary speaker; if a second voice starts, the model detects an interruption, and if both speak at once, it flags overlapping speech, responding in roughly 300ms on ultra-low latency settings.

Serving presets and capacity

Baseten exposes three serving presets: stateless batch requests, persistent WebSocket streams for live speaker tracking, and joint end-to-end transcription pipelines. Each latency level runs as its own instance, so developers can choose speed per request without redeploying. The real-time transcription option pairs the speaker tracker with NVIDIA Parakeet V2 ASR, a 600M English speech recognition model. The pipeline feeds per-speaker activity vectors directly into the initial encoder layers of Parakeet V2, transcribing overlapped audio channels in a single pass and preserving temporal alignment between words and speakers.

Baseten benchmarked capacity on a single NVIDIA RTX PRO 6000 under continuous load. The low-latency setting sustains up to 500 concurrent hour-long diarization streams, or 190 with transcription added. The optimized serving preset delivered about 1.35 times higher batch processing throughput than the NVIDIA reference setup. Over HTTP, a single GPU can continuously process 200 six-minute audio files every minute while maintaining steady response times. Latency stays flat near single-stream baseline values until hard capacity limits are reached. For live transcription, streaming and batch setups deliver virtually identical accuracy.

The model is available on Hugging Face and in the Baseten Model Library as batch, streaming, and real-time diarized transcription presets.