How speaker diarization works, and why your transcript still says “Speaker 2”

Speaker diarization answers one question about a recording: who spoke when. It doesn’t transcribe the words, and on its own it doesn’t know anyone’s name. It produces anonymous labels, Speaker 1, Speaker 2, with start and end times, which is why so many transcripts look the way they do.

TranscribeX, the app I build, uses the open-source pyannote pipeline for diarization, so this post uses pyannote as the worked example. It explains how pipelines like pyannote work, how accuracy is measured, why meetings are the hardest case, and how labels become names. All benchmark numbers are pyannote’s published results.

What is diarization?

Speaker diarization (from “diary”, a log of events) is the task of partitioning an audio recording into segments by speaker. The output is a timeline:

00:00.0 – 00:07.4   SPEAKER_00
00:07.4 – 00:09.1   SPEAKER_01
00:08.6 – 00:12.9   SPEAKER_00   (overlaps SPEAKER_01)
00:12.9 – 00:31.2   SPEAKER_02

Combine that timeline with a transcript’s word timestamps and you get the familiar “Speaker 1: …” format.

Diarization is often confused with two neighbours. Speech recognition turns audio into words and doesn’t care who said them. Speaker identification matches a voice against known people. A diarization system usually doesn’t know how many people are in the recording, let alone who they are, and has to work that out from the audio alone.

How a speaker diarization pipeline works

1. Segmentation: who is talking in this window?

The audio is cut into short, overlapping windows of a few seconds. A neural segmentation model looks at each window and outputs, frame by frame, which of a few local speakers is active, including more than one at once. pyannote’s model frames this as a “powerset” classification: instead of guessing each speaker independently, it picks from every combination (nobody, A, B, A and B, and so on), which makes overlapping speech a first-class case rather than an afterthought.

The labels are only local, though. “Speaker A” in one window has no connection to “Speaker A” in the next.

2. Embeddings: what does each voice sound like?

For each local speaker in each window, a speaker-embedding model turns their audio into a vector, a few hundred numbers that capture vocal characteristics and ignore the words. Two clips of the same person produce vectors that sit close together. pyannote community-1 uses a WeSpeaker embedding model for this.

3. Clustering: which voices are the same person?

Finally, all the embeddings from the whole recording are grouped, and each group becomes one global speaker. This stage also decides how many speakers there are, which is the step most likely to go wrong. pyannote community-1 uses VBx, a Bayesian clustering method that models speaker turns over time rather than treating each embedding as unrelated.

The result is a timeline of global speakers. Community-1 can also output an “exclusive” version where only one speaker is active at any moment, which is much easier to line up with a transcript’s word timestamps.

How diarization accuracy is measured: DER

Diarization error rate adds up three kinds of mistakes and divides by the total duration of speech:

• Missed speech: someone was talking and the system marked silence.
• False alarm: the system marked speech where there was none.
• Speaker confusion: speech was attributed to the wrong speaker.

Lower is better, and 0% is perfect. The catch is in the scoring rules. Many papers add a “forgiveness collar” of a quarter second around each speaker change and ignore regions where people overlap. Both make numbers look much better. pyannote publishes its results with no collar and with overlapping speech included, the strict way. When you see a DER claim, check how it was scored before comparing it with anything.

pyannote diarization benchmarks

DER, fully automatic, no forgiveness collar, overlapping speech included. From the pyannote/speaker-diarization-community-1 model card on Hugging Face (benchmark updated September 2025). Precision-2 is pyannoteAI’s commercial cloud model.

Community-1 is the open model you can run yourself, and it improved on 3.1 in most sets. The commercial precision-2 model is clearly better still, which is a fair reminder that open models aren’t the ceiling.

Why meetings are the hard case for diarization

The benchmark table tells the story: broadcast audio scores around 11% and meetings around 17–20%. A few reasons:

• Overlap. People interrupt, agree (“yeah, yeah”) and talk over each other. Overlap is where all three error types pile up.
• Distant microphones. Compare AMI’s headset (17.0%) and single distant mic (19.9%) rows: same meetings, worse audio, more errors. A laptop mic in a conference room is the distant case.
• Short turns. A two-word reply gives the embedding model very little audio to work with.
• Unknown speaker count. Clustering has to decide whether two similar voices are one person or two. Getting the count wrong cascades into confusion errors everywhere.
• Mixed audio in online calls. Recording a Zoom call usually means one mixed stream for everyone remote, sometimes with compression artifacts.

From “Speaker 2” to a name: speaker identification

Diarization stops at anonymous labels. To get names, you add identification: store an embedding for each known person (an “enrolment”), then compare each new speaker cluster against those stored voiceprints. If the distance is below a threshold, the label becomes a name.

The threshold is a real trade-off. Too strict and you get “Speaker 3” for someone you know; too loose and two colleagues with similar voices get merged. That’s why TranscribeX’s Voice Profiles let you set the match level to Strict, Normal or Low and save several samples per person, recorded on different days and microphones.

How to run speaker diarization

With pyannote in Python

This is the minimal example from the community-1 model card. You need a Hugging Face access token and must accept the model’s conditions on its page first.

from pyannote.audio import Pipeline

pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-community-1",
    token="YOUR_HUGGINGFACE_TOKEN")
output = pipeline("audio.wav")

for turn, speaker in output.speaker_diarization:
    print(f"{speaker} speaks between t={turn.start:.3f}s and t={turn.end:.3f}s")

If you already know how many people spoke, passing that to the pipeline removes the hardest decision from the clustering step. You’ll still need to align the output with a transcript yourself.

In a Mac app

Several Mac transcription apps now ship diarization with no setup:

• TranscribeX (Pro) runs the pyannote community-1 pipeline converted to Core ML on the Neural Engine, detects the number of speakers by itself with no upper limit, works live during meetings, and carries speaker labels into summaries and exports. How TranscribeX diarization works.
• MacWhisper (Pro) uses NVIDIA’s Nemotron diarization model for up to eight speakers, including live, since version 15.2.
• Superwhisper (Pro) separates speakers, with its docs noting the best results come from its cloud Nova models.

Tips for better diarization results

• Record each side separately if you can. In a call, your microphone and the remote audio are already separate streams; tools that keep them apart start with an easier problem.
• Avoid speakerphones and distant laptop mics in a room with several people. The AMI numbers show what distance costs.
• Give the system the speaker count when you know it.
• Enrol regular speakers so recurring meetings get names, not numbers.
• Fix one mistake, not fifty. If an app lets you merge or reassign speakers in bulk, correct the cluster rather than individual lines.