Workflows
What Is Speaker Diarization and Why Does It Matter for Video
Speaker diarization identifies who is speaking at every moment in a recording. With transcription, it turns a block of text into a conversation attributed to named people.
A transcript of a two-hour panel with every word correct still reads as one monologue. When the moderator asks about supply chain disruptions and three panellists answer, the text runs from one voice to the next with no boundary.
That transcript can be searched for what was said. It cannot be searched for what one person said. Speaker diarization adds that. It is a separate step from transcription, covered in the difference between diarization and transcription, and its quality depends on the audio, covered in what affects diarization accuracy.
Speaker diarization in plain terms
Diarization segments a recording by speaker. The system detects when one voice stops and another starts and labels each segment with a speaker tag. The output is a map of who spoke when.
The word comes from "diary": a chronological record, here of which speaker held each part of the timeline.
A diarized recording looks like this:
- Speaker 1 [00:00 - 00:43]: Opening remarks about Q3 performance
- Speaker 2 [00:44 - 01:12]: Response with revenue figures
- Speaker 1 [01:13 - 01:30]: Follow-up question about regional breakdown
- Speaker 3 [01:31 - 02:15]: Detailed regional analysis
Each segment has a speaker label, a time range and the transcript text for that range.
Welcome everyone, thank you for joining us today for our annual company conference.
Before we dive in, I want to acknowledge the incredible work everyone has done this past year.
Our company has grown by 35% year-over-year and we have expanded into three new international markets.
The migration patterns shifted dramatically. We tracked over two hundred species across the delta.
How the model separates voices
FrameQuery uses ECAPA-TDNN for diarization. The model reads the acoustic properties of a voice (pitch, timbre, speaking rate, spectral shape) and produces an embedding, a numeric summary of that voice.
Segments with similar embeddings are assigned to the same speaker. When the embeddings differ past a threshold, the model opens a new speaker. It needs no advance knowledge of who is in the recording or how many people spoke.
Voice embeddings are computed on your device and stay there. That matters for recordings of sensitive or privileged conversations.
Transcription and diarization are separate
Transcription converts speech to text. Diarization identifies who is speaking.
Transcription without diarization is what most tools produce: the full text with no attribution. It is fine for narration or a solo presentation.
Diarization without transcription is a timeline of when Speaker 1, 2 and 3 spoke, without the words. It is rare outside audio analysis.
A diarized transcript has both. Every sentence is tied to a speaker, and each speaker's lines can be searched on their own.
What attribution changes in search
Without diarization, searching "revenue projections" across a library of meeting recordings returns every instance from every speaker. You watch each one to find out who said it.
With diarization, you name the speakers (Speaker 1 becomes "Sarah Chen," Speaker 2 becomes "James Liu") and search with the @ syntax. "@Sarah revenue projections" returns only the moments where Sarah discussed revenue projections.
Where diarization matters most
Interviews. Documentary interviews, podcast episodes, journalist recordings: any recording where you need one voice isolated.
Meetings and calls. Finding what the CFO said about budget cuts, as opposed to what the COO said, requires speaker separation.
Multi-person recordings. Panels, depositions, focus groups, classroom recordings, event recordings.
Recurring speakers. When the same people appear across many recordings over months, naming them once makes their voices searchable across the library.
From labels to names
Raw diarization output is anonymous: Speaker 1, Speaker 2, Speaker 3. The labels are consistent within a recording and mean nothing on their own.
FrameQuery lets you name a speaker cluster. Once named, the voice embedding for "Sarah Chen" matches her in a boardroom recording from January and a phone interview from March.
Names also link to face recognition. When on-device face recognition (InsightFace Buffalo-L) identifies someone visually and ECAPA-TDNN identifies them by voice, the two identities can be joined. The person is then findable by face, by voice or by both.
Example
An editor cutting a corporate retrospective has 50 hours of board meetings, town halls and executive interviews from two years. The director wants every instance of the CEO discussing company culture.
Without diarization: search "company culture," get hundreds of results from dozens of speakers, and watch each one to find the CEO.
With diarization: search "@CEO company culture" and review the results.
Download FrameQuery to search your footage by speaker.