Most meeting assistants treat each calendar event as a fresh start. Despite attending the same weekly standup for six months, a user gets labeled "Speaker 0" in every session, forcing every downstream LLM to guess who committed to which deliverables. That gap between what the tool knows and what the user expects drives meeting assistant churn, and infrastructure fixes it. This article covers how cross-session speaker re-identification works technically, why it compounds product value over time, and the concrete implementation steps (embedding extraction, vector database design, confidence thresholding, edge case handling, and GDPR compliance) required to ship it on top of our async diarization pipeline.
Identifying participants across meeting series
Speaker diarization answers the question "who spoke when" within a single audio file. It is a session-scoped process that produces numerical labels tied to order of appearance in that recording. Speaker re-identification (Speaker Re-ID) is a different problem: it answers the question "is this the same person I heard in a previous session?", and solving it requires crossing the boundary between individual audio files and maintaining state across them.
Defining cross-meeting speaker identity
Technically, Speaker Re-ID maps high-dimensional voice embeddings extracted from a new meeting against a persistent database of embeddings linked to known user profiles. Where speaker diarization produces session-local output, Re-ID produces cross-session output that persists to the next call, the next week, and the next quarter.
| Feature |
Speaker diarization |
Speaker re-identification |
| Execution context |
Intra-audio (single file) |
Inter-audio (across multiple files) |
| Output labels |
Ephemeral local IDs (Speaker 0, Speaker 1) |
Persistent identifiers linked to user profiles |
| State persistence |
Session-only, resets after each call |
Cross-session, stored in vector database |
| Primary use case |
Transcript readability, per-utterance attribution |
Cross-session continuity for meeting history and participant tracking |
This distinction matters because the two systems share infrastructure at the audio layer but solve entirely different problems. Diarization is a prerequisite for Re-ID, not a substitute for it.
Impact on meeting assistant retention
When speaker attribution fails across sessions, the damage compounds downstream. Each utterance in a diarized transcript typically includes structured metadata like speaker IDs, timestamps, and confidence scores, and that structured output gives downstream LLMs the per-utterance context required to attribute action items to the correct speaker. Without cross-meeting continuity, those action items become orphans: correctly attributed in isolation, but disconnected from the participant's history across the series.
Claap demonstrates what happens when the foundational layer is accurate. By building on our infrastructure, the Claap team reached 1-3% WER in production, transcribing one hour of video in under 60 seconds, with improved prospect conversion following the switch from US-centric incumbents. The lesson is consistent: you cannot build reliable Re-ID on top of inaccurate diarization, because every misattributed segment in the base transcript creates a corrupted voice embedding that will misidentify the speaker in every future session.
Reducing churn with consistent speaker recognition
The business case for Speaker Re-ID is straightforward once you map the feature to retention mechanics. A meeting assistant that recognizes recurring participants across sessions graduates from "transcription tool" to "team intelligence system," and that shift in perceived value directly affects how often users return and how deeply they integrate the product into their workflow.
Tracking recurring meeting attendees
Without Re-ID, every meeting in a series requires either manual participant tagging or downstream LLM inference from names mentioned in conversation. Neither approach scales. Manual tagging adds friction that kills adoption, and LLM inference from context is unreliable when speakers discuss third parties or when names are not explicitly stated.
With Re-ID, the system resolves participant identity automatically from the audio signal itself, without relying on calendar metadata, email integration, or user input. A returning participant in session eight of a recurring sprint review can be matched to their stored profile once the system accumulates enough high-confidence speech. Subject to the same minimum segment duration thresholds discussed in the implementation steps below, and their profile centroid is updated with the new session's data without any action required from the user.
Personalized meeting summaries
Speaker diarization gives LLMs per-utterance attribution within a session. Re-ID extends that attribution across sessions, giving downstream LLMs the persistent speaker context needed to generate meaningfully richer outputs. The difference between "Action item assigned to Speaker 1: review the Q3 roadmap" and "Action item assigned to Alice, who has led this roadmap review in four of the last six sessions" is the difference between a transcript dump and a genuine productivity tool.
The audio-to-LLM pipeline that structured transcripts enable works only as well as the speaker attribution feeding it. When persistent speaker IDs replace ephemeral session labels, every summary, every "what you missed" digest, and every coaching score inherits the full context of that participant's history with the team.
Action item tracking per participant
Diarization attributes action items to the correct speaker within a session. Without Re-ID connecting those assignments across sessions, the item is correctly attributed in isolation but disconnected from the participant's history, making cross-session tracking unreliable. When a follow-up is assigned to the wrong speaker because the system confused two similar voices, the error propagates to CRM entries, project management tools, and coaching dashboards. Cross-meeting Re-ID closes this gap by ensuring that, for example, "Alice committed to this" in session three connects correctly to "Alice's open items" before session four begins.
Tracking participants over time
The long-term product value of a persistent speaker graph compounds with usage. After twenty sessions, the system can surface patterns — for example, "Bob consistently raises budget objections in the last ten minutes" or "three action items assigned to Carol in the last month remain unresolved." These second-order insights are unreachable without the underlying Re-ID layer, and they represent the stickiest, most defensible features a meeting assistant can offer.
How our engine maps speakers across sessions
The technical foundation of Speaker Re-ID is the voice embedding: a high-dimensional vector representation of the acoustic characteristics that make one person's voice distinct from another's. These vectors encode both physiological traits (vocal tract length, resonance patterns) and behavioral traits (speaking cadence, idiolect), producing a compact numeric signature that can be stored, queried, and compared efficiently.
How speaker embeddings identify users
Our async diarization engine, powered by pyannoteAI's Precision-2 model, processes the full recording before making speaker assignments. This full-context approach allows the model to analyze vocal characteristics across the entire audio file, producing more accurate speaker boundaries and fewer attribution errors. Our async pipeline delivers on average 3x lower DER than alternatives on conversational audio. That accuracy at the boundary level is what makes the resulting embeddings trustworthy enough to use as long-term identity anchors.
The build-vs-buy decision for the embedding extraction layer is significant:
| Metric |
Self-hosted open-source (raw pyannote) |
Managed API (our infrastructure) |
| Engineering effort |
Significant setup and tuning investment |
Under 24 hours to production |
| Performance |
Variable accuracy depending on configuration and audio conditions |
Optimized async pipeline tested across 74+ hours of conversational audio with open, reproducible benchmark methodology |
| Maintenance overhead |
Ongoing infrastructure and DevOps requirements |
Zero infrastructure maintenance |
| Total cost of ownership |
Compute and staff costs at scale |
Predictable per-hour billing ($0.20-$0.61/hr with diarization included) |
Teams migrating off self-hosted transcription setups report saving 20%+ of DevOps effort, engineering hours that go back to core product work rather than infrastructure maintenance. The diarization and embedding extraction layers carry their own operational overhead on top of that baseline.
Tracking users in recurring sessions
The matching workflow follows a clear pattern. After each meeting is transcribed and diarized, you extract embeddings from each diarized speaker segment, then query your vector database using cosine similarity. Cosine similarity computes the cosine of the angle between two embedding vectors: MATLAB's pretrained speakerEmbeddings model documents 0.25 as its empirically validated threshold for its specific ECAPA-TDNN/SpeechBrain implementation. Other embedding models require empirical tuning against your own enrollment and test audio before you can treat any threshold as production-ready. The formula is: s(φ_e, φ_t) = (φ_e · φ_t) / (||φ_e|| ||φ_t||).
Because embeddings from the same speaker cluster together in high-dimensional space while embeddings from different speakers remain distant, this approach scales to hundreds of speakers per organization without requiring acoustic fingerprinting of every participant in advance.
Maintaining identity in evolving audio
Voice characteristics shift over time. A participant joining from a different microphone, recovering from illness, or calling in from a noisy environment will produce embeddings that diverge from their stored reference profile. The mitigation is to treat the stored embedding as a centroid that updates incrementally with each new high-confidence match, rather than as a fixed reference. This allows the system to track gradual drift while rejecting sudden large deviations that suggest a different speaker entirely. The threshold logic for confidence scoring is covered in the practical implementation steps below.
Resolving speaker re-ID edge cases
Two categories of edge case require explicit handling in any production Re-ID pipeline.
- Shared microphones in physical rooms: When two participants share a conference room mic, their audio arrives on the same channel and their segments may be interleaved or overlapped. Overlapping speech creates the core technical challenge because embedding extraction requires clean, single-speaker audio segments. A common engineering approach is to prioritize segments with high diarization confidence scores and avoid using overlapped segments for embedding extraction where possible. The specific confidence threshold and overlap handling logic will depend on your embedding model and acoustic environment.
- Severe voice changes: A participant with laryngitis or a significantly different acoustic environment will produce embeddings far from their stored centroid. Rather than making a wrong assignment, the system should route those segments to human review or mark them as "unverified speaker" in the transcript output. Getting this wrong silently is worse than flagging it explicitly.
Practical steps for long-term speaker tracking
1. Build persistent speaker identity stores
The vector database is the stateful core of any Re-ID system. Options like Pinecone, Milvus, and pgvector all work for this purpose, and the choice depends on your existing stack rather than any Re-ID-specific requirement. What matters is that each stored vector links to a stable user ID (not a session-local speaker label), the index supports approximate nearest-neighbor search at millisecond latency, and the database supports deletion by user ID to comply with GDPR right-to-erasure requests.
Store multiple reference embeddings per speaker drawn from different sessions and acoustic conditions, then query against the centroid of the cluster. This makes matching more robust against day-to-day voice variation and different microphone setups. The vector database and embedding extraction layer sit outside our API and carry their own infrastructure costs, separate from the transcription and diarization layer.
2. Mapping new audio to known speakers
The pipeline for each new session runs as follows:
- Submit the recording to our async transcription API with diarization enabled.
- Receive the structured transcript with per-utterance speaker IDs, timestamps, and confidence scores.
- For each diarized speaker segment above your confidence threshold, extract a voice embedding.
- Query the vector database with that embedding using cosine similarity.
- If the score exceeds your acceptance threshold, which requires empirical tuning for your embedding model and audio conditions (MATLAB's pretrained speakerEmbeddings model documents 0.25 as its validated threshold for its specific ECAPA-TDNN/SpeechBrain implementation. Other embedding models require empirical tuning against your own audio), map the session-local speaker ID to the matched user profile.
- If similarity falls below threshold, create a provisional speaker entry for human review or automatic expiration (see guest speaker handling below).
- After human confirmation, promote the confirmed match to the persistent speaker index and update the centroid.
This pipeline runs entirely in post-processing. Because our speaker diarization is async-only, speaker identity resolution happens after the meeting concludes, not during it.
3. Manage low-confidence matches with human validation
A soft-match zone (where the embedding is close to a known profile but not close enough for automatic assignment) should route to a lightweight UI confirmation step rather than making an automatic assignment. Getting a silent misattribution into the speaker database is far more damaging than prompting a user for a one-tap confirmation.
The UI pattern is straightforward: after a session is processed, surface a confirmation card for any speaker segments that fell below the automatic threshold. For example, "Is this Alice Nguyen?" with a short audio clip and a confirm/deny action. Confirmed matches update the stored embedding centroid. Denied matches create a new speaker entry or link to a different profile. For automated systems with no human review, a conservative strategy is to require multiple consecutive segments from the same session to each independently score above the threshold before confirming a match, since single-segment matches are more susceptible to noise contamination.
As the system accumulates more confirmed matches per speaker, the centroid becomes more robust and the rate of ambiguous segments drops, reducing the frequency of review prompts over time.
Failure modes for cross-meeting speaker linking
Being honest about failure modes is part of building a reliable system. Re-ID pipelines break in predictable ways, and knowing where they break lets you instrument before it causes user-facing errors.
Handling acoustic ambiguity
Two participants with acoustically similar voices (same gender, similar age, similar regional accent) will produce embeddings that cluster close together in vector space. Raising your acceptance threshold for initial enrollment (requiring higher scores for the first match) reduces false positives for similar-sounding speakers. For ongoing sessions, temporal consistency checks help: if a speaker's embedding distance jumps dramatically between segments in the same session, the transition likely reflects a speaker change that diarization missed rather than the same person speaking.
The opening moments of a call can present acoustic challenges, including connection negotiation, background noise from participants joining, and audio level adjustments. Enforcing a minimum segment duration (MATLAB's pretrained speakerEmbeddings model documents 0.5 seconds as the floor for its specific ECAPA-TDNN/SpeechBrain implementation, with practical ranges of 1 to 10 seconds depending on your embedding model and acoustic conditions) of continuous, high-confidence speech before extracting any embedding filters out noisy frames and improves Re-ID accuracy on the remaining audio.
Managing guest speakers in recurring calls
One-time guests attending a recurring meeting should not generate permanent entries in the speaker database. The mitigation is a provisional speaker tier: new speakers from a given session are held in a temporary store for a configurable window. If the same voice appears in multiple sessions during that window, it graduates to a persistent profile. If not, the provisional entry expires and is purged. This keeps the database clean and reduces both storage costs and false match rates over time.
Security standards for speaker identity
Under GDPR Article 9, voice embeddings qualify as biometric data, a special category data that sets a high bar for lawful processing. Explicit consent from every meeting participant, including external parties, is a commonly applied principle for this type of processing, but no single EDPB opinion is scoped specifically to B2B meeting-recording Re-ID pipelines. The most-cited EDPB guidance on voice data addresses consumer voice assistants, a different processing context. Confirm the lawful basis and consent workflow with legal counsel before committing to a data lifecycle design for this use case.
Practical compliance requirements for any Re-ID pipeline:
- Encryption: All embeddings must be encrypted at rest and in transit, not just the raw audio.
- Explicit consent: Collect it before the session starts and record it in an auditable log.
- Data Protection Impact Assessment: A DPIA is required for large-scale processing of special category biometric data.
- Right to erasure: Your vector database must support deletion by user ID, and that deletion must propagate to all derived embeddings within the statutory window.
- Data retention controls: On Growth and Enterprise plans, customer audio is never used to retrain our models by default, with no opt-out action required.
Retention policy options vary by plan. HIPAA-regulated audio (clinical telehealth, for example) adds Business Associate Agreement requirements on top of the GDPR baseline. Our SOC 2 Type II, ISO 27001, HIPAA, and GDPR certifications cover the transcription and diarization layer, giving you what you need to satisfy legal review without a multi-month procurement process.
Implementation with our speaker diarization
Extracting speaker embeddings from API responses
Enabling diarization in our async API requires a single parameter in the transcription request. The response returns each utterance with a speaker field (an integer index assigned in order of first appearance). Depending on the features enabled in your request, utterances can also include word-level timestamps, per-utterance confidence scores, language tags, and channel information, as shown in the example payload below.
Here is a representative API response payload from our speaker diarization documentation:
```json
{
"transcription": {
"utterances": [
{
"words": [
{ "word": "hello", "start": 0.5, "end": 1.2, "confidence": 0.98 },
{ "word": "everyone", "start": 1.3, "end": 2.1, "confidence": 0.97 }
],
"text": "hello everyone",
"language": "en",
"start": 0.5,
"end": 2.1,
"confidence": 0.975,
"channel": 0,
"speaker": 0
},
{
"words": [
{ "word": "thanks", "start": 2.5, "end": 3.1, "confidence": 0.96 }
],
"text": "thanks",
"language": "en",
"start": 2.5,
"end": 3.1,
"confidence": 0.96,
"channel": 0,
"speaker": 1
}
]
}
}
```
The following Python snippet parses this response, groups segments by speaker, and prepares them for embedding extraction:
```python
import json
# Configuration: tune these against your own enrollment and test audio.
# Minimum segment duration: 0.5s is the documented floor for MATLAB's
# pretrained speakerEmbeddings model (ECAPA-TDNN/SpeechBrain). A practical
# starting range of 1-10s is reasonable depending on your embedding model
# and acoustic conditions. Tune against your own audio before deploying.
MIN_DURATION_SECONDS = 1.0
# Cosine similarity acceptance threshold: 0.25 is the empirically validated
# default for MATLAB's pretrained speakerEmbeddings model (ECAPA-TDNN/SpeechBrain).
# Other embedding models commonly sit in the 0.6-0.75 range depending on model,
# require empirical tuning against your own enrollment
# and test audio before deploying.
SIMILARITY_THRESHOLD = 0.6
# Load the API response
response_json = json.loads(response_body)
utterances = response_json['transcription']['utterances']
# Group high-confidence utterances by session-local speaker ID
speaker_segments = {}
for utterance in utterances:
speaker_id = utterance['speaker']
confidence = utterance.get('confidence', 0)
duration = utterance['end'] - utterance['start']
# Filter: only use utterances above the confidence threshold and
# at or above the minimum duration for reliable embedding extraction
if confidence < 0.90 or duration < MIN_DURATION_SECONDS:
continue
if speaker_id not in speaker_segments:
speaker_segments[speaker_id] = []
speaker_segments[speaker_id].append({
'text': utterance['text'],
'start': utterance['start'],
'end': utterance['end'],
'confidence': confidence
})
# For each speaker, concatenate segments for embedding extraction
for speaker_id, segments in speaker_segments.items():
combined_text = ' '.join([seg['text'] for seg in segments])
print(f"Speaker {speaker_id}: {combined_text}")
# Next steps (implement in your pipeline, not provided by Gladia):
# --> extract_voice_embedding(audio_file, segments) using your embedding model
# --> query_vector_db(vector, threshold=SIMILARITY_THRESHOLD) -> user_profile_id or None
```
Mapping speaker IDs across meetings
Once you have a vector for each diarized speaker, the cosine similarity query against your persistent store takes the following form: compute the dot product of the new embedding against each stored profile centroid, normalize by the product of their magnitudes, and return the profile with the highest score above your acceptance threshold. If no stored profile exceeds the threshold, route the segment to the provisional speaker queue described above.
The audio intelligence documentation covers how to combine diarization output with the broader enrichment pipeline (NER for Named Entity Recognition, sentiment, summarization) so that your Re-ID-resolved speaker IDs flow through to every downstream feature in a single structured payload.
Engineering costs for speaker tracking
Our per-hour pricing is based on audio duration, with diarization included in the base rate on Starter and Growth plans and no add-on fees for enabling pyannoteAI Precision-2 diarization at those tiers. Compare this to providers who charge separately for diarization, where processing 10,000 hours of meeting audio with diarization and NER can produce invoice surprises that were not visible in the headline rate.
| Monthly volume (hours) |
Starter ($0.61/hr) |
Growth (from $0.20/hr) |
Enterprise |
| 100 hrs |
$61 |
from $20 |
Custom |
| 1,000 hrs |
$610 |
from $200 |
Custom |
| 10,000 hrs |
$6,100 |
from $2,000 |
Custom |
For model selection in Re-ID pipelines: use Solaria-3 for European business audio across English, French, German, Spanish, and Italian, where it ranks #1 on real customer recordings in head-to-head benchmarks against AssemblyAI, ElevenLabs, Deepgram, Mistral, and Speechmatics. Use Solaria-1 for broader multilingual coverage and code-switching detection, and for any real-time streaming requirements, since Solaria-3 is async-only.
You can run both models in the same pipeline based on the participant's primary language, with Solaria-3 handling European business sessions and Solaria-1 covering global teams.
Start with €50 in free credits and have your integration in production in less than a day. Test Gladia on your own multilingual audio to see how it handles language detection, accent-heavy speech, and code-switching.
FAQs
What benchmarks define good speaker re-identification performance?
Our async diarization pipeline, powered by pyannoteAI Precision-2, is tested across 74+ hours of conversational audio using an open, reproducible benchmark methodology, which directly translates to cleaner embeddings and higher Re-ID accuracy on subsequent calls. The underlying metric is DER, which accounts for missed speech, false alarms, and speaker confusion errors.
How does the system maintain speaker identity when a user switches devices mid-series?
A switch from a laptop microphone to a Bluetooth headset changes the acoustic profile of the recording but not the underlying voice characteristics captured in the embedding. The cosine similarity score will drop slightly on the first session after the switch, which may trigger a soft-match flag, but the system resolves this within one or two human confirmations and updates the stored centroid to include the new acoustic context.
What data lifecycle policies apply to stored voice embeddings?
On Growth and Enterprise plans, customer audio is never used to train our models by default, with no opt-out required. Voice embeddings derived from transcripts are stored in your vector database (outside our infrastructure), so you control their lifecycle directly and must implement deletion by user ID to comply with GDPR right-to-erasure requests within the statutory window.
What are the privacy risks specific to speaker re-identification pipelines?
Voice embeddings are biometric data under GDPR, requiring explicit consent from every meeting participant before extraction and storage, a Data Protection Impact Assessment for large-scale processing, and encryption at rest and in transit for all stored vectors. HIPAA-regulated deployments additionally require a Business Associate Agreement with every processor in the audio pipeline, and our SOC 2 Type II and HIPAA certifications (detailed in the security standards section above) cover the transcription and diarization layer, though the consent workflow and vector database governance remain the responsibility of the product team building the Re-ID system.
Key terms glossary
WER (word error rate): The accuracy metric for transcription: the share of words a model gets wrong relative to a reference transcript. Lower is better.
DER (diarization error rate): The accuracy metric for diarization, accounting for missed speech, false alarms, and speaker confusion. Lower is better.
Code-switching: A mid-conversation change in language by the same speaker. A distinct failure mode from accented speech, and one that requires explicit handling in both diarization and Re-ID.
Async (batch) transcription: Processing a complete audio file after it's fully recorded, rather than streaming it live. Full-context processing is what lets our diarization engine analyze the entire recording before assigning speakers, which is why Re-ID depends on it.
BAA (Business Associate Agreement): The HIPAA-required contract between a healthcare provider and any processor in the audio pipeline that handles protected health information.
DPIA (Data Protection Impact Assessment): The GDPR-required risk assessment for large-scale processing of special category data, including the voice embeddings a Re-ID pipeline stores.
NER (named entity recognition): Automatic detection of structured entities, names, dates, amounts, in a transcript. Runs alongside diarization in the broader audio intelligence pipeline.
pyannoteAI Precision-2: The diarization model our async pipeline runs on to assign speaker labels within a session.
Speaker diarization: Answering "who spoke when" within a single audio file. Session-scoped, producing labels like Speaker 0 and Speaker 1 that reset with every new recording.
Speaker re-identification (Re-ID): Matching a voice heard in a new session against a persistent store of known speaker embeddings, so identity carries across a recurring meeting series instead of resetting each time.
