Voice Reference: how the system knows which voice is mine

Whisper can understand what was said.

But by itself it doesn’t know who said it.

Even if the system splits a conversation into:

SPEAKER_00 SPEAKER_01 SPEAKER_02

it still doesn’t know which of them is me.

For that, VoxBase has a separate layer designed for it: Voice Reference.

Its job:

separate the speakers › compare each voice with my reference › label Me / Other

First I recorded several samples of my voice:

• iPhone
• Mac microphone
• different recording conditions
• 1–3 minutes of speech each

From these recordings the system creates not text and not an audio copy but a voice embedding. It’s a numerical fingerprint of a voice.

reference = embed(my_voice)

for speaker in speakers:
    score = similarity(embed(speaker), reference)

me = speaker_with_highest_score

• embed() turns a voice into a set of numerical features
• similarity() compares each voice found with my reference
• the voice with the highest match gets the label Me
• the rest become Other

The full pipeline looks like this:

conversation recording › pyannote splits it by speaker › SpeechBrain creates embeddings › VoxBase compares them with my voice › Whisper adds the text

The output can then be something other than an impersonal transcript:

[00:00:04] Speaker 1: So I want to understand...
[00:00:09] Speaker 2: Yes, let's take a look.

Instead, a proper conversation:

[00:00:04] Me: So I want to understand...
[00:00:09] Other: Yes, let's take a look.

At the same time, the system shouldn’t pretend to be always sure.

The match score is saved in the metadata:

• high › confident match
• medium › Me, but with a warning
• low › best guess and low confidence

This is an important VoxBase principle: don’t hide uncertainty, keep it together with the result.

Voice Reference adds a new level.

First VoxBase learned to understand:

where there is speech

Then:

what was said

The next step:

who exactly said it

And here the transcript starts to preserve the structure of the conversation.

In the next post I’ll explain how ClipMind grew out of VoxBase: a layer that turns TikTok, YouTube and podcasts into the same kind of text artifacts.