VoxBase

I’ve long had a simple but unpleasant problem: after negotiations, meetings and discussions there are too many nuances left that you can’t properly hold in your head.

During the meeting itself everything seems clear.

• The client clarified something.
• Somewhere the meaning shifted.
• Somewhere an important phrasing came up.
• Somewhere it became clear what the person actually needs.
• Somewhere a small detail appeared that could later grow into a task, a decision or a separate project.

And then the meeting ends.

And the usual loss of context begins.

You can turn on the voice recorder on your iPhone. But an audio file by itself solves almost nothing. The problem is getting the sound to become working text quickly.

I needed to get a transcript within about 10 minutes of a conversation.

Not through a separate service that you have to go to, upload a file, wait, check, and download the result.

I needed a proper path:

conversation › recording › transcript › text artifact › further work

The built-in phone solutions didn’t work for me.

Quality, stability, control, output format, further processing: all of it quickly runs into limits.

A phone is convenient as a recording point, but not as a full speech processing factory.

There is a whole class of devices that record you all day and transcribe automatically. The idea is clear: clip on a small recorder, it listens, then a service turns your day into text. But I didn’t want to buy a separate device, pay for a subscription and hand all that logic to an external system.

I decided long ago: if something becomes part of my memory, my workflow and my personal infrastructure, I want to understand how it works.

That’s how VoxBase appeared.

At first it was the answer to a very practical need: automatically turning iPhone recordings into transcripts.

But it quickly became clear that the task is broader.

This isn’t only about a voice recorder.

• There are voice notes.
• There are negotiations.
• There’s YouTube.
• There’s TikTok.
• There are podcasts.
• There are long discussions from which you need to extract meaning, decisions and next steps.

These are all different sources, but they share one problem:

important information often arrives as sound or video, while my working system lives in text.

First the recording gets from the iPhone to the Mac. Then the system picks up the file on its own. Then the audio is processed. Then Markdown appears. Then that text can be read, searched, given to an LLM, turned into tasks, posts, notes and decisions.

That’s the first layer.

Other branches grew out of this idea: transcription of YouTube, TikTok, Apple Podcasts, and the next project, LiveContext, which is no longer about transcribing after a conversation but about real-time transcription while the conversation is still going.

But it all started with a simple thing.

I realised that after important conversations I don’t need an audio file.

I need memory.

I built VoxBase as a pipeline where each layer has its own job:

iPhone › iCloud › Mac worker › VAD › chunks › Whisper › Markdown + metadata

I record my voice on the iPhone with Just Press Record.

The file appears in iCloud, but VoxBase doesn’t process it right away.

First the system checks that the recording is really ready:

• the file is no longer .icloud
• its size has stopped changing
• the recording hasn’t been processed yet
• the worker isn’t running a second time
• the original can be read safely

In simplified form the logic looks like this:

for file in iphone_recordings:
    if not file.ready():
        continue

    if file in processed:
        continue

    copy = make_working_copy(file)

    speech = detect_speech(copy)
    chunks = build_chunks(speech, target_sec=60)

    text = transcribe(chunks)

    write_markdown(text)
    write_metadata(file, speech, chunks)

    archive_original(file)
    mark_processed(file)

file.ready() checks that iCloud has fully synced the recording • processed protects against processing the same file twice • make_working_copy() creates a copy so the original stays untouched • detect_speech() looks for actual speech in the audio • build_chunks() gathers speech sections into blocks of about 60 seconds • transcribe() sends those blocks to Whisper • write_markdown() creates the final .md file • write_metadata() saves technical information about the transcription • archive_original() puts the source .m4a into the archive • mark_processed() marks the recording as processed

The main principle: the original is never touched.

The source .m4a isn’t cut, cleaned or overwritten. VoxBase works with a copy.

The most important layer here isn’t Whisper, it’s the preparation before Whisper.

The system doesn’t send the whole audio file.

First it looks for speech.

For that it uses VAD, Voice Activity Detection.

long recording › speech map › chunks of ~60 seconds › Whisper

That way VoxBase doesn’t transcribe emptiness or feed the model garbage.

Then each chunk is brought to working form through ffmpeg:

• mono
• 16 kHz
• denoise
• loudnorm
• WAV for Whisper

After that the local whisper-cli with the large-v3 model turns speech into text.

The output is Markdown and metadata:

• source file
• recording length
• how much speech there was
• how much silence was skipped
• how many chunks came out
• which model was used
• which pipeline ran

So the result isn’t just a transcript.

What appears is a text artifact with a provenance.

voice › speech › chunks › transcript › Markdown › memory

An iPhone recording used to be the end of a process.

Now it has become the entry into a system.

In the next post I’ll explain why I decided to make this pipeline local rather than use a cloud transcriber.

Transcription is usually described like this:

audio › Whisper › text

But quality is often decided before Whisper starts.

The main question:

what exactly are we sending to the model?

A recording may contain an hour of sound but nowhere near an hour of speech.

Pauses. Waiting. Background noise. Random sounds. Long stretches where nobody says anything.

If you send it all, the model spends time on emptiness, loses context and sometimes starts finding words where there were none.

So VoxBase first builds a speech map.

audio › Silero VAD › speech islands › chunks › Whisper

VAD, Voice Activity Detection, doesn’t recognise words. It answers a simpler question: where is a person speaking, and where not?

segments = detect_speech(audio)
chunks = merge_segments(segments, target_sec=60)

for chunk in chunks:
    text = whisper(chunk)

• detect_speech() finds the start and end of each speech section
• merge_segments() gathers small sections into roughly one-minute blocks
• only after that is each block sent to Whisper

Why not transcribe each short segment separately?

Because on fragments of 1–2 seconds Whisper loses context and starts hallucinating.

So VoxBase gathers small lines into chunks of about 60 seconds of speech.

Not the whole file. Not separate scraps. Blocks big enough for the model to understand the context.

In one test a 26-minute recording contained 131 speech segments. VoxBase gathered them into just 4 proper chunks.

In another test:

62 minutes of audio › 574 speech segments › 42:50 of actual speech › 30.9% silence skipped

So almost a third of the file didn’t need transcribing at all.

Before Whisper, each chunk also goes through ffmpeg:

• converted to mono
• brought to 16 kHz
• cleaned of some of the noise
• normalised for loudness
• saved as a working WAV

And only then does the local large-v3 model start.

The main idea turned out to be simple:

good transcription doesn’t begin with choosing a model. It begins with not making the model listen to garbage.

VoxBase first separates speech from sound, restores a convenient context and only then asks Whisper to recognise the words.

raw sound › speech map › clean chunks › Whisper › transcript with original timecodes

So Whisper matters here, but it’s only the last worker on the line.

Before Whisper, each speech fragment goes through ffmpeg:

-af "highpass=f=80,lowpass=f=6000,afftdn=nf=-20,acompressor=threshold=-25dB:ratio=4:attack=5:release=50,loudnorm" \
-ac 1 -ar 16000

• highpass=80 cuts hum, vibration and excess low end
• lowpass=6000 removes high-frequency hiss
• afftdn suppresses constant background noise
• acompressor evens out quiet and loud phrases
• ratio=4 compresses peaks at roughly 4:1
• attack=5 reacts quickly to sharp sounds
• release=50 brings the level back smoothly
• loudnorm normalises the final loudness
• mono, 16 kHz bring the recording to Whisper’s format

It’s easy to overdo it here.

Heavy noise reduction starts removing word endings along with the background. Compression can raise the noise, and a noise gate can cut off quiet speech.

So the main mode cleans the recording but tries not to damage the voice.

For difficult conditions there’s a harsher Noisy preset.

As a result Whisper receives not the original iPhone recording but a prepared signal with more even loudness and less background noise.

In VoxBase, iCloud is not the brain of the system.

It works as a courier.

iPhone › iCloud carries the recording › the Mac processes it locally › Markdown goes back to iCloud

The transcription itself is done by my Mac:

• Silero VAD finds speech
• ffmpeg prepares the audio
• whisper-cli runs the large-v3 model
• the Metal GPU speeds up processing
• no external transcription API is needed

No paying by the minute. No request limits. No need to send a conversation to a third-party service.

But being local solves nothing by itself if every file has to be started by hand.

So VoxBase runs through launchd, the macOS system scheduler.

The worker configuration has simple logic:

<key>StartInterval</key>

<integer>60</integer>

This means that about once a minute macOS starts a check for new recordings.

The worker looks at:

• whether the file has fully downloaded from iCloud
• whether its size has stopped changing
• whether it has been processed before
• whether another instance is already running
• whether transcription can safely start

If everything is ready, the pipeline starts.

If not, the file simply waits for the next check.

I separately tested scenarios with interruptions:

iPhone without internet › Wi-Fi appears › the recording syncs › VoxBase makes a transcript

And another variant:

worker stopped › recordings pile up › worker starts › the queue is processed

So the system doesn’t require a perfect moment when the iPhone, the internet and the Mac are all available at once. It carries on when conditions become normal again.

Another important decision came after problems with CloudKit.

At first some working files were also in iCloud. But chunks, temporary audio, logs and constantly changing state began to bloat the local cloud cache.

After that the architecture became stricter.

In iCloud:

• finished Markdown transcripts
• metadata
• permanent copies of originals

Locally on the Mac:

• temporary processing
• chunks
• logs
• state
• errors
• locks and cache

The rule turned out simple:

iCloud should store artifacts, not the insides of a running machine.

A separate agent deletes temporary enhanced audio files at night. Originals are never deleted.

In the end the roles are separated:

iCloud carries › launchd watches › the Mac processes › Markdown keeps the result

VoxBase runs on its own not because there’s a lot of AI in it.

But because ordinary system details are assembled into a stable cycle: check, process, save, don’t repeat, and clean up what’s temporary.

In the next post I’ll talk about the next layer: how the system can determine not only what was said, but where exactly I’m the one speaking.

Whisper can understand what was said.

But by itself it doesn’t know who said it.

Even if the system splits a conversation into:

SPEAKER_00 SPEAKER_01 SPEAKER_02

it still doesn’t know which of them is me.

For that, VoxBase has a separate layer designed for it: Voice Reference.

Its job:

separate the speakers › compare each voice with my reference › label Me / Other

First I recorded several samples of my voice:

• iPhone
• Mac microphone
• different recording conditions
• 1–3 minutes of speech each

From these recordings the system creates not text and not an audio copy but a voice embedding. It’s a numerical fingerprint of a voice.

reference = embed(my_voice)

for speaker in speakers:
    score = similarity(embed(speaker), reference)

me = speaker_with_highest_score

• embed() turns a voice into a set of numerical features
• similarity() compares each voice found with my reference
• the voice with the highest match gets the label Me
• the rest become Other

The full pipeline looks like this:

conversation recording › pyannote splits it by speaker › SpeechBrain creates embeddings › VoxBase compares them with my voice › Whisper adds the text

The output can then be something other than an impersonal transcript:

[00:00:04] Speaker 1: So I want to understand...
[00:00:09] Speaker 2: Yes, let's take a look.

Instead, a proper conversation:

[00:00:04] Me: So I want to understand...
[00:00:09] Other: Yes, let's take a look.

At the same time, the system shouldn’t pretend to be always sure.

The match score is saved in the metadata:

• high › confident match
• medium › Me, but with a warning
• low › best guess and low confidence

This is an important VoxBase principle: don’t hide uncertainty, keep it together with the result.

Voice Reference adds a new level.

First VoxBase learned to understand:

where there is speech

Then:

what was said

The next step:

who exactly said it

And here the transcript starts to preserve the structure of the conversation.

In the next post I’ll explain how ClipMind grew out of VoxBase: a layer that turns TikTok, YouTube and podcasts into the same kind of text artifacts.

After voice notes, VoxBase got a second input.

Not my voice, but other people’s media.

I send a link from the iPhone through the Share Sheet.

Then ClipMind determines the source on its own:

if is_tiktok(url):
    process_tiktok(url)
elif is_youtube(url):
    process_youtube(url)
elif is_apple_podcast(url):
    process_podcast(url)

So one queue accepts different links but runs different strategies inside.

Because TikTok, YouTube and a podcast can’t be processed the same way.

TikTok

First ClipMind looks for captions and checks:

• whether the language is right
• whether there’s enough text
• whether these are lyrics
• whether these are junk auto-subtitles

If the captions don’t pass:

video › audio › ffmpeg › local Whisper › Markdown

YouTube

Here the principle is different:

don’t download the video if the text already exists.

YouTube URL › metadata › ready transcript › Markdown

No video, no audio and no Whisper if subtitles are available.

If there are none, ClipMind doesn’t start a heavy download blindly. It saves the metadata and a clear error status.

Apple Podcasts

This is the hardest source.

Apple Podcasts URL › iTunes Lookup › RSS › ready transcript

If there’s no text in the RSS:

RSS audio enclosure › local transcription

But a long podcast can’t be quietly sent to Whisper in the background. Once such a fallback created extra GPU load, so now it’s disabled for the automatic queue and only runs explicitly.

The most important part of ClipMind starts before transcription.

As soon as a link enters the queue, a task is created in Todoist.

Even if processing later ends in an error.

• the source link isn’t lost
• one task is updated as processing goes on
• no duplicates are created
• after completion the link is removed from the queue
• temporary video, audio and chunks are cleaned up
• only light final artifacts stay in iCloud

In the end three completely different sources converge into one form:

link › the right extraction strategy › Markdown › Todoist › memory

The main idea of ClipMind turned out not to be transcribing everything.

It’s finding the cheapest and most accurate path to text first.

Sometimes that’s ready subtitles. Sometimes RSS. Sometimes local Whisper. Sometimes an honest failure artifact.

Different inputs. Different routes. One result: media stops vanishing in the feed and becomes part of my working system.

I’ve told how voice notes, TikTok, YouTube and podcasts turn into Markdown and become part of my working memory.

But memory itself isn’t made only of transcripts.

You also need a system that can store context and the links between notes, projects and decisions.

So the next posts will be about my MCP server for Obsidian and how language models get access to the knowledge inside my base.

And then I’ll return to transcription.

Because the next step turned out to be far more interesting.

If VoxBase works after a conversation, LiveContext works during it.

But that’s a different story altogether, coming soon 💻