Building a voice agent platform, end to end

A voice agent lives or dies by one number: the pause between when you stop talking and when it starts. Three choices keep that pause under a second. The text-to-speech connection stays open for the whole session, because reconnecting on every turn costs 150 to 300 milliseconds of audible silence. The agent starts speaking after its first complete sentence instead of waiting for the full reply — a small buffer between the model and the speech engine flushes only on sentence boundaries, because feeding half-sentences to a speech engine breaks the intonation and sounds robotic. And audio chunks are scheduled back-to-back on the browser’s audio clock, so sentences join without gaps.

Every turn logs four timings — speech to transcript, transcript to first model token, first sentence to first audio, and end to end — into Postgres, with daily median and 95th-percentile views. A latency regression shows up as a number, not a feeling.

The hardest moment in a spoken conversation is the interruption — when you start talking before the agent has finished. Cutting its audio dead mid-word sounds like a glitch, so instead the agent fades to silence over a fraction of a second, the way a person trails off when you jump in. The stop reads as intentional rather than broken.

And rather than snapping straight into the new answer after that abrupt stop, it does what people do when they are cut off — a brief “okay” or “sure” in its own voice while the real reply is still forming behind it. That acknowledgment is synthesized once per session and slotted seamlessly in front of the first sentence, sharing the same gapless audio clock, so there is no pause and no overlap. This only happens on an interruption — on an ordinary turn a filler would sound odd. Neither touch changes when the agent stops or starts; they only change how those transitions sound, which is most of what makes a voice feel human.

The whole proof of concept runs on free API tiers, so hitting a quota is a when, not an if. Eight Gemini models sit in a priority chain tracked in the database. A single atomic Postgres function picks the best available model and increments its usage counter in one step, so two simultaneous requests can never double-spend the same quota. When a model runs out for the day, the system slides to the next one instead of failing.

The influence here was simple: a demo that dies mid-conversation is worse than one that quietly gets a little slower.

English uses the speech vendor’s server-side turn detection. Hindi isn’t supported by that mode, so the browser runs a small voice-activity model locally (Silero VAD compiled to WebAssembly) to detect when you’ve spoken, then sends each speech segment for transcription. Both paths sit behind one shared interface, so the rest of the app never knows which is active.

The decision driver: build to what each service actually supports, instead of forcing one elegant path that only works for one language.

The chat pipeline streams plain text — there is no native function-calling. So “tools” are sentinel tags like [END_CALL] and [TRANSFER_TO_HUMAN] that the agent is prompted to emit. The client strips them before they are ever spoken or shown, and acts on them: the call ends only after the goodbye has finished playing, and a tag that arrives split across two streamed tokens is held back until it completes.

This is exactly the kind of behaviour that is nearly impossible to test in a speech-to-speech model. Here it lives in text, so the evals can check it.

Agents can answer from uploaded documents. Retrieval runs two searches in parallel — semantic similarity over vector embeddings, and classic keyword search — then fuses the two rankings, because each catches what the other misses: keyword search nails exact names and codes, semantic search handles paraphrases. A lightweight model re-ranks the shortlist before it reaches the agent, and if that re-ranking step fails, the system degrades to the fused order rather than erroring.