Building voice agents that actually listen

Everyone benchmarks language models on accuracy. Almost nobody benchmarks them on the thing that decides a collections call: the pause. The gap between a borrower finishing their sentence and the agent starting its reply. Too long, and the person feels like they're talking to a machine. Too eager, and the agent talks over them. Trust lives in that half-second.

The latency budget

Every stage of the pipeline spends time — speech-to-text, the model's turn, text-to-speech. I treat the whole thing as a budget: a fixed number of milliseconds we're allowed before the borrower notices. Then I fight for every one of them.

• Stream, don't wait. Start speaking before the full response is generated.
• Predict the turn. Detect end-of-utterance early instead of waiting for silence to time out.
• Cheap first token. The first word out has to be instant; the rest can catch up.

Listening is a feature

An agent that interrupts is worse than one that's slow. We tune voice activity detection so the agent yields the moment the borrower starts talking — because a real conversation is the borrower's to lead.

The model being smart is table stakes. The model being present — fast, calm, and willing to listen — is the product.

That's the part I care about most, and it's the part you can only feel on a live call, never in a benchmark.