Nobody wants to talk to a phone agent. People want their question answered, and a business owner on the shop floor wants the call answered at all. A voice agent is worth building only if it is better than voicemail for the caller and better than a missed call for the business, and that bar is set by plumbing rather than by intelligence.
We have built two: a receptionist for a laboratory instrument reseller and a screener for a logistics office. The lessons were the same.
Ring a person first
The reseller’s agent does not answer the phone. The owner’s cell rings first. If nobody picks up, the agent answers, says what it is, and helps. This single design choice settled every argument about whether an agent would “replace” anybody. It replaces voicemail. The owner answers when he can, and the agent catches what he cannot.
The logistics screener goes the other way, by design: it answers first, because the point is to hang up on salespeople and pranksters before they reach a person. When a real caller needs someone who is not there, it takes a message and emails who called and what they wanted. Who answers first is a policy decision, and it should be written down with the client, not assumed.
No transcoding
Latency is the product. A caller who hears a pause of a second and a half after every sentence hangs up. The agents use a speech-to-speech model in a single session rather than a chain of transcription, text model and synthesis, and the audio is bridged between the telephony provider and the model without transcoding, because both speak the same eight-kilohertz telephone codec. There is no separate speech-to-text or text-to-speech service to pay for, deploy or wait on.
The practical effect is that the agent answers at conversational speed and can be interrupted mid-sentence. Interruption, or barge-in, is not a nicety. Callers interrupt constantly, and an agent that finishes its paragraph regardless feels like a recording.
One brain, two channels
The reseller also has a website chat. It is the same assistant: the same company facts in the prompt, the same catalogue search, the same booking tool. A caller who asks about a centrifuge and then checks the website gets the same answer, because it is the same answer. Two separate assistants drift within a week.
Everything the assistant knows comes from the reseller’s own site, scraped into a knowledge folder. Hours, services, warranty and financing sit in the prompt. The catalogue, several hundred items, is searched on demand through a tool, because a model that has the whole catalogue in its context will confidently describe items it has half-read.
Say what you are
The agent says it is an assistant. It does not claim to be a person, and it does not improvise a name. Beyond the ethics, this is practical: callers who know they are talking to an agent ask shorter questions and are more forgiving of a request to repeat something.
Measure faithfulness
The business owner’s real fear is that the agent will say something the site does not say: a warranty that does not exist, a price that is wrong. We do not answer that fear with reassurance. We answer it with a retrieval evaluation that asks the assistant a set of the reseller’s own questions, checks that the right facts were retrieved (context recall), and judges whether the answer stayed within those facts (faithfulness). The scores are reported, and when a change to the prompt or the knowledge folder moves them, we know.
What the screener does with salespeople
The logistics screener runs a policy: who to put through, who to hang up on, when to take a message. It is tested as text, without a phone, so the policy can be exercised on a hundred synthetic callers in a minute. The agent is polite to salespeople and ends the call. It answers the questions a customer actually asks, such as hours and whether a service is offered, from the business profile. Everything about the business lives in one file, so when the profile changed, the screener’s answers changed with it.
The checklist
If you are considering a voice agent, these are the questions that matter, in order.
- Who answers first, the agent or a person?
- What is the latency, and can the caller interrupt?
- Where does the knowledge come from, and is the catalogue searched or stuffed into the prompt?
- Does the web assistant share the same brain?
- Does the agent say what it is?
- How is faithfulness measured, and what is the number today?
A vendor who can answer all six has built one. A vendor who answers with the model’s name has not.
About the authors. Neugent Consulting designs and builds AI systems for operators. The examples in this piece come from client work; clients are anonymised and details are simplified.
