What is speech-to-speech, and why does it matter?

Speech-to-speech means the model takes in audio and produces audio directly, without separate speech-to-text and text-to-speech stages. It matters because each stage in a cascaded pipeline adds latency and loses tone — speech-to-speech is what lets the agent reply fast enough to feel human.

How fast does an AI phone agent reply?

The goal is for the agent to begin replying in well under a second after the caller stops speaking. Streaming audio both ways and running the model in-region keeps the round trip short enough that the conversation feels natural.

Which telephony providers does it work with?

Twilio and Exotel, or your own SIP trunk — including direct carrier peering with Jio, Vi, Airtel, Tata, and Servetel. You bring your own numbers and the agent answers and dials on them.

Can the agent make outbound calls, not just answer?

Yes. The same pipeline runs outbound for reminders, follow-ups, and campaigns, with a scheduler and a TRAI-compliant calling window, DND scrubbing, and opt-out handling.

What do I get after the call?

A recording and transcript, plus optional post-call extraction — sentiment, intent, action items, and a summary — delivered as a webhook event and queryable by API for CRM updates and routing.

16 April 2026ai phone agent · how it works

How AI phone agents work: the real-time voice pipeline

By V B · Reviewed by the Vanie AI team

The short answer: an AI phone agent works by bridging a phone call into a real-time model. Caller audio streams to Google Gemini Live, which reasons, calls in-call tools like knowledge-base search or calendar booking, and replies in voice natively — then writes a transcript and analytics after the call. Vanie AI runs this whole loop as one platform.

Step 1 — the call connects

A call arrives (or is placed) over a telephony carrier — Twilio, Exotel, or your own SIP trunk. Vanie AIbridges that audio stream into the agent. Nothing is pre-recorded; the live audio is what the model hears, and the model's audio is what the caller hears.

Step 2 — real-time speech-to-speech

The audio streams into Gemini Live, a speech-to-speech model. This is the part that makes the conversation feel human: there is no separate speech-to-text, then a text model, then text-to-speech. A cascaded pipeline like that stacks noticeable latency at every hop before the model even starts thinking. Speech-to-speech collapses those hops, so the agent can respond in real time.

Voice activity detection and barge-in

The runtime detects when the caller starts and stops speaking (VAD) and lets them interrupt the agent mid-sentence (barge-in) — exactly like a real conversation, where you can cut in without waiting for the other person to finish.

Step 3 — in-call tools

A useful agent does more than talk. Mid-call, the agent can use tools to get real work done:

Knowledge-base search — pull the right answer from your documents, live.
Calendar booking — check availability and book on Google Calendar, Outlook / Microsoft 365, or Cal.com.
Human transfer — connect the caller live to a person, to a single number or a routed set (support, sales).
Your own APIs — read or write a CRM, order system, or anything reachable by a signed webhook.

Step 4 — human handoff and end-of-call

When the agent should not handle something — a regulated topic, a complex complaint, or a caller request — it says a short hand-off line and connects the caller to a person. Otherwise it wraps the call cleanly and hangs up. Either way the call is recorded and the transcript, summary, and structured fields are written up afterward.

Step 5 — recording, transcript, and post-call analytics

After the call, Vanie AI stores the recording and a transcript, and (optionally) runs post-call extraction: sentiment, caller intent, action items, and a summary. That payload lands as a webhook event and is queryable by API, so CRM updates and ticket routing happen without a separate analysis pass. Recording and extraction are why the per-minute rate is bundled — see pricing.

Why speech-to-speech beats a cascaded pipeline

The older way to build a voice bot — wire a speech-to-text vendor to a text LLM to a text-to-speech vendor — works, but every hop adds latency and a place for errors to compound. A native speech-to-speech model hears tone and timing the text pipeline throws away, and it answers fast enough to feel like a person. For a head-to-head on the real-time models, see Gemini Live vs OpenAI Realtime.

The latency budget

Every hop in a phone call costs milliseconds, and they add up fast. Audio travels from the caller’s phone, through the carrier, into the Vanie AI bridge, into Gemini Live, back out as audio, and back through the carrier to the caller. The target is for the agent to start replying in well under a second of the caller finishing — fast enough that the pause feels natural rather than like a bad line. Speech-to-speech is what protects that budget: a cascaded transcribe-then-think-then-synthesise pipeline spends a chunk of it before the model has even read the words. Running the model in-region and streaming audio both ways, instead of waiting for complete utterances, is the rest of it.

Inbound and outbound use the same loop

The pipeline is identical whether the agent answers a call or places one. Inbound, a call arrives at your number and the agent picks up. Outbound, a campaign or a single API call dials the customer and the agent starts talking when they answer. The difference is mostly orchestration — outbound adds a scheduler, retry rules, and an India-specific compliance gate: calling only inside the TRAI 10 a.m.–9 p.m. window, scrubbing DND numbers, and honouring opt-outs. Once the call is live, the same speech-to-speech loop, tools, and analytics apply.

Languages, accents, and channels

Because the agent runs on Gemini Live, it speaks English plus 8 Indian languages with native voices, and can switch when the caller does — useful when someone opens in English and slips into Hindi or Tamil mid-sentence. And the loop is not phone-only: the same agent, knowledge base, and tools run on WhatsApp and Telegram, so a business answers wherever the customer reaches out without rebuilding anything.

The bottom line

An AI phone agent is a real-time loop: bridge the call, stream audio to a speech-to-speech model, let it use tools, transfer to a human when needed, and capture analytics after. Vanie AI ships that whole loop — the fastest way to understand it is to build a voice agent or read the bring-your-own-key guide.

Frequently asked questions

What is speech-to-speech, and why does it matter?: Speech-to-speech means the model takes in audio and produces audio directly, without separate speech-to-text and text-to-speech stages. It matters because each stage in a cascaded pipeline adds latency and loses tone — speech-to-speech is what lets the agent reply fast enough to feel human.
How fast does an AI phone agent reply?: The goal is for the agent to begin replying in well under a second after the caller stops speaking. Streaming audio both ways and running the model in-region keeps the round trip short enough that the conversation feels natural.
Which telephony providers does it work with?: Twilio and Exotel, or your own SIP trunk — including direct carrier peering with Jio, Vi, Airtel, Tata, and Servetel. You bring your own numbers and the agent answers and dials on them.
Can the agent make outbound calls, not just answer?: Yes. The same pipeline runs outbound for reminders, follow-ups, and campaigns, with a scheduler and a TRAI-compliant calling window, DND scrubbing, and opt-out handling.
What do I get after the call?: A recording and transcript, plus optional post-call extraction — sentiment, intent, action items, and a summary — delivered as a webhook event and queryable by API for CRM updates and routing.