Integrating Retell AI for Voice Ops: A Production Walkthrough

A voice agent that takes too long to reply is not a voice agent. It is awkward silence on a phone line. Here is how I ship Retell AI integrations that actually feel like conversations, with the architecture and the latency budget that makes it work.

The latency budget

Humans expect a reply within 800 ms on a phone call. Past 1.2 seconds, it feels weird. Past 2 seconds, the caller starts talking again. So our budget, from "user stops speaking" to "agent starts speaking" looks like this:

• Speech-to-text (handled by Retell): ~150 ms
• Webhook + LLM round trip: budget ~500 ms
• Text-to-speech first chunk: ~150 ms

So we have around 500 ms for our own logic + LLM. That sounds tight, but it is enough if you do not waste it.

Webhook lifecycle

Retell calls your webhook for every conversation turn. The endpoint must do four things, very fast:

1. Verify the signature.
2. Pull conversation context.
3. Call the LLM with streaming.
4. Stream the response back to Retell as it arrives — do not wait for the full reply.

// app/api/retell/webhook/route.ts
export async function POST(req: Request) {
  // 1. Verify signature
  const raw = await req.text();
  if (!verifyRetellSignature(req.headers, raw)) {
    return new Response('Unauthorized', { status: 401 });
  }
  const body = JSON.parse(raw);

  // 2. Pull context
  const ctx = await loadCallContext(body.call_id);

  // 3-4. Stream LLM tokens back to Retell as they come
  const encoder = new TextEncoder();
  const stream = new ReadableStream({
    async start(controller) {
      const llm = await openai.chat.completions.create({
        model: 'gpt-4o-mini',
        stream: true,
        messages: buildMessages(ctx, body.transcript),
        temperature: 0.5,
      });
      for await (const chunk of llm) {
        const token = chunk.choices[0]?.delta?.content;
        if (token) {
          controller.enqueue(encoder.encode(token));
        }
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
}

Prompt design for voice

Voice prompts are different from chat prompts. Voice prompts must enforce:

• Short sentences. Two clauses max. The caller is listening, not skimming.
• No markdown. Asterisks become "asterisk" when synthesised.
• Confirmations every 2–3 turns. "So just to confirm, you want X, correct?"
• Explicit escalation triggers. "If the user mentions 'manager' or 'human', call the handoff tool."

A snippet from a real prompt:

You are Asha, the booking assistant for {{businessName}}.
Speak in short, natural sentences. Never list with bullets.
After every two answers, confirm the booking detail back to the caller.
If the caller says "manager", "human", or sounds frustrated, immediately
call the escalate() tool with reason="user requested human".
Never invent prices or dates. If unsure, ask the caller to confirm.

Escalation: the part that builds trust

Good voice agents know when to give up. Bad ones loop forever. Implement escalation as an explicit tool the model can call:

const tools = [
  {
    name: 'escalate',
    description: 'Transfer the call to a human agent.',
    parameters: {
      type: 'object',
      properties: {
        reason: { type: 'string' },
        priority: { enum: ['low', 'medium', 'high'] },
      },
      required: ['reason'],
    },
  },
];

When the model calls escalate(), our webhook responds with Retell's transfer command. The caller hears "I am transferring you now" and is connected to a real agent.

State management

Voice calls are stateful. The model needs to remember the caller's name from turn 1 to turn 7. Retell ships per-call state, but I keep my own in PostgreSQL too — partly for replay, partly for analytics. Schema is boring:

CREATE TABLE call_events (
  id BIGSERIAL PRIMARY KEY,
  call_id VARCHAR(64) NOT NULL,
  event_type VARCHAR(32) NOT NULL,
  turn_index INT,
  role VARCHAR(16),
  content TEXT,
  tokens_in INT,
  tokens_out INT,
  latency_ms INT,
  created_at TIMESTAMPTZ DEFAULT NOW()
);
CREATE INDEX ON call_events (call_id, turn_index);

What broke in production (so it does not break for you)

Cost

For a small business doing ~120 calls/day at average 4 minutes each, expect:

• Retell platform fees (varies, check current pricing).
• LLM cost: ~USD 0.04 per call with GPT-4o-mini. That is ~USD 145/month for 120 calls/day.
• Telephony (Twilio or equivalent): ~USD 0.013 per minute.

Total typically lands well under USD 500/month for this size of operation. Compare with hiring one part-time receptionist.

The deliverable I hand over

• Working Retell agent on the client's account.
• A small Next.js dashboard for editing prompts, escalation rules, and reviewing call transcripts.
• Webhook + LLM + escalation code, deployed.
• An eval set of ~50 scripted conversations and a passing run.

If you want this built for your business — bookings, support, lead qualification — drop a brief in the contact section on the homepage. I have shipped this pattern several times now and the setup is well-understood.