Voice AI Development with Twilio and Amazon Lex

How Voice AI Works

When a user calls your business number, Twilio receives the call and streams the audio. Amazon Lex handles automatic speech recognition (ASR) to convert speech to text, then natural language understanding (NLU) to pull out intent. Your backend handles the logic, and Amazon Polly turns the response back into speech.

The flow: caller speaks → Twilio captures audio → Lex ASR + NLU → your logic → Polly TTS → Twilio plays the audio back.

That description sounds simple, but each arrow in that chain hides real decisions. Twilio streams audio as raw audio frames over a WebSocket. Lex processes it in near real-time and can handle interruptions — if a caller starts talking while the bot is speaking, Lex detects barge-in and cuts the playback. Your backend logic needs to respond within a few hundred milliseconds or callers hear dead air, which they interpret as a dropped call. Polly's Neural voices add roughly 80–120 ms of synthesis latency on top of that. In practice, building a voice AI that feels snappy requires profiling each segment of that chain, not just the NLU step.

Latency targets to aim for: end-to-end turn-around under 1.5 seconds from the caller finishing their sentence to hearing the bot's reply. Anything over 2 seconds starts feeling broken to most callers.

Setting Up Twilio Voice with Amazon Lex

Step 1: Create a Lex Bot

In the AWS Console, create an Amazon Lex bot with intents for your use case — BookAppointment, CheckStatus, and a Fallback intent for anything unrecognised. The Fallback isn't optional; it's what catches the long tail of real calls.

Think carefully about slot design before writing a single line of code. If your BookAppointment intent needs a date, time, and service type, you need to define how Lex should prompt for missing slots, what validation looks like, and what happens when a caller says "Thursday at three" without specifying AM or PM. Underdefined slots are where most voice bots break in production. Use slot type prompts like "Would that be 3 in the morning or 3 in the afternoon?" rather than re-asking the open-ended question.

For a 12-person law firm handling intake calls, a well-designed Lex bot might include intents like: ScheduleConsultation, AskAboutPracticeArea, RequestCallback, and OfficeHoursQuery. That covers roughly 70–80% of inbound call volume before a lawyer or paralegal needs to pick up.

Step 2: Create a Twilio Phone Number

In your Twilio console, buy a number and point the voice webhook to your Next.js API route at your domain.

If you are US-based, buying a local number (not a toll-free number) tends to improve answer rates for outbound use cases and looks more familiar on caller ID for inbound. Twilio's number provisioning is instant and costs around $1/month per number. For UK numbers, a standard geographic number (+44 XXXX XXXXXX) works well; Twilio supports those through the same console.

Step 3: Handle Inbound Calls

Create a Next.js API route that returns TwiML. Use twilio.twiml.VoiceResponse to greet callers with an Amazon Polly voice and connect to your Lex websocket endpoint.

Your TwiML should include a brief greeting that sets expectations: "Hi, you've reached [Business Name]. I can help you book an appointment or check your order status. What can I do for you today?" Keep it under 15 words after the business name — longer greetings get interrupted. Use <Gather input="speech dtmf"> rather than just <Gather input="speech"> so DTMF keypad input always works as a fallback.

Step 4: Handle Lex Fulfillment

When Lex recognises an intent with all required slots filled, it calls your fulfillment Lambda. The Lambda books the appointment, checks order status, or whatever the business logic is, then returns a spoken response.

The Lambda response format matters here. Lex v2 fulfillment expects a specific JSON structure with sessionState, messages, and optional requestAttributes. One common mistake is returning a response that looks correct but uses the Lex v1 format — the call fails silently and the caller hears the fallback phrase. Always test fulfillment responses with the Lex console's test chat before wiring up the phone number.

For order status checks, the Lambda needs to query your order management system — Shopify, an internal database, or an API — and respond with real data. "Your order 4421 shipped yesterday and is estimated to arrive Friday" is a useful answer. "Your order is being processed" is not. The difference between those two responses is usually just a database JOIN, but it determines whether customers trust the bot enough to use it again.

Production Considerations

• Use Amazon Polly Neural voices (Joanna, Matthew) for noticeably more natural-sounding TTS — the standard voices give the call away as a bot immediately
• Implement DTMF fallback — let users press numbers if speech recognition fails, especially in noisy environments
• Log every conversation for quality review; you'll want this the first time something goes wrong
• Add a human escalation intent that transfers to a live agent via Twilio TaskRouter

A note on what we've watched go wrong: skipping the fallback and DTMF paths because the demo worked. The demo always works. Real callers have accents, background noise, and bad signal — and an agent that can't gracefully escape to a human or a keypad will end the call abruptly enough to lose the customer.

Off-the-Shelf IVR vs Custom Twilio + Lex Build

This table helps put the build decision in context for business owners who are evaluating whether a custom voice AI is worth the investment compared to buying an off-the-shelf IVR product.

The break-even point on a custom build typically lands around 6–9 months when you factor in the reduced cost per call and the avoided per-seat licensing fees of many IVR products. For a business handling 500+ inbound calls per month, that math usually favours a custom build. Under 100 calls per month, an off-the-shelf option is probably fine.

What to Expect in Practice

The first version of a voice bot rarely sounds right until you've tested it with real callers. Internal testing produces a skewed sample — your team knows what to say and how to say it. Real callers ask the same question four different ways, go silent mid-sentence, or start by saying "Hello? Is anyone there?" before stating their actual request.

Plan for at least two rounds of tuning after launch. The first round fixes obvious failures in intent recognition. The second round addresses the subtler patterns: callers who never reach a successful intent, calls that escalate to a human faster than expected, and slots that are consistently misheard (phone numbers and postcodes are especially prone to ASR errors).

For a mid-sized UK dental group with three clinics, a voice bot handling appointment bookings typically needs about three weeks of live call monitoring before slot fill rates stabilise. During that period, the team reviews flagged calls daily — ones where the bot said "I'm sorry, I didn't understand that" more than twice — and either adds training utterances to Lex or adjusts slot prompts. After that initial tuning phase, the bot can reliably handle appointment bookings without staff involvement for roughly 65–70% of calls.

A concrete number worth tracking: intent recognition rate. If Lex correctly identifies the caller's intent on the first attempt less than 80% of the time, the bot needs more training utterances or better slot design. Above 85% first-attempt recognition, most callers will complete their interaction without needing human transfer.

Common Mistakes

Ignoring silence handling. If a caller pauses mid-sentence (very common when reading out a reference number), Lex may close the turn prematurely. Set the endpointingTimeout to at least 1200 ms for use cases involving numbers or addresses.

Skipping error budgets. Every voice bot needs a defined threshold: how many consecutive failures before the bot transfers to a human? Three failed attempts to recognise an intent should trigger escalation, not a fourth rephrasing.

Not testing on mobile. GSM audio compression sounds different from a landline or VoIP call. ASR accuracy can drop noticeably on compressed mobile audio. Run test calls from mobile networks during QA, not just from your office VoIP.

Launching without a monitoring dashboard. Twilio's logs show call duration and status codes. CloudWatch shows Lambda execution. But neither gives you a single view of: how many callers reached their intent, how many escalated, how many dropped. Build that dashboard before launch, not after the first complaint.

Talk to us if you want help shipping a Twilio + Amazon Lex voice AI system, or if you just want a second opinion on an architecture you've already drafted.

Related guides

• Voice AI for business: replacing hold music with real conversations
• Voice chatbot vs IVR: why businesses are switching
• What it takes to be a voice chatbot developer
• AI agents for appointment booking
• Our voice AI services

Frequently Asked Questions

How much does it cost to build a voice AI bot with Twilio and Amazon Lex?

Infrastructure costs are modest — Twilio charges around $0.0085 per minute for inbound calls, and Amazon Lex v2 charges $0.004 per speech request. A business handling 1,000 calls per month averaging 3 minutes each would pay roughly $35–$60/month in platform fees. Development cost is separate and depends on complexity: a single-intent bot (appointment booking only) typically takes 4–6 weeks; a multi-intent system with CRM integration takes 8–12 weeks.

Can Twilio + Amazon Lex handle British accents and UK phone numbers?

Yes, with configuration. Amazon Lex supports UK English as a separate locale (en_GB), which significantly improves ASR accuracy for British accents compared to the default en_US locale. UK phone numbers (11 digits) require custom slot validation because Lex's built-in phone number slot type defaults to US formats. Set up a custom slot type with a regex pattern for UK numbers during the initial build.

What happens when the voice bot cannot understand a caller?

If Lex fails to recognise an intent after repeated attempts, the bot should route the caller to a human agent via Twilio TaskRouter. You define the fallback threshold — typically two or three consecutive mismatches. A well-built system also lets callers say "speak to someone" or "transfer me" at any point, which triggers the escalation immediately rather than waiting for failure.

How long does it take to deploy a production-ready voice bot?

A simple bot — one or two intents, no backend integration — can be deployed in two to three weeks. A production system with appointment booking, CRM writes, and a human escalation path typically takes six to ten weeks including QA and call testing. Plan for another two to four weeks of post-launch tuning based on real call data before the bot performs consistently.

Is Amazon Lex or a GPT-based voice solution better for phone bots?

For structured, task-completion calls (book appointment, check order, update address), Lex performs reliably and costs less per call than LLM-based alternatives. For open-ended conversations where the caller's intent is unpredictable, a large language model layer makes sense — but it adds latency, cost, and more complex guardrails. Most business use cases work best with Lex handling structured intents and a GPT layer only for the genuine free-text edge cases.

Do callers know they are talking to a bot?

With Polly Neural voices, many callers do not immediately identify the system as automated — especially if the opening prompt is concise and the bot responds quickly. However, regulatory guidance in the UK (Ofcom) and in certain US states requires disclosure that a caller is interacting with an automated system, particularly for sales or debt-related calls. For inbound customer service, adding a brief disclosure ("You're speaking with our automated assistant") is good practice regardless of legal requirement.

Can the voice bot integrate with our existing CRM or booking system?

Yes. The Lex fulfillment Lambda is standard Node.js or Python code — it can call any API your CRM exposes. Common integrations include Salesforce, HubSpot, Google Calendar, Calendly, and custom-built booking systems. The main prerequisite is that your system has an API endpoint (REST or GraphQL) that returns data fast enough — Lambda responses need to arrive within about 800 ms to keep the conversation flowing naturally.