The demo worked. Live didn't. 3 voice AI pitfalls we fell into first

Most voice AI projects fail in the same places. And failures are well delayed to going live on the first busy Monday. Or the first model update.. Or the first week nobody has time to listen to AI calls.

Falling into each of them ourselves helped to map those.

This post is the start of a series. We go through the most common pitfalls for anyone who builds or installs voice agents. For each one: what it looks like, how it hit us, what we built, and the smaller things behind the scenes that made the difference.

In this post:

1. The demo works. The first Monday it doesn't.
2. The AI model changes under you.
3. Nobody listens to the calls.

Pitfall 1: The demo works. The first Monday doesn't.

What it looks like

Every voice AI vendor has a great demo. A clean line. A polite caller. A question the bot was built for. Testing by the customer representative is also fine. By multiple stakeholders five-ten minutes each, everyone nods.

Then the real customers calls. They interrupt. They ask a question nobody asked before. They say "yes" when they actually meant "wait". They call from a car, with the radio on. Or speak to both our AI and their kid. In the same sentence.

A demo shows the happy path. Real calls are mostly everything else.

How it hit us

Not gonna lie. We sold on demos too. After one demo, a lead told us their verdict: the agent "keeps talking when interrupted, and doesn't understand when you ask the same thing in a long and complex sentence."

They were right. Our demo was scripted. With limited amount of cases it had to cover. Their test was not so limited.

> What we learned: a demo proves the bot can work. Only testing shows how often it works.

What we built

Simulated callers. In the isLucid portal, you write test cases in plain words: who is calling, what they want, how they behave. An impatient customer. Someone who gives the wrong order number first. Someone who interrupts. The portal calls your agent with each one and grades the result as pass, fail or warn.

Pass criteria you can read. Each test has checks like "confirmed the delivery date" or "transferred to a person when asked". No code. If you can write a sales/onboarding script for a new employee - you can write a test easily.

Three ways to test before anyone real hears it:

• In your browser. Talk to the agent through your microphone. Do the heavy testing and editing without Tech Team being involved.
• Scripted. Write the caller's lines and run them. No microphone needed. Open the full trace afterwards. Use MCP to create such lines yourself
• On your own phone. Just add your number and let the agent call you. This is where most people hear the difference between "nice demo" and "I'd let it talk to my customers". The real testing on loud speaker, in the car or taking a shower ;) starts.

Compare versions on the same tests. A test suite doesn't belong to one version. Run it against the current version and the new draft. See which one is better before you switch.

What helped behind the scenes

• Test calls are kept separate from real calls. Your dashboards show customers, not your team testing on a Friday afternoon.
• Repeated runs. AI doesn't answer exactly the same way twice. Running a case several times shows whether a pass was luck.
• A testing guide built into the portal. What to cover, how to write a caller, how to read a failure. Most of it we learned the hard way.

> Try it yourself: write three callers for your own use case. A happy one, a confused one and an angry one. If your agent passes all three, you're ahead of most projects on the market.

Read more about the Voice AI agent and how we keep improving agents after go-live.

Pitfall 2: The AI model silently changes

What it looks like

Voice agents run on large AI models from providers like Google and OpenAI. Even the models we Built ourselves are constantly trained and changing. So no matter the decision what to use those models get updated. Often. Sometimes the update is better overall but worse at one thing your agent depends on.

Your instructions and logic didn't change. Your agent did.

Most teams find out the negative side effect of such change from a customer complaint.

How it hit us

At some cases we use Google Gemini models. This autumn we moved every live agent from one Gemini model version to a newer one. Newer is usually better. But not everywhere.

The newer model had one nasty side effect. It cut callers off in the middle of reading out their order number, and handed them to a human. Nothing in our instructions had changed. The model simply decided the caller had finished.

> What we learned: you don't own the model. You own the tests that tell you whether the model still does your job.

What we built

Every change is a version. The agent's instructions, voice, model and settings are saved together as a version. Production runs one version. You work on the next. Roll back in one click.

Compare versions side by side. See exactly what changed between two versions. Then run the same test suite on both.

Pinned building blocks. Prompts, tools, audio clips and knowledge sit in a shared library. Each agent pins the exact version it uses. Updating a prompt for one agent doesn't quietly change another.

Model choice per agent. Pick the provider and model for each agent. Some tasks need the newest model. Some need the fastest. Some need a model that runs on our own LOCAL servers because the data must not leave. You choose per agent, and you test before you switch. Read more about on-premise deployment.

What helped behind the scenes

• A second AI Agent just for numbers. Order IDs and phone numbers are where speech recognition failed after change the most. A separate AI agent listens for digits only. With two agents and a complex logic behind we increase the numbers capture rate back to where it had to be.
• The agent reads numbers back. "Did you say 1-7-4?" It costs two seconds. Yet it saves the whole call, as instead of failing silently, the model verified with the caller.
• Models page. See which models your workspace runs, where they run and when they are switched on.

The result for one of our clients, an electronics retailer: before these changes, about one in six "where is my order?" calls was resolved by the agent. In the two weeks after, it was 56%.

> Try it yourself: before you switch the model, run your test suite on the old version and the new one. If the new one fails a test the old one passed, you just saved yourself a Monday.

Pitfall 3: Nobody listens to the calls

What it looks like

The agent goes live. The team is busy. Someone listens to five calls on Friday. They sound fine.

Meanwhile, two hundred other calls happened. Couple of them went in two-three circles. Some switched to the wrong language halfway through.

Spot checks find spot problems. Patterns stay hidden.

How it hit us

When we started checking every single call automatically, our own agents surprised us. AI Agents repeating themselves. AI Agents getting stuck in a loop. AI Agents apologising for everything. One agent apologised so often that we marked the fix as critical.

None of these showed up in the first hundred of calls our team member had to listen to.

> What we learned: you can't improve what you don't hear. And no team can hear every call.

What we built

Automatic quality checks on every call. Each call is checked for problems a customer would notice. For example:

• The caller had to repeat themselves.
• The agent promised something it can't deliver.
• The agent made a claim it has no source for.
• A question was left unanswered.
• The agent ended the call too early.
• The agent slipped out of the caller's language.
• The agent went in circles.

The overview shows which checks fire most often. One click shows the calls behind each one.

Measurements against your own normal. Reply speed, recognition errors, failed tool calls. Each call is compared with your workspace's usual level, so "bad" means bad for you, not for an industry average.

Every agent call goes through Conversation Insights. The same analysis we run on recorded human calls now runs on the agent's own calls: topics, outcomes and your own custom questions. See Conversation Insights and AI quality monitoring.

A judge for the judge. A separate AI re-checks a sample of the analysis and scores how often it agrees. If the checker drifts, you see it.

What helped behind the scenes

• The call page. Recording, transcript, every action the agent took, and the quality flags, all on one page.
• Ask in plain words. Connect Claude or ChatGPT to the portal. Ask "which calls went wrong this week and why?" and get an answer with links to the calls.
• An audit trail. Every change, including changes an AI assistant prepared, is recorded with who made it and when.

> Try it yourself: after your first ten test calls, open the overview. Look at which check fired most. That's your first improvement.

The pattern behind all three

All three pitfalls have the same root. Voice AI looks finished long before it is.

None of this is glamorous. It is the work between a nice demo and an AI agent you trust with your customers. We did it the hard way first, so you don't have to.

Build your first voice agent that actually works

Sign up for free at portal.islucid.com and take the weekend.

• Describe your agent in plain words, or start from a few of your own call recordings.
• Test it with simulated callers, in your browser and on your own phone.
• Open the overview and see what the checks found.

Free for 14 days. One agent, 20 minutes of calls

Sign up for free →