An AI receptionist answers your business phone when you can’t, works out what the caller needs, and leaves you a record. A demo is chosen to sound good. Whether a product fits your business comes down to a few ordinary jobs, done under realistic conditions, with your rules.
This checklist gives you tests to run on any product, ours included. It doesn’t rank products. The tests need a list of situations, someone to make the calls, and a way to keep score.
Before you test
- Write down a dozen real situations. Look back at last month’s calls: the routine request, the change of plans, the urgent one, the price question, the caller who wants a person, the after-hours call.
- Write down what “right” looks like for each, before you call. Should it book directly or take a request for you to confirm? What should it say, what must it never say, and what should end up in the record? If you can’t answer, your rules aren’t clear yet, and no receptionist, human or AI, can follow them.
- Use made-up customers, on numbers and accounts you control. Test with fictional names and your own test phone numbers, inboxes and calendar, and keep real customers out of it. Everyone on a test call should know it’s a test and may be recorded. To test what happens when something fails, arrange it with the vendor in a test setup; don’t cause an unannounced outage on your live line.
- Have someone else make the calls. You know the script too well. Ask them to interrupt, change their mind and give details out of order.
- Plan the cleanup. When you’re done, cancel any test bookings and pending messages and delete the fictional customer records, so nothing from a test reaches a real calendar or customer.
The seven jobs to test
For each job, mark the result as pass, partial or fail, and note the evidence: the time in the recording, or a screenshot of the record. Check both ways it can fail: overstepping, like promising what you didn’t approve, and under-delivering, like refusing an ordinary question. Either one can cost you a customer.
1. Work out what the caller needs
Routine requests still have to be understood correctly, including vague ones, and captured in the caller’s own words.
A pass looks like
- Asks one or two useful questions when the caller is vague
- Gets the name, callback number and the problem right
- Recognizes a service you don’t offer and says so politely
A fail looks like
- Takes a message that says only “caller needs help”
- Asks the same question three times
- Pretends to offer a service you don’t
- How to test it
- Call with your most common request, asked the way customers ask it. Then call again with something vague: “Something’s wrong with my AC.” Then ask for a service you don’t offer.
- Ask the vendor
- What business information does it work from, and how do you update it? How long before a change takes effect?
2. Handle a change accurately
Customers change their minds mid-call. Whether the tool books directly or takes requests, it has to end with the right details and leave enough trace to sort out what changed.
A pass looks like
- The record matches what was finally agreed with the caller
- Enough trace to resolve the change later, such as a summary, transcript or history
- Tells a requested time apart from a confirmed booking, and your calendar matches what it said
A fail looks like
- The record, the calendar and what the caller was told disagree
- A detail is out of date, or changed with no explanation
- Says a time is booked when nothing is on your calendar, or books a slot it wasn’t allowed to book
- How to test it
- Ask for “Tuesday after 3,” then halfway through say “actually, after 6 works better.” Afterwards, check the record and your calendar against what the caller was told, and see whether you could tell what changed.
- Ask the vendor
- Can it book into your calendar directly, and under what limits, or does it take requests for you to confirm? When a caller changes a detail, where can you see what changed?
3. Stay inside your rules
Your rules on prices, arrival times and service area exist for a reason. The test is whether the AI follows them under a little pressure, and still makes the commitments you’ve allowed.
A pass looks like
- Quotes only the prices you’ve allowed, such as a service-call fee
- When a price isn’t one you’ve allowed, says so and offers the next step
- Commits only to arrival times your rules and availability support
A fail looks like
- Quotes a price or discount your rules don’t allow
- Promises an arrival time your rules or availability don’t support
- Accepts work outside the area you serve
- How to test it
- Ask “How much to replace a water heater?” and push once: “Just a ballpark.” Ask when someone can get there. Give an address outside your area.
- Ask the vendor
- Where do you set what it may and may not quote? What happens when a caller asks for something your rules don’t cover?
4. Bring you in when it should
Some calls must reach you, fast: an emergency under your rules, an upset customer, someone who insists on a person. The test is whether you actually get the alert.
A pass looks like
- Follows your urgent-call rule and alerts you the way you chose
- Tells the caller what happens next, honestly
- Lets a caller reach a person, or leave a message for one, when they ask
A fail looks like
- Treats an emergency like a routine request
- Says “I’ll transfer you” and nothing happens
- The alert arrives, but hours later or somewhere you don’t look
- How to test it
- Call with something your rules mark as urgent and time how long the alert takes to reach you. Call again and say “I just want to talk to the owner.”
- Ask the vendor
- How can it reach you: transfer, text, email, app? Can you set different rules for after hours? What does the caller hear if you don’t pick up?
5. Leave a record you can use
A call can sound great and still leave you a useless record. You should be able to read what happened in under a minute and know the next step.
A pass looks like
- Caller, need, what was requested and what was promised, in one place
- The next step, and who owns it
- The record lands where you already work, or exports cleanly
A fail looks like
- Only a transcript you have to read end to end
- No record at all for some calls
- Records you can’t export if you leave
- How to test it
- After each test call, time how long it takes you to find the record and understand it. Check that it reached the place you’d actually look.
- Ask the vendor
- Where do call records go? Can you export records and recordings, in what format, and do you keep them if you cancel?
6. Be honest about being an AI
Callers sometimes ask whether they’re talking to a person. A front desk that dodges or lies about it creates a trust problem for your business.
A pass looks like
- Says it’s an AI assistant when asked directly
- Still offers the caller a way to reach a person
A fail looks like
- Claims to be a person
- Changes the subject when asked
- How to test it
- Ask plainly: “Am I talking to a real person?” Listen to the answer, and to what it offers next.
- Ask the vendor
- Can you set how it introduces itself? If calls are recorded, does it play a recording notice, and can you change the wording? (See recording and consent, below.)
7. Fail safely
Everything breaks sometimes. What matters is what your caller experiences when it does, and whether you find out.
A pass looks like
- A clear fallback: voicemail, your cell, or a backup line
- You are told when calls fail or go unanswered
- A caller who isn’t understood gets a way forward
A fail looks like
- Calls ring out or drop with no record
- The caller is stuck in a loop
- You only learn about a problem from a customer
- How to test it
- Call from somewhere noisy and mumble a street name. For outages and other failures, ask the vendor how to trigger the fallback in a test setup you’ve arranged, rather than disrupting your live line.
- Ask the vendor
- What happens to calls during an outage? Do you publish a status page? Will you tell me when my calls fail?
Get these answers in writing
Some things you can’t learn from test calls. Ask the vendor, and keep the answers. A sales call is not a written answer; an email, the terms or a help page is.
| Ask | Why it matters | A useful answer covers |
|---|---|---|
| How is it priced? | Per minute, per call or flat monthly fees can differ a lot at your call volume. | The unit, what counts as a minute or a call, and the overage rate |
| Can I keep my phone number? | Moving your main number can be slow to undo. | Whether it works by call forwarding, porting, or a new number, and how to leave |
| Who can hear the recordings? | Calls contain customer details. | Who has access, how long recordings are kept, and whether you can delete them |
| Is our data used to train models? | Your customers’ calls may be used beyond your account. | A written yes or no, and how to opt out |
| What does it connect to? | A record in the wrong place is a record you won’t read. | Your calendar, CRM, email or text, and whether it writes or only reads |
| What’s the commitment? | A long contract makes a bad fit expensive. | Term length, cancellation terms, and what happens to your data when you cancel |
| Can I test it with my own calls first? | You need to see it handle your situations, not the demo’s. | A trial or a live walkthrough where you make the calls, its limits, and whether it has the same features as the paid plan |
Swipe the table sideways to see all three columns.
Comparing two or three products
- Same situations, same wording, same order. Otherwise you’re comparing your friend’s mood, not the products.
- Same setup effort. Give each product the same business information, entered the way its own instructions describe.
- Keep results by job. Don’t add them into one score. If after-hours emergencies are what keep you up at night, job 4 decides, whatever the totals say.
- Re-test carefully. Keep the first result. Work out whether the failure came from your setup or business information, or from the product. If you change the setup, write down what you changed and how long it took, make the equivalent change for every product you’re comparing, then run the same call again and record both attempts. Don’t keep adjusting one product until it passes.
A simple score sheet: one row per test call, with columns for the situation, what should happen, what it said, what it recorded, overstepped (yes or no), under-delivered (yes or no), and notes. Laid out this way, patterns are easier to spot.
Recording and consent
If the product records or transcribes calls, or calls and texts customers on its own, ask the vendor before you switch it on for real callers:
- Does it tell callers the call may be recorded, and can you change that wording?
- If it calls or texts customers, how does it get and keep their consent?
Recording and calling rules differ by state and by type of call. Check the ones that apply to your business; the sources below are a starting point, not legal advice.
Red flags
- It says a visit is “booked” when nothing is on your calendar.
- The record, the calendar and what the caller was told don’t agree.
- It quotes prices, discounts or arrival times your rules don’t allow.
- It claims to be a person when asked directly.
- An urgent call doesn’t reach you the way your rule says.
- The vendor can’t tell you in writing who can hear your recordings or how to get your data out.
What this checklist can’t tell you
It can’t tell you which product will pass. That takes testing, and when we publish comparisons they’ll follow our editorial policy: the same tasks for every product, results by task, and every claim labeled with its evidence and date. And a dozen test calls can’t show how often a product gets something right. They show how it handles the situations that matter to you.
Sources
- Introduction to the Reporter’s Recording Guide, Reporters Committee for Freedom of the Press.
- FCC ruling on AI-generated voices and the TCPA (2024), Federal Communications Commission news release, 8 February 2024.
Change log: October 2026, first version.