Skip to main content

AI agent pilot for a small business: how to test in four weeks whether it really brings customers

Small clinic reception desk with a phone showing a chat and a four-week planner
Contents

An AI agent pilot that starts without a single number written down in advance almost always ends with the same sentence: "Seems fine." Nobody can say how many enquiries the agent closed on its own, how many hot leads reached a person in time or how many appointments were booked thanks to it, because nobody decided before the start what "fine" means. Anyone searching for how to choose an AI agent for their business reaches this stage quickly: they are offered a chance to try, and they agree. This is the guide to those four weeks: what to decide before day one, what to check each week, and how to decide at the end whether to continue, change or stop, without gut feelings.

A pilot is not "let's connect it and see"

"Let's connect it and see" sounds careful, but in practice it has no end. The agent runs, someone glances at the conversations when there is time, and two months later the decision is postponed again, because there is nothing to compare against. A real pilot is built from four fixed things: one channel, one task you can count, one person responsible for whatever the agent does not close, and a decision date in the calendar.

If you have not yet chosen who to work with, ask the vendor your questions before that. We collected them in the guide to choosing an AI agent for your business. This article starts from the moment you have chosen and decide to test it in real life.

What to decide before day one

One channel, not five

Start where most enquiries already arrive today. In a clinic that is usually WhatsApp. In an online shop it is the website. In a business that advertises on Instagram it is the direct messages coming from the ads. When you connect five channels at once, every problem looks like a problem with "the agent", and you cannot tell whether it is in the answers, in the channel or in the person who is supposed to continue the conversation. One channel gives a clean picture. The rest you connect after you decide to continue.

One task you can count

An AI agent can answer, ask questions, book an appointment, send a photo or a document and hand a hot lead to a person. In a pilot you pick one main task and at most one secondary task. Examples of a main task:

  • Answer every enquiry within a minute, including evenings and weekends.
  • Ask two or three questions and mark who is really interested and who is only asking.
  • Book an appointment straight into the calendar, without a round of "when suits you?".
  • Hand every hot lead to a person with a summary, within the same hour.

One sentence at the top of the page is enough: "At the end of the four weeks we will know whether more evening enquiries turned into appointments."

A baseline

Before you switch anything on, count two weeks back. How many enquiries came in the channel you chose, how many got an answer within an hour, how many of them arrived in the evening or on Shabbat, and how many became an appointment, an order or a quote that was sent. In WhatsApp Business you can scroll through the chats and count by hand; it takes about half an hour and is worth every minute. Without these numbers, at the end of the pilot you will compare against memory, and memory is always optimistic.

Who gets what the agent does not close

A good agent knows where it stops: a complaint, a medical question, an unusual request, a customer who asks to talk to a human. Each of these needs a person's name, the hours that person is available and the way they get notified. If the agent writes to a customer "a representative will get back to you tomorrow morning", that is a promise of the business, and someone needs to know about it in the morning. How to connect this so a hot lead does not fall through the cracks, we covered in the article on handing a hot lead from an AI agent to a person.

Week one: read conversations, not reports

In the first week you do not look at charts. You read conversations, all of them if possible, a quarter of an hour in the morning and a quarter of an hour in the evening. What to look for:

  • A wrong answer, for example old opening hours or a service that no longer exists.
  • A question that came up several times and the agent did not know how to answer.
  • A tone that does not fit the business: too formal for a hair salon, too casual for a law firm.
  • A promise the agent made that nobody kept.
  • A moment where the customer disappeared mid-conversation, and what was written to them just before.

Whatever you find, fix it at the source: add the missing answer, update the document the agent learns from, sharpen when to hand over to a person. The checks to run before the agent meets real customers are in the checklist of tests before launching an AI agent. Week one is their continuation, this time with real customers and questions nobody thought of.

One important rule: do not change three things in one day. One fix, a day of conversations, then the next fix. Otherwise you will not know what helped.

Weeks two and three: what to count

From the second week the agent already knows the business, and you can start counting. Five numbers are enough:

  1. Time to first answer, separately for working hours and for evenings and weekends. This is usually where you see the first difference.
  2. How many conversations closed without a person: the customer got an answer, an appointment was booked, a link was sent. And next to them, how many went to a person and why.
  3. How many of the conversations handed to a person got an answer within the time promised to the customer. This is the number most pilots forget.
  4. How many appointments, orders or meetings came from conversations the agent ran, against the baseline you counted.
  5. How many customers asked for a human or left mid-conversation, and at which stage of the conversation.

How to read the numbers together: if the evening answer time dropped and appointments did not rise, look at number 3 first. Sometimes the agent does its part, and the lead waits for a person until noon. If many customers leave at the same stage, for example after the question about price, fix the answer at that stage, not the whole agent.

You count all of these from one place. If every conversation is saved on a customer card with the source, the summary and the status, counting takes minutes. If the conversations are scattered between the secretary's phone, the email inbox and notes, the pilot will be measured by feel.

Want to consult with us?

We can help you choose, build and deploy the perfect AI solution for your business. Leave your details and we'll get back to you.

Week four: continue, change or stop

On the day you set in advance, sit down with the page from day one and the five numbers. There are three possible outcomes, and each has a clear sign.

Continue

The main number improved against the baseline, the leads handed to a person got an answer in time, and few customers asked for a human. This is the time to connect a second channel or add a task, for example booking appointments once answering works.

Change

The answers are good, but the main number did not move. In most cases the problem is not the agent but around it: the wrong channel, a person who does not get back to leads, or a task that does not fit the business. Change one thing and run two more weeks with the same counting method.

Stop

Customers ask for a human again and again, and the same mistakes return even after fixes. That is a legitimate outcome of a pilot, and it is why you run one: better to find out after four weeks than after a year.

One thing you do not measure at all: how "smart" the agent sounds. A customer does not book because of nice wording, but because they got a fast and correct answer.

Five mistakes that ruin a pilot

  • Switching on every channel on day one. Afterwards you cannot tell what worked and what did not.
  • Comparing an unusual month to a regular one. A pilot in a holiday week or a month with a big sale says little about a regular month. If there is no choice, compare to the same period last year or extend the pilot.
  • Nobody owns the handoffs. The agent hands over a hot lead, and everyone is sure someone else will get back to them.
  • Judging by one conversation. One bad conversation that someone screenshotted and sent to the team group is a reason to fix, not a reason to decide.
  • Not telling the team. A secretary who does not know there is an agent answers in parallel, and the customer gets two different answers.

What this looks like with WhaleBiz

Anyone who searches Google for a "WhatsApp bot" or a "chatbot for business" finds plenty of tools that answer frequently asked questions. WhaleBiz is a technology company that wrote and runs its own system: an autonomous sales system in which AI agents on WhatsApp, the website, Instagram, Telegram and Facebook answer customers 24/7 in any language, ask the right questions, book appointments and hand to a person whatever needs a person. Every conversation is saved on a card in the built-in CRM, with the source and the summary, so the five numbers from the previous section sit in one place.

For a pilot this means starting with one channel and one task, as described here. A website agent is already in the entry plan; WhatsApp and calendar booking are in a higher plan. What each plan includes is set out on the pricing page. What the agent does in a sales conversation, from the first question to the hot lead handed to a person, you can see on the sales agent page.

And an honest note: even at the end of a successful pilot, a person stays in the picture. The agent answers most enquiries and clears the way, and a person takes the conversations that need judgement. That is how it works today, and that is how it is worth measuring.

Frequently Asked Questions

How long should an AI agent pilot last?

Four weeks is a good range for most small businesses: a first week for fixes, two weeks for counting and a week for the decision. Shorter than that, the numbers are still affected by the early fixes. Longer than that without a decision date, the pilot turns into "let's connect it and see".

What if the business gets only a few enquiries?

If only a few enquiries arrive each week, extend the pilot to six or eight weeks and read every conversation. When the numbers are small, one conversation changes the picture, so it matters to see what happened in it and not only to count.

Can an AI agent work without any person at all?

No, and it should not try. The agent answers most enquiries, asks questions, books appointments and sends what is needed, and hands complaints, questions that need judgement and customers who ask for a human to a person. A pilot tests exactly that connection.

What happens if the agent makes a mistake with a customer during the pilot?

A person continues the conversation and corrects it with the customer, and the cause is fixed at the source: the missing answer, the outdated document or the rule for when to hand over to a person. That is why you read every conversation in the first week.

How much does it cost to work with a WhaleBiz AI agent?

Prices and what each plan includes are set out on the pricing page of the site. For a pilot, start with the plan that covers the channel and the task you chose, and expand after the decision.

Free check-up