To know whether an AI agent is doing any good, you need to follow a small number of measures. But more measures are not better. A measure that looks good can hide a problem, and one that looks bad may have a simple explanation. This article goes through the measures that are most often useful, how to interpret them and what to do with them. There are no benchmarks here, because the right level depends on your business. What counts is how you develop against your own baseline.
Start with a baseline
A measure without a comparison says little. Before you start, measure how things look today, using the same method you will use later:
- how many enquiries come in, per channel
- what share is recurring questions
- how long one such question takes to answer
- how long it takes to the first reply
- how many enquiries go unanswered
Without this baseline you cannot tell an improvement from chance. If you do not have it, start by collecting it. It is the most valuable part of the whole measurement.
Five measures that are usually worth following
Volume
Number of conversations per week or month. Interpret the change together with traffic on the channel and customer outcomes. High volume is not a quality measure in itself: a campaign, a change on the website or a confusing text can raise it without anything getting better.
Answer quality
This measure requires a person to read. Take a sample of conversations regularly and judge whether the customer got a correct and useful answer. Note errors, wrong assumptions and gaps in the knowledge. It cannot be replaced by an automatic number, and it is often what gives the most to fix. How to keep the knowledge current is described in maintaining the knowledge base.
Resolved without handover
The share of conversations that end without a person being needed. But be careful: a low number of handovers does not prove that the issue was resolved or that time was saved. The customer may have given up. Check with samples that the result was a useful answer.
Handovers
Track why cases go to a person and whether the recipient gets enough context. There is no general normal level. The right level depends on risk, task and customer need. An agent that hands over the right cases at the right moment is doing its job, even if the share is high. Read about when AI should escalate to a human and how handover to staff works when it is configured.
Completed actions
Where the agent can carry something out, for example take a booking, count those that were actually completed. Compare against your baseline and include cancellations and no-shows where relevant. Available actions depend on platform, plan and configuration, so measure only what you have actually set up. See integrations for what applies.
Measures that easily mislead
| Measure | Why it can mislead | Complement with |
|---|---|---|
| Number of conversations | More volume is not better quality | Samples of answers |
| Low share of handovers | The customer may have given up | Review of closed dialogues |
| Short handling time | May mean an incomplete answer | Whether the issue was resolved |
| High share of handovers | May be right if cases are sensitive | Reasons and the context that came along |
Time and cost
To know whether it saves time, measure the same thing before and after: time spent on recurring questions, on follow-up of handed-over cases and on maintaining the knowledge. Count all the time, including your own. The cost side is described in what an AI agent costs, and when automating customer service pays off shows a way to calculate with your own numbers. Treat the result as a calculation and not as a promise.
Make measurement a routine
- Decide on the few measures you follow. Start narrow and add more only when you know what you will use them for.
- Read conversations regularly. Weekly during the pilot, then at a pace that suits the volume.
- Fix what you find. Errors and gaps in the knowledge are the fastest way to improve the outcome.
- Compare against the baseline. Same period, same method.
- Review again when things change. New channels, new knowledge or changed limits can change the picture.
Before launch, a test with real questions helps you get a first picture of the quality. More ways to calculate the benefit are in the guides. If you want to go through which measures suit you, you can contact us.
