AI Assistant in Business: How to Measure Whether It Really Works

An AI assistant that receives a lot of conversations isn't necessarily working well: it's simply being used. The difference shows up in the numbers behind that usage — how many requests it resolves without human intervention, in how much time, with what rate of acknowledged errors, and at what cost per conversation. Without these measurements, any judgment remains just an impression.

Why "everyone uses it" isn't a metric

The number of conversations started is the first figure anyone looks at when evaluating an AI assistant, and it's also the least useful. It tells you how many people started typing, not how many got a correct answer, nor how many still had to wait for an operator. An assistant that's heavily used but routes poorly ends up adding an extra step before the human operator, instead of removing one.

The right question isn't "how many people use it", but "how many requests get closed without needing anything else, and under what conditions". The same principle applies here as to any technical investment in a company: as discussed in where artificial intelligence really makes sense in a company, the starting point must be the problem to solve, not the tool itself.

The metrics that really tell you if it's working

To move beyond impressions, you need specific numbers. These six indicators, read together, give a realistic picture of how the assistant is performing:

None of these numbers is enough on its own. A high resolution rate achieved by giving generic answers, with no error checking, isn't a good sign: it's simply an assistant that closes conversations without actually resolving them.

How to measure the quality of answers, not just the volume

The volume of conversations handled doesn't tell you whether the answers are correct. That's why an editorial review, not just a technical one, is needed on what the assistant actually generates.

• Manual sampling. Reading a sample of real conversations every week, rather than relying solely on aggregate metrics, is the most direct way to understand whether the answers are actually useful.

• Explicit user feedback. A simple question at the end of the conversation, such as "was this answer helpful?", with a low response rate that's still indicative of a recurring problem.

• Conversation logs. Keeping the history, in compliance with data protection regulations and checking with whoever handles the data how long it should be retained, makes it possible to reconstruct what was said in case of a dispute.

• Acknowledged hallucinations. How many times the assistant made up information, a price, an availability, a policy, that didn't exist in the data available to it. This is the hardest error to spot from aggregate metrics alone, because a made-up answer can sound just as fluent and confident as a correct one.

Three types of assistant, three ways of judging them

An informational assistant, a customer care one, and one connected to an ecommerce store aren't evaluated the same way, because they serve different purposes.

• Informational assistant (hours, services, frequently asked questions). The main metric is the resolution rate without escalation: the task is to answer questions that have a known, stable answer.

• Customer care. The main metric is the first response time and the quality of routing to the right operator, not total resolution: many support requests, by nature, remain a matter for human handling.

• Ecommerce assistant (product search, purchase support). The main metric is the conversion rate to contact or order, not the conversation itself. A concrete case applied to a PrestaShop store shows how the goal changes when the assistant is directly connected to the product catalog.

When a rule-based chatbot beats a language model

Not every assistant is worth building with a language model (LLM). For repetitive and predictable requests, such as order status, opening hours, a question with an always-the-same answer, a rule-based flow (a classic chatbot with predefined answers) often remains the most solid choice: it is one hundred percent predictable, has an almost fixed cost independent of volume, and does not risk making up an answer.

An LLM with information retrieval, technically a RAG (retrieval augmented generation) architecture, becomes useful when questions are phrased freely and unpredictably, or when there's a need to interpret a large, constantly updated catalog. The same principle applies to a CMS with built-in artificial intelligence: the right technology depends on the type of question it needs to handle, not on the "AI" label on the product. In practice, many companies end up using both approaches together, with the rule-based flow handling predictable cases and the LLM taking over for the rest.

The costs you don't see in the subscription fee

In some solutions, such as Keidea's AI assistant, the application fee may not be included: the costs of API calls for the chosen language model, which grow with the number of conversations handled, remain the responsibility of the user. This is a cost item that needs to be monitored month by month, not estimated only once at activation time: an assistant handling a few dozen conversations a day has a very different spending profile from one handling thousands, and the cost per conversation (the metric seen above) is the only way to keep it under control over time.

The same applies to an AI assistant applied to online sales: the larger the catalog retrieved for each response, the more the context to be processed grows with each conversation, and with it the variable cost.

Supervision, logs and privacy: what to check before saying it works

A high resolution number is not enough if the assistant has no control mechanism for when it gets things wrong. Before considering it reliable, check three things.

• Fallback and escalation. When the assistant is not sure of the answer, it says so and passes the request to an operator, instead of answering anyway with the risk of making up information.

• Logging. Every conversation is tracked, with date, content and, if provided for, the outcome (resolved, escalated, manually corrected): without logs it is not possible to reconstruct an error or calculate the metrics seen above.

• Retention and data handling. Check with the service provider where conversations are stored, for how long, and who has access to them. If you want to see an example of how a provider publicly documents these choices, Keidea's page on transparency about artificial intelligence shows what you should expect to find, regardless of the provider you choose.

A minimal 4-step method to start measuring

You don't need a complex analytics platform to get started. These four steps, in order, are enough:

1. Set a baseline before activating it. Document the volumes, times and current channel of the requests the assistant will have to handle: without a point of comparison, any subsequent number is meaningless.

2. Choose 3-4 metrics, not 15. Resolution rate, response time, escalation rate and cost per conversation are almost always enough to start with; adding more dilutes attention without producing any additional decisions.

3. Manually sample conversations every week, for the first 6-8 weeks. Read a real sample of exchanges, not just the aggregated numbers: it's the only way to catch hallucinations or answers that are technically correct but useless.

4. Review after a minimum cycle, not after a few days. A low volume in the first few weeks makes any percentage unreliable: wait until the sample is large enough before changing the configuration or judging the project.

If you want to see these numbers calculated for your specific case, the most direct way is a demo of the AI assistant configured on your real content, not on a generic scenario.

Request a Demo of the AI Assistant →