Guest blog by Francoise Tourniaire at FT Works

Support chatbots are near ubiquitous and support leaders often boast of the large volume of queries they handle. But actual success is more than mere volume.

We set out to explore the actual customer experience. Holding 600 conversations with 60 chatbots of SaaS software companies, we found that the chatbots fully resolved just 42% of our queries, and not one of the bots succeeded on all 10 questions we posed.

How the study was conducted

Using AI agents, we asked 10 questions, two variations each of five common topics ranging from password reset to fixing a 403 error. The agent acted as a customer, reading each reply and deciding what to say next just like a real person would, providing clarification and pushing for more precise answer. We then graded the conversation against a rubric and assigned a  category of resolved, partial answer, routed to a person, gated behind a contact form or login, or dead end or wrong answer.

What we found

1. Fewer than half of conversations ended with a full answer

Only 42% of conversations were fully resolved, and another 23% got a partial answer. And this was for relatively easy questions.

2. Success depended a lot on the topic

Success rates ranged from 24% (for queries about cancellations) to 54% (for questions about how to set alerts). Success was much higher for precise queries. Since customers don’t always use “correct” terminology or phrasing, that’s a concern.

3. The chatbot platform is not destiny

On the very same chatbot platform, one company resolved eight of our ten questions while another resolved none. Technology by itself isn’t enough: the quality of the content makes the difference.

What good looks like

We found some solid best practices across the best answers. Here are a few.

• A good bedside manner helps. Don’t launch in a long intro if the customer alresy posed a question. Don’t repeat the same answer. Don’t end with a cheerful send-off if you provided no help.
• Ask clarifying question early. For instance, if the customer asks about changing password, start by asking whether they use SSO.
• Don’t gate the whole experience. Let anyone ask a question, and reserve logins for account-specific actions..

What you can do now

Ramp up your chatbot testing. It needs to be done regularly, not just at launch time. And make sure to use terminology and intents that your customers would actually use.

Take action on what you learn. Properly structuring knowledge base documents makes a huge difference.

Read the full study for more details: How good are SaaS support chatbots? Results from 600 real conversations

Francoise Tourniaire is the co-founder of isalo, a Support and Customer Success expert, and the author of The Art of Support. You can reach her at FT@isalo.ai.