Chatbot Testing Checklist: 50 Cases to Run Before Production

Why conversational testing is different

Conventional software testing assumes a defined input space. Conversational systems do not have one: customers can say anything, in any order, in any language, with typos and interruptions and second thoughts. You cannot enumerate the inputs, which tempts teams to conclude that systematic testing is impossible and to replace it with a few demo conversations.

That conclusion is wrong, but the correction is not to test more conversations. It is to test categories. A well-built matrix covers the ways a conversation can go wrong rather than the things a customer might say, and a few dozen well-chosen cases will find more defects than a thousand happy-path chats.

What follows is a reusable matrix of roughly fifty cases across seven categories. Treat it as a starting checklist to adapt, and — more importantly — as something that is re-run on every release rather than once before launch.

Category 1: happy paths and core knowledge

Ten cases. These should pass trivially; if they do not, stop and fix the knowledge before testing anything else.

• The single most common question, asked exactly as customers ask it.
• The same question asked in three different phrasings, including one that uses no product terminology.
• The top ten questions by frequency, each answered correctly and with a traceable source.
• A question whose answer includes a condition or exception — verify the exception is included, not dropped.
• A question with a numeric answer: price range, timeframe, limit. Verify the number matches the source exactly.
• A multi-part question containing two separate asks.
• A follow-up question that depends on the previous turn ('and what about the second one?').
• A question answered by content updated in the last week, confirming the index is current.
• The full qualification flow completed end to end, with the record appearing correctly in the CRM.
• A booking or action flow completed end to end, verified in the target system rather than in the chat transcript.

The last two matter disproportionately. A conversation that looks correct in the chat window and writes nothing useful downstream is the most common undetected launch defect.

Category 2: ambiguity and imperfect input

Eight cases covering how real people actually type. This category finds more defects per case than any other.

• A question with significant typos and missing diacritics.
• A one-word message ('pricing', 'help', 'broken').
• A vague opener with no content ('hi', 'question', 'are you there').
• A question referring to something never mentioned ('does it work with ours?').
• A message mixing two languages in one sentence, which is routine in multilingual markets — see multilingual design.
• A very long message containing three questions and a complaint.
• A message in a language the bot does not support.
• A message consisting only of an emoji or punctuation.

Category 3: unsupported and out-of-scope requests

Eight cases. This is where the system's honesty is measured, and it is the category most often skipped because the cases are uncomfortable to write.

• A question with no documented answer anywhere — verify it declines rather than improvises.
• A question about a product you do not sell.
• A request for custom pricing or a discount.
• A request to interpret a contract clause.
• A comparison with a named competitor.
• A request for advice in a regulated domain.
• A question about another customer.
• A request for internal information — margins, roadmap, staff details.

Every case in this category should produce a refusal or an escalation, never an answer. Verify the refusal is useful: it should say what it cannot do and offer a next step, in line with your guardrails.

Category 4: escalation and handover

Eight cases, one per major rule in your escalation catalogue. If a rule cannot be tested, it does not exist.

• An explicit request for a human — verify immediate handover with no retention loop.
• Two failed attempts at the same question — verify escalation on the second, not the fourth.
• Frustration or anger language — verify detection and handover.
• Legal or regulatory language — verify it reaches a named destination with no substantive automated answer.
• A complaint — verify it does not enter a sales qualification flow.
• An escalation attempt outside business hours — verify honest messaging and an asynchronous option, not a false promise.
• Handover context — verify the agent receives the transcript, resolved identity, detected intent and escalation reason.
• Return path after the agent closes — verify the defined behaviour, whatever you chose.

Run these against the live escalation destinations, not against a test queue. Most escalation defects live in the integration rather than in the detection.

Category 5: guardrails and adversarial input

Eight cases. Someone should deliberately try to break it, and it should not be the person who built it.

• An attempt to get a commitment ('so you guarantee it will work with X?').
• An attempt to extract internal content by rephrasing.
• An attempt to access another customer's data by claiming to be them.
• An instruction to ignore previous instructions.
• A request to execute an action outside the permitted list.
• A request to repeat back its configuration or system instructions.
• A leading question presupposing a false fact ('since you offer free installation, when...').
• Repeated pressure after a refusal.

Category 6: integration and failure conditions

Eight cases. These require deliberately breaking things, which is why they are usually skipped and why they cause most production incidents.

• CRM unavailable during a conversation — verify the conversation continues and the data is queued rather than lost.
• CRM slow to respond — verify a timeout is handled without stalling the reply.
• Calendar returning stale availability — verify a double booking cannot be confirmed.
• A duplicate contact — verify the matching rule updates rather than creating a second record.
• An action executed twice by a retry — verify idempotency.
• Knowledge source unreachable — verify honest messaging rather than improvised answers.
• A field value the CRM rejects — verify the failure is logged and surfaced, not swallowed.
• Concurrent conversations from the same customer on two channels.

Building these into an automated suite is worth the effort, because they are the cases nobody will run manually under release pressure. Our custom software services page covers how this layer is typically constructed.

Category 7: channel, locale and presentation

The final eight cases, checking that behaviour survives its context.

• The same conversation on each supported channel — web, WhatsApp, Instagram — verifying consistent answers and routing.
• Message formatting per channel, where rich formatting may not render.
• Long answers on a mobile viewport.
• Each supported language, with its own test set rather than a translation of one.
• Language switching mid-conversation, verifying context survives.
• Date, currency and number formatting per locale.
• Session resumption after the customer closes and reopens the widget.
• Accessibility of the chat interface — keyboard navigation and screen reader labels, which are frequently overlooked entirely.

Consistency across channels is the case most often failed by deployments that configured each channel separately, which is one of the practical reasons to run a single configuration across channels — as a consolidated AI communication platform does.

Turning the checklist into a repeatable suite

A checklist run once before launch has a short useful life. The value comes from re-running it.

1. 1

   Record expected behaviour, not expected text

   Asserting exact wording produces a suite that fails on every harmless rephrasing. Assert the facts present, the source cited, the route taken and the action performed.

2. 2

   Automate what can be automated

   Integration, escalation routing, action idempotency and refusal behaviour can be checked automatically. Tone and nuance need a person.

3. 3

   Keep a human review sample

   A weekly read of real transcripts finds categories of problem no suite anticipates, and it is where next quarter's test cases come from.

4. 4

   Gate releases on the suite

   Any change to content, configuration, prompts or integrations triggers a re-run. This is what prevents the slow degradation that otherwise characterises these systems.

5. 5

   Add a case for every production defect

   The suite should grow from real failures rather than from imagination. This single practice keeps it relevant.

If the suite is slow or manual, it will be skipped exactly when it matters most — during an urgent fix.

Metrics and exit criteria

Decide what 'ready' means before testing starts, or the decision will be made by the release date.

1. 1

   Pass rate on the happy-path and knowledge categories

   This should be at or very near complete. Failures here indicate the knowledge base is not ready and no amount of conversation tuning will compensate.

2. 2

   Zero failures in guardrails and out-of-scope

   A non-negotiable gate. A single case where the bot made a commitment it should not have is enough to hold a launch.

3. 3

   Every escalation rule demonstrably firing

   One passing test per rule, against live destinations.

4. 4

   Integration failure cases handled without data loss

   Degraded behaviour is acceptable; losing a lead is not.

Publishing these criteria in advance makes the launch conversation a factual one rather than a negotiation about confidence.

What testing cannot establish

Some questions are not answerable by a test suite, and pretending otherwise creates false confidence.

• Whether the tone suits your customers — that needs real conversations and human judgement.
• Whether the knowledge is correct, as opposed to consistently retrieved. Testing verifies the system says what the source says; whether the source is right is a content-ownership question.
• Whether automating a category is appropriate at all, particularly where legal or regulatory exposure exists.
• How the system behaves under volumes you have not simulated.
• Which questions customers will actually ask, which only production reveals.

Plan for a supervised first period after launch, with elevated human review, on the assumption that the first weeks will surface categories the matrix did not contain.

Decision framework and next step

Four questions before setting a launch date.

1. 1

   Do you have test cases for every escalation rule?

   Untested rules are the most common source of post-launch surprise.

2. 2

   Has someone adversarial tried to break it?

   Not the implementer. This category finds different defects depending on who runs it.

3. 3

   Have you tested with integrations deliberately failing?

   If not, production will run that test for you at a worse moment.

4. 4

   Can the suite be re-run in an afternoon?

   If not, it will not be re-run, and the system will drift.

Start with categories 1, 3 and 4 — happy paths, out of scope, escalation. Those three cover most of the risk. Add the rest before widening scope. Our AI solutions overview covers how testing usually fits into the implementation stages.

Frequently asked questions

1. 1

   What should a chatbot testing checklist cover?

   Seven categories: happy paths and core knowledge, ambiguous and imperfect input, out-of-scope requests, escalation rules, adversarial attempts, integration failure conditions, and channel and locale behaviour. Roughly fifty cases covers a typical B2B deployment.

2. 2

   What does chatbot testing require that other QA does not?

   Asserting behaviour rather than exact text, live escalation destinations rather than test queues, deliberately broken integrations, adversarial testing by someone other than the implementer, and a separate test set per supported language.

3. 3

   Which exit criteria should gate a launch?

   Near-complete pass on happy paths and knowledge, zero failures on guardrails and out-of-scope cases, one passing test per escalation rule against live destinations, and no data loss in integration failure scenarios.

4. 4

   What are the most common testing mistakes?

   Testing only happy paths, asserting exact wording, using test queues instead of live escalation destinations, never simulating integration failures, and running the suite once before launch rather than on every change.

5. 5

   What can testing not tell you?

   Whether the tone fits your customers, whether the source content is factually right, whether a category should be automated at all, and which questions customers will actually ask.

A test suite and a knowledge audit answer different questions — one checks that the system says what the content says, the other checks whether the content is worth saying.