The 8 questions to ask before you buy an AI chatbot

An AI chatbot is easy to demo and hard to procure well. Every vendor’s pilot works — that’s the point of a pilot — and the real differences only appear months later when volume, edge cases and integrations catch up.

Here are the eight questions I would ask before signing anything. Ask them in writing.

1. Show me an answer where the bot didn’t know

Get the vendor to demo a question the knowledge base cannot answer. Watch what the bot says. If it fabricates a plausible answer, walk away — that’s what will happen to your customers, too, and you won’t find out until reviews start showing up.

A good bot refuses cleanly and offers escalation.

2. Every answer must show its source. Yes or no?

Not “we support citations.” Every answer. If a claim can’t be traced back to a document you loaded, the model made it up. In regulated verticals this is the single most important control.

Anserra makes this mandatory by default. Most vendors treat it as opt-in and hide the toggle.

3. What happens when I delete a source?

The answer you want: “The document and its embeddings are removed immediately, and the chatbot no longer draws from it.”

Answers you don’t want:

• “It’s queued for deletion overnight.” (Answers keep coming from deleted content for hours.)
• “The embeddings persist, but we filter them at query time.” (One misconfiguration and you leak deleted content.)

Ask them to explain the exact sequence.

4. Can I export everything?

Ask for:

• All chatbot configurations as JSON.
• All uploaded documents as their original files.
• Every conversation as a transcript.
• Every lead the bot captured.

If any of those aren’t available on a paid plan, you’re renting your data.

5. How does hybrid retrieval work here?

Semantic search alone fails on exact strings — SKUs, error codes, account IDs, model numbers. Ask specifically:

• Do you combine semantic and keyword retrieval?
• What algorithm fuses the ranks?
• Can I configure the weighting?

Vague answers here mean the bot will feel dumb when customers paste real strings into it.

6. What roles exist, and what can each do?

The right answer is at least: Owner, Admin, Manager, Agent, Viewer.

The one you should test: can an Agent (typically a support person) read conversations but not change the system prompt? If everyone with access can change the bot’s behaviour, you’ll wake up to answers that don’t match your policies.

7. What limits does the free/entry plan impose, and how do they escalate?

Trap questions like:

• “How many messages before I have to upgrade?”
• “Is there a document count limit? A crawl page limit?”
• “What happens when I hit the ceiling — does the widget just stop, or does it fall back?”

Some vendors block the widget mid-conversation. That is the worst possible customer experience. Others meter and warn. Others let you overage. Know which.

8. Show me an outage post-mortem

Every real product has had outages. Vendors that have publicly written up how they handled one are vendors that will handle the next one better. Vendors that have never published a post-mortem are vendors whose ops story you should probe harder.

The bonus question that separates real vendors

Who owns the vector database?

If the answer involves a hosted vector store you can’t see into, migrating your embeddings is going to be a rebuild. If the answer involves a database you can query — Postgres with pgvector, for example — you own the tokens you paid to compute.

What honest limits look like

Anserra’s pricing page tells you exactly what the free plan does and doesn’t include, including where the built-in model is a mock and where it’s live. We say when a chatbot is in demo mode. Nobody buys a support tool that lies to them; there’s no reason a support tool should be built by a company that lies about how it works.

If this checklist changes your shortlist, start a trial. No credit card required.