What does it actually know?
Ask specifically: does it read my website automatically, or do I have to write every answer? A chatbot that only knows what you manually feed it will always lag behind your real content — new products, updated policies, seasonal changes — because someone has to remember to update it by hand.
The honest way to test this is to sign up, connect a real page, and see what the bot says about something on that page you didn't type in yourself. If it gets it right, the crawling actually works. If it can't answer basic questions from your own homepage, no amount of manual FAQ writing will fully compensate.
How does it handle what it doesn't know?
A chatbot that confidently makes up an answer when it's unsure is worse than no chatbot, because a visitor has no way to tell a fabricated answer from a real one. Test edge cases deliberately — ask it something genuinely outside your site's content and see whether it says so honestly, offers to connect you with a human, or fabricates a plausible-sounding answer anyway.
This single test tells you more about a chatbot's reliability than a dozen feature comparisons. Vendors rarely advertise how their product handles uncertainty, because 'sometimes it makes things up' isn't a selling point — you have to check for yourself.
Can you see what it's actually saying?
You should be able to review real conversation transcripts, not just a satisfaction score or a dashboard of vague engagement metrics. This is how you catch a bad answer before ten more visitors get the same one, and it's the difference between a chatbot you actively manage and one you've quietly lost visibility into.
If a vendor's dashboard only shows aggregate numbers — total chats, average response time — and hides the actual conversation content behind a higher tier or doesn't offer it at all, that's worth noting before you commit.
What happens if you outgrow the plan — or want to leave?
Check the upgrade path and whether there's lock-in. A widget that's one script tag to install is usually one script tag to remove, which is a reasonable sign the vendor isn't relying on switching costs to keep you rather than genuine usefulness. Ask what happens to your data — your conversation history, your crawled content — if you cancel.
Also check whether upgrading actually requires a new conversation with sales, or whether it's a self-serve plan change. The latter is a good sign the company expects to earn your continued business rather than trap it.
Putting it together
None of these checks require technical expertise, and none of them take long — you can run this entire evaluation for a vendor in under an hour, most of it spent just using the free trial like a real visitor would. That's usually more revealing than any spec sheet the vendor publishes themselves.
