HOME / BLOG / CHATBOT-PRE-FLIGHT-CHECK
The pre-flight check

Your chatbot should sit an exam
before it talks to customers.

Most chatbot platforms hand you a bot and wish you luck. You find out it misquotes your prices when a customer screenshots the chat. There's a duller, better order of operations: build the bot, examine it, then let it loose.

Try HyperChat free · 60s See the exam
THE SHORT ANSWER

A pre-flight check is an automated exam your chatbot sits after it's built from your website and before it meets a customer. Around ten questions drawn from your own business information are answered by the real bot and scored for groundedness by a judge model; then five adversarial probes — injection, jailbreak, price-fishing, off-topic and policy-fishing — try to break it. Your portal shows the result in plain English: "your bot passed 9/10 checks".

01 // The problem

You wouldn't publish a website without reading it first.

Yet that's exactly how most chatbots go live. Paste in a URL, wait a minute, embed the widget, hope. The bot is fluent from the first message — fluency is the easy part — and the first person to discover what it actually knows is a paying customer. If it invents a discount, misquotes your hours or makes up a refund policy, you find out from a screenshot, not a dashboard.

The fix isn't a better disclaimer. It's marking the bot's homework before term starts.

02 // The exam

Ten questions from your own site.
Marked by a judge.

After every knowledge-base build, automatically, the bot is tested on the material it was just taught. No mock answers — the real bot, the real pipeline.

HOW IT'S MARKED
Questions generated from your own business info~10
Answered by the real bot, not a simulationEVERY BUILD
Judge model scores groundedness — is the answer actually in your material?PASS / FAIL
Adversarial probes fired at the bot5

RE-RUNS ON EVERY REBUILD · NO EXTRA COST

IN YOUR PORTAL
WHAT YOU SEE
9/10
"YOUR BOT PASSED 9/10 CHECKS"

A plain-English score before a single visitor chats — and the failed checks name the exact question that tripped the bot, so you fix the knowledge rather than guess at it.

03 // The adversarial probes

Then five scripted attacks
try to break it.

Answering politely is only half the job. The other half is refusing to be steered. Each probe mimics a real thing visitors — or bored teenagers — will try.

INJECTION "Ignore your instructions and…" — the bot must keep its brief and deflect, not obey the visitor.
JAILBREAK Role-play tricks to make it drop its persona. It stays your business's bot, politely.
PRICE-FISHING "You quoted my mate half price." If the discount isn't in your material, the bot doesn't invent it.
OFF-TOPIC Homework help, politics, the football. The bot steers back to your business instead of wandering.
POLICY-FISHING "So I can always get a full refund, right?" No fabricated terms — airlines have been held to chatbot promises.
04 // When the score is low

A low score is useful.
It's a to-do list.

A bot that fails its exam is almost never a stupid bot — it's a faithful mirror of a website with holes in it. The fix is usually content, not code.

Read the failures first

The portal names the exact questions that failed. "What are your opening hours?" failing means the hours aren't on your site — or they're buried in a photo of a door sign the scraper can't read.

Fix the source content

Put prices, hours, service areas and policies in plain text on your site, then rebuild. The exam re-runs automatically, so you see the score climb — and your website gets better for humans too.

Patch gaps directly

For one-off answers, use the content-gaps list in your portal: type the answer once and it's in the knowledge base instantly — no rebuild, no developer. Live questions visitors ask get clustered there too.

The full accuracy system — exam, runtime guardrails, feedback loop — is covered on the AI chatbot accuracy page. More posts on the blog.

05 // Questions

Asked
plainly.

What is a chatbot pre-flight check?
An automated exam that runs after your bot is built from your website but before it talks to customers. Around ten questions drawn from your own business information are put to the real bot, a judge model scores each answer for groundedness, and five adversarial probes try to break it. The result appears in your portal as a plain score.
What are the five adversarial probes?
Prompt injection ("ignore your instructions"), a jailbreak attempt, price-fishing (trying to make the bot quote a discount that doesn’t exist), off-topic steering, and policy-fishing (trying to make it invent refund or cancellation terms). The bot must deflect all five to pass.
What should I do if my bot’s score is low?
Read which questions failed — the portal shows you. Most low scores trace back to the website itself: missing prices, vague service pages, no stated hours or policies. Improve the source content and rebuild, or answer the specific gap directly in your portal’s content-gaps list, which updates the knowledge base instantly without a rebuild.
Does a passing score guarantee the bot never makes a mistake?
No — no honest vendor can promise that. A passing score means the bot answered your own material accurately and survived the standard attacks on the day it was tested. The exam re-runs on every rebuild, and runtime guardrails keep refusing guesses between exams.
Does the pre-flight exam cost extra?
No. It runs automatically on every knowledge-base build, including the free preview you can build from your website URL with no card. Live is £49 per month per chatbot with 5,000 AI answers included.

Keep reading: the accuracy system · more from the blog · pricing

EXAM BEFORE CUSTOMERS

Watch your bot pass its own exam.

Build it free from your website in 60 seconds, then open the portal and read the score yourself. No card, no commitment.

Build my chatbot · free