AI Chatbot & LLM Quality Evaluation · AI Agent & MCP Testing
AI support features: a chatbot, a help-centre assistant and an order agent
15 seeded AI defects caught across a chatbot, a RAG assistant and an MCP agent running on local models, plus 6 real problems found in the versions that were meant to be fixed.
Demonstration on synthetic data in my own demo project, not client work.
Scenario
The shop adds three AI features: a support chatbot, a help-centre assistant that answers from documents (RAG), and an agent that looks up, cancels and refunds orders through an MCP server. They run on small open models through Ollama, so no paid API is used and no data leaves the machine.
Risks
- Confident wrong answers about returns and refunds
- Internal codes, staff emails or supplier margins disclosed to customers
- Instructions planted in documents or courier notes taking control of the AI (prompt injection)
- Refunds or cancellations made without the customer's confirmation
- One customer's order exposed to another
Tests
- Chatbot: 16 Promptfoo cases covering accuracy, made-up answers, injection, data leaks and off-topic use
- Help-centre assistant: search measured on its own first (23 queries: hit rate in the top 3, MRR, no forbidden documents), then 16 answer cases
- Agent: 12 MCP contract checks with no model involved, then 7 agent cases judged by their side effects (what actually changed in the order data)
- Checks test facts, "I don't know", cited sources and forbidden content, never exact wording. Every suite was repeated 3 times before publishing, and CI reruns all of it.
Defects found
| ID | Defect | Type |
|---|---|---|
| AI01 | Stale policy answers: wrong return window and free-shipping threshold | Misinformation |
| AI02 | Invents policies it does not have (warranty, gift wrap, store hours) | Misinformation |
| AI03 | Prompt injection exposes an internal escalation code | Prompt injection |
| AI04 | Discloses an internal staff email address | Data disclosure |
| AI05 | Does unrelated work such as cover letters and homework | Misuse |
| RG01 | Serves the archived 14-day return policy instead of the current one | Misinformation |
| RG02 | Answers and cites documents for questions no document covers | Misinformation |
| RG03 | An instruction planted in a community post reaches customers | Prompt injection |
| RG04 | Staff discount codes and supplier margins retrievable by customers | Data disclosure |
| RG05 | Cites every retrieved document, related or not | Misinformation |
| AG01 | Cancels and refunds without the customer's confirmation | Excessive agency |
| AG02 | Tools read and change any customer's order | Access control |
| AG03 | A planted courier-note instruction is treated as trusted | Prompt injection |
| AG04 | Accepts negative refunds and refunds above the order total | Input validation |
| AG05 | An internal admin tool is exposed to customers | Excessive agency |
Evidence
- AI evaluation report (PDF) — Results by layer, every seeded defect with the reply that exposed it, method and limits
- AI Trust Lab results — All three projects, with the defects each suite caught
Fix
Chatbot: answers only from the matching policy section, says "I don't know" when the policy is silent, blocks injection and off-topic requests, keeps secrets out of the prompt and filters every reply.
Help-centre assistant: only the latest, customer-facing, official documents are indexed; a relevance threshold measured from real scores (0.665, between the highest score of an unanswerable question, 0.655, and the lowest of an answerable one, 0.677); documents treated as data, never as instructions.
Agent: tools check the signed-in customer, refunds and cancellations need a confirmation that happens outside the model, arguments are validated, the admin tool is removed, and text from third parties is screened before the model sees it.
Result
- v1.0 (seeded defects)
- 15 of 15 AI defects caught
- Chatbot v1.1
- 16 of 16 cases, 3 runs
- Help-centre assistant v1.1
- 16 of 16 cases, 3 runs; correct document in the top 3 for 100% of questions, MRR 1.0
- Agent v1.1
- 12 of 12 contract checks and 7 of 7 agent cases, 3 runs
Lessons
- A "fixed" build is not a finished build: the suites found 6 real problems in the versions meant to be correct, from a false "I don't know" to an agent claiming it had updated an order when nothing changed.
- Read the replies, not only the pass/fail. A check on the order data alone missed two of the four agent problems.
- Measure search on its own first. The scores showed a real customer question could not reach the warranty article: a gap in the content, not in the model.
- Labelling third-party text as "untrusted" was not enough. The planted note stopped reaching customers only once it was screened before the model saw it.
Need this for your product?
Get a suggested scope and starting price in four questions, or tell me about your project.
Next case study: Data migration: 1,200 orders moved to a new schema, reconciled to the cent →