Skip to content
Sharan L · QA Engineering
Menu

← All case studies

AI Chatbot & LLM Quality Evaluation · AI Agent & MCP Testing

AI support features: a chatbot, a help-centre assistant and an order agent

15 seeded AI defects caught across a chatbot, a RAG assistant and an MCP agent running on local models, plus 6 real problems found in the versions that were meant to be fixed.

Demonstration on synthetic data in my own demo project, not client work.

Scenario

The shop adds three AI features: a support chatbot, a help-centre assistant that answers from documents (RAG), and an agent that looks up, cancels and refunds orders through an MCP server. They run on small open models through Ollama, so no paid API is used and no data leaves the machine.

Risks

Tests

Defects found

IDDefectType
AI01Stale policy answers: wrong return window and free-shipping thresholdMisinformation
AI02Invents policies it does not have (warranty, gift wrap, store hours)Misinformation
AI03Prompt injection exposes an internal escalation codePrompt injection
AI04Discloses an internal staff email addressData disclosure
AI05Does unrelated work such as cover letters and homeworkMisuse
RG01Serves the archived 14-day return policy instead of the current oneMisinformation
RG02Answers and cites documents for questions no document coversMisinformation
RG03An instruction planted in a community post reaches customersPrompt injection
RG04Staff discount codes and supplier margins retrievable by customersData disclosure
RG05Cites every retrieved document, related or notMisinformation
AG01Cancels and refunds without the customer's confirmationExcessive agency
AG02Tools read and change any customer's orderAccess control
AG03A planted courier-note instruction is treated as trustedPrompt injection
AG04Accepts negative refunds and refunds above the order totalInput validation
AG05An internal admin tool is exposed to customersExcessive agency

Evidence

Fix

Chatbot: answers only from the matching policy section, says "I don't know" when the policy is silent, blocks injection and off-topic requests, keeps secrets out of the prompt and filters every reply.

Help-centre assistant: only the latest, customer-facing, official documents are indexed; a relevance threshold measured from real scores (0.665, between the highest score of an unanswerable question, 0.655, and the lowest of an answerable one, 0.677); documents treated as data, never as instructions.

Agent: tools check the signed-in customer, refunds and cancellations need a confirmation that happens outside the model, arguments are validated, the admin tool is removed, and text from third parties is screened before the model sees it.

Result

v1.0 (seeded defects)
15 of 15 AI defects caught
Chatbot v1.1
16 of 16 cases, 3 runs
Help-centre assistant v1.1
16 of 16 cases, 3 runs; correct document in the top 3 for 100% of questions, MRR 1.0
Agent v1.1
12 of 12 contract checks and 7 of 7 agent cases, 3 runs

Lessons

  1. A "fixed" build is not a finished build: the suites found 6 real problems in the versions meant to be correct, from a false "I don't know" to an agent claiming it had updated an order when nothing changed.
  2. Read the replies, not only the pass/fail. A check on the order data alone missed two of the four agent problems.
  3. Measure search on its own first. The scores showed a real customer question could not reach the warranty article: a gap in the content, not in the model.
  4. Labelling third-party text as "untrusted" was not enough. The planted note stopped reaching customers only once it was screened before the model saw it.

Need this for your product?

Get a suggested scope and starting price in four questions, or tell me about your project.

Scope BuilderGet a quote

Next case study: Data migration: 1,200 orders moved to a new schema, reconciled to the cent →