Skip to content
Sharan L · QA Engineering
Menu

Proof Lab

I built ShopLab, a small online shop with 14 deliberately seeded defects (v1.0) and the same shop with every defect fixed (v1.1). My test suites catch every defect on v1.0 and verify every fix on v1.1 — automatically, on every change.

Demonstration with synthetic data — not client work. Source and CI results: GitHub repository (shared on request while private).

58

API + database tests (pytest, JSON schema, SQL)

v1.1 all pass v1.0: 10/10 caught

12

Browser tests (Playwright, phone, axe-core WCAG 2.2)

v1.1 all pass v1.0: 4/4 caught

~3 ms → ~1.5 s

k6, 50 users: p95 on v1.1 vs v1.0 (threshold 800 ms)

v1.0 slowdown caught

The 14 seeded defects

The kind of problems clients actually fear — and the test that catches each one.

IDSeverityDefect in v1.0Caught byv1.1
D06CriticalAny customer can read another customer's orderAPI authorization testFixed ✓
D01HighAccount never locks after repeated wrong passwordsAPI testFixed ✓
D03HighOne discount code can be applied twiceAPI testFixed ✓
D05HighQuantity accepts 0 and negative numbersAPI boundary testsFixed ✓
D08HighStock goes negative under simultaneous ordersConcurrency test + SQL checkFixed ✓
D14HighCustomer name runs as code in the browser (stored XSS)Playwright testFixed ✓
D02MediumSign-up accepts "a@b" and 1-character passwordsData-driven API testsFixed ✓
D04MediumTax truncated instead of roundedAPI + UI testsFixed ✓
D07MediumMalformed input returns a 500 with a stack traceAPI negative testsFixed ✓
D09MediumListed price differs from the databaseAPI + SQL comparisonFixed ✓
D10MediumProduct list slows to ~1.5 s at 50 usersk6 thresholdsFixed ✓
D11Medium"Place order" off-screen on phonesPlaywright phone testFixed ✓
D12MediumUnlabelled fields and low-contrast pricesPlaywright + axe-coreFixed ✓
D13LowSecurity headers missingAPI testFixed ✓

AI Trust Lab

ShopLab has three AI features, each built twice: v1.0 with seeded AI defects and v1.1 with them fixed. They run on small open models served locally by Ollama (qwen3:1.7b and nomic-embed-text), so no paid API is used and no data leaves the machine. Checks test facts, "I don't know", cited sources, forbidden content and what the AI actually changed, never exact wording. Every suite was repeated three times before these results were published.

Project 1 · Promptfoo

Support chatbot

v1.1: 16/16, 3 runs v1.0: 5/5 defects caught

Caught: stale policy answers, invented policies, prompt injection exposing an internal code, internal-email disclosure and off-topic misuse. The suite also found a real over-refusal in the fixed build (a false "I don't know"), now fixed.

Project 2 · Retrieval metrics + Promptfoo

Help-centre RAG assistant

v1.1: 16/16, 3 runs v1.0: 5/5 defects caught

Search is tested on its own first (hit rate at 3: 100%, MRR 1.0, no archived, internal or community document ever retrieved), then the answers. Caught: an archived policy served, answers with no supporting document, an instruction planted in a community post, internal codes leaked, and citations that cannot be traced. Measuring retrieval scores also found a content gap in the warranty article, now fixed.

Project 3 · MCP contract + agent tests

MCP order agent

v1.1: 12/12 + 7/7, 3 runs v1.0: 5/5 defects caught

The agent acts on orders through an MCP server. The tools are tested directly (12 contract checks, no model), then the agent is judged by its side effects. Caught: refunds and cancellations without the customer's confirmation, another customer's order exposed, a courier-note injection, negative and oversized refunds, and an admin tool exposed to customers. Reading every reply also found four real problems in the first fixed build, such as claiming an order was updated when nothing changed and relaying the planted note to the customer. All four are fixed and now tested.

Download the AI evaluation report (PDF, 4 pages) →

Mapped to the OWASP Top 10 for LLM Applications 2026: LLM01 prompt injection (direct and indirect), LLM02 sensitive information disclosure, LLM05 improper output handling, LLM06 excessive agency, LLM07 misinformation and LLM08 vector and embedding weaknesses.

API integration lab

ShopLab's partner API is what a warehouse or accounting system connects to: API keys with scopes, a documented OpenAPI 3.1 contract, refunds, and signed webhooks. The tests check the contract from the provider's side and from a consumer's side, every key and scope, and every webhook delivery, using a local receiver that can fail or answer slowly on purpose.

36 tests

Contracts, authentication and authorization, webhook delivery and security

v1.1: 36/36, 3 runs v1.0: 10/10 caught

Consumer contract

A contract written from the warehouse app's side is replayed against the API: one renamed money field breaks it

5 SSRF targets

Cloud metadata, ShopLab's own admin endpoint, loopback, private network and IPv6: all refused by v1.1

IDSeverityDefect in v1.0Caught byv1.1
AU01CriticalRead-only API keys can create refundsScope test (OWASP API5:2023)Fixed ✓
CT01HighOrder total sent as dollars with no currencyConsumer contract (warehouse app)Fixed ✓
CT03HighCustomer name and postal code sent to partnersData-minimisation check (API3:2023)Fixed ✓
AU02HighRevoked API keys still workRevoked-key test (API2:2023)Fixed ✓
AU03HighA partner can approve its own refundMass-assignment test (API3:2023)Fixed ✓
WH01HighWebhooks are not signedHMAC signature checkFixed ✓
WH02HighFailed deliveries (HTTP 5xx) never retriedReceiver answers 500 onceFixed ✓
WH04HighWebhook URLs may point to internal addresses (SSRF)5 internal URLs (API7:2023)Fixed ✓
CT02MediumErrors ignore the documented error formatOpenAPI Error schemaFixed ✓
WH03MediumA retry gets a new delivery idSlow receiver forces a retryFixed ✓

Data migration lab

ShopLab's legacy order database (300 customers, 1,200 orders, 2,966 order lines) moves to a new schema: a bulk load at a snapshot, then a delta load for changes made while it ran. The first version of the migration has 8 seeded data defects; the fixed version has none. The checks are plain SQL that work out the expected value from the source on their own, so a mistake in the migration cannot hide itself in the test.

16 SQL checks

Row counts, money to the cent, transformation rules, delta load and re-runs (30 pytest tests)

v1.1: 16/16 pass v1.0: 8/8 defects caught

176 → 0

Orders lost without a trace on v1.0. On v1.1: 1,197 migrated and 3 unreadable ones rejected with a reason

$245,963 → $0

Order value missing after the v1.0 bulk load. On v1.1 both systems hold $554,646.51

IDSeverityDefect in v1.0Caught byFound
DM01HighRows that fail conversion are skipped silentlyRow counts: source = target + rejected176 of 1,200 orders and 546 lines lost, nothing recorded
DM02HighMoney truncated when converted to centsRow-level amounts + checksum139 order totals, 9 prices, 1,006 line prices a cent short
DM04HighDay and month swapped in datesDate rule on every row371 orders, 106 customers
DM07HighRe-running the delta load duplicates order linesDuplicate-key check + second run50 duplicate lines, more on each retry
DM08HighLate changes to existing orders are ignoredLate-data check20 delivered orders still "shipped"
DM03MediumAccented names corrupted ("Noémie" → "No?mie")Row-level name comparison72 customers
DM05MediumUnknown status spellings become "placed"Mapping table vs source246 orders
DM06MediumLower-case product codes lose their product linkLine → product check96 order lines

Two checks pass on the broken build on purpose: a status check (the wrong status is still a valid one) and an orphan check (lost orders took their lines with them). Constraint checks prove data is valid; only comparing with the source proves it is correct. Read the case study →

Automation framework demos

  • Java + Selenium + TestNG — page objects, data-driven, BDD (Cucumber), Chrome/Firefox/Edge, Selenium Grid in Docker, Allure. 55/55 green on CI.
  • Playwright + TypeScript — custom fixtures, saved login, 5 browsers + phones, visual regression, network mocking, WCAG 2.2.
  • Python + pytest — page objects, parametrised negative cases, BDD, HTML/Allure reports.

Next

See how these results become a Go/No-Go decision →