Proof Lab
I built ShopLab, a small online shop with 14 deliberately seeded defects (v1.0) and the same shop with every defect fixed (v1.1). My test suites catch every defect on v1.0 and verify every fix on v1.1 — automatically, on every change.
Demonstration with synthetic data — not client work. Source and CI results: GitHub repository (shared on request while private).
58
API + database tests (pytest, JSON schema, SQL)
v1.1 all pass v1.0: 10/10 caught
12
Browser tests (Playwright, phone, axe-core WCAG 2.2)
v1.1 all pass v1.0: 4/4 caught
~3 ms → ~1.5 s
k6, 50 users: p95 on v1.1 vs v1.0 (threshold 800 ms)
v1.0 slowdown caught
The 14 seeded defects
The kind of problems clients actually fear — and the test that catches each one.
| ID | Severity | Defect in v1.0 | Caught by | v1.1 |
|---|---|---|---|---|
| D06 | Critical | Any customer can read another customer's order | API authorization test | Fixed ✓ |
| D01 | High | Account never locks after repeated wrong passwords | API test | Fixed ✓ |
| D03 | High | One discount code can be applied twice | API test | Fixed ✓ |
| D05 | High | Quantity accepts 0 and negative numbers | API boundary tests | Fixed ✓ |
| D08 | High | Stock goes negative under simultaneous orders | Concurrency test + SQL check | Fixed ✓ |
| D14 | High | Customer name runs as code in the browser (stored XSS) | Playwright test | Fixed ✓ |
| D02 | Medium | Sign-up accepts "a@b" and 1-character passwords | Data-driven API tests | Fixed ✓ |
| D04 | Medium | Tax truncated instead of rounded | API + UI tests | Fixed ✓ |
| D07 | Medium | Malformed input returns a 500 with a stack trace | API negative tests | Fixed ✓ |
| D09 | Medium | Listed price differs from the database | API + SQL comparison | Fixed ✓ |
| D10 | Medium | Product list slows to ~1.5 s at 50 users | k6 thresholds | Fixed ✓ |
| D11 | Medium | "Place order" off-screen on phones | Playwright phone test | Fixed ✓ |
| D12 | Medium | Unlabelled fields and low-contrast prices | Playwright + axe-core | Fixed ✓ |
| D13 | Low | Security headers missing | API test | Fixed ✓ |
AI Trust Lab
ShopLab has three AI features, each built twice: v1.0 with seeded AI defects and v1.1 with them fixed. They run on small open models served locally by Ollama (qwen3:1.7b and nomic-embed-text), so no paid API is used and no data leaves the machine. Checks test facts, "I don't know", cited sources, forbidden content and what the AI actually changed, never exact wording. Every suite was repeated three times before these results were published.
Project 1 · Promptfoo
Support chatbot
v1.1: 16/16, 3 runs v1.0: 5/5 defects caught
Caught: stale policy answers, invented policies, prompt injection exposing an internal code, internal-email disclosure and off-topic misuse. The suite also found a real over-refusal in the fixed build (a false "I don't know"), now fixed.
Project 2 · Retrieval metrics + Promptfoo
Help-centre RAG assistant
v1.1: 16/16, 3 runs v1.0: 5/5 defects caught
Search is tested on its own first (hit rate at 3: 100%, MRR 1.0, no archived, internal or community document ever retrieved), then the answers. Caught: an archived policy served, answers with no supporting document, an instruction planted in a community post, internal codes leaked, and citations that cannot be traced. Measuring retrieval scores also found a content gap in the warranty article, now fixed.
Project 3 · MCP contract + agent tests
MCP order agent
v1.1: 12/12 + 7/7, 3 runs v1.0: 5/5 defects caught
The agent acts on orders through an MCP server. The tools are tested directly (12 contract checks, no model), then the agent is judged by its side effects. Caught: refunds and cancellations without the customer's confirmation, another customer's order exposed, a courier-note injection, negative and oversized refunds, and an admin tool exposed to customers. Reading every reply also found four real problems in the first fixed build, such as claiming an order was updated when nothing changed and relaying the planted note to the customer. All four are fixed and now tested.
Download the AI evaluation report (PDF, 4 pages) →
Mapped to the OWASP Top 10 for LLM Applications 2026: LLM01 prompt injection (direct and indirect), LLM02 sensitive information disclosure, LLM05 improper output handling, LLM06 excessive agency, LLM07 misinformation and LLM08 vector and embedding weaknesses.
API integration lab
ShopLab's partner API is what a warehouse or accounting system connects to: API keys with scopes, a documented OpenAPI 3.1 contract, refunds, and signed webhooks. The tests check the contract from the provider's side and from a consumer's side, every key and scope, and every webhook delivery, using a local receiver that can fail or answer slowly on purpose.
36 tests
Contracts, authentication and authorization, webhook delivery and security
v1.1: 36/36, 3 runs v1.0: 10/10 caught
Consumer contract
A contract written from the warehouse app's side is replayed against the API: one renamed money field breaks it
5 SSRF targets
Cloud metadata, ShopLab's own admin endpoint, loopback, private network and IPv6: all refused by v1.1
| ID | Severity | Defect in v1.0 | Caught by | v1.1 |
|---|---|---|---|---|
| AU01 | Critical | Read-only API keys can create refunds | Scope test (OWASP API5:2023) | Fixed ✓ |
| CT01 | High | Order total sent as dollars with no currency | Consumer contract (warehouse app) | Fixed ✓ |
| CT03 | High | Customer name and postal code sent to partners | Data-minimisation check (API3:2023) | Fixed ✓ |
| AU02 | High | Revoked API keys still work | Revoked-key test (API2:2023) | Fixed ✓ |
| AU03 | High | A partner can approve its own refund | Mass-assignment test (API3:2023) | Fixed ✓ |
| WH01 | High | Webhooks are not signed | HMAC signature check | Fixed ✓ |
| WH02 | High | Failed deliveries (HTTP 5xx) never retried | Receiver answers 500 once | Fixed ✓ |
| WH04 | High | Webhook URLs may point to internal addresses (SSRF) | 5 internal URLs (API7:2023) | Fixed ✓ |
| CT02 | Medium | Errors ignore the documented error format | OpenAPI Error schema | Fixed ✓ |
| WH03 | Medium | A retry gets a new delivery id | Slow receiver forces a retry | Fixed ✓ |
Data migration lab
ShopLab's legacy order database (300 customers, 1,200 orders, 2,966 order lines) moves to a new schema: a bulk load at a snapshot, then a delta load for changes made while it ran. The first version of the migration has 8 seeded data defects; the fixed version has none. The checks are plain SQL that work out the expected value from the source on their own, so a mistake in the migration cannot hide itself in the test.
16 SQL checks
Row counts, money to the cent, transformation rules, delta load and re-runs (30 pytest tests)
v1.1: 16/16 pass v1.0: 8/8 defects caught
176 → 0
Orders lost without a trace on v1.0. On v1.1: 1,197 migrated and 3 unreadable ones rejected with a reason
$245,963 → $0
Order value missing after the v1.0 bulk load. On v1.1 both systems hold $554,646.51
| ID | Severity | Defect in v1.0 | Caught by | Found |
|---|---|---|---|---|
| DM01 | High | Rows that fail conversion are skipped silently | Row counts: source = target + rejected | 176 of 1,200 orders and 546 lines lost, nothing recorded |
| DM02 | High | Money truncated when converted to cents | Row-level amounts + checksum | 139 order totals, 9 prices, 1,006 line prices a cent short |
| DM04 | High | Day and month swapped in dates | Date rule on every row | 371 orders, 106 customers |
| DM07 | High | Re-running the delta load duplicates order lines | Duplicate-key check + second run | 50 duplicate lines, more on each retry |
| DM08 | High | Late changes to existing orders are ignored | Late-data check | 20 delivered orders still "shipped" |
| DM03 | Medium | Accented names corrupted ("Noémie" → "No?mie") | Row-level name comparison | 72 customers |
| DM05 | Medium | Unknown status spellings become "placed" | Mapping table vs source | 246 orders |
| DM06 | Medium | Lower-case product codes lose their product link | Line → product check | 96 order lines |
Two checks pass on the broken build on purpose: a status check (the wrong status is still a valid one) and an orphan check (lost orders took their lines with them). Constraint checks prove data is valid; only comparing with the source proves it is correct. Read the case study →
Automation framework demos
- Java + Selenium + TestNG — page objects, data-driven, BDD (Cucumber), Chrome/Firefox/Edge, Selenium Grid in Docker, Allure. 55/55 green on CI.
- Playwright + TypeScript — custom fixtures, saved login, 5 browsers + phones, visual regression, network mocking, WCAG 2.2.
- Python + pytest — page objects, parametrised negative cases, BDD, HTML/Allure reports.
Next
See how these results become a Go/No-Go decision →