How to Test AI Systems (For SaaS Teams)

Testing AI systems is not testing software. The output is non-deterministic; correctness is fuzzy; quality drifts with model changes. Here is a practical framework.

The Four Layers of AI Testing

1. Evals — input/expected-output pairs run on every prompt or model change.
2. Regression suite — replay real production inputs and compare outputs over time.
3. Observability — log every prompt, response, latency, and cost for debugging.
4. User-feedback loops — thumbs up/down, edits, escalations as ground-truth labels.

Layer 1: Evals

The minimum: 10-20 input/expected-output pairs per feature. Run them on every prompt change. Tools: Promptfoo, Braintrust, in-house Python script.

Scoring: exact match for structured output, LLM-as-judge for open-ended. LLM-as-judge introduces its own non-determinism — pin the judge model and prompt.

Layer 2: Regression Suite

Capture 100-500 real production inputs. When you change a prompt or upgrade a model, replay them and diff the outputs. Manually review changes. The diff is the test.

Run on every model release from your provider. "Anthropic released Sonnet 4.5" should trigger a regression run before you flip the production model.

Layer 3: Observability

Log every request: input, output, model version, prompt version, latency, token cost. Without this, debugging "why did the bot say that" is impossible.

Tools: Helicone, LangSmith, Braintrust, or a Postgres table you query yourself. The tool matters less than having it.

Layer 4: User Feedback

Thumbs up/down on every AI response. Capture edits when users override AI output. These are the real ground truth labels for what "good" looks like in your context. Feed them back into your eval set monthly.

Common Pitfalls

• Treating one bad output as a failure. AI is non-deterministic. Use sample sizes (100+) for quality claims.
• Trusting LLM-as-judge without calibration. Compare judge scores to human scores on a 50-sample set before relying on the judge.
• No model-version pinning. Provider auto-upgrades change behavior. Pin model versions explicitly.
• No cost tests. A prompt change can 5x your tokens. Test cost like you test correctness.

Test Strategy by Stage

• Pre-launch: 20 evals, manual smoke tests.
• Post-launch month 1: add observability, build regression suite from real inputs.
• Month 3+: monthly regression runs, LLM-as-judge for scaled scoring, user-feedback dashboards.

What to Do Next

If you have an AI feature in production with no evals: stop adding features for one day. Write 20 evals. The discipline pays back the next time you change a prompt and break three of them silently.