How to Build AI Tools (For SaaS Builders)

Building an AI tool inside a SaaS product is now a one-week project for the v0.5 and a six-month project to make it actually work in production. Most teams nail the first and stall on the second. Here is what the full path looks like.

The Layers You Actually Need

• Model layer — the LLM call. Pick one provider, one default model. Add fallback later.
• Prompt layer — system prompt, few-shot examples, output format. Lives in code, not in a config file you never update.
• Eval layer — a small set of test cases with expected outputs. Run on every prompt change.
• Routing layer — which model handles which request, cost-aware. Add this only when you have multiple models worth comparing.
• Caching layer — request-level caching for identical prompts. Saves tokens and latency.
• Observability layer — log every prompt, response, cost, and latency. Without this, debugging production AI bugs is impossible.

Model Choice

Default to Claude or GPT-4o. Pick one. Switching models is cheap once your eval suite exists; expensive before. Use a smaller model (Haiku, GPT-4o-mini) only if your eval shows it passes — never as a default optimization.

Open source (Llama, Mistral) makes sense only if you have specific data residency, compliance, or cost requirements that the hosted APIs cannot meet. For most SaaS teams, the time-to-launch tradeoff favors hosted.

Prompt Engineering That Holds Up

• System prompt declares role, tone, constraints. Specific. Not "helpful assistant."
• Few-shot examples beat instructions for consistency. 3-5 examples is the sweet spot.
• Output format specified explicitly — JSON schema, headings, length. Models hold formats reliably when asked.
• Refusal handling — what does the tool do when the input is out of scope? Specify in the prompt.

Evals — The Step Most Teams Skip

Without evals, you cannot iterate. The minimum eval suite: 10-20 test inputs with expected outputs (or scoring rubrics), run on every prompt change. Tools: Promptfoo, Braintrust, or a 50-line in-house script.

The first time you change the prompt and break three test cases without noticing is the moment you understand why evals exist.

Cost Math at Scale

Run unit economics at 100, 1,000, and 10,000 daily active users before you launch. A feature that costs $0.05 per use is $50/day at 1k users — fine. At 10k, it is $500/day. Decide pricing tier and rate limiting before you scale, not after the bill shocks you.

Caching with prompt-prefix caching (Anthropic) or content-cache (OpenAI) routinely cuts cost 50-70% for chat-style features. Implement before launch, not after.

Common Mistakes

• No evals. Means you cannot tune the prompt with confidence. Means quality regresses silently.
• No observability. Means the first production bug takes a week to debug.
• Switching models for marginal gain. The 5% quality bump from a new model is rarely worth the eval-suite invalidation.
• Pricing flat when costs are variable. Cap usage per tier, or pass token costs through.

What to Do Next

If you are starting today: ship a v0.5 with one model, one prompt, no evals, full logging. One week of work. Then add evals before you ship to all users. If you have a feature in production with no evals: stop adding features and write evals first. The discipline pays for itself in the first month.