How AI Tools Are Developed
The development cycle for an AI feature in a SaaS product has stabilized. Here is what it actually looks like, in practice.
Phase 1: Problem Framing (1-3 days)
- Pick a workflow where AI compresses time 5x or more.
- Define inputs and outputs precisely.
- Write 5-10 example inputs and the outputs you would expect.
- Decide whether this needs RAG, tool use, or just a strong prompt.
Phase 2: Prototype (1-3 days)
- Pick a model (Claude or GPT-4 default).
- Write the system prompt.
- Run your example inputs through it manually.
- Iterate until the prompt produces acceptable output on 80% of examples.
Phase 3: Eval Harness (1-2 days)
- Convert your example inputs into formal evals with expected outputs.
- Write a script that runs the prompt against the evals and scores.
- This becomes your regression suite for every prompt change going forward.
Phase 4: Production Build (3-7 days)
- Wire the prompt into the product UX.
- Add observability (log every prompt, response, cost, latency).
- Add error handling and fallbacks.
- Add streaming if user-facing chat.
- Implement caching where prompts have stable prefixes.
Phase 5: Beta and Iterate (2-4 weeks)
- Release to a small user cohort.
- Capture user feedback (thumbs up/down, edits).
- Mine production logs for failure cases.
- Add new evals from real failures.
- Tune the prompt against the expanded eval set.
Phase 6: GA and Maintain
- Roll out to all users.
- Monitor cost and quality dashboards weekly.
- Run the regression suite on every model upgrade.
- Re-tune the prompt monthly as user behavior reveals new edges.
Where Teams Get Stuck
- Skipping the eval harness. Then prompt changes are guesswork.
- Skipping observability. Then production debugging is impossible.
- Treating AI features as one-time builds. They are living systems that drift with model upgrades.
What to Do Next
If you are starting an AI feature: budget two weeks for v0 plus four weeks of beta. Anything faster is missing eval discipline. Anything slower is overengineering.