How AI Tools Are Made (The Honest Stack)

Most AI tools you use are surprisingly thin. Once you understand the layers, you can evaluate any tool in five minutes.

The Stack

1. Foundation model — the LLM provider (Anthropic, OpenAI). Almost no AI tool trains its own model.
2. Prompt layer — system prompt, few-shot examples, output formatting. The product's secret sauce, kind of.
3. Retrieval / context layer — for tools that ground answers in customer data: vector DB, document chunking, retrieval logic.
4. Tool-use layer — for agents: the functions the model can call (search, query, post).
5. UX layer — chat UI, sidebar, in-app embed. Where the product actually lives for the user.
6. Eval layer — quality test suite. The mature tools have one. The new tools mostly do not.
7. Observability — logging, cost tracking, debug tools. Internal-only.

What Vendors Hide

• Most "our AI" pitches mean a thin wrapper around GPT-4 or Claude.
• The real moat is rarely the model. It is data, distribution, or workflow integration.
• The system prompt is often the only meaningful IP. A 200-token system prompt is the difference between two competing tools.

How to Evaluate a Tool's Stack

• Whose model? Anthropic, OpenAI, or self-hosted. Self-hosted is rare and usually worse unless they have a domain-specific reason.
• Where is the prompt? Configurable by you, fixed by them, or hidden? Configurable is better for power users.
• Is there RAG? If the tool answers using your data, ask how chunking and retrieval work. Bad RAG is a leaky abstraction.
• Are evals public? Few are; some publish accuracy on benchmarks, more should.

Build vs Buy

Looking at the stack, you can build a basic AI tool in a week. Most teams should still buy. The buying logic: a good vendor maintains the eval suite, the latency, the cost optimization, and the model upgrades. That is the work.

Build only if your prompt or retrieval needs are domain-specific in ways no vendor maps to. Even then, build on the model APIs, not your own model.

What This Means for Picking Tools

• The tool with the better UX usually wins, not the one with the bigger model.
• The tool with the eval discipline beats the tool that ships fast and breaks quality.
• The tool that gives you the prompt is more flexible than the tool that hides it.

What to Do Next

Next time you evaluate an AI tool, ask: which model, what prompts, what data goes in, where do logs live, what is the eval story? Five questions. The answers tell you whether the tool will hold up at scale.