Problem Interviews Without Overengineering (AI SaaS)

AI SaaS teams have an extra temptation: build evaluation infrastructure before talking to anyone. Eval pipelines, scoring rubrics, golden datasets, A/B harnesses. All useful eventually. None of it earns its place before the first ten customer conversations. Here is the lightweight version.

Five Calls, Three Questions, One Notebook

Find five people who plausibly have the workflow you want to improve. Bring three open-ended questions about their actual past. Take notes in a single doc. Run another five next week. After ten you will see patterns. After fifteen you can summarize the customer in three sentences.

That is enough to stop interviewing and ship the first version of whatever AI integration you are considering. No eval suite required to make this decision.

What You Can Skip

Skip the evaluation framework. Skip the model bake-off comparison spreadsheet. Skip the prompt versioning system. Skip the synthetic test set. Skip the latency dashboard. Skip the cost-per-token tracker. None of those are bad. They are infrastructure for teams shipping AI products to thousands of users. You have ten interviewees.

You can also skip the AI ideology survey. Do not ask people what they think about AI in general. Ask them about their workflow. Whether AI shows up is a fact you discover.

What You Cannot Skip

You cannot skip the calls. The reason eval frameworks exist is to verify AI behavior at scale, after you know what behavior is even worth verifying. The interview round is what tells you what to build evals for.

You also cannot skip writing things down. Even minimal notes - direct quotes, the trigger event, the workaround, the trust boundary - are non-negotiable.

The Three Questions for AI SaaS

Slightly tweaked from the generic version because AI workflows have specific texture worth capturing.

One: walk me through the last time you did X. What were you trying to do, what did you actually do, what did the output look like?

Two: have you ever tried an AI tool for this? What happened?

Three: where in this workflow do you check the output by hand? Why there?

Question three is the AI-SaaS-specific one. The trust boundary the customer drew is the design constraint your product has to respect.

The Engineering-Brain Trap

Technical AI SaaS founders often want to validate the model before validating the problem. They run benchmarks, compare GPT-4 to Claude to local models, build eval harnesses. None of that is wasted exactly, but it is sequenced wrong. The model question only matters once you know which workflow you are inserting into.

The ten customer interviews tell you the workflow. The model selection happens after. Doing it in the other order is a common, expensive mistake.

When Heavier Tooling Earns Its Place

Past about thirty users in beta, you will need eval infrastructure. By then you will know exactly what to evaluate, because the users will have told you. Build the evals against real failure cases the users surfaced, not against a synthetic test set you invented.

Same logic for prompt versioning, latency dashboards, and cost tracking. They earn their place when the product is in front of users producing failure modes. Premature is procrastination.

The Honest Trade

Two to four weeks of upfront interviews. About 25 to 30 hours total. You skip the eval-framework build for now. You ship the first version against a workflow you actually understand. You add infrastructure as the product earns it.

The alternative is building elaborate AI evaluation infrastructure for a product nobody has yet adopted. The math is bad. The lightweight version saves the work that does not yet need doing.