Your agent can build anything. Is it building the right thing?
With a coding agent, the first version takes an afternoon. The agent won't tell you whether you picked the right problem, whether your evals measure anything the business cares about, or when to stop tuning and change the plan. Those questions decide whether a project reaches production. I learned them across more than a hundred AI use cases, and this book is the method.
The framing layer above evals, frameworks, and architecture
Evals are the inner loop: did the model produce a good output. The outer loop, the harder one, is whether you framed the right problem at all. Most teams have a great inner loop and no outer loop. This book is the outer loop, drawn from more than a hundred AI use cases across traditional ML, generative AI, and agents.
- Chapter 1First Principles. Why projects fail, and the thinking habits that catch it early.
- Chapter 2Framing the Agent Task. The GOATS Loop end to end, on one project.
- Chapter 3Knowing If It's Working. Diagnostic tests for a system that's already running.
- Chapter 4The Decision Point. When to persist, pivot, or stop.
- Part 2Diagnostics in Depth. Separate sections for ML, GenAI, and agents.
- AppendixThe four artifacts. The one-page documents the method produces, filled in.
Frame it, diagnose it, then decide
The same questions work for a churn model, a RAG bot, and an autonomous agent. What changes from one to the next is the tactics inside each stage.
The GOATS Loop
Five questions to answer in order before you build: the goal, what has to be true, what else could work, how you'll choose, and how you'll know you're wrong.
Three questions, in order
Is there signal? Is the system cheating? Is the metric real? Each answer traces back to the assumption that broke, which tells you whether to tune the model or change the frame.
Persist, pivot, or stop
Make the call against a stop criterion you wrote down before launch. When the answer is pivot, five reframing moves cover most of what teams actually change.
Print these for your next kickoff
Short, printable versions of the method. No email wall.
Run the method on your own project
I turned the book into two free skills for Claude Code and other coding agents. frame-use-case interviews you before you build. diagnose-use-case works on a system that's struggling, and it is allowed to tell you to stop.
Watch it before you read it
Three short talks that cover the whole arc, from the framing questions to what your evals are missing.
Before You Build an AI Project, Answer These 3 QuestionsTest the decision, the approach, and the evidence before you commit to the build.
Your AI Evals Passed. Your Project Can Still Fail.Good model metrics don't tell you whether the project is helping users.
ChatGPT Loves Your AI Idea. That's the Problem.The same project run through ChatGPT and through a framing skill built to ask what the chatbot skips.Bring a real project
The book teaches the questions. Getting good at answering them takes practice on a project with real stakes, with someone pushing back.
AI Problem Framing
You bring a project and run it through each stage in live sessions, with recorded lessons, worksheets, and a database of 200+ AI case studies. Students rate it 4.9 on Maven.
Team workshops
One team, its own projects, run privately so people can put confidential details on the table. The team leaves with a shared process it reuses on every project after.
“We are all becoming AI managers, which means our decision-making skills are our most valuable (and marketable) skills. This course hones those within the context of applied AI.”
Chad Harness, Head of Technical Enablement, Snorkel AI