模型不是难点，harness 才是：构建 AI Agent 的工程经验

Product Management Agentic Ai Machine Learning Artificial Intelligence Towards Ai The Model Wasn’t the Hard Part. The Harness Was. What building AI agents taught me about loops, permissions, memory, evals, and the economics of autonomy Neelesh Tanwar 7 min read · Aug 6, 2025 -- Listen Share Press enter or click to view image in full size A graphical representation of a Horse being a model and harness being its Saddle, image generated by the author using ChatGPT For a while, most AI product conversations started with the same question: Which model should we use? GPT, Claude or Gemini? Frontier model or smaller model? Fine-tuning or prompting? Bigger context window or better retrieval? Those are legitimate questions. However, Once you move from an AI feature that answers something to an agent that is expected to do something, the center of gravity shifts. The model might decide that a customer record needs to be updated. Something else has to determine whether it is allowed to update it. (Guardrails) The model might decide to call an API. Something else has to authenticate that request, validate the parameters, handle the timeout and decide whether retrying is safe. (Identity & Authentication) The model might realize halfway through a task that its first approach was wrong. Something has to preserve enough state for it to recover instead of starting over. (Memory) That “something else” is increasingly called the AI harness. A useful simplification is: Agent = Model + Harness Databricks describes the harness as the infrastructure surrounding an LLM that provides tools, memory, execution environments and guardrails. At the center is a repeating cycle: the model reasons, the harness executes an action, the result is observed, and the model decides what to do next. As a Product Manager, I find the idea important for a different reason. The harness is where AI stops being a model problem and starts becoming a product problem. The Demo Usually Hides the Harness A typical agent demo looks wonderfully simple. “Find the issue, fix it and tell me when you are done.” The agent investigates, calls a few tools, corrects itself and returns the answer. What the user sees is intelligence. What the product team should see is an execution system. Behind that seemingly simple interaction, somebody had to decide: What tools can the agent use? Which data can it read? Which systems can it modify? How many times can it retry? What happens when a tool fails? What should it remember? When should it ask for human approval? How do we know that “done” actually means done? Press enter or click to view image in full size A graphical representation of a the Demo being trip of the Iceberg, image generated by the author using ChatGPT That last question turns out to be surprisingly important. Get Neelesh Tanwar’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in I have seen versions of the same problem in conventional products: a team optimizes the visible part of the experience while the real failure is happening one layer underneath. With agents, this effect becomes much stronger. A model can produce a perfectly reasonable plan and still create a terrible product experience because the surrounding execution system is weak. The model may not be the problem at all. The Harness Manages the Loop A chatbot gets one shot. An agent doesn’t. Ask it to investigate why a dataset failed, and it might reason, query logs, inspect the result, form a hypothesis, run another query, find that hypothesis wrong, try a different path, and only then declare the job done. Databricks describes this as a reason-act-observe cycle; production harnesses add tests and validators on top so the agent isn’t the only judge of its own success. This changes how I write requirements. For ordinary software I still describe the happy path, but for an agent I spend more time on the recovery path: can it tell a timed-out API from a permissions error, how many retries are safe before a retry becomes a duplicate transaction, and when does it hand the problem to a human instead of guessing again. Press enter or click to view image in full size A graphical representation of an Agentic loop, image generated by the author using ChatGPT I once worked on a multi-step process that failed often enough that the first instinct was to make the model smarter. Looking closer, the model’s decisions were usually reasonable — it just wasn’t getting enough structured feedback after something broke. Fixing the loop did more than fixing the reasoning would have. Memory Is a Product Decision, Not Just a Database Models don’t carry durable memory on their own — the harness decides what gets stored, retrieved, summarized, and injected back into context. “Give the agent memory” sounds like one requirement, but it’s really several: what’s worth remembering, for how long, whether it’s a user preference or just an assumption the agent generated about itself, and whether the user can see and correct it. The real risk isn’t too little memory, it’s the wrong kind. On a past project we assumed more retained context would make the system smarter, and sometimes it did the opposite — old history competed with what the user actually wanted right now. A remembered preference makes an agent better; a remembered false assumption makes it worse in a way that’s much harder to spot. So the requirement isn’t “remember previous interactions.” It’s what’s worth remembering, where it came from, how long it should live, and under what conditions it gets forgotten. The Next Problem Is Agents Talking to Agents One agent is complicated enough. Now give it coworkers — one that understands the objective, one that retrieves information, one that analyzes it, one that executes the action, one that verifies the result. Standards like A2A exist so agents built on different frameworks can discover each other’s capabilities without exposing their internals, which is a real technical achievement. Press enter or click to view image in full size A graphical representation of Agents talking to Agents, image generated by the author using ChatGPT But the handoffs are what interest me as a PM. If Agent A delegates to Agent B, who owns the final answer, what permissions travel with the request, and how does A confirm B actually finished rather than just reported that it did? Anyone who has run a cross-functional project has lived the human version of this: “can your team take care of this” sounds simple until three days later everyone discovers they interpreted “this” differently. Agents won’t fix that instinct — they’ll just execute it faster, which is why agent-to-agent systems need explicit contracts around capabilities, ownership, and what counts as done. Evals Become the Product’s Nervous System We usually use metrics after launch to ask whether the product is working. With agents, evals have to answer something more basic first: what does “working” even mean? An agent can reach the right answer through a dangerous path, cite the wrong source convincingly, or technically succeed while burning ten times the tool calls a cleaner run would need. Anthropic recommends judging the full trajectory — tool calls, intermediate state, not just the final output — which is why this is harder to evaluate than a single model response. Press enter or click to view image in full size A graphical representation of Evals, image generated by the author using ChatGPT I think Product should own the test cases even if engineering builds the infrastructure. A useful eval set covers the normal cases, the edge cases, the adversarial ones where tools return misleading information, whether the agent refuses actions it shouldn’t take, whether it can recover from a failed step, and whether it got there efficiently. Every real production failure should leave something behind in that suite — otherwise a model upgrade can look like progress while quietly breaking a workflow three steps downstream. OpenAI’s harness-engineering work points at the same shift: environments, automated tests, and feedback loops that let agents do more while humans still steer. The eval suite is what tells you how much of the model’s capability you can actually trust with real work. Agentic ROI Is Not Token Cost Eventually every AI product conversation reaches the same question: is this actually worth it? Measuring agents by inference cost alone is misleading. An agent that costs 20 cents a run but succeeds 60% of the time can be more expensive, in practice, than one that costs 50 cents and succeeds 95% of the time with almost no human correction — because the cheaper run still needs someone to catch and fix the other 40%. I’ve found it more useful to think in cost per accepted outcome rather than cost per token or even cost per run: business value created, minus execution cost, minus human review time, minus the cost of the failures that get through unnoticed. McKinsey makes a similar point — as agents get more complex, the completed task becomes the economically meaningful unit, not the token, because retries, verification, and different execution paths create real cost variance underneath a single price tag. Which raises the question worth asking before any of this: why is this an agent at all? If a deterministic workflow solves it reliably, use the workflow. If one model call solves it, stop there. Anthropic makes the same argument — add agentic complexity only when it demonstrably changes the outcome. Autonomy should have to earn its complexity, not just its budget. The Harness Is Where the Product Actually Lives The industry spent a few years teaching us to obsess over the brain — parameters, benchmarks, reasoning, context windows. Those still matter. But the next generation of AI products will be differentiated by what surrounds the brain: whether the system can recover, whether it knows who it’s acting for, whether it holds the minimum permissions it needs, whether it can remember what’s useful without accumulating garbage, and whether we can actually observe and evaluate what it did before and after launch. That’s harness engineering, and despite the technical name, most of its hardest questions are product questions. I’d sum up the PM’s job in an agentic product this way: define how much freedom the system gets, build the feedback loops that let it earn more, and make sure every additional unit of autonomy creates more value than risk. The model provides the intelligence. The harness decides whether we can trust it with the work.