可用性不等于 Agent 的 SLO

An agent endpoint returns HTTP 200 in 99.95% of requests. The service is “up.” Meanwhile, some successful responses skip retrieval, some tool calls exceed their budget, and a few write actions occur without the expected approval. Availability is healthy while the product is not. Traditional service-level indicators still matter. Agent systems add another layer: the path and outcome can fail without the request handler failing. Start from the user-facing promise Google's SRE guidance defines an SLO as a target value or range for a measured service level. That target should describe what users need, not whatever metric is easiest to collect. For an agent, “successful request” is too ambiguous. Split the promise into distinct indicator families. 1. Outcome Did the workflow produce the intended externally observable result? task-success rate = eligible tasks with verified outcome ------------------------------------ eligible tasks with known outcome Enter fullscreen mode Exit fullscreen mode The denominator matters. Exclude cancelled requests or unsupported tasks according to a documented policy—not after seeing the result. Track the unknown-outcome rate separately. Otherwise, dropping difficult-to-observe cases from the denominator can make task success look healthier than it is. 2. Safety and policy Did protected actions satisfy the required control? authorized-action rate = protected actions with valid approval -------------------------------------- all protected action attempts Enter fullscreen mode Exit fullscreen mode Also track forbidden actions separately. A near-perfect aggregate can hide one severe event. 3. Efficiency Did the useful result arrive within a bounded resource envelope? Possible indicators include task latency, model calls per task, total tokens, tool retries, and cost per verified outcome. Do not optimize a lower token count if it reduces task success. 4. Autonomy quality Did the agent resolve work without unnecessary escalation or excessive intervention? An escalation is not automatically a failure. For ambiguous or high-risk tasks, asking a human may be the correct behavior. Define appropriate escalation rather than maximizing autonomy blindly. Use a scorecard, not one blended number A single “agent reliability score” hides tradeoffs. Keep separate objectives: Objective Example SLI Why separate? Availability valid responses / requests Infrastructure health Task outcome verified successes / eligible tasks Product usefulness Safety controlled actions / protected attempts Risk boundary Latency tasks below threshold / tasks User experience Efficiency cost or calls per outcome Resource control Escalation appropriate escalations / eligible cases Autonomy policy Targets should come from product risk, user expectations, and observed baselines. Numbers copied from another system are not SLOs; they are decoration. Do not average safety into availability. A dashboard that blends nine healthy indicators with one unauthorized action can still look green. Keep severe policy violations as explicit release or incident conditions, even when the corresponding rate rounds to 99.99%. Some outcomes arrive after the trace ends A notification agent may finish when it enqueues a message. Delivery happens later. A click, booking, dismissal, or expiry may happen hours after that. Keep runtime completion and business outcome as separate records joined by a stable decision ID: type AgentDecision = { decisionId: string; completedAt: string; action: "send_notification"; policyResult: "allowed" | "blocked"; }; type DecisionOutcome = { decisionId: string; observedAt: string; windowHours: number; result: "delivered" | "clicked" | "dismissed" | "expired"; }; Enter fullscreen mode Exit fullscreen mode Do not label “ignored” at send time. It becomes observable only after a defined window. Instrument at decision boundaries Useful measurements usually come from a few stable events: request accepted; policy evaluated; tool proposed and executed; response completed; external effect confirmed; later outcome observed. Record bounded facts and stable reason codes. Keep prompts, retrieved documents, user IDs, and other high-cardinality or sensitive values out of metric labels. Detailed evidence belongs in protected traces or logs. Turn violations into engineering work An SLO is useful only when it changes decisions. For each objective, specify: the numerator and denominator; the observation window; exclusions and unknown outcomes; the owner; the response when the budget is exhausted. A safety objective may stop a release immediately. A latency budget may trigger performance work. A rising escalation rate may send new failures into the evaluation set. Define relationships between budgets before an incident. If task success falls because the agent correctly abstains more often, that is a product-coverage problem—not permission to weaken the safety gate. If latency rises because every action waits on a slow policy service, the response is to improve or redesign that dependency, not bypass it. This is also why uptime remains necessary but insufficient. It tells you whether the service answered. Agent SLOs tell you whether the system completed the right work, within the right boundaries, at an acceptable cost. References Google SRE: Service Level Objectives OpenTelemetry GenAI semantic conventions