120万美元的SRE自动扩缩容事故：为何自回归Agent在基础设施写边界上失败

DevOps AWS Software Engineering Artificial Intelligence System Design Concepts The $1.2M SRE Autoscaling Disaster: Why Autoregressive Agents Fail at Infrastructure Write Boundaries How an autonomous agent mistook a PostgreSQL lock for a traffic surge, provisioned 400 GPU instances across 3 AWS regions, and why infrastructure control planes require deterministic circuit breakers. Maya Lin 7 min read · Just now -- Listen Share Press enter or click to view image in full size Modern platform engineering teams are rapidly adopting agentic workflows to handle tier-1 site reliability tasks: triaging alerts, summarizing stack traces, and triggering remediation runbooks. The prevailing design pattern couples a reasoning agent (typically a fine-tuned LLM or a ReAct loop) directly to cloud infrastructure orchestration APIs — such as Kubernetes horizontal pod autoscalers (HPA), Terraform cloud runners, or the AWS SDK/Boto3 client. If the model determines that system latency exceeds predefined SLOs, it is authorized to call programmatic infrastructure scaling endpoints to mitigate client-facing degradation. Over a single 48-hour weekend window, this architecture cost a venture-backed tech company $1,240,000 in un-budgeted compute. Here is the AST- and network-level post-mortem of how an unindexed database query fooled an LLM into an exponential provisioning loop, why system prompt constraints fail catastrophically during localized thread deadlocks, and how to engineer deterministic execution gateways to govern autonomous cloud operations. 1. Incident Breakdown: The Lock Contention Cascade The agent was configured as an autonomous SRE triage node with delegated IAM permissions to adjust cluster capacity across multi-region Elastic Kubernetes Service (EKS) clusters. The Trigger On Friday at 11:14 PM UTC, an un-optimized background migration initiated an exclusive table lock on a shared PostgreSQL primary. A critical microservice (svc-auth-router) attempted to acquire a connection from its local pool. The pool was exhausted within 400 milliseconds. The HTTP health check probe (/healthz) began timing out at the Kubernetes ingress layer, increasing average service latency from 45ms to 845ms. CPU utilization across the existing pods remained below 12%, but inbound HTTP request queues began backing up. The Agentic Misdiagnosis The SRE agent received a Prometheus webhook notifying it of elevated P99 latency and ingress 504 Gateway Timeouts. Operating under a standard ReAct paradigm, the agent queried recent telemetry. Because the CPU load was low but latency was spiking, the model’s in-context heuristic matched the pattern against an unexpected distributed traffic spike (DDoS or flash mob event) rather than thread-pool starvation. The model initiated a remediation sequence: { "action": "scale_node_group", "parameters": { "cluster_name": "prod-us-east-1", "nodegroup_name": "gpu-accelerated-inference-nodes", "desired_capacity_delta": 50, "instance_type": "p4d.24xlarge", "region": "us-east-1" } } The model selected high-throughput, GPU-accelerated compute instances (p4d.24xlarge) because its system prompt explicitly prioritized "minimizing request queue latency for high-throughput enterprise workloads." The Compounding Autoscaling Loop Zero Impact on Root Cause: The new nodes joined the cluster, and Kubernetes scheduled 150 new replicas of svc-auth-router. However, every single new pod immediately attempted to connect to the deadlocked PostgreSQL instance, further saturating connection limits. Escalation Loop: In the next monitoring tick (5 minutes later), P99 latency had worsened from 845ms to 1,920ms. The agent determined that the initial capacity increase was insufficient. Multi-Region Burst: It repeated the tool call across three failover regions (us-east-1, us-west-2, eu-west-1), scaling on-demand instances to maximum provision quotas. The Spend Acceleration: Over 400 on-demand GPU instances were provisioned and kept running in un-converged boot/drain cycles for 46 hours until on-call engineers intervened on Sunday evening. Every single AWS API call returned HTTP 200 OK. Every Kubernetes manifest parsed without a syntax error. The infrastructure broke because the reasoning layer lacked a deterministic boundary between diagnosis and API execution. 2. The Theoretical Failure Mode: Why System Prompts Cannot Protect Cloud Budgets When engineering leads review autonomous agent incidents, their initial reaction is almost invariably prompt hardening: You are a Principal Site Reliability Engineer. CRITICAL OPERATIONAL RULES: 1. Always maintain cluster spend below reasonable budget parameters. 2. Verify root causes before scaling out compute infrastructure. 3. Never provision more than 10 additional nodes without human sign-off. 4. If scaling does not reduce P99 latency within 5 minutes, rollback immediately. This intervention fails because it attempts to solve an asynchronous systems control problem with synchronous autoregressive token prediction. Asynchronous Latency vs. The Token Generation Context An LLM context window is stateless between execution ticks unless explicitly fed current telemetry. However, cloud billing telemetry is inherently asynchronous: AWS Cost Explorer and CloudWatch billing metrics often lag real-time resource allocation by 4 to 24 hours. Get Maya Lin’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in When the agent evaluates its next token prediction: The model has direct visibility into real-time latency (which is spiking), but zero real-time visibility into instantaneous burn rate. In the distribution of SRE incident documentation, restoring uptime is overwhelmingly prioritized over cost reduction. As a result, the token probability for “remediate and scale” dominates the token probability for “throttle and fail.” The Prompt Disappearance at the Network Edge A system prompt lives in the inference memory of the LLM. It does not exist inside the outbound HTTP request. Once the model generates a valid JSON payload targeting an AWS STS-authenticated endpoint, the underlying cloud API does not evaluate whether the request is “reasonable” or “cost-effective.” The AWS control plane simply validates that the calling IAM role has the ec2:RunInstances or eks:UpdateNodegroupConfig permission and executes the command. Prompts cannot act as network firewalls. 3. The Architectural Solution: Deterministic Infrastructure Gateways To safely deploy autonomous agents with infrastructure remediation capabilities, the model must never hold direct write credentials to the cloud control plane. All agent-generated actions must be intercepted by a deterministic gateway layer that evaluates system-level invariants before any API call is serialized and dispatched. ┌─────────────────────────────────────────────────────────────┐ │ Autonomous SRE Agent │ │ (Diagnostic & Plan Generation) │ └──────────────────────────────┬──────────────────────────────┘ │ HTTP POST (Proposed Infrastructure Delta) ▼ ┌─────────────────────────────────────────────────────────────┐ │ Deterministic Cloud Governance Gateway │ │ │ │ 1. Telemetry Invariant Cross-Verification │ │ - If P99 Latency > Threshold AND CPU Utilization Flag Database Contention (Scale Lockout) │ │ │ │ 2. Spend-Velocity Derivative Circuit Breaker │ │ - Reject if: d(EstimatedCost)/dt > $2,000 / hour │ │ - Reject if: Delta(InstanceCount) > 10% Cluster Size │ │ │ │ 3. Ephemeral Cryptographic Execution Tokens │ │ - Mint short-lived IAM credentials ONLY upon passing │ │ deterministic invariant checks │ └──────────────────────────────┬──────────────────────────────┘ │ Authorized API Payload Only ▼ ┌─────────────────────────────────────────────────────────────┐ │ Cloud Infrastructure Surface │ │ (AWS / EKS / Terraform API) │ └─────────────────────────────────────────────────────────────┘ Step 1: Telemetry Invariant Verification The gateway intercepts the proposed tool payload and queries live APM metrics independently of the agent’s context window. It runs programmatic invariant checks to determine whether horizontal scaling is a valid remediation path for the observed failure mode: def validate_scaling_preconditions(cluster_id, nodegroup_id, proposed_delta): # Fetch real-time out-of-band metrics directly from Prometheus cpu_utilization = get_metric_average(cluster_id, "node_cpu_utilization_percent", window="5m") db_lock_count = get_metric_average(cluster_id, "pg_stat_activity_exclusive_locks", window="5m") # Invariant: Never scale compute if the bottleneck is lock contention if db_lock_count > 0 and cpu_utilization < 20.0: raise CircuitBreakerViolation( "Scaling blocked: Low CPU utilization accompanied by database lock contention. " "Horizontal compute scaling will not resolve downstream connection starvation." ) return True If the invariant fails, the gateway drops the payload, halts the agent ReAct loop, and dispatches a prioritized PagerDuty notification to human engineers with the exact metric mismatch. Step 2: Spend-Velocity Circuit Breakers The gateway evaluates the second derivative of infrastructure cost: the rate of change of spend velocity. Rather than relying on static budget thresholds that get checked once per day, the gateway calculates the financial impact of the requested delta over a rolling 72-hour window: If d(Cost)/ dt exceeds an empirical standard deviation from the 30-day baseline (e.g., more than $500/hour of sudden infrastructure change), the gateway trips a hard circuit breaker. The transaction requires out-of-band human sign-off via Slack or SMS before the write key is authorized. Step 3: Zero-Trust Write Tokenization Under this model, the autonomous agent never possesses permanent IAM write credentials (such as an Administrator or PowerUser ARN). The agent only holds permissions to submit proposed state mutations to the gateway. If and only if the gateway validates all mathematical, operational, and budget invariants, it generates a single-use, cryptographically signed token that executes the requested infrastructure adjustment via a sandboxed runner. The Production Reality Autoregressive models are exceptional tools for diagnosing complex stack traces, parsing log dumps, and proposing infrastructure fixes. But letting an LLM write directly to cloud APIs based purely on prompt-based instructions is architectural negligence. In production infrastructure, the model should propose; deterministic software must gate. Until your cloud control plane incorporates out-of-band circuit breakers and invariant solvers, keep your AI agents out of your root account. Published by Maya Lin On the team at Claire: AI Orchestration & Digital Labor By The Algorithm