规模化下的工程债：生产级 AI 系统中的三类结构性故障

New Story Engineering Debt at Scale: Three Structural Failures in Production AI Systems by Brindal P September 18th, 2026 TLDR Your browser does not support the audio element. byBrindal P@b101010 byBrindal P|@b101010 Senior Software Engineer and Senior Architect Story's Credibility Most of what breaks AI systems in production has nothing to do with the model. I spend a good chunk of my job reviewing code: production systems and open-source contributions as an engineer, plus the code behind systems and software papers as an academic peer reviewer. No matter the context, the pattern is the same: state-of-the-art math wrapped in software that can't survive a Tuesday afternoon of real traffic. Basic engineering discipline seems to evaporate the moment import torch shows up. Based on hundreds of these reviews, the failures cluster into three recurring architectural flavors. Here's what they look like, why they happen, and the fixes that actually hold up under load. Flavor Root Cause What You'll See in Prod Notebook-Driven State Global model/cache objects, undetached tensors OOM crashes after N requests Happy-Path Networking No timeouts, no circuit breakers, synchronized retries Thread exhaustion, cascading outages Dependency Anarchy Unpinned transitive deps, implicit CUDA coupling Clean installs that break the moment someone else runs them 1. Notebook-Driven State (Memory Bleed) In a notebook, global state is a feature: load a 14 GB checkpoint into cell two, keep it resident, mutate variables downstream forever. Port that habit into a long-running FastAPI worker or Celery consumer and it becomes a liability. Those processes are expected to survive days, not a Restart Kernel. MODEL = None METRICS_HISTORY = [] # unbounded growth def predict(x): global MODEL if MODEL is None: MODEL = torch.load("weights.pt").to(DEVICE) out = MODEL(x) METRICS_HISTORY.append(out) # keeps the full autograd graph alive return out Every tensor a forward pass produces carries its grad_fn unless you strip it. Appending that tensor to a global list doesn't store numbers. It pins the entire computation graph in memory, and a few hundred requests later CUDA throws an OOM and takes the worker down with it. Treat models and caches as scoped, injected resources, never module-level singletons. Constructor injection for models, devices, and allocators: nothing hides in module globals @torch.inference_mode() (or .detach()) on anything you retain past the request Bounded caches: an LRU ring buffer, never an unbounded list or dict You can see this same discipline win out at the infrastructure layer, too. PagedAttention became the default architecture for open-source LLM serving because vLLM stopped pre-allocating memory for worst-case context length and started paging the KV cache the way an OS pages RAM. It's the same scoped-allocation idea above, just enforced one layer down the stack. 2. The Happy-Path Network Trap Modern AI systems are never isolated: vector databases, model routers, remote inference endpoints. A shocking number of repositories still assume the network is local, cheap, and infinite. def fetch_embeddings(texts): r = requests.post(VECTOR_DB_URL, json={"texts": texts}) # no timeout return r.json()["vectors"] One hung connection is a shrug. Multiply it across 32 worker threads and your thread pool is exhausted within seconds. Naive for attempt in range(3): retries without jitter make it worse: when the upstream service recovers, every queued client fires at the same instant and knocks it straight back offline. The failure boundary matters as much as the retry logic. Earlier this year vLLM patched an advisory (CVE-2026-54236) after unsanitized error messages in its API router were found leaking raw memory addresses back to clients. It's a textbook case of an exception bubbling past a service boundary instead of failing safely. The same blind spot shows up in unattended retry logic: an agent stuck in a loop overnight doesn't crash, it just quietly runs up a bill until someone happens to check the dashboard the next morning. Nothing trips, because nothing is watching. A network call without a timeout is a bet you didn't know you were making. Explicit connect/read/write timeouts on every call, no exceptions Pooled, persistent clients (httpx.Client, urllib3.PoolManager) instead of a new connection per request Circuit breakers that fail fast and return a cached or degraded response Exponential backoff with jitter, so retries scatter instead of synchronizing 3. Dependency Anarchy & Non-Deterministic Builds You pull down a repo that topped the benchmarks six months ago, and it will not build. The Python packaging ecosystem is already fragile; layering in CUDA toolkits, native C++ bindings, and platform-specific wheels turns it into a minefield. The usual sins: a flat requirements.txt with bare torch or transformers, no locked transitive tree, and an assumption that everyone's host runs the exact CUDA minor version the wheels were compiled against. One silent numpy or pydantic point release later, serialization formats shift and the repo is bricked for the next person who tries to run it. A repo you can't rebuild the same way twice isn't really done. Commit a hashed, cross-platform lockfile (uv.lock, poetry.lock), not just a pyproject.toml Run CI across a real Python × CUDA/architecture matrix, not just ubuntu-latest Degrade gracefully on unmapped hardware (custom kernel → Triton → CPU) instead of segfaulting uv has become the default choice here for good reason: it resolves and locks a universal, hashed dependency tree far faster than pip ever did. But it doesn't own the non-Python binary layer, which is exactly why teams shipping heavy CUDA dependencies still lean on conda or explicit driver-matrixed CI alongside it, not instead of it. One Function, Three Fixes Here's how all three flavors show up in a single retrieval-augmented inference handler, and the refactor that fixes it. Before: the antipattern # anti_pattern_inference.py import requests import torch MODEL = None GLOBAL_CACHE = {} def process_and_infer(batch_texts): global MODEL if MODEL is None: MODEL = torch.load("model.pt").cuda() # hardcoded CUDA, crashes without GPU res = requests.post("https://api.internal/embed", json={"texts": batch_texts}) # no timeout embeddings = res.json()["embeddings"] tensors = torch.tensor(embeddings).cuda() predictions = MODEL(tensors) for text, pred in zip(batch_texts, predictions): GLOBAL_CACHE[text] = pred # retains the full computation graph return predictions After: production architecture # robust_inference.py import logging, random, time from collections import OrderedDict from typing import Dict, List, Optional import httpx import torch logger = logging.getLogger(__name__) class ResilientInferenceEngine: def __init__( self, model: torch.nn.Module, device: torch.device, embed_endpoint: str, http_client: Optional[httpx.Client] = None, max_cache_size: int = 10_000, ): self.device = device self.model = model.to(device).eval() self.embed_endpoint = embed_endpoint self.max_cache_size = max_cache_size self._cache: "OrderedDict[str, List[float]]" = OrderedDict() self._client = http_client or httpx.Client( timeout=httpx.Timeout(connect=2.0, read=5.0, write=5.0, pool=10.0) ) def _fetch_embeddings(self, texts: List[str], max_retries: int = 3) -> List[List[float]]: for attempt in range(max_retries): try: r = self._client.post(self.embed_endpoint, json={"texts": texts}) r.raise_for_status() return r.json()["embeddings"] except (httpx.RequestError, httpx.HTTPStatusError) as exc: if attempt == max_retries - 1: logger.error("Embedding API failed after %d retries.", max_retries) raise sleep_time = (0.5 * (2 ** attempt)) + random.uniform(0.05, 0.2) logger.warning("Retrying in %.2fs after %s", sleep_time, exc) time.sleep(sleep_time) def _cache_get(self, text: str) -> Optional[List[float]]: if text not in self._cache: return None self._cache.move_to_end(text) # touch: mark as most recently used return self._cache[text] def _cache_put(self, text: str, vector: List[float]) -> None: if text in self._cache: self._cache.move_to_end(text) elif len(self._cache) >= self.max_cache_size: self._cache.popitem(last=False) # evict the true least-recently-used entry self._cache[text] = vector def infer(self, batch_texts: List[str]) -> torch.Tensor: results: Dict[int, List[float]] = {} miss_positions, misses = [], [] for i, text in enumerate(batch_texts): cached = self._cache_get(text) if cached is not None: results[i] = cached else: miss_positions.append(i) misses.append(text) if misses: raw = self._fetch_embeddings(misses) with torch.inference_mode(): tensors = torch.tensor(raw, dtype=torch.float32, device=self.device) computed = self.model(tensors).detach().cpu().tolist() for pos, text, vector in zip(miss_positions, misses, computed): self._cache_put(text, vector) results[pos] = vector return torch.tensor([results[i] for i in range(len(batch_texts))], dtype=torch.float32) def close(self): self._client.close() None of this changes what the function does, only how it survives contact with production. The model now comes in through the constructor instead of hiding in a global. The HTTP call runs through a pooled client with real timeouts and jittered backoff instead of a bare requests.post. The cache is actually consulted before any work happens, a repeated text skips the network call and the forward pass entirely, and eviction removes the genuinely least-recently-used entry instead of just whatever was inserted first. inference_mode() stays scoped tightly around the forward pass, so the tensor infer() hands back is a plain, ordinary tensor a caller can safely slice or mutate, not one still tagged by an active inference context. Same functionality, fewer reasons for anyone to get paged at 3 a.m. Production Readiness Checklist Checkpoint Target State Failure Indicator Memory isolation Forward passes wrapped in @torch.inference_mode(); zero graph allocation in monitoring hooks Process memory grows monotonically over a 1,000-iteration stress loop Deterministic lifecycles Resources passed via constructors; explicit close() or context management Reliance on __del__ or module-level singletons for GPU contexts Failure boundaries Every external call has bounded pools, hard timeouts, jittered retries An unreachable endpoint hangs the client process indefinitely Reproducible isolation pyproject.toml paired with a hashed, platform-pinned lockfile Fresh install in a clean environment breaks on a sub-dependency bump Device agnosticism Explicit device targeting with a working CPU fallback .cuda() called directly without checking driver availability Architecture Is the Multiplier A breakthrough model wrapped in a leaking, unguarded script is still just an experiment: promising in a demo, unusable in production. None of this asks teams to slow down. It asks that model code get the same engineering hygiene we've expected from databases and network protocols for years. Treat memory as finite, design for the network to fail, and make your build reproducible, and the payoff shows up less in adoption metrics and more in who doesn't get paged at 3 a.m.. ← Previous Designing Reliable LLM Agents With Deterministic Control Flow About Author byBrindal P|@b101010 Senior Software Engineer and Senior Architect Read my storiesAbout @b101010 TOPICS Software Engineering #artificial-intelligence#ai#software-architecture#code-review#machinelearning#production-ai-systems#production#cloud-computing