部署代理式AI前，对合同数据进行四层审计

A Four-Layer Audit for Contract Data Before You Deploy Agentic AI by Eshaan Jain September 3rd, 2026 TLDR Your browser does not support the audio element. byEshaan Jain@eshaanjain26 byEshaan Jain|@eshaanjain26Senior Product Manager at T-Mobile US Eshaan Jain is Lead Product Owner for Salesforce / Vlocity CPQ at Mphasis, working on the T-Mobile account in Bellevue, WA. Story's Credibility Most enterprises audit the model. Almost no one audits the contract data underneath it. Ninety-five percent of organizations investing in generative AI are getting zero return on it, according to MIT NANDA's July 2025 report on the state of AI in business, which tracked $30 to 40 billion in enterprise GenAI spend. I've spent roughly 15 years building on Salesforce CPQ and CLM, first at Amazon on a clause-extraction system, now at T-Mobile leading a CPQ and CLM transformation through Mphasis. In that time, I've watched a lot of agent deployments get evaluated on the agent and almost never on the data underneath it. Everyone audits the model. Nobody audits the contract. A few numbers worth holding onto before the four layers: MIT NANDA, July 2025: 95% of organizations get zero return on $30 to 40 billion in enterprise GenAI investment. Gartner, February 2025: 63% of organizations lack the data management practices AI needs, and Gartner predicts 60% of AI projects will be abandoned by the end of 2026 without them. CUAD benchmark, NeurIPS 2021: the best clause-extraction model tested against 510 real contracts hit 47.8% AUPR, dropping to 17.8% precision at a 90% recall target. World Commerce & Contracting and Deloitte: contract value erosion sits at 8.6% on average, barely down from 9.2% a decade earlier, with contract data spread across roughly two dozen systems. That's backward. Quote-to-cash runs on contract and clause data, and that data is some of the messiest structured-adjacent information any enterprise holds: negotiated redlines, template drift across business units, clause language that means the same thing three different ways depending on who wrote it. An agent built on top of that inherits the mess. It just inherits it faster and with more confidence than a human would. Here's a four-layer audit I use before recommending an agentic system for quote-to-cash or contract workflows. It answers the one question that predicts whether the deployment works: Is the data ready? Layer 1: Clause taxonomy coverage Before you extract anything, you need a consistent way to name what you're extracting. Most legal teams don't have one. Contracts get filed by deal, not by clause type, and "limitation of liability" in one template is "liability cap" in another. The best public benchmark for this problem is CUAD, a 2021 NeurIPS dataset built from 510 contracts and over 13,000 expert annotations across 41 clause categories. The best model tested against it, DeBERTa-xlarge, hit an AUPR of 47.8%. At a precision target of 90% recall, precision dropped to 17.8%. That's not a criticism of the model. It's a measurement of how hard clause extraction is even with a clean, purpose-built taxonomy and expert labels. If your organization doesn't have a taxonomy at all, you're starting from a worse position than the benchmark. Score this layer honestly: do you have a documented, versioned clause taxonomy that every contract template maps to? If the answer is "we're working on it," the agent isn't ready, no matter how good the model is. Layer 2: Structural consistency Taxonomy tells you what to look for. Structure tells you whether you can find it reliably. At Amazon, I helped co-build a clause-extraction system for a supply-chain contract portfolio worth more than $40 billion annually. We got it to 95% accuracy, but not by pointing a model at raw PDFs. Most of the accuracy gain came before the model ever ran: normalizing formats, flagging which templates were still in active negotiation versus locked, and building a confidence score for every document based on how far it had drifted from its source template. That's the layer most teams skip. They measure model accuracy on a clean validation set and assume production documents will look similar. They don't. A contract that's been through four rounds of redlines with outside counsel no longer structurally resembles the template it started as. Score this layer by sampling 50 to 100 live contracts and checking what percentage still match their source template within a defined tolerance. If it's under half, the model's accuracy numbers from your pilot won't hold in production. Layer 3: Source-of-truth fragmentation Research from World Commerce & Contracting and Deloitte shows that the average organization's contract value erosion is 8.6%, down only 0.6 percentage points from 9.2% a decade earlier, despite a decade of investment in contract lifecycle management software. The same research found contract data spread across roughly two dozen systems in a typical organization. Twenty-four systems means 24 places a clause can exist in a slightly different version. An agent that queries one of them and treats it as truth is going to be confidently wrong some percentage of the time, and you won't know what percentage until it's making decisions at scale. Score this layer by counting your actual systems of record for contract data: CLM, CRM, shared drives, email attachments, deal desk spreadsheets. If there's more than one place a live contract can be the "real" version, that's the gap an agent will fall into first. Layer 4: Outcome traceability The last layer is whether you can trace a contract's terms forward to what actually happened. Gartner's October 2025 survey of 104 general counsel found only 9% of legal departments prioritized advanced contract analytics, and 37% reported low confidence using the analytics tools they already had. Most legal and revenue teams can tell you what a contract says. Far fewer can tell you whether the deal performed the way the contract predicted. This layer is the hardest to fix and the one that matters most for agentic systems, because it's the feedback loop. Without it, an agent has no way to learn which extracted terms were correct and which ones led to a dispute, a renegotiation, or a write-off. You're running the model blind on the one signal that would make it better. At T-Mobile, the CPQ and CLM work I lead has cut contract creation time in half. That number came from fixing layers 1 through 3 first: a shared clause taxonomy, structural normalization at intake, and a single system of record for active quotes. Layer 4 is still under construction. The scoring rubric Score each layer 0 to 3. Zero means the layer doesn't exist. One means it exists informally, undocumented, inconsistent across teams. Two means it's documented and applied to new contracts but not retrofitted to the existing portfolio. Three means it's documented, applied retroactively, and measured on a rolling basis. A total score under 6 out of 12 means an agentic deployment will be evaluated on a false pilot: clean, hand-picked data for the demo, in a production environment that looks nothing like it. In the deployments I've seen up close, that gap explains most of the MIT NANDA number. The model gets credit when a pilot works and blame when production fails, and the difference between the two is almost always the data underneath it. Run this audit before the first prompt gets written. Most teams can score all four layers against 50 to 100 live contracts in a week. Finding out mid-pilot that the taxonomy doesn't exist, or that the same clause lives in 24 different systems, costs a lot more than that week does. ← Previous Token Detox for the Enterprise: Cutting Wasteful Inference in Workflows Already in Production Up Next → How to Run a Sandboxed LLM in a School Lab With No Cloud Bill About Author byEshaan Jain|@eshaanjain26 Senior Product Manager at T-Mobile US Eshaan Jain is Lead Product Owner for Salesforce / Vlocity CPQ at Mphasis, working on the T-Mobile account in Bellevue, WA. Read my storiesLearn More TOPICS ai-and-ml #contract-management#enterprise-ai#ai#cpq#data-quality#agentic-ai#quote-to-order-system#ai-model-audit