如何衡量运行规范与其执行结果之间的一致性

In my previous article, I explored executor independence: the idea that different execution mechanisms should be able to satisfy the same operational specification. By executor independence, I mean that the operational specification remains stable even when the mechanism performing the work changes. That gives us a useful architecture: Specification ↓ Executor ↓ Evidence But it leaves an important question unanswered: how do we decide whether the evidence produced by an execution actually satisfies the specification? That's a conformance problem. A deployment can finish successfully from the executor's point of view and still violate the operational contract. A remediation agent can complete every action it planned and still leave the system outside acceptable limits. A restore operation can finish without errors and still fail to restore the required data. This is why a generic status like: success = true isn't enough. We need a way to compare what should have happened with what was actually observed and produce a result that's explicit, explainable, and useful. In this article, I'll show you how to model operational conformance, evaluate individual constraints, distinguish pass, fail, and unknown outcomes, handle tolerances, work with missing evidence, assign severity, calculate a useful conformance summary, and avoid turning conformance into a misleading score. The examples use TypeScript and a deployment scenario, but the same approach applies to infrastructure changes, incident response, backup restoration, data pipelines, and AI-operated systems. The central idea is this: conformance isn't whether the executor says it succeeded. Conformance is whether observed evidence satisfies the operational specification. Prerequisites You should be comfortable with: key software architecture concepts TypeScript or a similar language basic testing observability CI/CD concepts operational specifications executor independence You don't need a particular deployment platform, and the examples are deliberately small so the evaluation model stays visible. Table of Contents Prerequisites What Conformance Means in This Context Start with a Specification and Evidence Evaluate Constraints Individually Don't Reduce Everything to Pass or Fail Treat Missing Evidence as Unknown Use Tolerances Where Exact Comparison Is Wrong Add Severity to Constraints Separate Hard Constraints from Advisory Constraints Build a Reusable Conformance Evaluator Calculate a Conformance Summary Carefully Why a Single Score Can Be Dangerous Preserve the Evidence Behind the Verdict Conformance Should Be Reproducible How This Applies to Multiple Executors How This Applies to AI Agents A Small End-to-End Example A Practical Conformance Workflow What Conformance Can't Prove From Execution Success to Verifiable Operations Conclusion What Conformance Means in This Context In ordinary language, conformance means complying with a rule, standard, or requirement. Here, I use the term more specifically: Operational conformance is the evaluation of whether, and where, observed execution evidence satisfies the requirements defined by an operational specification. In this context, evidence is the set of observable facts collected from the system that can be compared against the specification. It's not a success verdict. It's the data used to produce that verdict. That definition has three parts. 1. The Specification This defines what should be true. For example: version = v42 available replicas >= 3 error rate = spec.minReplicas && evidence.errorRate = spec.minReplicas, }, { name: "error-rate", passed: evidence.errorRate ({ name: constraint.name, status: constraint.evaluate(evidence), severity: constraint.severity, mode: constraint.mode, }) ); } The specification owns what should be evaluated. The evaluator collects the results. It doesn't need to know the meaning of each constraint. It simply executes every constraint's evaluation function and preserves the resulting status, severity, and mode. That keeps the evaluation engine generic while the specification remains responsible for the operational rules. The expected and observed fields are optional in this minimal example, but in a production system I would populate them, or store references to the underlying evidence, so the final verdict remains explainable and auditable. Calculate a Conformance Summary Carefully Individual results are useful, but sometimes you also need a summary. type ConformanceSummary = { total: number; passed: number; failed: number; unknown: number; conformant: boolean; }; Then: function summarize( results: ConstraintResult[] ): ConformanceSummary { const required = results.filter( (result) => result.mode === "REQUIRED" ); const passed = results.filter( (result) => result.status === "PASS" ).length; const failed = results.filter( (result) => result.status === "FAIL" ).length; const unknown = results.filter( (result) => result.status === "UNKNOWN" ).length; const conformant = required.every( (result) => result.status === "PASS" ); return { total: results.length, passed, failed, unknown, conformant, }; } Overall conformance now requires all required constraints to pass. Notice what that means for UNKNOWN: because every() requires a required constraint to have status PASS, a required constraint with missing evidence makes the overall result non-conformant rather than silently passing. Advisory constraints still appear in the report. They just don't determine the final verdict. Why a Single Score Can Be Dangerous It's tempting to create one single score: conformance = 92% But that can be misleading. Suppose: 9 low-risk constraints pass 1 critical security constraint fails A naïve score says: 90% conformant Operationally, that may be completely unacceptable. If you calculate a score, you should preserve: constraint-level results severity required/advisory status unknown evidence failure reasons The score can summarize, but it shouldn't erase the evidence. Preserve the Evidence Behind the Verdict A conformance result shouldn't only say FAIL. It should preserve what produced that verdict. { "constraint": "minimum-replicas", "expected": ">= 3", "observed": 2, "status": "FAIL", "severity": "CRITICAL" } This lets someone answer: What did we expect? What did we observe? Which rule was applied? Why did it fail? That is especially important when conformance decisions trigger automated recovery. Conformance Should Be Reproducible Suppose an operation executes at 10:00. At 10:01, the error rate is: 0.6% At 14:00, the dashboard shows: 2.1% If you reevaluate the old operation using current telemetry, you may get a different result. Conformance should therefore be evaluated against evidence associated with the execution window. By execution window, I mean the period during which the operation ran and its immediate effects were being verified. For a deployment that finished at 10:00, for example, you might evaluate health using the telemetry collected between 10:00 and 10:05 rather than whatever the dashboard happens to show several hours later. The point is to bind the verdict to the evidence that was available for that specific execution. Otherwise, later changes in the system can retroactively change the apparent result of an operation that already happened. A reproducible record might preserve: specification version execution ID evidence snapshot evaluation rules timestamp For example: { "executionId": "deploy-812", "specVersion": "3", "evaluatedAt": "2026-10-06T10:01:00Z", "evidence": { "version": "v42", "replicas": 3, "errorRate": 0.006, "p95LatencyMs": 290 } } Now the decision can be reconstructed later. How This Applies to Multiple Executors Suppose two executors run the same specification. Executor A: version PASS replicas PASS error-rate PASS latency PASS Executor B: version PASS replicas FAIL error-rate PASS latency PASS The question is no longer: Which executor is better? The immediate question is: Which execution satisfied the specification? You answer that by running the same conformance evaluator against the evidence produced by each executor. The specification and evaluation rules stay the same; only the evidence changes. In the example above, Executor A satisfies the specification because every required constraint evaluates to PASS. Executor B does not, because its replica constraint evaluates to FAIL. That gives us a common basis for comparison without requiring the two executors to use the same internal steps. Over time, executor performance can be compared using conformance rate, failure categories, latency, cost, recovery frequency, and unknown evidence rate. But conformance remains the contract-level evaluation. How This Applies to AI Agents AI agents make explicit conformance especially important. An agent may choose different plans for the same objective. Agent execution 1: scale deploy observe route Agent execution 2: deploy canary observe expand Agent execution 3: create parallel environment switch traffic Because the agent can choose a different plan each time, comparing executions step by step isn't very useful. One run might use a canary rollout while another creates a parallel environment, and both can still be valid ways to reach the same objective. What stays stable is the specification: the objective, constraints, and evidence requirements don't change just because the agent chooses a different path. After each run, the system collects evidence about the resulting state. The conformance evaluator then compares that evidence with the same specification used for every other run. This lets the agent vary the execution plan while keeping the definition of success outside the agent itself. Specification ↓ AI Agent ↓ Dynamic Plan ↓ Execution ↓ Evidence ↓ Conformance The agent can propose actions. It shouldn't be the only authority deciding whether those actions achieved the intended outcome. A Small End-to-End Example Start by defining the evidence: type DeploymentEvidence = { version?: string; replicas?: number; errorRate?: number; p95LatencyMs?: number; }; Now define the constraints: const constraints: Constraint[] = [ { name: "version", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) => { if (!evidence.version) { return "UNKNOWN"; } return ( evidence.version === "v42" ) ? "PASS" : "FAIL"; }, }, { name: "replicas", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) => { if ( evidence.replicas === undefined ) { return "UNKNOWN"; } return ( evidence.replicas >= 3 ) ? "PASS" : "FAIL"; }, }, { name: "error-rate", severity: "CRITICAL", mode: "REQUIRED", evaluate: (evidence) => { if ( evidence.errorRate === undefined ) { return "UNKNOWN"; } return ( evidence.errorRate { if ( evidence.p95LatencyMs === undefined ) { return "UNKNOWN"; } return ( evidence.p95LatencyMs <= 350 ) ? "PASS" : "FAIL"; }, }, ]; Create the specification: const deploymentSpec: OperationalSpec = { name: "deploy-orders-v42", constraints, }; Evaluate: const evidence: DeploymentEvidence = { version: "v42", replicas: 3, errorRate: 0.004, p95LatencyMs: 370, }; const results = evaluateConformance( deploymentSpec, evidence ); const summary = summarize(results); The result might be: version PASS REQUIRED replicas PASS REQUIRED error-rate PASS REQUIRED latency-target FAIL ADVISORY And: conformant = true Every required constraint passed. The advisory latency target failed, so the operation should still surface a warning. A Practical Conformance Workflow If you want to introduce conformance evaluation into an existing system: Pick one operation. Define the specification. Classify constraints as required or advisory. Assign severity. Define the evidence model. Decide how missing evidence becomes UNKNOWN. Evaluate constraints individually. Produce a summary. Preserve the evidence snapshot. Version the specification. Revisit rules that produce unexpected failures or unknowns. Don't simply weaken the specification to make results pass. What Conformance Can't Prove Conformance can tell you: The observed evidence satisfies the specification. It can't automatically tell you: The specification is correct. The specification may be wrong, and the evidence may also be wrong. So conformance depends on: specification quality evidence quality evaluation correctness It's a verification mechanism, not absolute truth. From Execution Success to Verifiable Operations Traditional automation often ends here: execute ↓ success / failure A conformance-based model adds more structure: Specification ↓ Executor ↓ Execution ↓ Evidence ↓ Constraint Evaluation ↓ Conformance Now you can ask: Which requirements were satisfied? Which failed? Which could not be evaluated? What evidence supports the verdict? If the system is conformant immediately after execution, you now have a baseline. From that baseline, you can keep observing the system over time. And then ask: What happens when reality starts to diverge from the specification after the operation has already succeeded? That's operational drift. Conclusion A successful executor isn't enough. Neither is a green pipeline or a completed agent plan. What matters is whether the resulting system satisfies the operational specification. That requires three things: Specification Evidence Evaluation A useful conformance model should preserve: constraint-level results PASS / FAIL / UNKNOWN severity required vs advisory rules observed values expected values evidence snapshots specification version The executor performs the work, evidence describes what happened, and the specification defines what should have happened. Then the conformance evaluator compares the two. That gives us a stronger foundation for traditional automation and for increasingly autonomous systems. But conformance at one point in time isn't the end of the story. A system can conform now and drift later. The next question is: How do we detect when observed reality stops matching operational intent? That's where I want to go next.