LLM 评估 104：为什么 AI 应用需要多条评估管线

Artificial Intelligence LLM Machine Learning Large Language Models LLM Evaluation 104: Why Your AI Application Needs Multiple Eval Pipelines Part 4 of the LLM Evaluation series — picking up right where “LLM Evaluation 103” left off. Momina Ather 7 min read · Just now -- Listen Share Press enter or click to view image in full size Image by the author TL;DR One single “eval score” can never tell you if your AI app is actually safe to ship. A real AI application (say, a RAG chatbot) is made of several moving parts — a retriever, a generator, a memory module, maybe an agent that calls tools. Each part can pass its own test perfectly, and the whole app can still fail the moment they’re wired together. That’s why you need multiple, parallel evaluation pipelines: one for each component, one for the workflow that connects them, and one for the finished application as the user experiences it. A quick recap first In the last two posts of this series, we covered two things: why “vibe checking” a model’s output isn’t a real evaluation strategy (Evaluation 101), and the difference between evaluating a raw model versus evaluating the full application built on top of it (Evaluation 102). Then in 103, we walked through the end-to-end workflow of actually evaluating an LLM app. This post answers the natural next question: if I already have one eval pipeline, why do I need more than one? Think of your AI app like a factory line, not a single machine Picture a typical Retrieval-Augmented Generation (RAG) setup. It’s not one block — it’s a chain: The Retriever — searches your knowledge base and pulls the most relevant chunks of text. The Generator (the LLM) — takes those chunks plus the user’s question and writes the final answer. (Sometimes) A Memory or Agent layer — keeps context across turns, or decides which tool to call. Each of these is its own mini-system with its own way of failing. A retriever can pull the wrong documents. A generator can take the right documents and still write a wrong answer. If you only test the final output of the whole chain, you have no idea which link actually broke. Independent success does not guarantee combined success Here’s the part that trips up most people building their first AI product: each component can individually pass its own evaluation, and the full application can still fail. That sounds counterintuitive, so let’s make it concrete with an example that mirrors the “8-week ML course duration” scenario below. Imagine a course-assistant chatbot for an ed-tech platform. A student asks: “How long is the Machine Learning course?” The retriever does its job correctly — it fetches the right page of the course catalog, which clearly states the course is 8 weeks long. Retriever eval: pass. The generator takes that retrieved chunk and, while phrasing a friendly answer, misreads the number and tells the student the course is 12 weeks long. That is a failure the user actually experiences. If you only ran a retrieval eval, you would conclude everything is fine, because the correct document was fetched. If you only ran a “does the LLM sound fluent and helpful” eval, that would also pass, because the answer reads perfectly naturally. Neither test catches the actual bug: a wrong fact reaching the user. Only a test that checks the final workflow output against the ground truth — 8 weeks, not 12 — catches this. This is exactly why one eval pipeline isn’t enough. You need checks at more than one level. The 3 levels of evaluation There are three distinct levels here, and it’s worth treating each one as its own pipeline rather than mashing them into a single test. Component-level evals test each piece in isolation — is the retriever pulling the right chunks (precision and recall)? Is the generator, given a perfect context, producing a coherent answer? This tells you where inside the machine something is broken. Workflow-level evals test the hand-off between components. Does the final answer, once the retriever and generator have both done their jobs together, actually match the ground truth? This is the level that caught the “8 vs 12 weeks” bug above — in software terms, this is very similar to integration testing: individual units can pass their own unit tests while the integrated system still breaks. Application-level evals zoom out even further. This is the full user-facing experience: latency, safety, cost, and whether the entire session — not just one turn — actually achieved what the user came for. A workflow can be factually correct and still fail at the application level if it’s too slow, too expensive, or leaks something it shouldn’t. Skipping straight to application-level testing without the component and workflow layers means that when something breaks, you’re left guessing which of the ten moving parts caused it. Testing bottom-up — component, then workflow, then application — gives you a paper trail. It’s not just about being “right” — 3 pillars of risk Once you accept that correctness needs to be checked at multiple levels, the next question is that correct and safe isn’t the whole picture either. Everything an AI application needs to be evaluated against falls into three pillars. Quality asks: is the answer accurate, relevant, complete, and well-formatted? This is the pillar most people evaluate first, and the only one many teams ever get to. Safety asks: does the app resist producing toxic content, leaking private or system information, or getting jailbroken into ignoring its instructions? Operations asks: does the app respond fast enough, and does it cost a sane amount of money per request, especially under real production load? Get Momina Ather’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in A chatbot that gives perfect answers but leaks your system prompt on request, or one that’s accurate but takes 40 seconds and costs $2 per reply, is not production-ready either. You need a pipeline watching each pillar, because a fix in one area can quietly break another — adding more context to improve quality, for example, often increases latency and cost. Press enter or click to view image in full size Image by the author Different app types, different metrics A single generic “accuracy score” doesn’t translate well across use cases either. The kind of app you’re building changes what “good” even means. Summarization apps care about faithfulness — no hallucinated facts — and information coverage, meaning nothing important gets left out. RAG apps care about retrieval relevance and answer groundedness: is the answer actually supported by the retrieved text? Chatbots care about coherence across multiple turns, tone, and whether the conversation actually resolves the user’s need. Agents care about task completion, correct tool selection, and whether the agent recovers gracefully when a tool call fails. Each of these needs its own metric set and, often, its own pipeline — a summarizer’s tests won’t catch an agent calling the wrong tool, and an agent’s tests won’t catch a chatbot losing the thread of a conversation. Press enter or click to view image in full size Image by the author The safety guardrail matrix On the safety pillar specifically, there are three guardrails every pipeline should check for before an AI app goes live. Toxicity: does the model ever produce harmful, biased, or offensive output? Leakage: can a user trick the model into revealing its system prompt, internal data, or other users’ information? Jailbreak resistance: can the model’s safety instructions be bypassed with clever prompting? These need adversarial, “red-team style” testing — a completely different pipeline from the one checking factual accuracy, because the inputs you use to test them (deliberately tricky, manipulative prompts) are nothing like the inputs you’d use to test quality. Operational audits: the pipeline most teams skip The last pillar, operations, is where a lot of otherwise-good AI products quietly lose money or users. Two numbers matter most here. Latency under load: how does response time hold up when 100 or 1,000 users hit the app at once, not just in a single clean demo run? Cost per request: what does one interaction actually cost once you multiply out tokens times model price times retries? This can make or break a product’s unit economics. Neither of these shows up if your only eval pipeline is checking answer quality on a handful of test questions. They need their own monitoring pipeline, usually running continuously in production rather than as a one-off pre-launch check. Putting it together: what a real eval setup looks like Putting it all together, the picture is clear: a production-ready AI application isn’t guarded by one test suite; it’s guarded by a small portfolio of pipelines running in parallel — component pipelines for the retriever, generator, and memory; a workflow pipeline that checks whether the assembled answer matches ground truth; an application pipeline for full-session success, latency, and cost; a safety pipeline for toxicity, leakage, and jailbreak red-teaming; and task-specific metric pipelines depending on whether you built a summarizer, a RAG app, a chatbot, or an agent. Each one answers a different question, and each one can fail independently of the others. That’s the entire argument for why “multiple eval pipelines” isn’t over-engineering—it’s the only way to know where your AI application will break before your users find out for you. Quick self-check before you move on Before reading the next part of this series, it’s worth pausing and asking about your own AI project. If your app’s final answer was wrong today, could you tell whether the retriever, the generator, or the hand-off between them caused it? Do you have any pipeline testing for toxicity, prompt leakage, or jailbreaks — or only for “is the answer good”? Do you know your app’s cost per request and latency under real concurrent load, or only in a single-user demo? If the honest answer to any of these is “no,” that’s exactly the gap this post is pointing at. Next up in this series: LLM Eval Methods — LLM-as-a-Judge vs. reference-based evaluation.