说‘本文档中没有’的RAG必须展示四种证据

The most useful answer a RAG system can give is sometimes “that is not in this document.” But when asked a question, a model tends to answer anyway, so the honest “not found” has to be built in, not hoped for. Getting the system to say no, and to be right when it does, is harder than getting it to say yes. This article is a bonus in Enterprise Document Intelligence, a series that builds an enterprise RAG system from four bricks. It justifies the “I don’t know”: a confident wrong answer is a bug, a bare no-answer is almost as bad, and each of the four bricks has one piece of evidence to show. 🧭 New to the series? Every article in this series sits on our two Towards Data Science author pages, Angela Shi and Kezhan Shi. That is the shortest way to see what is covered and where this one sits. where this article sits in the series: a bonus article alongside the numbered spine - Image by author 📓 Runnable companion notebooks are on GitHub: doc-intel/notebooks-vol1. The public companion-code repo at doc-intel/notebooks-vol1 - Image by author Ask the corporate chatbot “How much electricity does AI consume globally?” and it returns “I cannot find an answer in this document.” Rephrase and ask again: same answer. At that point most users give up and go to Google. The system did the right thing. The document it runs on is the World Bank’s Commodity Markets Outlook (April 2025 issue; CC BY 3.0 IGO, as declared on the World Bank Open Knowledge Repository terms of use which states the default license for pre-2023 publications), a 63-page quarterly report on oil, agriculture, and metals prices. AI electricity consumption is not in there. There is no answer to find. The system was not lying. But the user did not learn anything useful. They cannot tell whether the pipeline looked everywhere it should have, or whether it gave up after one shallow embedding search. “I cannot find an answer” is a verdict that needs to come with a defense. Otherwise it reads as failure, not as a fact. The series argued that structured output (Article 8) makes “yes” answers verifiable, by forcing the model to cite its evidence: a quote, a page number, a confidence. The same logic applies to “no” answers, and the same four bricks contribute, one piece of evidence each. This article walks the four bricks once more, on a question whose right output is no, and shows what each brick has to produce so the no-answer is defensible. 1. The four bricks, four kinds of evidence The argument compresses to one table. Four bricks, four kinds of evidence, none of them optional for a defensible verdict - Image by author The rest of the article walks each row in order, then ties them together on the CMO / AI electricity-consumption case. 2. Parsing: the relational tables are the evidence The cost of a missing parse is asymmetric. A false positive (extracting noise) makes retrieval messier but is recoverable downstream. A false negative (failing to extract a real token) is silent: the no-answer verdict will look correct, but it is wrong. The answer was there; the pipeline just never saw it. The good news is that the parsing brick (Article 5B, the relational data model) already produces exactly the data the absence claim needs. The output of parse_pdf is not a wall of text; it is a small relational set of DataFrames: one per kind of thing the document contains. line_df is the central table: one row per text line across the whole document, with its page number, bounding box on the page, and the detected line type (prose, heading, table row, figure caption). The other tables hang off it. Parsing emits a relational set of DataFrames; the primary sweep uses three of them - Image by author For a no-answer verdict, we do not write a new parsing function. We aggregate over the tables that already exist: python out = parse_pdf(pdf_path) # gives line_df, image_df, page_df, toc_df, ...# The parsing brick's piece of evidence is a few aggregations over the tables.parse_coverage = { "pages_total": len(out["page_df"]), "pages_with_text": out["line_df"]["page_num"].nunique(), "images_total": len(out["image_df"]), "toc_entries": len(out["toc_df"]), "cross_refs_total": len(out["cross_ref_df"]), "objects_registered": len(out["object_registry"]),}# On CMO April 2025: 63 pages, 63 with text, 426 image registrations,# 9 TOC entries, 76 in-body cross-references. The aggregates name the# surfaces an absence claim must walk -- text in line_df is the easy# one, the other rows are the audit trail for image text and named# objects (figures, tables, annexes). The three places an absence claim still needs to check beyond line_df: Images that carry text: Charts often hold their axis labels and legends as image-embedded text rather than typographic spans. The image_df registry lists every such image; an OCR pass adds an ocr_text column to that table without touching line_df. The parsing brick exposes both before and after states; the coverage report names how many images were OCR’d. Tables: A number in a cell parses as a line whose neighbours are the other cells of the same row, not the words above it on the page. If the answer is “AI consumed 460 TWh in 2024” and the figure sits in a table without the word AI in any nearby cell, a sweep on AI misses it. object_registry flags which pages host tables so the retrieval brick can do a column-aware sweep there (Article 5 section on table parsing). Cross-references that did not resolve. cross_ref_df lists every “see annex B”, “figure 3.D”, “table 11.E” mention in the body. Any row whose target is not in object_registry is a parsing gap: the body promises an object the extraction did not find. Each gap is a candidate spot where the answer could be hiding. The parsing brick’s piece of evidence is therefore a small summary derived from the tables it already produces, not a new component. The no-answer verdict can state: “63 pages parsed, 63 with text, 0 unresolved cross-references.” That is enough for the user to trust the parsing layer did its work. 3. Question parsing: enumerate the vocabulary The retrieval step is only as good as the keywords it sweeps on. If the expert lists AI but forgets artificial intelligence, the retrieval misses every line that spells the concept out. The work of the question-parsing brick is to push the keyword set to exhaustion, with the expert as the source of truth and an algorithmic safety net underneath. 3.1 The expert knows the vocabulary In enterprise RAG, the expert almost always knows the keywords of their domain. This is the simple case and it covers most questions. They list the concepts that the answer would mention, and for each concept they enumerate the variants: synonyms, acronyms, abbreviations, English / French / German forms if the corpus is multilingual. For the CMO / AI electricity example, the expert provides this table: The concept_keywords_df schema from Article 6 applied to a ‘no answer’ question - Image by author This is exactly the concept_keywords_df schema from Article 6. The same data structure that drives normal retrieval also drives the absence claim. The signature is identical; only the interpretation of the output changes. 3.2 Concept clustering as a safety net The rare case is the expert who isn’t sure they listed every variant. The safety net is algorithmic: cluster the corpus tokens by embedding similarity, then expose the clusters that the expert’s keywords land in, and ask whether any cluster member should be added. python def expand_keywords_via_clustering( seed_keywords: list[str], corpus_tokens: list[str], *, n_neighbors: int = 10, cosine_floor: float = 0.85,) -> list[KeywordCandidate]: """For each seed keyword, return its top-N nearest tokens in the corpus above the cosine floor. The expert reviews the candidates and accepts or rejects each one.""" seed_vecs = {k: embed(k) for k in seed_keywords} corpus_vecs = {t: embed(t) for t in corpus_tokens} candidates = [] for seed, sv in seed_vecs.items(): scored = [(t, cosine(sv, tv)) for t, tv in corpus_vecs.items()] scored.sort(key=lambda x: -x[1]) for t, sim in scored[:n_neighbors]: if sim >= cosine_floor and t not in seed_keywords: candidates.append(KeywordCandidate(seed=seed, candidate=t, similarity=sim)) return candidates The output is a list, not a decision. The expert decides what to add. The candidates are a suggestion to consider, not a replacement for the expert’s judgment. The series’ editorial position holds: amplify the expert, do not replace them. The clustering reveals what the expert may not have thought of; the expert validates what belongs in the keyword set. The keyword set, signed off by the expert, is the question-parsing brick’s piece of evidence. 4. Retrieval: sweep, not top-k For a normal question (Article 7), retrieval returns top-k: the few pages or lines most likely to contain the answer. For a no-answer claim, that framing fails. The system cannot prove the answer is absent by looking at the top-10 pages. It has to look at every page that mentions any of the concepts. The shape of the retrieval call therefore changes: python def sweep_for_absence( line_df: pd.DataFrame, concept_keywords_df: pd.DataFrame,) -> pd.DataFrame: """Return one row per (concept, page, line) match across the corpus. NOT top-k. NOT scored. Every hit on every variant of every concept.""" rows = [] for concept_row in concept_keywords_df.itertuples(): pattern = r"\b(" + "|".join(map(re.escape, concept_row.variants)) + r")\b" matched = line_df[line_df["text"].str.contains(pattern, regex=True, case=False, na=False)] for line in matched.itertuples(): rows.append({ "concept": concept_row.concept, "variant": extract_first_match(pattern, line.text), "page": line.page_num, "line": line.line_num, "snippet": line.text[:120], }) return pd.DataFrame(rows) The return value is a DataFrame, not a ranked list. Each row is an EVIDENCE entry: “on page P, line L, the variant V of concept C was found in the snippet S”. The cardinality of the result is what the no-answer verdict turns on: Zero rows for some concept (no variant appears anywhere) → strong evidence the document does not cover that concept at all. Several rows but never co-located (concepts appear on different pages, never together) → medium evidence: the concepts exist but no passage talks about their intersection. Co-located rows (multiple concepts in the same page or paragraph) → weak evidence of absence. Generation will have to look at the snippets and decide whether they answer the question or only touch it. On the CMO April 2025 case, the result is unambiguous. The full sweep on the keyword set above returns: Real sweep over the 7,829 lines of the report: zero AI hits, a handful on electricity, none co-located - Image by author The hit-list is the retrieval brick’s piece of evidence. On this question and this document, the verdict writes itself. 5. Generation: structured “no answer” with justification The last brick takes the parse coverage report, the validated keyword set, and the hit-list, and turns them into the actual response the user sees. The schema mirrors AnswerWithEvidence from Article 8, but the active fields are different. python class AbsenceJustification(BaseModel): """Returned when the system cannot answer from the corpus. Companion to AnswerWithEvidence from Article 8.""" answer: None = None # always None for this schema reason: Literal[ "concept_not_found", # no variant of any concept hit anything "concept_found_but_off_topic", # concepts exist but never co-locate "ambiguous_question", # question itself is too vague to sweep ] searched_concepts: list[ConceptSearch] # what was looked for, with hit counts closest_mentions: list[Mention] # any nearby snippets, with why-not parse_coverage: ParseCoverage # what the parsing brick covered suggestion: str | None = None # optional reformulation or off-corpus pointerclass ConceptSearch(BaseModel): concept: str variants: list[str] pages_with_hits: list[int] line_hits: intclass Mention(BaseModel): page: int line: int snippet: str why_not_an_answer: str # one sentence : why this passage is near but does not answer The generation step does not invent any of the fields. The schema’s first three fields come straight from the previous bricks (parse coverage, the keyword set, the hit-list). The model’s job is narrower: pick the reason, choose which mentions to flag as closest, write the one-sentence “why this is not an answer” for each, and optionally suggest a reformulation. On the CMO / AI electricity case, the model’s output looks like this: json { "answer": null, "reason": "concept_not_found", "searched_concepts": [ { "concept": "AI", "variants": [ "AI", "artificial intelligence", "machine learning", "deep learning", "neural network", "GenAI", "generative AI", "LLM", "large language model" ], "pages_with_hits": [], "line_hits": 0 }, { "concept": "electricity consumption", "variants": [ "electricity", "power consumption", "energy use", "kWh", "MWh", "TWh", "electrical load", "power demand", "electricity demand" ], "pages_with_hits": [30, 31], "line_hits": 7 }, { "concept": "data center", "variants": [ "data center", "data centre", "datacenter", "server farm", "hyperscaler", "cloud infrastructure" ], "pages_with_hits": [], "line_hits": 0 } ], "closest_mentions": [ { "page": 30, "line": 89, "snippet": "growing power demand in EMDEs is expected to", "why_not_an_answer": "Mentions a country's rising electricity needs in a coal-demand context ; the AI sector is not the subject and no figure is given for AI consumption." } ], "parse_coverage": { "pages_total": 63, "pages_with_text": 63, "pages_ocred": 0 }, "suggestion": "The Commodity Markets Outlook covers oil, agriculture, metals, and fertilizers ; it does not discuss AI sector energy demand. For figures on AI electricity consumption, see the IEA 'Electricity 2024' report or a dedicated AI-energy outlook."} A user reading this output learns three things at once. The system did look for AI under nine different names and found zero hits anywhere in 63 pages. There is one electricity-related passage, but it concerns India’s coal-driven power demand, not AI. And if they want the answer, the CMO is the wrong corpus to ask. The no-answer is now a useful response. 6. Where this stops Three edge cases the framework above doesn’t cover cleanly, in order of importance. Partial answers: A question may have part of its answer in the corpus and part outside it. “How does the EU regulate AI under the AI Act, and how does that compare to U.S. policy?” on an EU-only legal corpus. The retrieval brick will return hits for the EU side and nothing for the U.S. side. The right response is neither a confident yes nor a clean no; it is a structured partial answer that shows what was found and is explicit about what was not. The schema for that is a third sibling of AnswerWithEvidence and AbsenceJustification, with both answer_partial: str and missing_concepts: list[ConceptSearch] fields. Ambiguous questions: “What about coverage?” asked without context. The system cannot sweep because there is no defined concept set. The right response is a clarification request, not a no-answer. The signal that distinguishes the two is the parsing of the question (Article 6): if the question parser cannot extract concepts, the question is the problem, not the corpus. Hostile or out-of-scope questions: “What is the meaning of life?” on an insurance-policy corpus. The system can correctly say no-answer, but spending effort on the parse-coverage report and the closest-mentions is wasted. The pipeline should have an upstream “is this question in scope” check that short-circuits the full sweep when the question parser flags the concepts as unrelated to any corpus tag. Pipeline cost matters when this kind of query is frequent. These cases share a structure: the no-answer schema above is the right shape for absence claims, but it is not the right shape for every “I cannot help” output. Treating it as one of three siblings (yes / partial / no) rather than a single fallback handles the variation cleanly. 7. Conclusion A defensible “no answer” is not a single sentence; it is a chain of evidence the four bricks produce together. Parsing reports coverage, question parsing reports the expert keyword set, retrieval reports the sweep, generation reports the closest mention and why it does not answer. The user sees the work and can dispute the parse, the keyword set, the closest mention. The symmetry with the rest of the series is the bigger point. Yes answers are verifiable when the schema forces the model to cite its evidence (Article 8); no answers are verifiable when the schema forces the pipeline to expose its search. It is the same discipline on the same bricks, and both outputs land structured. 8. Sources and further reading The canonical benchmark establishing “no answer” as a first-class output is Rajpurkar et al. (SQuAD 2.0, ACL 2018). The framing where many questions are unanswerable inside the given passage is Choi et al. (QuAC, EMNLP 2018). The model-side reflection-token analogue of the pipeline-side sweep used here is Asai et al. (Self-RAG, ICLR 2024). The article’s framing: the three-sibling answer schema (yes / partial / no) with a defensible evidence chain on the no-answer branch, parse-coverage report, expert-keyword sweep, closest-mentions, every step auditable. Earlier in the series: Document Intelligence: series intro. What the series builds, brick by brick, and in what order. What works, what breaks Baseline Enterprise RAG, from PDF to highlighted answer. The four-brick pipeline end to end: PDF in, highlighted answer out. Embeddings Aren’t Magic: The Predictable Failure Modes of RAG Retrieval. Where embedding similarity wins (synonyms, typos, paraphrase), where it predictably breaks (unknown terms, negation, term-vs-answer relevance), and how to use it anyway. RAG is not machine learning, and the ML toolkit solves the wrong problem. Why chunk-size sweeps and finetuning optimize the wrong thing; route by question type instead. From regex to vision models: which RAG technique fits which problem. Two axes, document complexity and question control, that pick the technique for each case. 10 common RAG mistakes we keep seeing in production. Ten production mistakes, organized brick by brick, with the fix for each. Document parsing Building Document Structure with Loop Engineering: Recovering a PDF’s Outline from Body Typography for RAG. Rebuilding the outline from body typography when the PDF ships no contents page at all: six signals, one bounded loop. Before Full Agentic RAG: Know How You Decide, and the Parsing Methods You Pick From. The parsing methods as a catalogue, and the decision of which to run, before handing the loop to an agent. Generation Most RAG Hallucinations Are Extraction Errors: Seven Patterns for a Typed Generation Contract. Seven recurring ways a model gets the extraction wrong, and the typed contract that catches each one. Loop engineering for RAG generation: an LLM cascade from a cheap local model up to a hosted flagship. Starting on a cheap local model and escalating only when the answer does not hold up, measured. One-document pipelines Cut an Enterprise RAG Pipeline’s Latency and Cost by Calling the LLM Less, Not by Buying a Faster Model. Cutting a pipeline’s latency and cost by calling the model less often and cheaper, not by buying a faster one. RAG workflow and loop engineering: the dispatcher that decides when to loop and when to stop. Feedback loops, bounded iteration, and the dispatcher, composed into one workflow. Loop engineering for RAG: the small loops inside each step, the big loops across the pipeline. The two scales of loop: small bounded loops inside each brick, big generation-triggered loops across them. One doc to a corpus Three Kinds of RAG Corpus, and What It Costs to Build for the Wrong One. Three questions that tell you which shape a document collection has, and the price of building for the wrong one. Drilling Into AI’s Financial Sustainability Artificial Intelligence Budgets for AI tokens can’t be infinite, no matter how much hyperscalers wish they were Stephanie Kirmer June 16, 20267 min read Building a Smart Travel Itinerary Suggester with LangChain, Google Maps API, and Gradio (Part 1) Large Language Models Learn how to build an application that might inspire your next road trip Robert Martin-Short September 26, 202315 min read Fine-Tuning BERT for Text Classification Deep Learning A hackable example with Python code Shaw Talebi October 17, 20246 min read LLMs Are Randomized Algorithms Large Language Models A surprising connection between the newest AI models and a 50-year old academic field Udayan Kanade November 13, 202515 min read Your RAG System Retrieves the Right Data — But Still Produces Wrong Answers. Here’s Why (and How to Fix It). Large Language Models Even with perfect retrieval, conflicting context breaks answers. This is the hidden failure mode most production RAG systems ignore — and how to fix it. Emmimal P Alexander April 18, 202616 min read ChatLLM Presents a Streamlined Solution to Addressing the Real Bottleneck in AI Artificial Intelligence By consolidating access to GPT-5, Claude, Gemini, DeepSeek, Grok, Llama, and more into one subscription, ChatLLM helps budget-conscious users replace multiple AI tools with a single, flexible platform Marcus Dawson December 22, 20257 min read How to Fine-Tune an LLM: An End-to-End Guide Large Language Models A hands-on guide to fine-tuning LLMs for the real world Sam Black August 20, 202626 min read Deep Dive into Sora's Diffusion Transformer (DiT) by Hand ✍︎ Large Language Models Explore the secret behind Sora's state-of-the-art videos Srijanie Dey, PhD April 2, 202412 min read The Build vs. Buy Dilemma for GenAI Applications Strategy A strategic guide to build vs. buy for GenAI Anurag Bhagat January 7, 20256 min read