我试图证明DocLang优于Markdown用于PDF转LLM，数据却相反

Press enter or click to view image in full size Image by Author, generated by AI AI Artificial Intelligence Technology Machine Learning Data Science I Tried to Prove DocLang Beats Markdown for PDF→LLM. The Data Said Otherwise. A converter-independent experiment on a real 15-page RFP — and an honest look at where each format actually wins. Akshay Kokane 6 min read · 5 hours ago -- Listen Share If you’re building anything that feeds PDFs to an LLM — RAG, document Q&A, contract review — you hit the same fork early: what text representation do you hand the model? The default answer is Markdown. Parse the PDF, export_to_markdown(), stuff it in the context window. It's cheap, it's readable, every tokenizer likes it. The newer answer is DocLang — an XML-based, “AI-native” markup for unstructured documents. It keeps headings, lists, tables, reading order, and even bounding-box coordinates, all in a schema designed to be read directly by a language model. On paper it should dominate Markdown: same content, but structured. I went in wanting DocLang to win. I built the experiment to show it. Here’s what actually happened. The setup Source document: a real “General Conditions and Fee Request for Proposal” — a 15-page construction RFP with a project-info section and six exhibits (A–F). It’s a scanned-style PDF, the kind you actually get in the wild. The critical design choice — one parse, many renderings. My first attempt at this experiment had a fatal flaw: I compared my hand-written PDF→DocLang converter against Docling’s mature, years-tuned Markdown exporter. That’s not “Markdown vs DocLang.” That’s “my weekend project vs a library with hundreds of contributors.” So I rebuilt it. The PDF is parsed once into a DoclingDocument. Then every representation is produced by docling-core's own first-party serializers on that single shared object: Press enter or click to view image in full size No hand-written converter anywhere in the loop. Any difference between the arms is now attributable to the notation, not to converter maturity. The model: gpt-5.4-mini via Azure OpenAI, JSON-mode, whole document in the system prompt. The questions: two sets, each graded by an LLM judge against a gold answer and a gold location (“which section/exhibit is this in?”). Fact set — 15 questions × 3 trials: scope, budget, dates, contacts, exhibit lookup, one deliberately not-in-the-document negative. Structure set — 8 questions × 2 trials: enumerate the exhibits, “which exhibit governs requirement X?”, “is this heading a real section or a repeated page header?”, cross-references, structural negatives. Everything is in the repo: doclang-impl/experiment/ — prepare_inputs.py builds the arms, run_ci.py runs the comparison. Result 1: The answers are identical Press enter or click to view image in full size Answer accuracy — fact set” bar chart, all three arms at 100% Press enter or click to view image in full size “Answer accuracy — structure set” bar chart, all three arms at 87.5% With the whole document in context, all three representations produced the same answers — 100% on facts, 87.5% on structure questions. The one structure-set miss (“is there a substantive Exhibit G?”) was shared by every arm: it’s a text-extraction gap, not a format gap. Get Akshay Kokane’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in The notation did not change whether the model could find and state the answer. Not for flat facts, not for “which exhibit is this in,” not for cross-references. Result 2: DocLang costs 1.5×–2.6× the tokens Press enter or click to view image in full size Document size fed to the model Press enter or click to view image in full size Context-size overhead vs Markdown Here’s where it gets expensive: DocLang native: 16,232 tokens vs Markdown’s 6,348–2.56×. The bounding-box elements — four per content block — are the bulk of it. DocLang without locations: 9,288 tokens — still 1.46×. Even stripped bare, … and … wrappers are ~46% overhead over Markdown's ## and blank-line paragraphs. That’s a per-call cost you pay on every single query, for zero measured accuracy gain on this document. Result 3: Latency follows token count Press enter or click to view image in full size Average answer latency (fact set) Press enter or click to view image in full size scatter plot of prompt tokens (x) vs answer latency (y), three colored clusters, latency rising left to right No surprise, but worth showing: more input tokens → slower first token. Markdown was fastest in every cell. The scatter makes it plain — the three arms form three vertical bands by token count, and latency climbs with them. Result 4: On localization, Markdown was as good or better This is the one I expected DocLang to win — “which section is this answer in?” is exactly what structure should help with. Press enter or click to view image in full size Cited-location accuracy — fact set Press enter or click to view image in full size Cited-location accuracy — structure set Native DocLang tied Markdown on facts (86.7%) — the page numbers seem to give it a small assist. Strip the locations and it drops to 73.3% — same flat content as Markdown, more syntax noise, nothing gained. On structure questions, the bbox spam actively distracted the model (62.5% vs Markdown’s 75%). The per-question grid shows where the misses land: Press enter or click to view image in full size grid: rows = 3 arms, columns = questions; green/red square = answer correct/wrong, green/red dot = cited location correct/wrong The answer squares are almost all green. The red dots — wrong cited location — cluster on “DocLang (no locations).” Why? The DocLang for this PDF is flat Here’s the thing I didn’t expect. When I looked at what export_to_doclang() actually produced: Wind Point Partners (WPP) ... General Conditions and Fee Request for Proposal ... November 21, 2025 Every heading is level="2". There are no containers wrapping the exhibits. The running page header ("Wind Point Partners (WPP)") is repeated ~12 times as a real element. This isn’t a serializer bug. Docling’s layout model parsed this scanned PDF and didn’t recover the exhibit hierarchy — so there’s no nested structure in the DoclingDocument for export_to_doclang() to serialize. The DocLang output faithfully reflects a flat parse. Which means: on this document, DocLang is Markdown’s exact information content, wrapped in XML that costs 1.5–2.6× the tokens. The structural advantage I was counting on requires structure the parser never extracted. What changed from my first (broken) experiment My first write-up compared two hand-rolled DocLang converters against Docling’s Markdown exporter and concluded “flat DocLang is worse, my structured version closes the gap.” That comparison was confounded — I was measuring my converter, not the format. Removing the confound, the conclusion held and got sharper: the official DocLang serializer is also flat for this PDF, ties on answers, and loses on cost and latency. The format didn’t fail. The parser didn’t give the format anything to work with. So when should you use PDF→DocLang? This experiment stress-tests exactly one regime: one small document, full context, single-doc Q&A. In that regime, Markdown wins — cheaper, faster, equal accuracy. Full stop. DocLang’s value is in the regimes I didn’t test: 1. When the parser actually recovers structure. A born-digital PDF with a real outline, or a pipeline that post-processes the DoclingDocument tree to rebuild sections before serializing. Then export_to_doclang() carries nesting that Markdown flattens away. 2. Retrieval, when the corpus doesn’t fit in context. DocLang gives you addressable units — , label — as natural chunk boundaries. Split Markdown on ## and your running page headers bleed into every chunk. This is DocLang's home turf and it's completely untested here. 3. Round-trip fidelity. Tables as , formulas, forms as , checkboxes, reading order, RTL — Markdown drops all of it. And if you need to cite "page 12, this bounding box" back to the source pixels, the elements are the entire point. That 2.6× token cost is buying you provenance. 4. Scale. ~46–156% overhead per call here. Across a retrieval pipeline with clean unit boundaries, that math can flip. The recommendation For a single document you can fit in context, doing Q&A: use Markdown. DocLang from the same parse gives identical answers at 1.5–2.6× the cost. Reach for DocLang when you need provenance for pixel-accurate citations, the document has tables/forms Markdown mangles, or you're doing retrieval over a corpus too big for full context and want structural chunk boundaries. Don’t judge DocLang by this test. It measures the one setting where the format has the least to offer.