PDF 抽取文本中哪些换行是真实的？基于字符坐标重新拼接折行

Extract text from a PDF and you get far too many line breaks. pdfium (the engine behind Chrome's PDF viewer) returns a break after every visual line, so sentences are cut in the middle of paragraphs. Feed that to an LLM, a search index or a summarizer and quality drops. Remove every break instead, and bullet lists and tables melt into one long sentence. This post is about how the "Export text for AI" feature of PDF Privacy Checker (a Windows app that finds hidden text in PDFs) decides which line breaks are soft wraps inside a paragraph, using only character coordinates. It is the follow-up I promised at the end of the previous post (I rebuilt my "export PDF text for AI" feature three times in one day). This is not a write-up of rules that worked the first time. I measured them on real PDFs for this post, found they were broken, fixed them and measured again. The measurements led to three fixes in the app. TL;DR pdfium inserts a line break after every visual line. Removing all breaks erases paragraph boundaries; joining by character class or by sentence-ending punctuation melts lists and tables. Rebuild the visual lines from character boxes, then check each pair of neighbouring lines against eight conditions. Join only when all of them hold; when in doubt, keep the break. On my own 10 articles printed to PDF (3,751 labelled line breaks), it restores 89.8% of the soft wraps, and 95.2% of the joins are correct. Almost all wrong joins are lines of source code; only 4 are not. On 29 real business PDFs (invoices, quotes, vehicle papers), it was gluing table rows together. One extra condition fixed that. Some errors remain, such as short stacked form fields. Line breaks from a PDF are not usable as-is Here is a made-up store closing checklist I built for this post (the script at the end generates the PDF). This is what pdfium returns for page 1: Closing Checklist This sheet lists the checks to do in the last five minutes before closing. If you follow them in order, you will not forget to lock up, and the sales total will match the cash in the register. When you are done, sign the back. 1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager. 2 Record the temperature of the fridge and the freezer. If one is too warm, circle it in red for the morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 (rest omitted) Enter fullscreen mode Exit fullscreen mode The sample contains a title, a four-line paragraph, two numbered items with a hanging indent, three bullets, and a small cash-count table (denomination, count, amount). The breaks inside the paragraph and inside the numbered items should be joined. The breaks after the title, between the bullets and between the table rows should stay. The goal is to tell these two kinds apart. A PDF has no notion of "this is the middle of a paragraph". It only says where each glyph is drawn. The meaning of a line break has to be inferred from positions. The obvious approaches each break something These are real outputs of simple rules, run on the sample PDF that the companion script generates. 1. Remove every line break Closing Checklist This sheet lists the checks … sign the back. 1 Count the cash in the register … call the manager. 2 Record the temperature … morning shift. - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter … Enter fullscreen mode Exit fullscreen mode The title merges into the text, the bullets become one sentence and the whole table becomes one line. Is 45.00 the amount for the $5 bills, or the start of the next row? Neither a person nor an LLM can tell any more. 2. Join unless the line ends with a period (or ! ? :) Closing Checklist This sheet lists the checks to do in the last five minutes before closing. … - Manager: extension 11 - Security company: number on the receipt - When in doubt, use the paper checklist Denomination Count Amount $20 bills 12 240.00 $5 bills 9 45.00 Quarters 40 10.00 Pull the shutter down last … Enter fullscreen mode Exit fullscreen mode It fails in both directions. Titles, bullets and table rows have no period, so they all merge. And whenever a sentence happens to end exactly at the right margin inside a paragraph, the paragraph is cut there. The first failure is dangerous; the second is merely ugly. Japanese, with 。 as its full stop, fails the same way. 3. Join when the next line starts with a lowercase letter 1 Count the cash in the register and compare it with the total on the Daily Sales Sheet. If it does not match, call the manager. Enter fullscreen mode Exit fullscreen mode This works well on this page, and its failures are mostly misses rather than false merges: it cannot join a wrap that is followed by a capital letter or a digit, such as a document name, a person's name, an amount or a date. Business text is full of those. It also has no equivalent for Japanese, which has no letter case; the closest thing there, joining when both sides are CJK characters, merged the title into the text and the table header into its first row on my Japanese version of this page. Letter case and character class tell you how to join (with or without a space), not whether to join. 4. Same baseline means same line This one is about the earlier step of merging fragments. PDFs from Word's character spacing or from CAD tools often draw one visual line as many small strings, and pdfium inserts a break between them. Matching baselines sounds like the fix, but a line where the number "5" is set in a different size from the text has different baselines. I use vertical overlap instead. All four look only at the string. The meaning of a line break lives in where the lines are, not in the text. Rebuilding visual lines from character boxes pdfium gives a box for every character: FPDFText_GetCharBox (the tight glyph bounds) and a "loose" box built from the advance width and the font height. I use the union of both, because with some fonts one of them collapses (an example appears later). The pipeline: PDF ↓ pdfium (characters and boxes; a break after every visual line) character boxes ↓ step 1: merge fragments of the same visual line visual lines (left, right, top, bottom, median character height h) ↓ step 2: check each pair of neighbouring lines against 8 conditions each break: soft wrap or real break ↓ join (no space between CJK characters, one space in English) text for the LLM (the report's full-text appendix keeps the original breaks) Enter fullscreen mode Exit fullscreen mode Every threshold is expressed in units of h, the character height, so the same rules work for any font size. h is about 1.1 times the font size. Threshold Step Meaning vertical overlap ≥ 0.5h 1 on the same height next fragment within 2.5h to the right 1 continuation of the same line gap 2.5 * h and (a["R"] - a["L"]) 0.3 * h: return False # 3. different text size (heading, note) if not (b["R"] > a["L"] and a["R"] > b["L"]): return False # 4. no horizontal overlap (column, cell) if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3): return False # 5. column gap (a table row) block = [x["R"] for x in lines if abs(x["L"] - a["L"]) = 0.5 * hmin and -0.3 * hmin 2.5 * h and (a["R"] - a["L"]) 0.3 * h: return False # 3. different text size (heading, note) if not (b["R"] > a["L"] and a["R"] > b["L"]): return False # 4. no horizontal overlap (column, cell) if "gap" in rules and (max_gap(a) >= 3 or max_gap(b) >= 3): return False # 5. column gap (a table row) block = [x["R"] for x in lines if abs(x["L"] - a["L"]) >", "", ">"] kids = [] for ops in (en, en2): data = "\n".join(ops).encode("latin-1") objs.append(f">\nstream\n".encode() + data + b"\nendstream") objs.append(f"> >> >>") kids.append(len(objs)) objs[1] = f">" out, offs = b"%PDF-1.7\n", [] for k, o in enumerate(objs, 1): offs.append(len(out)) out += f"{k} 0 obj\n".encode() + (o if isinstance(o, bytes) else o.encode()) + b"\nendobj\n" xref = len(out) out += f"xref\n0 {len(objs) + 1}\n0000000000 65535 f \n".encode() out += b"".join(f"{o:010d} 00000 n \n".encode() for o in offs) out += f"trailer >\nstartxref\n{xref}\n%%EOF\n".encode() open(path, "wb").write(out) if __name__ == "__main__": sys.stdout.reconfigure(encoding="utf-8") if len(sys.argv) > 1: for page in pdfium.PdfDocument(sys.argv[1]): print(rejoin(page_lines(page), {"cjk", "gap"}), end="\n\n") sys.exit() build_pdf("article10_sample_en.pdf") doc = pdfium.PdfDocument("article10_sample_en.pdf") for no, page in enumerate(doc, 1): lines = page_lines(page) print(f"===== page {no}: pdfium line breaks as-is =====") print(page.get_textpage().get_text_range().replace("\r\n", "\n")) print(f"===== page {no}: first rules (indent rule for any script, no column-gap check) =====") print(rejoin(lines, set())) print(f"===== page {no}: current rules =====") print(rejoin(lines, {"cjk", "gap"})) print() Enter fullscreen mode Exit fullscreen mode Wrap-up The meaning of a PDF line break lives in where the lines are, not in the text. Rebuild visual lines from character boxes and judge each break from its neighbours. Join only when every condition holds. A missed wrap is ugly; a wrong join changes the meaning. The rules only showed how they break once I measured them on real PDFs. The indent rule fired on English tables; "the line is full" fired on table rows. Prose and business documents give completely different numbers for the same rules. When you report numbers, say what kind of documents you measured. Measuring for this post led to three fixes in PDF Privacy Checker: table rows are no longer glued together, a line-end hyphen no longer triggers a false control-character warning, and "一" and "—" in PDFs saved from Chrome are no longer flagged as tiny text. All three are in v1.15.0, which is now on the Store. PDF Privacy Checker is on the Microsoft Store (detection is free; the text export for AI is part of a paid add-on). Files never leave your PC; it works fully offline. The source code is not public. https://apps.microsoft.com/detail/9PLRJHFTPS53?hl=en-us&gl=US Next, I plan to write about using the same coordinates to detect heading lines, so the text for AI keeps the document's structure. About this article — The implementation and the writing were done together with Claude (Anthropic's AI). Most of the prose was drafted by Claude and checked by me. The decisions about which errors to fix in the product and which to leave as limitations, and the real business documents, are mine. About the author Okinawa Software Lab. I lead in-house digital transformation at a small company in Okinawa, Japan. I build the tools we need ourselves, and I publish PDF apps on the Microsoft Store that follow the same principle: everything happens on your own PC. Website: https://okinawasoftwarelab.com/en/