pdfmux confidence scoring explained: how we measure extraction quality

pdfmux assigns a confidence score between 0 and 100 to every page it processes. Those scores aren’t cosmetic — they drive the routing decisions that send bad pages through OCR, Docling, or Gemini. Understanding how they work is the fastest way to predict what pdfmux will do with any given PDF, and the clearest onramp for contributors who want to improve the extractor pipeline.

Why PDF extraction quality is hard to measure

The obvious metric is “did we get text out?” That bar is too low. PyMuPDF can return text from a scanned page — technically successful, practically useless. The output looks plausible but contains garbage characters from bitmap noise, misread glyphs, or copy-protection artifacts.

The opposite failure is equally common. A page with a complex table might yield syntactically correct text but lose every column relationship. The words are real. The structure is gone.

Solid pdf extraction quality metrics have to catch both failure modes. That means measuring not just whether text came out, but whether the text is coherent, complete, and structured correctly for the page type.

pdfmux’s approach is to run five independent checks on the output of every extraction pass. Each check produces a sub-score. Those combine into a single page confidence value. Pages below 80% get flagged and re-routed. Pages above 95% are done.

This is what self-healing pdf extraction looks like in practice: the system detects its own failures and escalates without you writing any retry logic.

The five quality checks

Every page goes through the same audit regardless of which extractor ran first. On a clean digital PDF, the audit adds single-digit milliseconds per page — you won’t notice it.

1. Character yield

This check estimates how much text a page should contain, then measures how much was actually extracted. For digital PDFs, PyMuPDF exposes the font objects on each page. We count the total character slots in those font structures. If extraction returned 200 characters and the font structures account for 800, yield is 25% — a red flag.

For image-heavy pages with no embedded fonts, yield is estimated from the pixel density of dark regions. This is a rougher estimate, but it still catches total failures like “the entire page was a scanned image and we returned nothing.”

2. Printable character ratio

Raw extraction output sometimes contains control characters, null bytes, private-use Unicode codepoints, and other non-printable garbage. This is common with encrypted PDFs that have partial copy protection — PyMuPDF can read the font table but the character mapping is scrambled.

This check counts what fraction of extracted characters fall in the printable Unicode range, including CJK, Cyrillic, Arabic, and other extended blocks. A ratio below 0.85 — meaning more than 15% of characters are garbage — drops page confidence significantly.

Language detection interacts with this check in subtle ways. A page of Korean text has a very different character distribution than English, but both should have near-100% printable ratios. A Korean OCR failure often shows as valid Korean characters interspersed with garbage — still caught, though at a less extreme ratio than a full extraction failure. This is the root of issue #10 on GitHub, where MCP server mode was triggering OCR fallback on Korean documents and producing garbled output. The scoring flagged the problem correctly; the bug was in how the code path selected extractors in server mode, not in the scoring itself.

3. Word recognition rate

Individual characters can be printable without forming real words. This check runs a fast dictionary lookup on a sample of extracted tokens. We don’t tokenize the whole page — that would be too slow. Instead we sample 100–200 tokens from distributed positions across the page and check them against a language-aware word list.

The check adapts to document type. PDFs with lots of numbers — financial reports, scientific papers — would score poorly on strict word recognition, so numeric tokens, units, and common abbreviations get a pass. Code listings and URLs get the same exemption.

A word recognition rate below 0.6 contributes to a lower confidence score. Combined with a low printable character ratio, it’s a near-certain indicator that OCR fallback is warranted.

4. Spatial coverage

For digital PDFs, PyMuPDF returns the bounding boxes of every text block it detected. This check compares the union of those bounding boxes against the page’s content area. If text blocks cover only 20% of a region where text was expected, something is wrong.

This catches a specific failure mode: PDFs where the text layer exists but is positioned incorrectly due to a bad coordinate transform. The characters are there. The layout is broken. Spatial coverage detects the breakage even when character-level checks pass.

For scanned pages with no text layer at all, spatial coverage returns 0% — which immediately flags the page for OCR.

5. OCR engine confidence

When OCR runs — RapidOCR, Surya, or Gemini — the engine returns per-character or per-word confidence values alongside the recognized text. We aggregate those into a page-level OCR confidence score.

This check only applies when an OCR extractor ran. For pages extracted cleanly by PyMuPDF, it’s not computed and doesn’t affect the score. When OCR does run, this score carries high weight. An OCR engine that’s uncertain about its own output is a strong signal that the result is unreliable.

RapidOCR (our default OCR extractor) returns confidence per detected text region. Surya returns it per line. Gemini’s response includes a structured confidence field in the JSON it returns. All three get normalized to the same 0–1 scale before being incorporated into the page score.

How scores drive self-healing pdf extraction

The five check scores combine into a single page confidence value. The combination isn’t a simple average — checks are weighted based on how reliable each one is for the specific page type. Spatial coverage gets lower weight on pages already determined to be image-only. OCR confidence gets higher weight when an OCR extractor ran.

Here’s what happens at each confidence band:

EMPTY is a special case. Pages that return no text — blank pages, pure-image pages, protected pages with no fallback — get classified as EMPTY rather than BAD. They’re re-routed differently: EMPTY pages go straight to OCR, while BAD pages with partial text might go to Docling first for layout-aware re-extraction.

The routing continues up the extractor hierarchy until either a passing confidence score is achieved or all extractors have been tried. If everything fails, the page is included in output with its best available extraction and a low confidence score so your pipeline can handle it explicitly.

This is the mechanic behind the 90/10 insight: 90% of pages in a typical PDF corpus pass the audit on the first PyMuPDF pass. Only 10% need further processing. You pay OCR costs only for the pages that actually need it.

Reading per-page pdf confidence scoring in Python

extract_json returns a DocumentResult dict with full per-page metadata including confidence scores and quality classifications. This is the primary way to access pdf confidence scoring in Python.

import pdfmux

data = pdfmux.extract_json("annual-report.pdf")

for page in data["pages"]:
    print(
        f"Page {page['page_number']}: "
        f"{page['quality']} — {page['confidence']:.0f}% confidence"
    )

Typical output for a mixed document:

Page 1: GOOD — 97% confidence
Page 2: GOOD — 94% confidence
Page 3: BAD — 61% confidence
Page 4: GOOD — 96% confidence
Page 5: EMPTY — 12% confidence

Page 3 triggered OCR. Page 5 had no recoverable text even after escalation. Your pipeline can branch on these values directly:

low_confidence = [
    p for p in data["pages"]
    if p["confidence"] < 80
]

if low_confidence:
    print(f"{len(low_confidence)} pages need manual review")
    for page in low_confidence:
        print(f"  Page {page['page_number']}: {page['confidence']:.0f}%")

The quality field is the string form of PageQuality — "GOOD", "BAD", or "EMPTY". The confidence field is a float from 0 to 100. Both are always present in the output schema, even for EMPTY pages.

The document-level confidence in DocumentResult is the mean confidence across all non-empty pages. Use it as a quick signal — if it’s above 90%, the extraction is clean without needing to inspect every page.

The CLI view

You don’t need to write code to inspect confidence scores. The analyze subcommand gives you a per-page quality triage directly in the terminal.

pip install pdfmux
pdfmux analyze report.pdf

Output:

report.pdf — 47 pages
  45 GOOD   (avg 96.2% confidence)
   1 BAD    (page 23, 61% confidence — OCR applied)
   1 EMPTY  (page 31 — no recoverable text)

Document confidence: 94.1%

This is the fastest way to sanity-check a file before wiring it into a pipeline. If you’re processing a batch and want to triage which files need attention, run pdfmux analyze across the batch and filter on document confidence below your threshold.

The bench subcommand goes deeper — it runs every available extractor on the file and shows the confidence score each one achieves per page.

pdfmux bench report.pdf

That gives you extractor-by-extractor results so you can see the scoring differential between, say, PyMuPDF and RapidOCR on a specific problem page. Useful when a page is consistently scoring below threshold and you want to understand why.

Edge cases that break naive scoring

No scoring system is perfect. Here’s where ours strains.

Dense mathematical notation. Pages full of LaTeX-rendered equations score poorly on word recognition because most tokens aren’t dictionary words. The check is tuned to be less aggressive when spatial layout suggests a math-heavy document, but this is ongoing calibration. If you’re processing scientific papers with heavy equation content, quality="high" with Gemini tends to produce better results than the default pipeline.

Copy-protected PDFs with fake text layers. Some PDFs embed a scrambled text layer specifically to defeat extraction. Characters are printable, yield looks adequate, but words are nonsense. The word recognition check catches this — though at lower confidence than a clean failure. A document where 50% of tokens are nonsense scores around 60–65%, which triggers OCR. OCR on a clear digital PDF is wasteful, but the output is correct.

Scanned handwriting. RapidOCR struggles with handwriting. Surya handles it better. Gemini handles it best. Confidence scoring correctly identifies these pages as needing escalation — OCR confidence comes back low from RapidOCR — but the escalation chain needs to reach Surya or Gemini to produce usable output. If you only have pdfmux[ocr] installed, these pages end up with low-confidence output and no further escalation path. Install pdfmux[llm] for documents with handwritten content.

Mixed-language documents. A document that mixes English body text with Korean headers can confuse the word recognition check if the sampling happens to hit a concentration of non-English tokens. This is a known rough edge. The printable character ratio still catches outright OCR failures, but the confidence value on mixed-language pages is noisier than on single-language documents.

Tuning extraction for your use case

The quality parameter controls how aggressively pdfmux escalates through the extractor hierarchy.

# Fast mode — PyMuPDF only. Audit still runs, no OCR triggered.
text = pdfmux.extract_text("report.pdf", quality="fast")

# Standard mode — default. OCR on bad pages, Docling for tables.
text = pdfmux.extract_text("report.pdf", quality="standard")

# High mode — Gemini Flash first, OCR and PyMuPDF as fallback.
text = pdfmux.extract_text("report.pdf", quality="high")

quality="fast" still runs the audit and returns confidence scores — it just doesn’t act on them by triggering OCR. Use it when you want to assess a file before deciding whether to invest in full extraction.

quality="standard" handles most PDFs correctly. It activates OCR on BAD and EMPTY pages, and uses Docling when table structures are detected.

quality="high" inverts the routing: Gemini Flash runs first, with cheaper extractors as fallback. This costs more per page but produces the highest accuracy on complex layouts, heavily formatted documents, and anything with mixed content types.

The confidence threshold for re-routing is not currently user-configurable through the public API. If your use case needs a different threshold — stricter for high-stakes pipelines, looser for speed — open an issue on GitHub. It’s a reasonable feature request and straightforward to expose.

What the numbers mean for production pipelines

When you’re processing documents at scale, the per-page confidence data is operationally useful beyond “did this page succeed.”

Low-confidence pages clustered across many documents can signal a systematic problem: a batch of scanned files that needed pdfmux[ocr] installed, a class of PDFs with copy protection, a document type that benefits from quality="high". The aggregate confidence distribution tells you where to invest in better tooling before you discover it through downstream data quality issues.

Document-level confidence below 80% is worth routing to a manual review queue in high-stakes pipelines. pdfmux tells you when it’s uncertain. Using that signal rather than ignoring it is the difference between a pipeline that degrades silently and one that surfaces problems while they’re still recoverable.

The benchmarking post covers how different extractors perform across document classes. Confidence scoring is the bridge between those benchmark results and per-page routing decisions. It’s what makes the extractor hierarchy useful rather than just a list of tools — the system knows which pages need escalation and acts on it without waiting for you to notice the output was wrong.

Dive in

pip install pdfmux
pdfmux your-file.pdf

Join the discussion on GitHub. Contributions welcome.