Certify Anything: verify any PDF extractor for silently dropped pages

Direct answer: pdfmux now ships Certify Anything — a free auditor that verifies any extractor’s output, not just its own. Run pdfmux verify --source doc.pdf --extracted reducto.json, or paste your extraction into the free /audit tool, and you get a certification manifest: per-page coverage, alignment, hallucination-risk, silent-drop detection, table/heading integrity, and a PASS / REVIEW / FAIL verdict. It reuses the same 5-check per-page audit that powers pdfmux’s own self-healing pipeline — pointed at somebody else’s output. It’s in the CLI, the Python API (verify_extraction), and the MCP server (verify_extraction tool). MIT licensed, CPU-only, $0 per page.

The bug every extraction pipeline has and can’t see

Your extractor returned exit code 0. It handed back some JSON. The pipeline moved on.

That is the exact moment most RAG and agent pipelines go wrong. Exit code 0 does not mean “I extracted the document.” It means “I did not crash.” A parser can return success and still have dropped page 7 entirely, flattened a financial table into a wall of whitespace-separated numbers, or read a two-column layout straight across so every row is scrambled. Nothing in the return value tells you. The bytes look fine. The index fills up. Three weeks later a user asks about a number that lived on the page that never made it in, and your agent confidently invents one.

The failure is invisible because the only thing that could catch it — comparing the extraction back against what the source PDF actually contains — is exactly the step nobody runs. You trusted the extractor. That was the whole point of using it.

We know, because our own tool did it

We are not describing a hypothetical. The first version of pdfmux did this to a real customer.

We ran 433 real customer documents — technical sheets and safety data sheets, mixed digital and scanned, some with corrupted text encodings. The naive early pipeline (an early pdfmux CLI, pypdf fallback, no OCR) reported success and produced output for the batch. It had silently dropped 16 of them. Eleven of those had no log line at all — no warning, no error, no trace. The documents were simply not there, and nothing said so.

That failing run was pdfmux’s own early pipeline — our tool failing at the exact thing it promises. We rebuilt it around a per-page audit and a budgeted OCR cascade: every unrecoverable page is now flagged, not dropped. On the same 433 documents, the rebuilt pipeline processed all 433 with zero silent failures. (The full story is in the real-world PDF benchmark.)

The audit we built to stop ourselves from silently dropping pages turned out to be the more valuable half of the product. So we pulled it out and pointed it at everyone else’s output. That’s Certify Anything.

What it actually does

Point pdfmux verify at a source PDF plus an external extraction — produced by Reducto, Mistral OCR, LlamaParse, Docling, an in-house parser, anything — and it re-derives what the source page actually contains, then scores the extraction against it.

It is not a new, hand-wavy “AI judge.” It reuses the same source-of-truth per-page extraction and the same 5-check confidence score that pdfmux uses to audit itself during multi-pass extraction. We are not inventing a new referee; we are pointing the referee we already own at somebody else’s game.

For every page it produces four hard signals:

On top of the per-page signals it detects silent drops (the source page has real text; the extraction returned nothing), flags table and heading integrity problems, and rolls everything up into a single PASS / REVIEW / FAIL verdict plus a SHA-256-anchored, timestamped manifest you can diff across runs and check into version control.

Three ways to run it

1. The CLI

pip install pdfmux

# Certify an external extraction you already have
pdfmux verify --source doc.pdf --extracted reducto.json --engine-name reducto

# Or have pdfmux run an engine and certify the result in one shot
pdfmux verify --source doc.pdf --engine pdfmux

# Batch a whole directory of source PDFs against a directory of outputs
pdfmux verify --source ./pdfs/ --extracted ./outputs/ -o report.json

--extracted takes .json, .md, or .txt (use --extracted-format to force a hint). Write the machine-readable manifest with -o manifest.json and a human-readable summary with --report report.md. Console output defaults to a table; --format json or --format markdown if you’re piping it somewhere.

2. CI gating

The reason silent drops survive is that nothing fails when they happen. --strict fixes that:

pdfmux verify --source contract.pdf --extracted out.json --strict
# exit 0 = ran (check the verdict)
# exit 2 = usage error
# exit 3 = --strict gate failed (verdict was not PASS)

Drop that in your ingestion CI and a regression in your extractor — a dependency bump that quietly breaks table parsing, a new document type your parser can’t handle — fails the build instead of poisoning your index. This is the check your pipeline never had.

3. Inside your agent (MCP)

The MCP server exposes verify_extraction alongside the extraction tools, so an agent in Claude Desktop, Cursor, or Windsurf can certify its own work before it trusts it:

“Extract q3-report.pdf, then verify the extraction and tell me if any pages were silently dropped before you answer.”

The agent calls convert_pdf, then verify_extraction, sees that page 9 came back FAIL on coverage, and tells you it can’t answer the question about page 9 — instead of hallucinating a number. Silent failure is the enemy of a trustworthy agent, and this is the tool that makes the failure loud.

4. The free web tool

No install, nothing to configure: paste or upload at pdfmux.com/audit. Same audit engine, in the browser, free. Bring the output from whatever extractor you’re paying for and get a second opinion in a few seconds.

Honest limits

A verifier that quietly passes things it can’t actually check is worse than no verifier. So Certify Anything only certifies what it can cheaply re-derive from the source. A scanned page with no digital text layer is marked unverifiable, not silently passed — we refuse to certify what we ourselves cannot read. That honesty is the point of the whole feature; it would be absurd to abandon it in the tool’s own output.

Why it’s free

Because the wedge isn’t the certificate — it’s being the thing that watches everything else.

pdfmux competes with Marker, Docling, and LlamaParse on raw extraction quality, and it does well there: 0.903 overall on opendataloader-bench — #2 of all tools and #1 among free, open libraries, within 0.6 points of the paid #1. But “use a better extractor” is a migration, and migrations are slow. “Keep your extractor, and get a free second opinion on it” is a five-second pip install. The verifier meets you where you already are, on the pipeline you already trust, and shows you the pages it’s been dropping.

Every engine that Certify Anything audits — including the paid ones — becomes a reason to have pdfmux installed. That’s worth more to us than charging for the audit.

The 30-second version

pip install pdfmux
pdfmux verify --source yourfile.pdf --engine pdfmux

Run it against whatever you’re using now. If it comes back all PASS, you’ve lost thirty seconds and gained a signed record. If it doesn’t, you just found the pages your pipeline has been quietly losing — the ones you’d otherwise have discovered from a user, three weeks too late.

The extractor that admits when it failed is the one you can build on. Now it admits it for everyone else’s extractor too.

FAQ

Does Certify Anything only work on pdfmux’s output?

No — that’s the entire point. Point it at the output of any extractor: Reducto, Mistral OCR, LlamaParse, Docling, AWS Textract, Azure Document Intelligence, or your own in-house parser. --engine-name just labels which engine is under audit in the manifest.

How is this different from a confidence score?

A confidence score is the extractor grading its own homework — it only knows about the pages it thinks it processed. Certify Anything re-derives the source independently and compares, so it catches the pages the extractor never reported on at all. Silent drops are invisible to self-reported confidence by definition.

Can it produce false confidence on scanned PDFs?

It’s explicitly built not to. Pages with no digital text layer are marked unverifiable rather than passed, so a scanned document can’t sneak through with a green verdict it didn’t earn. You’ll see exactly which pages were checked and which couldn’t be.

Can I gate my CI on it?

Yes. pdfmux verify --strict exits 3 unless the overall verdict is PASS, so a broken extraction fails the build. Exit 0 means it ran (read the verdict), exit 2 is a usage error.

Is the manifest tamper-evident?

The free CLI writes a SHA-256-anchored, timestamped manifest — a content hash you can diff and check into version control to prove an extraction hasn’t changed between runs. Cryptographically-signed, third-party-verifiable manifests (Ed25519) are a separate, patent-pending layer on the Cloud roadmap; the auditing itself is free and MIT-licensed forever.

How does the audit work under the hood?

It reuses pdfmux’s self-healing pipeline — the same per-page audit and source-of-truth extraction that decide when to re-extract during normal processing. Certify Anything runs those signals against an external extraction instead of against pdfmux’s own draft.