Detect and redact PII in PDFs before sending them to an LLM (Python)

Direct answer: Redact PII from a PDF after you extract its text and before that text reaches a model — and do the extraction locally so raw PII never leaves your machine in the first place. Detection has three tiers you combine, not choose between: regex catches structured PII with high precision (emails, credit cards validated by the Luhn check, IBANs, phone numbers) but misses names; NER (spaCy or a transformer model) catches unstructured PII — person names, organizations, locations — that no pattern can match; Microsoft Presidio wraps both plus checksum validators and context words into one library and is the pragmatic default. The order that matters: extract text locally (with pdfmux or any offline parser — no API call), detect, redact or tokenize, then send the cleaned text to the LLM. Skip the “local extraction” step and you’ve already leaked the document to a cloud OCR service before your redaction code ever runs. This guide builds the full pipeline in Python.

Why redact before the model, not after

Once text enters an LLM API, you’ve lost control of it. It may be retained in provider logs, surface in a prompt-injection leak, or — on some tiers and older agreements — influence future training. For regulated data (GDPR, the DPDP Act, HIPAA-adjacent workflows) that’s not a hypothetical risk, it’s a compliance finding. Redacting the model’s output is too late; the raw PII already crossed the wire in the input.

So the boundary you defend is the moment before the API call. Everything upstream — extraction, detection, redaction — has to happen on infrastructure you control.

The pipeline order that actually matters

The mistake that quietly defeats an otherwise-correct redaction pipeline is the extraction step. If your PDFs are scanned and you reach for a cloud OCR API to read them, you have shipped the entire unredacted document to a third party before your Presidio code runs. The redaction is real, but it’s guarding a door you already walked the data out of.

The safe order:

1. Extract text  ──►  LOCAL only. No network. (this is where most leaks happen)
2. Detect PII    ──►  regex + NER + validators
3. Redact/tokenize ─► mask, hash, or remove
4. Send to LLM   ──►  cleaned text only

Step 1 is why an offline-by-default extractor matters for privacy specifically. pdfmux runs entirely on your machine — no API keys, no GPU, no network call — including its OCR path for scanned pages. The document’s raw contents never leave the host, so the first thing your PII pipeline touches is text that’s still inside your trust boundary:

from pdfmux import process

# Runs locally. Scanned pages are OCR'd on-device, not shipped to a cloud API.
result = process("customer-contract.pdf", quality="standard")
raw_text = result.text        # PII is still here — and still on your machine

Now the text is local, and detection can begin.

Tier 1: regex for structured PII

Structured PII has predictable shapes, and regex nails them with high precision. The one non-obvious rule: validate, don’t just match. A 16-digit number isn’t a credit card unless it passes the Luhn checksum, and matching without validating floods you with false positives (order numbers, tracking IDs).

import re

EMAIL = re.compile(r"\b[\w.+-]+@[\w-]+\.[\w.-]+\b")
PHONE = re.compile(r"\b(?:\+?\d{1,3}[\s-]?)?(?:\(?\d{2,4}\)?[\s-]?){2,4}\d{2,4}\b")
IBAN  = re.compile(r"\b[A-Z]{2}\d{2}[A-Z0-9]{11,30}\b")
CARD  = re.compile(r"\b(?:\d[ -]*?){13,16}\b")

def luhn_ok(number: str) -> bool:
    digits = [int(d) for d in re.sub(r"\D", "", number)]
    if len(digits) < 13:
        return False
    checksum, parity = 0, len(digits) % 2
    for i, d in enumerate(digits):
        if i % 2 == parity:
            d *= 2
            if d > 9:
                d -= 9
        checksum += d
    return checksum % 10 == 0

def find_structured_pii(text: str) -> list[tuple[str, str, int, int]]:
    hits = []
    for label, rx in (("EMAIL", EMAIL), ("PHONE", PHONE), ("IBAN", IBAN)):
        hits += [(label, m.group(), m.start(), m.end()) for m in rx.finditer(text)]
    for m in CARD.finditer(text):
        if luhn_ok(m.group()):                       # validate before flagging
            hits.append(("CREDIT_CARD", m.group(), m.start(), m.end()))
    return hits

Regex is fast, deterministic, and auditable — you can point to exactly why something matched. Its ceiling is that it can only find PII with a shape. It will never find “the claimant, Priya Sharma, of Jumeirah.”

Tier 2: NER for names, orgs, and places

Person names, company names, and locations have no fixed pattern — you need a model that learned what they look like from context. Named Entity Recognition (NER) fills exactly the gap regex leaves.

import spacy

# en_core_web_lg is a solid CPU default; en_core_web_trf is more accurate, slower.
nlp = spacy.load("en_core_web_lg")

PII_LABELS = {"PERSON", "GPE", "LOC", "ORG", "NORP", "FAC"}

def find_named_pii(text: str) -> list[tuple[str, str, int, int]]:
    doc = nlp(text)
    return [
        (ent.label_, ent.text, ent.start_char, ent.end_char)
        for ent in doc.ents
        if ent.label_ in PII_LABELS
    ]

NER’s tradeoff is the mirror image of regex: it catches the unstructured PII regex can’t, but it’s probabilistic. It misses unusual names, over-flags common words as organizations, and its accuracy drops on the garbled or mis-spaced text that broken PDF extraction produces — another reason the extraction quality upstream matters. Never run NER alone; run it with regex so each covers the other’s blind spot.

Tier 3: Microsoft Presidio (regex + NER + validators, batteries included)

Rather than hand-assemble the two tiers, Microsoft Presidio does it for you: it ships regex recognizers with checksum validators (it runs Luhn on card candidates itself), wraps a spaCy NER model, and adds context-word boosting — a number near the word “SSN” scores higher. It’s the pragmatic default for anything beyond a toy.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import OperatorConfig

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def redact_with_presidio(text: str) -> str:
    results = analyzer.analyze(
        text=text,
        language="en",
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER",
                  "CREDIT_CARD", "IBAN_CODE", "LOCATION", "IP_ADDRESS"],
    )
    return anonymizer.anonymize(
        text=text,
        analyzer_results=results,
        operators={"DEFAULT": OperatorConfig("replace", {"new_value": "[REDACTED]"})},
    ).text

Presidio is extensible — add a recognizer for domain-specific IDs (an Emirates ID, a policy number, an MRN) with a regex and an optional checksum, and it slots into the same pass.

Which tier catches what

The row that matters most for compliance is false negatives — PII you missed. Regex misses names; NER misses odd-shaped IDs. Only the combination (which Presidio gives you out of the box) gets coverage across both axes.

Redaction strategies: mask, tokenize, or remove

Detection tells you where the PII is. What you do with it depends on whether you ever need it back.

• Mask — replace with [REDACTED] or [PERSON]. Irreversible, simplest, safest. Use when the model never needs the real value.
• Tokenize (pseudonymize) — replace each unique value with a stable, reversible token so the LLM can still reason about relationships (“does PERSON_1 appear in both documents?”) without seeing the real name. You keep the mapping in a local vault.
• Remove — delete outright. Cleanest, but destroys sentence structure and can hurt an LLM’s comprehension.

Tokenization is usually the right call for RAG, because it preserves referential structure:

import hashlib

class Tokenizer:
    def __init__(self, salt: str):
        self.salt = salt
        self.vault: dict[str, str] = {}   # token -> original, kept LOCAL

    def token_for(self, value: str, label: str) -> str:
        h = hashlib.sha256((self.salt + value).encode()).hexdigest()[:8]
        token = f"[{label}_{h}]"
        self.vault[token] = value          # store for later re-identification
        return token

def tokenize(text: str, spans, tk: Tokenizer) -> str:
    # Replace right-to-left so earlier offsets stay valid.
    for label, value, start, end in sorted(spans, key=lambda s: s[2], reverse=True):
        text = text[:start] + tk.token_for(value, label) + text[end:]
    return text

The same person becomes the same token everywhere, so retrieval and reasoning still work, but the model only ever sees [PERSON_1a2b3c]. Re-identify from the local vault after the model responds, if you need to.

Don’t forget tables and forms

PII loves to hide in structure. A bank statement’s account number sits in a table cell; a claim form’s applicant name sits in a form field. If your extraction flattens tables into a mangled text blob, regex offsets and NER both degrade. Extract tables as tables and run detection cell-by-cell — you get cleaner matches and you can redact a whole “SSN” column by header rather than hoping the pattern fires on every row.

Common pitfalls

• Cloud OCR before redaction. The single most common leak. If a scanned page goes to a hosted OCR API, the unredacted document is already gone. Extract locally.
• Regex-only pipelines. They feel complete because structured PII is what you see in test files. Names slip through in production. Always add NER.
• Trusting one pass silently. Log a count of detected entities per document and alert on documents that came back with zero — a legal contract with no detected names is a detection failure, not a clean file.
• Over-redaction. Replacing every capitalized word destroys the context RAG needs. Tune entity types to what you actually must protect.
• Forgetting the vault is PII. A reversible token map is a re-identification key. Encrypt it and control access like the original data.

The rule of thumb

Extract locally so PII never leaves your machine, detect with regex and NER together (or Presidio, which is both), tokenize rather than delete when you want RAG to keep working, and gate on the count of detected entities so a silent detection failure doesn’t sail through. The redaction code is the easy part; the discipline is making sure the raw document is still inside your trust boundary when that code runs.

For the extraction foundation this all sits on, see PDF extraction for RAG pipelines — clean, local text is the prerequisite for clean redaction.