Garbled PDF text in Python: fixing CID fonts, ligatures, and missing spaces

Direct answer: Garbled PDF text almost always traces to one of three root causes, and each has a different fix. (1) CID/Type0 fonts with no ToUnicode CMap return the wrong characters or empty strings — the glyph-to-Unicode map the extractor needs simply isn’t in the file, so switch extractors, because pdfminer.six, PyMuPDF, and pdfplumber reconstruct missing CMaps differently. (2) Missing spaces (“thequickbrownfox”) happen because PDFs position individual glyphs and often store no space characters at all — you rebuild word boundaries from the horizontal gap between character boxes. (3) Ligature soup (fi, ff, or private-use codepoints where “fi” should be) is fixed with Unicode NFKC normalization. The fastest way to know which one you have — and to route bad pages automatically — is to score the extraction: run the file through pdfmux, read result.confidence per page, and only hand-repair the pages that score low. This guide shows how to diagnose and fix all three by hand, and when to stop.

What “garbled” actually means

“My PDF text is garbled” is three different bugs wearing the same coat. Before you fix anything, classify what you’re looking at, because the wrong fix does nothing:

• Mojibake — Ø=Ün where a word should be, or long runs of □/�. The bytes came out, but the character mapping is wrong. This is a font/encoding problem (cause 1).
• Glued words — Netincomeforthequarter. The characters are correct; the spaces are gone. This is a layout/positioning problem (cause 2).
• Ligature holes — nal report where “final report” should be, or a stray fi glyph. Correct-ish characters, but multi-letter glyphs collapsed or vanished. This is a Unicode normalization problem (cause 3).

A quick classifier tells you which bucket a page falls into:

import unicodedata

def diagnose(text: str) -> str:
    if not text.strip():
        return "empty — likely a scan or a font with no extractable mapping"
    replacement = text.count("�") + text.count("□")  # � and □
    if replacement / max(len(text), 1) > 0.05:
        return "mojibake — font/encoding problem (cause 1)"
    words = text.split()
    long_runs = sum(1 for w in words if len(w) > 25)
    if words and long_runs / len(words) > 0.10:
        return "glued words — missing spaces (cause 2)"
    pua = sum(1 for ch in text if 0xE000 <= ord(ch) <= 0xF8FF)
    ligatures = sum(text.count(c) for c in "fffiflffifflſtst")
    if pua or ligatures:
        return "ligature/PUA codepoints — normalization problem (cause 3)"
    return "clean"

Run this per page, not per document. A 60-page filing is often clean for 58 pages and garbled on the two that embedded a subset font.

Cause 1: CID fonts and the missing ToUnicode CMap

Most garbled-text pain comes from CID-keyed (Type0) fonts. When a PDF embeds a subset of a font — only the glyphs the document actually uses — it maps character codes to glyph IDs internally, and it’s supposed to also ship a ToUnicode CMap that maps those codes back to real Unicode. That CMap is what your extractor reads to turn glyph 42 back into the letter “A”.

The problem: the ToUnicode CMap is optional, and plenty of PDF producers omit it or write it wrong. When it’s missing, the extractor has glyph IDs with no idea what characters they represent. Some libraries then emit the raw code points (mojibake), some emit the font’s internal encoding (wrong letters), and some return nothing.

The single most effective fix is to try a different extractor, because they disagree on how to recover:

# Same page, three engines — compare the output, keep the sane one.
import fitz              # PyMuPDF
import pdfplumber
from pdfminer.high_level import extract_text as pdfminer_text

def extract_three_ways(path: str, page: int = 0) -> dict:
    out = {}

    doc = fitz.open(path)
    out["pymupdf"] = doc[page].get_text()
    doc.close()

    with pdfplumber.open(path) as pdf:
        out["pdfplumber"] = pdf.pages[page].extract_text() or ""

    out["pdfminer"] = pdfminer_text(path, page_numbers=[page])
    return out

pdfminer.six tends to be the most forgiving with broken CMaps because it falls back to the font’s built-in encoding tables; PyMuPDF is fastest and usually correct when the CMap exists; pdfplumber (which wraps pdfminer) gives you the per-character boxes you need for cause 2. There is no single winner — the right engine is document-dependent, which is exactly why picking one parser for a whole corpus produces intermittent garbage. (This is the core argument for routing each page to the right extractor.)

When no engine recovers readable text, the font genuinely has no usable mapping. At that point the only reliable path is to rasterize and OCR the page — treat it as if it were scanned, because functionally it is:

import fitz, pytesseract
from PIL import Image
import io

def ocr_fallback(path: str, page: int) -> str:
    doc = fitz.open(path)
    pix = doc[page].get_pixmap(dpi=300)
    img = Image.open(io.BytesIO(pix.tobytes("png")))
    doc.close()
    return pytesseract.image_to_string(img)

That fallback is the whole idea behind self-healing extraction: detect that the text layer is broken, then re-derive the text from pixels instead of trusting a font that lied.

Cause 2: missing spaces (PDFs don’t store words)

A PDF does not contain words. It contains instructions like “draw glyph N at coordinate (x, y).” Word boundaries are an illusion created by the gaps between glyphs. Many producers — especially LaTeX, some InDesign exports, and print drivers — never emit an actual space character; they just move the cursor. A naive extractor that only reads the glyph stream gets Netincomeincreased12%.

The fix is to reconstruct spaces from geometry: if the gap between one character’s right edge and the next character’s left edge exceeds a fraction of the font size, insert a space.

import pdfplumber

def extract_with_reconstructed_spaces(path: str, page: int = 0,
                                      space_ratio: float = 0.25) -> str:
    with pdfplumber.open(path) as pdf:
        chars = pdf.pages[page].chars

    if not chars:
        return ""

    out = []
    prev = None
    for ch in chars:
        if prev is not None:
            gap = ch["x0"] - prev["x1"]
            # A gap wider than 25% of glyph height ≈ a word break.
            if gap > space_ratio * ch["height"]:
                out.append(" ")
            # A downward jump means a new line.
            if ch["top"] - prev["top"] > prev["height"] * 0.5:
                out.append("\n")
        out.append(ch["text"])
        prev = ch
    return "".join(out)

Tune space_ratio per document class — condensed fonts need a lower threshold, wide-tracked display type a higher one. PyMuPDF’s get_text("words") and get_text("dict") modes apply a similar heuristic internally and are a good default before you hand-roll this; reach for the character-box version when the built-in one still glues specific columns together.

Cause 3: ligatures and private-use codepoints

Typographic ligatures combine letters into one glyph: fi, fl, ffi, ff, ft. When a font maps these to the Unicode ligature block (U+FB00–FB06), you get a stray fi that breaks search and tokenization. Worse, some fonts map ligatures into the Private Use Area (U+E000–F8FF), so “final” extracts as “\uE0Anal” — a codepoint with no standard meaning at all.

Standard ligatures are a one-liner: NFKC normalization decomposes them back to their component letters.

import unicodedata

raw = "The final affidavit was flagged."   # fi, ffi, fl
clean = unicodedata.normalize("NFKC", raw)
print(clean)   # The final affidavit was flagged.

NFKC handles the standard ligature block, full-width Latin, and many compatibility characters in one pass, so run it on every extraction as cheap insurance — it’s idempotent and won’t harm clean text.

Private-use-area ligatures are harder because the mapping is font-specific and non-standard. There’s no general rule; you build a small remap table for the fonts that actually appear in your corpus:

PUA_FIXUPS = {
    "\uE000": "fi",   # example codepoints — the real ones are font-specific
    "\uE001": "fl",
    "\uE002": "ffi",
}

def fix_pua(text: str) -> str:
    for bad, good in PUA_FIXUPS.items():
        text = text.replace(bad, good)
    return unicodedata.normalize("NFKC", text)

Building that table by hand is tedious, which is the point at which most teams decide the per-font archaeology isn’t worth it and switch to an extractor that scores its own output and flags the pages that need attention.

Which extractor handles what

No single library is best at all three failure modes. This is the practical breakdown:

The takeaway isn’t “always use X.” It’s that the correct engine changes page to page, and you can’t know in advance which pages will break. That’s an argument for measuring extraction quality rather than assuming it.

Detect garbled output automatically

The dangerous failure isn’t garbled text — it’s garbled text you don’t notice, because it flows straight into a RAG index and quietly returns wrong answers weeks later. Gate on a quality signal instead of eyeballing samples.

A crude but effective heuristic is the ratio of “sane” characters plus a dictionary hit-rate check:

import re

def looks_garbled(text: str) -> bool:
    if not text.strip():
        return True
    printable = sum(ch.isprintable() or ch.isspace() for ch in text)
    if printable / len(text) < 0.90:
        return True
    words = re.findall(r"[A-Za-z]{2,}", text)
    if not words:
        return True
    vowelless = sum(1 for w in words if not re.search(r"[aeiouAEIOU]", w))
    return vowelless / len(words) > 0.30   # real English words mostly have vowels

This is the manual version of what a scoring extractor does for you. pdfmux computes a per-page confidence score from several such signals and routes low-confidence pages to a stronger backend automatically:

from pdfmux import process

result = process("report.pdf", quality="standard")
for page in result.pages:
    if page.confidence < 0.80:
        print(f"page {page.number}: {page.confidence:.0%} — flagged for review")

Instead of running three extractors by hand and writing your own garbled-text detector, you read one number per page and only intervene where it’s low. The pages that scored 0.95 you never touch; the two that scored 0.40 get flagged before they poison anything downstream.

Common pitfalls

• Fixing the wrong cause. Running NFKC on a missing-spaces problem does nothing. Diagnose first (the diagnose() function above), then fix.
• Assuming empty means scanned. An empty extract can be a scan or a CID font with no mapping. Check whether the page has image objects before you reach for OCR.
• Normalizing too aggressively. NFKC is safe; NFKD (which strips accents) is not — it will turn “café” into “cafe” and corrupt non-English text. Use NFKC.
• Trusting one extractor’s silence. A library returning clean-looking text on a broken font is worse than an obvious crash, because it’s invisible. This is why we now audit every extraction rather than trust a single pass.

The rule of thumb

Classify before you fix: mojibake is a font problem (switch extractors, then OCR as fallback), glued words are a positioning problem (rebuild spaces from character geometry), ligature holes are a Unicode problem (NFKC). Run NFKC on everything as free insurance. And past a few documents, stop hand-diagnosing — gate on a confidence score so the clean pages cost you nothing and only the genuinely broken ones reach your desk.

Once the text comes out clean, the next decision is how to split it — see PDF chunking strategies for RAG for turning clean pages into good retrieval.