Multilingual PDF to Markdown: extract non-English documents in Python

Direct answer: To convert a non-English PDF to Markdown in Python, detect each page’s language first, then route it: pages with a digital text layer go straight to a Markdown formatter, and scanned pages go to OCR with the correct language model loaded. Tesseract 5 supports 100+ languages via per-script trained data, and you pass the codes it needs (for example deu for German or jpn for Japanese) so it does not fall back to English and produce garbage. pdfmux collapses this into one call — it detects the language, selects the engine, preserves headings and tables, and returns Markdown for documents in more than 100 languages without any per-file configuration.

Why “just extract the text” fails on non-English PDFs

Tooling built and benchmarked on English documents makes three assumptions that break the moment you feed it another language:

• It assumes Latin script. OCR trained only on English will read Cyrillic, Greek, CJK, Devanagari, or Arabic as noise. You have to load the right language model per page, not rely on a default.
• It assumes left-to-right, single-script pages. Real documents mix scripts: a Japanese contract with English clause references, a German invoice with a French address block, an Arabic form with Latin product codes. Each run needs correct ordering.
• It throws away structure. A raw text dump loses the headings, tables, and lists that make a document useful to a search index or an LLM. For a retrieval pipeline you want PDF to Markdown for RAG, not a flat string.

The fix is a routing pipeline: detect language, choose the engine, preserve structure. Here is how each stage works.

Step 1: Detect the language per page

Do not assume the whole document is one language. Multi-page packets often switch — a cover letter in one language, an appendix in another. Detect language on the extracted text of each page. The langdetect port of Google’s language-detection library covers 55 languages and is fast enough to run on every page:

import fitz  # PyMuPDF
from langdetect import detect, DetectorFactory

DetectorFactory.seed = 0   # make detection deterministic across runs

doc = fitz.open("contract.pdf")
for i, page in enumerate(doc):
    text = page.get_text("text").strip()
    if len(text) >= 20:
        lang = detect(text)              # e.g. 'de', 'ja', 'ru'
        print(f"page {i}: {lang} ({len(text)} chars)")
    else:
        print(f"page {i}: scanned — needs OCR")

For scanned pages there is no text to detect from yet. In that case, either OCR with a multi-language chain and detect afterward, or use document metadata and filename conventions to seed the guess.

Step 2: Map language codes between libraries

This is where most multilingual pipelines quietly go wrong. langdetect returns ISO 639-1 two-letter codes (de, ja, zh), but Tesseract expects its own three-letter codes (deu, jpn, chi_sim). Pass the wrong one and Tesseract silently loads nothing and falls back to English. Keep an explicit map:

LANG_MAP = {
    "de": "deu", "fr": "fra", "es": "spa", "it": "ita",
    "ru": "rus", "ja": "jpn", "ko": "kor", "zh-cn": "chi_sim",
    "ar": "ara", "hi": "hin", "el": "ell", "pt": "por",
}

def tesseract_lang(iso_code: str) -> str:
    return LANG_MAP.get(iso_code, "eng")

You will need the corresponding trained-data files installed. Tesseract ships more than 100 language packs; the full list and install steps live in the tesseract-ocr repository on GitHub.

Step 3: OCR scanned pages with the right model

Now render each scanned page and OCR it with the detected language chain. Include eng as a secondary so embedded English (URLs, product codes, part numbers) is still recognised:

import io, pytesseract
from PIL import Image

def ocr_page(page, lang: str = "eng", dpi: int = 300) -> str:
    pix = page.get_pixmap(dpi=dpi)
    img = Image.open(io.BytesIO(pix.tobytes("png")))
    chain = f"{lang}+eng" if lang != "eng" else "eng"
    return pytesseract.image_to_string(img, lang=chain, config="--psm 6")

A few language-specific notes:

• CJK (Chinese, Japanese, Korean) needs 300 DPI minimum; dense strokes blur below that.
• Devanagari and Thai use stacked marks that need clean scans to segment correctly.
• Cyrillic and Greek are usually reliable at standard DPI because their glyph shapes are well separated.

Step 4: Convert to structured Markdown

Text alone is not the goal — structure is. After OCR (or after reading the digital text layer), reconstruct headings, tables, and lists so the output is usable downstream. The mapping is straightforward for well-behaved documents:

The hard part is detecting those elements reliably across scripts and layouts, which is where a layout-aware extractor earns its keep. Multi-column pages in particular need column detection before you serialise them, or the reading order interleaves the columns — see multi-column PDF extraction in Python for that specific case.

The one-call version with pdfmux

The four-step pipeline — detect, map codes, OCR, structure — is what pdfmux runs internally. You do not maintain a language-code map or install language packs by hand:

pip install pdfmux
pdfmux convert japanese-report.pdf --output report.md

from pdfmux import convert

result = convert("multilingual-contract.pdf")
print(result.markdown)   # per-page language handled automatically

pdfmux detects each page’s language, routes digital pages and scanned pages down the right path, and returns Markdown with headings and tables intact — across 100+ languages including right-to-left scripts. If your documents are specifically Arabic, the focused guide on Arabic PDF OCR covers reshaping and bidi ordering in detail.

Common failure modes and fixes

• Everything comes out as gibberish English. The Tesseract language code is wrong or the pack is not installed. Verify the ISO-to-Tesseract map and check tesseract --list-langs.
• Mixed-script lines reorder incorrectly. You skipped the bidirectional pass for right-to-left runs. Apply it after OCR.
• Tables collapse into one line. You used a plain text extractor instead of a layout-aware one. Reconstruct columns before serialising.
• Detection flips between two languages on similar pages. Set DetectorFactory.seed for deterministic results and detect on the longest text block, not a short header.
• Accented characters mojibake. Read and write with explicit utf-8 encoding end to end.

Summary

Multilingual PDF extraction is a routing problem, not a single OCR call. Detect the language per page, translate the code to whatever your OCR engine expects, load the right model, and reconstruct structure into Markdown. Do it manually with langdetect plus Tesseract and an explicit code map, or hand the whole thing to pdfmux and get Markdown for 100+ languages from one function call. Either way, the principle is the same: never assume the document is English, and never throw away the structure.