Multilingual PDF to Markdown: extract non-English documents in Python

Direct answer: To convert a non-English PDF to Markdown in Python, detect each page’s language first, then route it: pages with a digital text layer go straight to a Markdown formatter, and scanned pages go to OCR with the correct language model loaded. Tesseract 5 supports 100+ languages via per-script trained data, and you pass the codes it needs (for example deu for German or jpn for Japanese) so it does not fall back to English and produce garbage. pdfmux collapses this into one call — it detects the language, selects the engine, preserves headings and tables, and returns Markdown for documents in more than 100 languages without any per-file configuration.

Why “just extract the text” fails on non-English PDFs

Tooling built and benchmarked on English documents makes three assumptions that break the moment you feed it another language:

• It assumes Latin script. OCR trained only on English will read Cyrillic, Greek, CJK, Devanagari, or Arabic as noise. You have to load the right language model per page, not rely on a default.
• It assumes left-to-right, single-script pages. Real documents mix scripts: a Japanese contract with English clause references, a German invoice with a French address block, an Arabic form with Latin product codes. Each run needs correct ordering.
• It throws away structure. A raw text dump loses the headings, tables, and lists that make a document useful to a search index or an LLM. For a retrieval pipeline you want PDF to Markdown for RAG, not a flat string.

The fix is a routing pipeline: detect language, choose the engine, preserve structure. Here is how each stage works.

Step 1: Detect the language per page

Do not assume the whole document is one language. Multi-page packets often switch — a cover letter in one language, an appendix in another. Detect language on the extracted text of each page. The langdetect port of Google’s language-detection library covers 55 languages and is fast enough to run on every page:

import fitz  # PyMuPDF
from langdetect import detect, DetectorFactory

DetectorFactory.seed = 0   # make detection deterministic across runs

doc = fitz.open("contract.pdf")
for i, page in enumerate(doc):
    text = page.get_text("text").strip()
    if len(text) >= 20:
        lang = detect(text)              # e.g. 'de', 'ja', 'ru'
        print(f"page {i}: {lang} ({len(text)} chars)")
    else:
        print(f"page {i}: scanned — needs OCR")

For scanned pages there is no text to detect from yet. In that case, either OCR with a multi-language chain and detect afterward, or use document metadata and filename conventions to seed the guess.

Step 2: Map language codes between libraries

This is where most multilingual pipelines quietly go wrong. langdetect returns ISO 639-1 two-letter codes (de, ja, zh), but Tesseract expects its own three-letter codes (deu, jpn, chi_sim). Pass the wrong one and Tesseract silently loads nothing and falls back to English. Keep an explicit map:

LANG_MAP = {
    "de": "deu", "fr": "fra", "es": "spa", "it": "ita",
    "ru": "rus", "ja": "jpn", "ko": "kor", "zh-cn": "chi_sim",
    "ar": "ara", "hi": "hin", "el": "ell", "pt": "por",
}

def tesseract_lang(iso_code: str) -> str:
    return LANG_MAP.get(iso_code, "eng")

You will need the corresponding trained-data files installed. Tesseract ships more than 100 language packs; the full list and install steps live in the tesseract-ocr repository on GitHub.

langdetect vs fastText for language detection

langdetect is the easiest starting point — pure Python, no model download, covers 55 languages, and the seeded version above is deterministic. It struggles on two cases: very short strings (under ~20 characters, which is why the snippet skips them) and closely related languages, where it will occasionally confuse Croatian and Serbian, or Indonesian and Malay.

If you are processing high volumes or need better accuracy on short fragments, swap in Meta’s fastText language-identification model instead. The official lid.176.bin model covers 176 languages in a single ~126 MB file and, being a linear classifier rather than a full statistical model, is fast enough to run on every page without a noticeable pipeline slowdown. The trade-off is an extra dependency and a model file to vendor with your deployment — for most document pipelines, langdetect is the right default and fastText is the upgrade once misclassifications start showing up in production.

Step 3: OCR scanned pages with the right model

Now render each scanned page and OCR it with the detected language chain. Include eng as a secondary so embedded English (URLs, product codes, part numbers) is still recognised:

import io, pytesseract
from PIL import Image

def ocr_page(page, lang: str = "eng", dpi: int = 300) -> str:
    pix = page.get_pixmap(dpi=dpi)
    img = Image.open(io.BytesIO(pix.tobytes("png")))
    chain = f"{lang}+eng" if lang != "eng" else "eng"
    return pytesseract.image_to_string(img, lang=chain, config="--psm 6")

A few language-specific notes:

• CJK (Chinese, Japanese, Korean) needs 300 DPI minimum; dense strokes blur below that.
• Devanagari and Thai use stacked marks that need clean scans to segment correctly.
• Cyrillic and Greek are usually reliable at standard DPI because their glyph shapes are well separated.

Step 4: Convert to structured Markdown

Text alone is not the goal — structure is. After OCR (or after reading the digital text layer), reconstruct headings, tables, and lists so the output is usable downstream. The mapping is straightforward for well-behaved documents:

The hard part is detecting those elements reliably across scripts and layouts, which is where a layout-aware extractor earns its keep. Multi-column pages in particular need column detection before you serialise them, or the reading order interleaves the columns — see multi-column PDF extraction in Python for that specific case.

The one-call version with pdfmux

The four-step pipeline — detect, map codes, OCR, structure — is what pdfmux runs internally. You do not maintain a language-code map or install language packs by hand:

pip install pdfmux
pdfmux convert japanese-report.pdf --output report.md

from pdfmux import convert

result = convert("multilingual-contract.pdf")
print(result.markdown)   # per-page language handled automatically

pdfmux detects each page’s language, routes digital pages and scanned pages down the right path, and returns Markdown with headings and tables intact — across 100+ languages including right-to-left scripts. If your documents are specifically Arabic, the focused guide on Arabic PDF OCR covers reshaping and bidi ordering in detail. For agent-facing pipelines that need to call this from Claude or Cursor rather than a script, see how to give your AI agent the ability to read any PDF.

RTL beyond Arabic: Hebrew and Urdu

Arabic gets most of the attention because it is the most common right-to-left script in business documents, but Hebrew and Urdu need the same bidi handling and neither uses Arabic’s contextual letter-shaping. Two differences to build for:

• Hebrew has no letter-shaping problem — each character has one glyph regardless of position — but numbers and Latin-script product codes embedded in Hebrew text still need the bidirectional (BiDi) reordering pass, or they render reversed.
• Urdu is written in a Nastaliq or Naskh style Arabic script and inherits Arabic’s contextual shaping and its bidi requirements, but Tesseract’s urd model is trained separately from ara — using the Arabic pack on Urdu text degrades accuracy noticeably, so map the detected language to its own Tesseract code rather than collapsing every RTL script to ara.

Route both through the same OCR-plus-bidi pipeline as Arabic; just make sure the language map from Step 2 sends each script to its own trained-data file.

Validating output when you don’t read the language

The hardest part of a multilingual pipeline is knowing whether the output is actually correct when nobody on the team reads Japanese, Thai, or Amharic. A few checks that don’t require fluency:

• Character-set ratio. If you detected Cyrillic but the output is more than a few percent Latin characters (excluding numbers and punctuation), the OCR fell back to the wrong language pack.
• Encoding round-trip. Write the output to UTF-8, read it back, and diff — a silent mojibake bug usually shows up as a byte-count mismatch, not an exception.
• Confidence scoring, not just text. A plain OCR call gives you a string with no signal about which pages it struggled on. pdfmux’s verify_extraction and certification tooling — covered in how to certify any PDF extractor’s output and fixing garbled PDF text in Python — score each page so low-confidence pages get flagged for review instead of shipping silently wrong text into a downstream index or contract pipeline (see contract data extraction in Python for a case where that matters).

Common failure modes and fixes

• Everything comes out as gibberish English. The Tesseract language code is wrong or the pack is not installed. Verify the ISO-to-Tesseract map and check tesseract --list-langs.
• Mixed-script lines reorder incorrectly. You skipped the bidirectional pass for right-to-left runs. Apply it after OCR.
• Tables collapse into one line. You used a plain text extractor instead of a layout-aware one. Reconstruct columns before serialising.
• Detection flips between two languages on similar pages. Set DetectorFactory.seed for deterministic results and detect on the longest text block, not a short header.
• Accented characters mojibake. Read and write with explicit utf-8 encoding end to end.
• A document mixes a digital text layer with scanned appendix pages. Do not OCR the whole file uniformly — detect per page whether page.get_text("text") returns real content; only fall back to OCR for pages where it’s empty or near-empty, otherwise you throw away the higher-accuracy digital layer for the pages that had one.
• Right-to-left text extracts in the wrong visual order even after a bidi pass. Check that the bidi algorithm is being applied to logical order, not to already-reordered visual text — running it twice un-reverses a correct result.

When language detection isn’t worth building

If every document in your pipeline is contractually guaranteed to be one language — a single company’s invoices, a government form template — skip per-page detection entirely and hardcode the Tesseract language chain. Detection adds a dependency and a failure mode (misclassification) for zero benefit when the language never varies. Reach for the routing pipeline in this post only when you genuinely don’t know what’s coming in: aggregated intake from multiple sources, user uploads, or archives spanning several countries.

Summary

Multilingual PDF extraction is a routing problem, not a single OCR call. Detect the language per page, translate the code to whatever your OCR engine expects, load the right model, and reconstruct structure into Markdown. Do it manually with langdetect plus Tesseract and an explicit code map, or hand the whole thing to pdfmux and get Markdown for 100+ languages from one function call. Either way, the principle is the same: never assume the document is English, and never throw away the structure.