Converting research papers and arXiv PDFs to Markdown for RAG

Direct answer: For a digital, text-layer arXiv PDF, pdfmux.extract_text() handles the two-column body in correct reading order and returns clean Markdown — no GPU, no LaTeX re-typesetting. For equation-heavy papers where you need math expressions preserved as LaTeX rather than read as plain text, pdfmux isn’t the right tool; use Marker (Apache 2.0 code, GPU recommended) or Meta’s Nougat (MIT code, CC-BY-NC weights) instead. For citation and bibliography metadata specifically — not body text — GROBID is the purpose-built tool. And whenever the paper’s LaTeX source is available, pulling that directly beats extracting from any rendered PDF.

Why academic papers are a distinct extraction problem

A research paper PDF combines several of the harder extraction problems in one document:

• Two-column layout — the dominant academic format, and the classic reading-order trap: a naive extractor interleaves a line from the left column with a line from the right column instead of reading each column top-to-bottom before moving to the next. See multi-column PDF extraction for the general mechanics of this failure.
• Inline citations — [12], (Smith et al., 2024), or superscript numbers that need to stay attached to the sentence they modify, not get stripped as noise.
• Equations — inline math ($x^2$-style expressions rendered as glyphs or embedded images) and display equations with their own numbering, neither of which a text-only extractor represents faithfully.
• Footnotes and a multi-page reference list — content that belongs in a different structural bucket than body prose, and that a naive extractor often just appends wherever it falls in the page’s content stream.
• Figures and tables with captions — where keeping the caption attached to the right figure, not floating loose in the text, is what makes the output usable downstream. See extracting images and figures from a PDF for the caption-pairing problem specifically.

None of these are unique to academic papers individually, but the combination — two columns, dense citations, real equations, and a long reference section, all in one document — is what makes “just run it through a generic PDF-to-Markdown tool” underperform here more than on, say, a single-column business report.

Prefer the LaTeX source when it exists

Before extracting anything from a PDF, check whether the paper has LaTeX source available. For arXiv papers, the source is one request away:

import httpx

arxiv_id = "2511.01234"
source_url = f"https://arxiv.org/e-print/{arxiv_id}"

response = httpx.get(source_url, follow_redirects=True)
# response.content is a gzipped tarball of the LaTeX source, when available

If the source download succeeds, parsing the .tex files directly gives you exact equations, exact citation keys, and exact figure/table markup — there’s no extraction risk because nothing was rendered and re-read. Reserve PDF extraction for the cases where source isn’t available: papers published outside arXiv behind a PDF-only paywall, older preprints predating consistent source submission, or any scanned document with no digital source at all. That’s most of what the rest of this post covers.

Basic extraction for a digital arXiv PDF

pip install pdfmux

import pdfmux

text = pdfmux.extract_text("attention-is-all-you-need.pdf", quality="standard")
print(text[:2000])

On a digital, text-layer PDF — which covers the large majority of papers from the last ~15 years — this handles the two-column reading order automatically and returns Markdown with headings preserved, which matters for downstream chunking by section (Abstract, Introduction, Related Work, Method, Results) rather than by arbitrary token count. See PDF chunking strategies for RAG for the heading-based approach this feeds into.

For per-page structure and a confidence signal — useful when batch-processing a corpus of papers of uneven quality — use extract_json() instead:

data = pdfmux.extract_json("paper.pdf", quality="standard")

print(f"{data['page_count']} pages, confidence {data['confidence']:.0%}")
print(f"Pages that needed OCR: {data['ocr_pages']}")

Older scanned preprints or low-quality photocopied papers will show up as lower confidence and a non-empty ocr_pages list — that’s the signal to route them to manual review or a higher-quality OCR pass rather than trusting the extraction blindly.

Pulling LLM-ready chunks directly

For a RAG pipeline over a paper corpus, load_llm_context() returns section-aware chunks with token estimates in one call, instead of extracting Markdown and chunking it yourself:

chunks = pdfmux.load_llm_context("paper.pdf", quality="standard")

for chunk in chunks:
    print(f"{chunk['title']}: ~{chunk['tokens']} tokens (p.{chunk['page_start']}-{chunk['page_end']})")

This is a reasonable default for most papers. For a corpus where retrieval quality matters enough to justify tuning chunk boundaries yourself (very long papers, survey papers with deep subsection nesting), see the heading-based chunking approach in PDF chunking strategies for RAG instead.

When pdfmux isn’t the right tool

pdfmux reads what’s in the PDF — text, embedded images, table structure — fast and without a GPU. It does not re-typeset equations as LaTeX, and it does not parse the reference list into structured citation metadata (author, year, venue, DOI per reference). Three specialized tools exist for exactly those jobs:

Pick based on what you’re actually trying to get out of the document. If you need the paper’s prose for a RAG index, pdfmux is the fast default. If you need the equations themselves (a math-heavy literature review, a dataset of worked problems), Marker or Nougat are built for that transcription job specifically — both route the page through a vision model and cost real GPU time per page. If you need a structured citation graph — who cites whom, with parsed author/year/venue fields — GROBID is the purpose-built tool; it isn’t trying to give you clean body-text Markdown at all. Nougat’s underlying model is frozen (version-pinned, not under active retraining), which is useful if you need reproducible output across runs — pin the version and expect the same transcription in a year that you got today.

Batch processing a paper corpus

from pathlib import Path
import pdfmux

paths = list(Path("papers/").glob("*.pdf"))

for path, result in pdfmux.batch_extract(paths, output_format="llm", workers=4):
    if isinstance(result, Exception):
        print(f"FAILED {path.name}: {result}")
        continue
    # result holds the chunked output for this paper; write to your vector store

batch_extract yields results as each paper completes rather than blocking on the whole corpus — useful when a literature review corpus mixes recent digital arXiv PDFs (fast) with older scanned preprints (slow, OCR-bound).

What’s measured and what isn’t

The public opendataloader-bench — pdfmux’s 0.903 overall, #2 of 8 engines — draws its 200 documents from four categories: financial filings, academic papers, legal contracts, and government documents. Academic papers are genuinely in the mix, but the published score is blended across all four categories, not broken out per category. There is no academic-papers-only subscore to cite for pdfmux or for any of the other engines in that benchmark. Treat the overall number as directional for papers specifically, not as a papers-specific result — if extraction accuracy on your own paper corpus matters enough to make a tooling decision on, measure it on a sample of your own papers rather than trusting a blended benchmark number to represent one of its four input categories.

For the reading-order mechanics behind two-column extraction, see multi-column PDF extraction. For turning extracted papers into retrieval-ready chunks, see PDF chunking strategies for RAG. For the full engine comparison this post’s table draws from, see PyMuPDF vs Marker and the 200-PDF benchmark.