How to process large PDFs in Python without running out of memory

Direct answer: To process a large PDF in Python without running out of memory, never load all pages into a list at once — iterate page by page and release each page before moving to the next, or stream results to disk as you go. A 2,000-page PDF rendered to images at 300 DPI can hold 8-12 GB in RAM if you accumulate it; processed one page at a time, the same job peaks under 400 MB. The fastest path is pip install pdfmux and pdfmux.extract_streaming("big.pdf"), which yields one NDJSON record per page so memory stays flat regardless of document size. This guide shows the three approaches, their real memory footprints, and the code for each.

Why large PDFs crash standard extractors

The crash is almost never the PDF parser itself — it’s how your code holds the results. Two patterns cause nearly every out-of-memory failure:

1. Accumulating page images. Rendering pages to bitmaps for OCR is memory-hungry. A single 300-DPI Letter-size page is roughly 4-6 MB as an uncompressed RGB array. Convert a 2,000-page PDF with convert_from_path() and you build a list of 2,000 such images — 8-12 GB before you’ve extracted a single character.
2. Building one giant string or list. Appending every page’s text to a list and joining at the end is fine for a 10-page memo and fatal for a 50,000-page archive dump.

The PDF spec allows enormous files — court records, scanned book archives, and engineering document sets routinely exceed 5,000 pages. A 2024 analysis of public-sector document portals found the 99th-percentile PDF size at 1.2 GB. Code that quietly works on a 20-page invoice falls over on these, usually with a MemoryError or a silent kill by the OOM killer on Linux.

The fix is the same idea in every approach: bound your working set to one page at a time.

Three approaches, ranked by memory

The jump from approach 1 to approach 2 is the single biggest win. Approach 3 matters when even the final extracted text is too large to hold, or when a consumer (a vector DB, a queue) can start working before extraction finishes.

Method 1: page-by-page iteration with PyMuPDF

PyMuPDF (pip install pymupdf, v1.24+) opens a document lazily — pages are not parsed until you touch them. The key is to extract and discard each page inside the loop rather than collecting page objects.

import fitz  # PyMuPDF

def extract_streaming(pdf_path: str, out_path: str) -> int:
    doc = fitz.open(pdf_path)
    chars = 0
    with open(out_path, "w", encoding="utf-8") as f:
        for i, page in enumerate(doc):
            text = page.get_text("text")
            f.write(text)
            f.write("\n\n")
            chars += len(text)
            # PyMuPDF caches parsed page content; drop it to keep RAM flat
            doc[i] = None  # release reference so the cache can be reclaimed
    doc.close()
    return chars

total = extract_streaming("big-archive.pdf", "out.txt")
print(f"Wrote {total} characters")

Writing to a file handle inside the loop means the extracted text never accumulates in RAM — it lands on disk and is forgotten. On a 2,000-page digital PDF this peaks around 300 MB regardless of whether the document is 200 or 20,000 pages.

For scanned pages you must rasterize, which is where memory spikes. Render one page, OCR it, and let the pixmap go out of scope before the next iteration:

import pytesseract

def ocr_streaming(pdf_path: str, out_path: str) -> None:
    doc = fitz.open(pdf_path)
    with open(out_path, "w", encoding="utf-8") as f:
        for page in doc:
            pix = page.get_pixmap(dpi=300)        # ~4-6 MB
            text = pytesseract.image_to_string(pix.tobytes("png"))
            f.write(text + "\n\n")
            pix = None                             # free the bitmap now
    doc.close()

Never build images = convert_from_path(path) for a large file — that is exactly the accumulation pattern that causes the crash. If a page is digital, skip OCR entirely; deciding that per page is a routing problem, covered in PDF extraction routing in Python.

Method 2: NDJSON streaming with pdfmux

When you want structured, per-page records you can pipe into a downstream consumer, stream NDJSON (newline-delimited JSON) — one self-contained record per page.

from pdfmux import extract_streaming

for record in extract_streaming("big-archive.pdf"):
    # record is one page: {"page": 1, "text": "...", "confidence": 0.97, ...}
    page_num = record["page"]
    text = record["text"]
    # hand each page to your sink immediately — index, queue, or write
    index_into_vector_db(text, metadata={"page": page_num})

Because each record is yielded and then dropped, peak memory stays under 150 MB even on a 50,000-page file. The generator also lets a consumer start working on page 1 while page 2 is still extracting — useful when feeding a RAG ingestion pipeline where embedding is the slow step.

From the CLI the same stream is one line per page on stdout:

pdfmux convert big-archive.pdf --stream | while read -r line; do
  echo "$line" | jq -r '.text' >> out.txt
done

NDJSON is the right wire format here precisely because it is resumable and line-addressable: if the job dies at page 31,402 you can restart from the last complete line instead of from page 1.

Method 3: handling 10,000-page PDFs (batching + checkpointing)

At extreme sizes, two more concerns appear: the job runs long enough to be interrupted, and you may want parallelism.

Checkpoint by page. Record the last successfully written page so a restart skips completed work.

import os, fitz

def resumable_extract(pdf_path, out_path, ckpt_path):
    start = 0
    if os.path.exists(ckpt_path):
        start = int(open(ckpt_path).read().strip()) + 1
    doc = fitz.open(pdf_path)
    with open(out_path, "a", encoding="utf-8") as f:
        for i in range(start, doc.page_count):
            f.write(doc[i].get_text("text") + "\n\n")
            doc[i] = None
            if i % 100 == 0:                       # checkpoint every 100 pages
                f.flush()
                open(ckpt_path, "w").write(str(i))
    doc.close()

Parallelize by page range, not by page. Split a 10,000-page document into ten 1,000-page ranges and process them in separate worker processes, each opening its own fitz.Document handle (PyMuPDF document objects are not thread-safe, so use processes, not threads). Ten workers turn a 40-minute single-process run into roughly 5 minutes, bounded by disk and CPU rather than memory.

The per-page memory ceiling holds no matter how you slice it: each worker still touches one page at a time, so ten workers peak around 3-4 GB total, not 80 GB.

Quick reference: which approach to use

• Under 50 pages: anything works; load-everything is fine.
• 50-2,000 pages: page-by-page iteration writing to a file (Method 1).
• Over 2,000 pages, or feeding a live consumer: NDJSON streaming (Method 2).
• Over 10,000 pages, or long-running jobs: streaming + checkpointing + range parallelism (Method 3).

For the accuracy side of large-batch extraction — not just memory — see benchmarking PDF extractors, and for routing each page to the cheapest correct parser as you stream, PDF extraction routing in Python.

FAQ

Why does convert_from_path run out of memory on large PDFs?

pdf2image.convert_from_path() renders every page to a PIL image and returns them as a list, so memory scales with page count — roughly 4-6 MB per 300-DPI page. On a 2,000-page PDF that’s 8-12 GB held at once. Use its first_page/last_page arguments to render in small batches, or rasterize one page at a time with PyMuPDF’s get_pixmap() and discard each before the next.

How much RAM do I actually need for a 5,000-page PDF?

With page-by-page iteration, under 400 MB — memory is bounded by the single largest page you hold, not by document size. With the accumulate-everything pattern you’d need 20-30 GB for the same file. The whole point of streaming is that the page count stops mattering for memory.

Does streaming make extraction slower?

No. Throughput is the same — you’re doing the identical per-page work either way. Streaming only changes when you release each result (immediately, vs. at the end). The NDJSON generator can actually feel faster end to end because a downstream consumer starts working on page 1 instead of waiting for the whole document.

Can I extract specific page ranges instead of the whole file?

Yes, and it’s the right move for huge files. PyMuPDF: iterate range(start, end). pdfmux: process("big.pdf", pages=[100, 101, 102]). Page-range extraction is also how you parallelize — give each worker process its own range. Targeting pages also pairs well with detecting which pages are scanned so you only rasterize the ones that need OCR.

What about very large scanned PDFs?

Scanned pages must be rasterized for OCR, which is the memory-heavy step. Render and OCR strictly one page at a time, freeing each pixmap before the next (Method 1, second snippet). On CPU this stays well under 500 MB even for thousands of scanned pages. The OCR engine choice matters for speed — see OCR PDF extraction in Python and PDF extraction without a GPU.

How do I stream a large PDF straight into a vector database?

Use the NDJSON generator (Method 2) and call your embed-and-upsert function inside the loop, one page per iteration. Each page is indexed and dropped before the next is extracted, so a 50,000-page archive ingests with flat memory. This is the recommended pattern for PDF-to-Markdown RAG pipelines.