Running PDF Extraction Locally with Gemma 4: Zero API Costs, Full Privacy

TL;DRReplace cloud LLM extraction with Gemma running locally via Ollama. Zero API costs, full privacy, 140+ language OCR. Model sizing, hardware guidance, and setup.

Direct answer: Gemma 4 (released April 2, 2026, Apache 2.0 license) can replace a cloud LLM as pdfmux’s fallback extractor, running fully on local hardware with zero API costs and no data leaving the machine. It ships in four sizes (E2B, E4B, 26B MoE, 31B dense), supports multimodal input across 140+ languages, and runs on Ollama with one command. The tradeoff versus a frontier cloud model is quality on the genuinely hard pages — handwriting, degraded scans, faxed forms — in exchange for full privacy and no per-page cost. pdfmux has not benchmarked the two head to head, so the section below tells you how to measure that tradeoff on your own corpus instead of quoting you a number. For law firms, healthcare, finance, and regulated industries where documents cannot leave the premises, the privacy side of that trade usually decides it on its own.

Why local matters now

pdfmux’s default pipeline uses a cloud LLM as the last-resort fallback for pages that PyMuPDF and OCR cannot handle. It is cheap — a fraction of a cent per page at current Google API rates — and accurate. But for three categories of users, cloud is a non-starter no matter the price:

Law firms processing privileged client documents. Sending a confidential merger agreement to Google’s API is malpractice.
Healthcare processing PHI under HIPAA, or patient records under GDPR Article 9. Third-party processors require Business Associate Agreements and an audit trail that most teams would rather avoid.
Finance and regulated industries where data residency rules (UAE PDPL, Saudi PDPL, Swiss FADP, EU GDPR) prohibit processing outside specific jurisdictions.

For these users, the right answer is not “use cloud and hope for the best.” It is “run the model locally.” The open-weight options have historically lagged the cloud frontier on extraction quality by enough to matter; each generation narrows it.

What shipped on April 2, 2026

Google DeepMind released Gemma 4 under the Apache 2.0 license, which means commercial use is permitted with no royalty and no usage restrictions. The release included four model sizes, all weights downloadable. Figures below are from the Gemma 4 model card:

Model	Parameters	Architecture	Context
Gemma 4 E2B	2.3B effective (5.1B with embeddings)	Dense	128K
Gemma 4 E4B	4.5B effective (8B with embeddings)	Dense	128K
Gemma 4 26B A4B	25.2B total, 3.8B active	Mixture of experts (8 of 128 experts active)	256K
Gemma 4 31B	30.7B	Dense	256K

A fifth size, a 12B unified multimodal model (11.95B, dense), was added in June 2026.

All sizes are natively multimodal. They accept image inputs alongside text, which is what makes them usable as OCR engines. The model card states multilingual support across 140+ languages, with out-of-the-box support for 35+.

The MoE model is the interesting one for document extraction: 25.2B total parameters for quality but only 3.8B active at inference, so it runs far closer to E4B speed than its total parameter count suggests.

Memory sizing follows from the parameter count and your quantization, not from anything pdfmux measures — as a rough guide, a 4-bit quantized model needs a little over half a byte per parameter plus context overhead, so the 26B MoE fits comfortably in 24 GB of VRAM and the 31B dense wants closer to 48 GB. Check the actual file size on the Ollama model page before you pull.

How pdfmux uses Gemma

pdfmux’s pipeline routes pages by difficulty:

PyMuPDF digital extraction for clean digital pages. Fast, free, CPU-only.
Tesseract or EasyOCR for scanned pages with Latin-script text.
LLM multimodal extraction for pages that fail the first two (handwriting, complex layouts, tables bleeding across columns, non-Latin scripts at low DPI).

Which backend fills step 3 is selected by provider, on the CLI or through the environment:

# Local, via Ollama
pdfmux convert doc.pdf --llm-provider ollama --llm-model gemma4:e4b

# Cloud
pdfmux convert doc.pdf --llm-provider gemini

The same settings are available as environment variables, which is usually what you want in a server deployment:

export PDFMUX_LLM_PROVIDER=ollama
export PDFMUX_LLM_MODEL=gemma4:e4b

A provider is only usable when its SDK is installed and its credentials are present. Cloud providers read their key from the environment (GEMINI_API_KEY / GOOGLE_API_KEY for Google, and the equivalents for the others), so a machine with no cloud credentials cannot reach a cloud endpoint regardless of configuration. That property is what makes the offline guarantee in the next section real rather than advisory.

If no LLM provider is configured at all, the extractor chain falls through to native extraction (PyMuPDF) rather than failing — you get the free path, not an error.

Hardware requirements per model size

pdfmux does not publish a throughput matrix, and you should be suspicious of anyone who does — tokens/sec depends on your quantization, context length, batch size, and what else is resident in memory, so a table of numbers measured on someone else’s box tells you very little about yours.

What holds regardless of hardware is the sizing rule:

Hardware class	Model that fits
Laptop, 8–16 GB unified memory	E2B, or E4B quantized
Laptop / desktop, 32 GB	E4B comfortably
Workstation GPU, 24 GB VRAM (e.g. RTX 4090)	26B MoE quantized
Server GPU, 48 GB+ (e.g. A6000, H100)	31B dense

Two things matter more than raw tokens/sec for a document pipeline:

Only a fraction of pages ever reach the LLM. pdfmux routes digital pages to PyMuPDF and scanned Latin-script pages to OCR first; the vision model is the fallback. What that fraction is depends entirely on your corpus — a stack of born-digital invoices barely touches it, an archive of faxed forms hits it constantly. Measure yours before sizing hardware, because it drives the answer more than the GPU does.
Throughput is usually not the binding constraint. For most legal, medical, and financial workflows the local path is fast enough, and the reason to choose it is that documents never leave the building.

To get real numbers for your setup, time a representative batch:

time pdfmux convert sample.pdf --llm-provider ollama

Run that across 20–50 documents from your actual corpus and you will have a throughput figure that applies to you.

Quality tradeoff: local Gemma vs cloud Gemini

pdfmux has not run a head-to-head Gemma-vs-Gemini benchmark on a fixed corpus — the head-to-head numbers any vendor would publish are highly sensitive to the document mix, and a single number rarely survives contact with a different mix. The honest tradeoff most teams see when they swap a local open-weight model in for a frontier cloud model on PDF extraction:

Clean digital PDFs — gap is negligible. Both extract markdown cleanly. Pick on cost and privacy, not accuracy.
Good-quality scans — small gap, usually a few percentage points on character error rate. Often acceptable.
Handwriting, degraded scans, faxed forms — gap widens. Frontier multimodal models (Gemini 2.5 Pro, Claude Sonnet, GPT-4o) still beat open-weight VLMs on the genuinely hard pages. If your corpus is dominated by these, the local path will leak content silently — exactly the failure mode the audit-correctness harness is built to surface so you find out at ingest time, not in production.

The bench you should actually run is your own. Pipe ~50 representative documents through pdfmux with --llm-provider ollama (local) and the same set through --llm-provider gemini (cloud), compare the manifest.json output for confidence distribution and per-page audit verdicts, then decide. The eval harness is designed so this takes an afternoon, not a quarter.

For a law firm choosing between “send privileged documents to a cloud API” and “accept some quality delta on local,” the calculus is rarely “is local good enough on average?” — it’s “what is the cost of the documents the local path will silently miss?” Confidence scoring is the lever that makes that question answerable: you can route low-confidence pages to a human reviewer (or to the cloud model under privilege rules) and keep the bulk of the corpus local. Same shape for healthcare under HIPAA and finance under banking secrecy rules.

Ollama setup: full installation

The full local path takes about 15 minutes on a first-time setup.

Step 1: install Ollama

macOS:

brew install ollama
brew services start ollama

Linux:

curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable --now ollama

Windows: download the installer from ollama.com.

Step 2: pull the right Gemma 4 size

Pick based on your hardware:

# Laptop class
ollama pull gemma4:e2b
ollama pull gemma4:e4b

# Workstation class
ollama pull gemma4:12b
ollama pull gemma4:26b

# Server class
ollama pull gemma4:31b

Apple-silicon machines also have -mlx variants of each tag. Download sizes on the gemma4 model page currently span roughly 6.5 GB to 20 GB depending on the tag — check the page rather than trusting a figure written into a blog post, since Ollama re-quantizes and the numbers move.

Quantized variants trade a small amount of quality for a substantially smaller memory footprint. For extraction work that trade is usually worth taking — but if you are near a quality cliff, run both against the same batch and compare, rather than assuming the loss is negligible.

Step 3: install pdfmux with local support

pip install "pdfmux[llm-ollama]"

The llm-ollama extra pulls the Ollama client library.

Step 4: point pdfmux at the local model

Set the provider and model in the environment:

export PDFMUX_LLM_PROVIDER=ollama
export PDFMUX_LLM_MODEL=gemma4:26b

Or save them as a named profile you can reuse (~/.config/pdfmux/profiles.yaml, XDG-aware) so you are not retyping flags on every run:

pdfmux profiles list

The offline guarantee is an environment property, not a config flag. pdfmux does not have a “disable cloud” switch, and you should not go looking for one — a provider is only usable when its SDK is installed and its credentials are present. So the hard gate for a regulated workflow is:

install only the local extra (pdfmux[llm-ollama]), not pdfmux[llm-all]
keep GEMINI_API_KEY, GOOGLE_API_KEY, OPENAI_API_KEY, ANTHROPIC_API_KEY, and MISTRAL_API_KEY out of the process environment

With no cloud SDK and no cloud credentials, there is no code path that reaches a cloud endpoint. That is a stronger guarantee than a config flag, because it does not depend on the flag being read correctly — and it is auditable from outside the process. For air-gapped deployments, the network itself is the third layer.

Step 5: run a test extraction

pdfmux convert contract.pdf --output contract.md

Watch ollama ps in another terminal to confirm the model is being invoked locally. No network traffic, no API calls, no logs to Google. The document is processed on your machine and the output is written to disk.

Cost comparison over a year

The shape of this comparison is simple enough to work out for yourself, and you should — per-token prices change often enough that any table published here is stale by the time you read it.

A worked hypothetical. Assume 50,000 pages per year, and that a page costs you P dollars through a cloud provider:

Cloud annual cost   = 50,000 × P
Local annual cost   = $0 (on hardware you already own)
                    = one-time hardware cost, then $0

Break-even against a new workstation is hardware_cost / P pages. At a tenth of a cent per page that is a lot of pages; at a cent per page it arrives quickly.

To fill in P, read the current rate from the provider directly — Google, OpenAI, Anthropic — and multiply by your average tokens per page, which you can get from a sample batch. For reference, pdfmux’s Gemma provider estimates cost from Google’s published per-million-token rates rather than a per-page figure, because pages vary enormously in token count.

The cost argument is usually not the deciding factor anyway. The privacy and compliance argument is.

When to use which mode

Scenario	Recommended mode
Personal side project, occasional PDFs	Cloud (Gemini Flash default)
SaaS product processing public documents	Cloud
Solo developer on a laptop, privacy-conscious	Local E4B
Legal, healthcare, finance, regulated	Local 26B MoE on workstation
High-volume enterprise (>500K pages/yr)	Local 31B on server
Air-gapped environment (gov, defense)	Local, with no cloud SDK and no cloud credentials installed

For teams already running the PDFMux MCP server in Claude Desktop or Cursor, switching to local Gemma 4 is a one-line config change. The MCP tools are identical. Only the extraction path underneath changes.

Known limitations

Three things Gemma 4 does less well than the frontier cloud models today:

Handwritten text across multiple mixed languages. Gemma 4 handles printed multilingual well; handwritten Arabic or Chinese on low-quality scans is where the gap against frontier cloud models is widest.
Extremely long documents processed as a single context. The context windows are large, but dense documents near the limit tend to degrade. Chunk-and-merge is the safer pattern for long files regardless of model.
Structured extraction with complex nested schemas. Cloud models still have the edge on deeply nested JSON. For flat schemas — invoices, Bills of Lading, forms — the gap narrows considerably.

None of these are blockers, and all of them are things you can check on your own corpus in an afternoon using the comparison recipe above.

Conclusion

Gemma 4 is good enough to serve as pdfmux’s extraction fallback for a large share of real document work. The trade — some quality on the hardest pages, against zero API costs, full privacy, and no data leaving the machine — is the right one for most regulated workflows, and it is a trade you can measure on your own corpus rather than take on faith. Ollama installs in minutes, and pointing pdfmux at the local model is two environment variables.

For law firms, healthcare providers, banks, and anyone else who cannot send documents to a third-party API, this is the setup that finally works.