PDF to Pinecone: a production ingestion pipeline in Python

Direct answer: Extract PDFs to Markdown with pdfmux, chunk by heading, embed each chunk, and upsert into a Pinecone serverless index with confidence and source_file stored as metadata — not just the vector. Pinecone has no self-hosted option, so the tradeoff versus pgvector or Qdrant is entirely about ops: no server to run, namespaces give you free per-tenant isolation, and metadata filtering is built into the query call. The extraction and chunking steps are identical across all three pipelines; this post covers what’s specific to Pinecone.

Why Pinecone over self-hosted options

The pgvector post and the Qdrant post both assume you’re willing to run infrastructure — a Postgres extension or a separate ANN service. Pinecone removes that choice: it’s serverless-only as of its current SDK, billed on stored vectors plus read/write units, with no self-hosted or on-prem path. That’s the entire tradeoff. You give up infrastructure control and the option to run air-gapped; in exchange you get a managed index that scales without capacity planning, and namespaces — a first-class partition inside one index that’s the natural fit for multi-tenant RAG (one namespace per customer or per document set), with zero cross-namespace query cost.

If keeping documents on infrastructure you control matters (regulated data, on-prem requirements, or just not wanting a new vendor bill), pgvector or Qdrant are the better fit. If you want to stop thinking about the vector store entirely, Pinecone is built for exactly that.

Setup

pip install pdfmux pinecone openai watchdog

from pinecone import Pinecone, ServerlessSpec

pc = Pinecone(api_key="YOUR_PINECONE_API_KEY")

pc.create_index(
    name="document-chunks",
    dimension=1536,
    metric="cosine",
    spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)

index = pc.Index(name="document-chunks")

dimension=1536 matches OpenAI’s text-embedding-3-small, same as the pgvector and Qdrant setups — swap it for whatever embedding model you’re using. ServerlessSpec is the only index type in the current SDK; the old pod-based indexes are legacy and shouldn’t be used for new projects.

Step 1: extract with confidence

from pdfmux import process

result = process("annual-report.pdf", quality="standard")

print(f"Confidence: {result.confidence:.0%}")
print(f"Extractor used: {result.extractor_used}")

Same self-healing pipeline as every other ingestion path in the pdfmux docs: fast-extract, audit each page, re-extract failures with OCR or a table-aware backend, return one confidence score. On the opendataloader-bench of 200 real-world PDFs this pipeline scores 0.903 overall — full numbers in the benchmark writeup.

Step 2: chunk by heading

Reuse the same heading-based chunker as the pgvector and Qdrant pipelines — the chunking decision doesn’t change based on which vector store you’re loading into:

import re

def chunk_by_headings(markdown: str) -> list[dict]:
    sections = re.split(r'\n(?=#{1,3} )', markdown)
    chunks = []

    for section in sections:
        if len(section.strip()) < 10:
            continue
        lines = section.strip().split('\n')
        heading = lines[0].lstrip('#').strip() if lines[0].startswith('#') else None
        chunks.append({"text": section.strip(), "heading": heading})

    return chunks

For chunk-size tradeoffs and overlap strategy, see PDF chunking strategies for RAG.

Step 3: embed and upsert, gated on confidence, namespaced by tenant

A document extracted at 40% confidence still produces chunks, and if those chunks land in the index next to clean ones, nothing distinguishes them at query time until a user notices a wrong answer. Gate before the upsert, not after. And since Pinecone namespaces are free to create and query in isolation, use one per tenant from the start rather than retrofitting a tenant_id metadata filter later:

from openai import OpenAI

MIN_CONFIDENCE = 0.6
openai_client = OpenAI()


def embed(text: str) -> list[float]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small", input=text
    )
    return response.data[0].embedding


def chunk_id(source_file: str, index_num: int) -> str:
    return f"{source_file}:{index_num}"


def ingest(pdf_path: str, tenant: str):
    result = process(pdf_path, quality="standard")

    if result.confidence < MIN_CONFIDENCE:
        print(f"SKIPPED (confidence {result.confidence:.0%}): {pdf_path}")
        return

    chunks = chunk_by_headings(result.text)
    vectors = [
        {
            "id": chunk_id(pdf_path, i),
            "values": embed(chunk["text"]),
            "metadata": {
                "source_file": pdf_path,
                "heading": chunk["heading"] or "",
                "content": chunk["text"],
                "confidence": result.confidence,
            },
        }
        for i, chunk in enumerate(chunks)
    ]

    index.upsert(vectors=vectors, namespace=tenant)
    print(f"Ingested {len(vectors)} chunks from {pdf_path} into namespace '{tenant}' (confidence {result.confidence:.0%})")

upsert batches naturally — pass a list of vector dicts and Pinecone writes them in one call. For large batches (thousands of chunks), split into batches of a few hundred rather than one giant call, to stay under the request size limit.

Step 4: query with a confidence filter, scoped to a namespace

def search(query: str, tenant: str, min_confidence: float = 0.6, top_k: int = 5):
    query_embedding = embed(query)

    results = index.query(
        vector=query_embedding,
        namespace=tenant,
        top_k=top_k,
        include_metadata=True,
        filter={"confidence": {"$gte": min_confidence}},
    )
    return results.matches


for match in search("what was Q3 revenue?", tenant="acme-corp"):
    m = match.metadata
    print(f"[{match.score:.2f}] {m['source_file']} — {m['heading']} (confidence {m['confidence']:.0%})")

The namespace argument on query() is what makes multi-tenant isolation free: a query against tenant="acme-corp" physically cannot see vectors upserted under a different tenant’s namespace, so there’s no risk of a filter bug leaking one customer’s documents into another’s search results — the kind of bug a WHERE tenant_id = ? clause in pgvector can’t rule out by construction.

Bulk ingestion with pdfmux watch

Identical pattern to the pgvector and Qdrant pipelines — point a watcher at the source directory and call ingest() on each new file:

from watchdog.observers import Observer
from watchdog.events import PatternMatchingEventHandler

class NewPdfHandler(PatternMatchingEventHandler):
    def on_created(self, event):
        ingest(event.src_path, tenant="acme-corp")

observer = Observer()
observer.schedule(NewPdfHandler(patterns=["*.pdf"]), "./incoming", recursive=False)
observer.start()

Run pdfmux watch ./incoming -o ./extracted/ --profile bulk-rag alongside it if you also want extracted output persisted to disk — the extraction inside ingest() hits the same content-hash cache, so it returns fast rather than re-running the full pipeline on a file that’s already been processed.

Handling document updates

Because chunk_id() derives a stable ID from source_file + chunk index, re-ingesting an unchanged document just overwrites the same vectors — upsert is idempotent on ID. But if the new version has a different chunk count (a section was added or removed), stale trailing chunks from the old version won’t get overwritten, only replaced up to the new chunk count. Delete by metadata filter before re-inserting to avoid that:

def reingest(pdf_path: str, tenant: str):
    index.delete(
        namespace=tenant,
        filter={"source_file": {"$eq": pdf_path}},
    )
    ingest(pdf_path, tenant)

Delete-by-metadata-filter is a serverless-index feature — it isn’t available on the legacy pod-based index type, another reason to default to ServerlessSpec for new projects.

Pinecone vs pgvector vs Qdrant — quick reference

Cost model

Pinecone serverless bills on three dimensions — stored vectors, read operations, and write operations — plus a monthly plan minimum, rather than a flat per-page rate like an extraction API. That makes it structurally different from the $0/marginal-cost pgvector and Qdrant self-hosted setups: even an idle Pinecone index with data sitting in it costs storage, and every query() call costs read units regardless of extraction volume. For current per-unit rates, check pinecone.io/pricing directly — Pinecone has changed its pricing structure more than once, so a number quoted here would likely be stale by the time you read it. The practical takeaway: budget for ongoing query volume, not just ingestion volume, when comparing Pinecone’s total cost against a self-hosted option where the only cost is the server itself.

FAQ

Do I need a separate index per tenant, or is a namespace enough? Namespace is enough for the common case — it’s designed exactly for this and query latency doesn’t degrade based on how many namespaces exist in an index. Reach for a separate index only if tenants need different dimension or metric settings, which namespaces can’t vary within one index.

What happens if I upsert a vector with an ID that already exists in the same namespace? Pinecone overwrites it. That’s what makes the chunk_id() scheme above safe for re-ingestion — same file, same chunk index, same ID, so a repeat ingest() call updates in place instead of creating a duplicate.

Can Pinecone embed the text for me instead of calling OpenAI separately? Yes, via upsert_records() with Pinecone’s integrated inference models, which skips the separate embedding call entirely. This pipeline uses explicit embed() + upsert() instead so the same embedding model and code path are shared across the pgvector, Qdrant, and Pinecone pipelines — useful if you ever migrate between them.