PDF to Weaviate: hybrid search ingestion in Python

Direct answer: To get PDFs into Weaviate, extract clean Markdown with pdfmux (pip install pdfmux), chunk it by heading boundary, embed each chunk, and insert it into a Weaviate collection with the extraction confidence stored as a numeric property. The reason to choose Weaviate over Chroma, pgvector, or Pinecone is hybrid search: collection.query.hybrid() blends vector similarity with BM25 keyword matching in a single call, which matters for PDF content full of exact-match terms — invoice numbers, SKUs, named entities — that pure vector search alone tends to miss. The extraction and chunking steps are identical to every other pdfmux ingestion pipeline; what’s specific to Weaviate is the collection schema and the hybrid query.

Why hybrid search matters for PDF content

Pure vector search is good at “find me something semantically similar” and bad at “find me the document containing exactly this part number.” A PDF corpus — invoices, contracts, compliance filings, technical manuals — is full of tokens where exact match is the right behavior: a customer ID, a clause number, a product SKU. Embed those into a dense vector and semantically-similar-but-wrong results can outrank the one chunk that contains the literal string. BM25 keyword scoring catches exactly these cases; it’s what full-text search engines have used for decades because it rewards exact term overlap.

Weaviate bakes both into one query path. collection.query.hybrid(query=text, vector=embedding, alpha=0.5) runs a BM25 search and a vector search against the same object set and fuses the ranked lists, with alpha controlling the blend — alpha=1.0 is pure vector, alpha=0.0 is pure keyword. Chroma, pgvector, and Qdrant all support vector search natively; getting the same fused ranking out of them means standing up a separate full-text index and reranking client-side yourself. If your PDFs are full of exact-match identifiers, that’s the gap Weaviate closes without extra infrastructure.

Setup

pip install pdfmux weaviate-client openai

Run Weaviate locally with Docker for development:

docker run -d -p 8080:8080 -p 50051:50051 \
  -e QUERY_DEFAULTS_LIMIT=25 \
  -e DEFAULT_VECTORIZER_MODULE=none \
  cr.weaviate.io/semitechnologies/weaviate:latest

DEFAULT_VECTORIZER_MODULE=none matters: pdfmux’s pipeline produces the text, and you’re bringing your own embeddings (via OpenAI or any other model), so Weaviate shouldn’t try to vectorize anything itself.

import weaviate
from weaviate.classes.config import Configure, Property, DataType

client = weaviate.connect_to_local()

client.collections.create(
    name="DocumentChunk",
    properties=[
        Property(name="content", data_type=DataType.TEXT),
        Property(name="heading", data_type=DataType.TEXT),
        Property(name="source_file", data_type=DataType.TEXT),
        Property(name="confidence", data_type=DataType.NUMBER),
        Property(name="extractor_used", data_type=DataType.TEXT),
    ],
    vectorizer_config=Configure.Vectorizer.none(),
)

collection = client.collections.get("DocumentChunk")

Swap connect_to_local() for weaviate.connect_to_weaviate_cloud(cluster_url=..., auth_credentials=Auth.api_key(...)) to use Weaviate Cloud instead — the collection schema and every query below stay the same.

Step 1: extract with confidence

from pdfmux import process

result = process("annual-report.pdf", quality="standard")

print(f"Confidence: {result.confidence:.0%}")
print(f"Extractor used: {result.extractor_used}")

Same self-healing pipeline as the rest of this series: fast-extract with PyMuPDF, audit each page, re-extract anything that fails the audit with OCR or a table-aware backend, and return one confidence score for the document. On the opendataloader-bench of 200 real-world PDFs this pipeline scores 0.903 overall — full numbers in the benchmark writeup.

Step 2: chunk by heading

The same heading-based chunker as the pgvector, Pinecone, Qdrant, and Chroma pipelines — the chunking logic doesn’t change based on the destination store:

import re

def chunk_by_heading(markdown: str, source_file: str) -> list[dict]:
    sections = re.split(r"(?m)^(#{1,3}\s+.+)$", markdown)
    chunks = []
    current_heading = "Introduction"
    for part in sections:
        if re.match(r"^#{1,3}\s+", part):
            current_heading = part.lstrip("#").strip()
            continue
        text = part.strip()
        if len(text) < 40:
            continue
        chunks.append({
            "heading": current_heading,
            "text": text,
            "source_file": source_file,
        })
    return chunks

Step 3: embed and insert, gated on confidence

A document extracted at low confidence still produces chunks — the text is just garbled or partial where extraction failed. Inserted alongside clean chunks, nothing distinguishes them in a plain similarity ranking, and in a hybrid search they’re just as capable of winning the BM25 half on a keyword match even when the surrounding context is wrong. Gate at insert time:

from openai import OpenAI

openai_client = OpenAI()
MIN_CONFIDENCE = 0.6

def embed(text: str) -> list[float]:
    response = openai_client.embeddings.create(
        model="text-embedding-3-small",
        input=text,
    )
    return response.data[0].embedding

def ingest(pdf_path: str, collection) -> None:
    result = process(pdf_path, quality="standard")

    if result.confidence < MIN_CONFIDENCE:
        print(f"SKIPPED (confidence {result.confidence:.0%}): {pdf_path}")
        return

    chunks = chunk_by_heading(result.text, pdf_path)
    if not chunks:
        print(f"No chunks extracted from {pdf_path}")
        return

    with collection.batch.dynamic() as batch:
        for chunk in chunks:
            batch.add_object(
                properties={
                    "content": chunk["text"],
                    "heading": chunk["heading"],
                    "source_file": chunk["source_file"],
                    "confidence": result.confidence,
                    "extractor_used": result.extractor_used,
                },
                vector=embed(chunk["text"]),
            )

    print(f"Ingested {len(chunks)} chunks from {pdf_path} (confidence {result.confidence:.0%})")

collection.batch.dynamic() batches the inserts and handles retries internally — worth using over individual collection.data.insert() calls as soon as you’re loading more than a handful of chunks at a time.

Step 4: hybrid query with confidence filtering

from weaviate.classes.query import Filter, MetadataQuery

def search(query: str, collection, min_confidence: float = 0.7, limit: int = 5):
    response = collection.query.hybrid(
        query=query,
        vector=embed(query),
        alpha=0.5,
        limit=limit,
        filters=Filter.by_property("confidence").greater_or_equal(min_confidence),
        return_metadata=MetadataQuery(score=True),
    )
    for obj in response.objects:
        props = obj.properties
        print(f"[{obj.metadata.score:.2f}] {props['source_file']} — {props['heading']} "
              f"(extraction confidence {props['confidence']:.0%})")


search("invoice number INV-20394", collection, min_confidence=0.7)

alpha=0.5 weights vector and keyword scoring evenly — raise it toward 1.0 for queries that are mostly conceptual (“what did the report say about supply chain risk”), lower it toward 0.0 for queries built around an exact token like the invoice number above. Filter.by_property("confidence").greater_or_equal(...) applies to hybrid, pure-vector (near_vector), and pure-keyword (bm25) queries identically — the confidence gate travels with the data regardless of which query mode you reach for.

Combining filters

Real queries usually need more than one condition — “only this customer’s documents, and only chunks above a confidence floor.” Weaviate’s Filter objects compose with & and | directly in Python, instead of a separate filter-building mini-language:

combined = (
    Filter.by_property("confidence").greater_or_equal(0.7)
    & Filter.by_property("source_file").equal("annual-report.pdf")
)

response = collection.query.hybrid(
    query="revenue guidance for next quarter",
    vector=embed("revenue guidance for next quarter"),
    alpha=0.5,
    limit=5,
    filters=combined,
    return_metadata=MetadataQuery(score=True),
)

& is a logical AND, | is a logical OR, and they nest, so (Filter.by_property("confidence").greater_or_equal(0.7)) & (Filter.by_property("source_file").equal("a.pdf") | Filter.by_property("source_file").equal("b.pdf")) is valid if you need to scope a query to a specific set of source documents above a confidence floor.

Picking alpha in practice

There’s no single correct alpha for a whole corpus — the right value tracks how identifier-dense the query is, not the document type. A question like “summarize the risk factors section” is almost entirely conceptual and benefits from alpha closer to 1.0, where vector similarity dominates. A query built around a specific string — an invoice number, a contract clause reference, a part number — needs alpha closer to 0.0 so BM25’s exact-match scoring isn’t diluted by semantically-close-but-wrong vector neighbors. If you don’t want to tune this per query, a reasonable default is alpha=0.5 and a one-line heuristic: if the query contains a token matching an identifier pattern you care about (a regex for invoice-number formats, SKUs, case numbers), drop alpha toward 0.2; otherwise leave it at 0.5.

Scaling past a single Docker container

The local Docker setup above is a single-node Weaviate instance — fine for development and for corpora that fit comfortably on one machine. Production workloads typically move to either a multi-node self-hosted cluster (Weaviate supports horizontal scaling with replication) or Weaviate Cloud, which runs that cluster for you. Neither migration touches the collection schema or the query code above: connect_to_local() becomes connect_to_weaviate_cloud(), and everything from the Property definitions through the hybrid() call is unchanged. The confidence gate on every object means that migration also doesn’t require re-auditing old data — a chunk inserted at 55% confidence stays filterable out of results after the move, exactly as it was before it.