AutoGen + pdfmux: a PDF tool your agents can trust

Direct answer: AutoGen has no built-in PDF reader — you either write a plain function tool or reach for a community wrapper that returns raw text and nothing else. Wrap pdfmux instead: extract with process(), return the Markdown plus a 0–100 confidence score, and register it with register_function() so both your assistant agent and your executor agent can call it. The agent then has a number to reason about, not just a string it has to take on faith — the difference between a multi-agent pipeline that silently acts on a garbled scan and one that routes the bad page back for review.

Why “just write a function” isn’t the whole answer

AutoGen’s selling point is that any Python function becomes a tool — decorate it, register it, done. For PDFs, that’s exactly the trap. A one-line function that calls PyPDF2.extract_text() or pdfplumber’s default reader and returns a string will work in your demo, because demos use clean, digital-native PDFs. It falls over the first time a real document — a scanned contract, a dense multi-column report, a low-resolution fax — comes through the pipeline, because the function has no way to tell the calling agent that the extraction was bad. The agent gets a string back and treats it as ground truth, because that’s what a tool result is supposed to be.

This is worse in a multi-agent setup than in a single-agent one. A CrewAI crew or an AutoGen GroupChat passes extracted text between agents — a researcher summarizes it, a critic checks it, a writer drafts from it. If the first agent’s PDF read is garbage, every downstream agent reasons from that garbage with no signal that anything went wrong. The failure doesn’t look like an error. It looks like three agents confidently agreeing on a wrong answer.

The fix is the same one that works for LangChain loaders, CrewAI tools, and MCP clients: give the tool a way to report its own uncertainty, and give the agent a rule for what to do with that number.

Setup

pip install pdfmux pyautogen

pdfmux ships pure-Python extraction backends by default — no GPU, no API key required to get started. See the confidence scoring deep-dive for how the 0–100 score is actually computed.

Writing the tool function

AutoGen doesn’t need a class — a typed function with a docstring is enough for the LLM to understand when and how to call it:

from pdfmux import process


def read_pdf(file_path: str, quality: str = "standard") -> dict:
    """Extract text from a PDF file and report extraction confidence.

    Args:
        file_path: Path to the PDF file to read.
        quality: Extraction quality tier — "fast", "standard", or "high".

    Returns:
        A dict with "text" (the extracted Markdown), "confidence" (0-100),
        and "tier" (the quality tier actually used).
    """
    result = process(file_path, quality=quality)
    return {
        "text": result.text,
        "confidence": round(result.confidence * 100),
        "tier": quality,
    }

Registering it on two agents

AutoGen’s function-calling pattern splits a tool into a caller (the LLM agent that decides to invoke it) and an executor (the agent that actually runs the code). register_function wires both sides at once:

from autogen import ConversableAgent, register_function

assistant = ConversableAgent(
    name="pdf_reader",
    system_message=(
        "You read PDFs using the read_pdf tool. If the returned confidence "
        "is below 60, say so explicitly and ask whether to retry at a higher "
        "quality tier instead of answering from the low-confidence text."
    ),
    llm_config={"config_list": [{"model": "gpt-4o"}]},
)

executor = ConversableAgent(
    name="executor",
    human_input_mode="NEVER",
    code_execution_config=False,
)

register_function(
    read_pdf,
    caller=assistant,
    executor=executor,
    name="read_pdf",
    description="Extract text from a PDF and report a 0-100 confidence score.",
)

The system message is doing real work here — it’s the instruction that turns a bare confidence number into a behavior change. Without it, most models will cite the number if asked but won’t act on it unprompted.

An escalating version

Rather than making the calling agent decide whether to retry, you can bake the retry logic into the tool itself and only return once you’ve hit a confidence floor or run out of tiers:

def read_pdf_escalating(file_path: str, min_confidence: int = 60) -> dict:
    """Extract text from a PDF, automatically retrying at higher quality
    tiers until confidence clears min_confidence or tiers run out."""
    tiers = ["fast", "standard", "high"]

    for attempt, tier in enumerate(tiers, start=1):
        result = process(file_path, quality=tier)
        confidence = round(result.confidence * 100)
        if confidence >= min_confidence or tier == tiers[-1]:
            return {
                "text": result.text,
                "confidence": confidence,
                "tier": tier,
                "attempts": attempt,
            }

    return {"text": "", "confidence": 0, "tier": "high", "attempts": len(tiers)}

This trades latency for reliability: high quality routes more pages through OCR and, for the hardest pages, a BYOK vision LLM backend — slower, and potentially not free. For a GroupChat where a human is waiting on the conversation, that’s usually the right trade. For a batch job, it’s the wrong one, which is the next section.

Batch processing outside the agent loop

Don’t route bulk extraction through agent tool calls. An LLM reasoning about “should I call read_pdf on document 1,847 of 3,000” is slower and more expensive than a plain loop, and it adds nothing — the decision isn’t one that benefits from reasoning. Extract everything first, then hand the agents only the documents that actually need judgment:

import glob
from pdfmux import process


def preprocess_batch(pdf_dir: str, min_confidence: float = 0.6):
    clean, needs_review = [], []

    for path in glob.glob(f"{pdf_dir}/*.pdf"):
        result = process(path, quality="standard")
        entry = {"path": path, "text": result.text, "confidence": result.confidence}
        (clean if result.confidence >= min_confidence else needs_review).append(entry)

    return clean, needs_review

Feed needs_review into a GroupChat where an agent’s judgment — retry at higher quality, flag for a human, discard — actually earns its cost. Feed clean straight into whatever the pipeline does next (summarization, indexing, structured extraction) without spending a single agent turn on documents that already extracted fine.

Using pdfmux’s MCP server instead of a custom function

If you’d rather not maintain a hand-rolled tool function, pdfmux also ships an MCP server exposing the same pipeline as convert_pdf, analyze_pdf, and extract_structured. AutoGen’s autogen-ext package includes an MCP tool adapter that can pull in an MCP server’s tools directly, which saves you from re-declaring argument schemas by hand and keeps you current with pdfmux’s tool surface as it evolves. The tradeoff is the same one as with CrewAI’s MCP adapter: less control over exactly how the confidence number is framed to the agent. Worth it if you’re already wiring up several MCP-compatible tools in one pipeline; less worth it if the system-message framing above is the entire point of the exercise.

Comparing the options

FAQ

Does the agent actually respect the confidence threshold? Only as reliably as any other instruction in a system message — LLMs follow prompted rules probabilistically, not deterministically. For a threshold that must never be crossed (compliance, financial reporting, anything with a human downstream who assumes the text is accurate), enforce it in the function itself rather than trusting the agent to read a number and obey it.

Can multiple agents in the same GroupChat call the same tool function? Yes — register_function can register the same Python function on multiple caller/executor pairs. The function itself holds no per-agent state, so there’s no cross-talk risk.

What about a 200-page PDF that blows past the context window? Chunk before the agent ever sees the text rather than relying on the model to skim a huge extraction in one call. See PDF chunking strategies for RAG for a heading-aware chunker — the same approach used in the pgvector ingestion pipeline — and have the tool return a chunk index the agent can request by section instead of the full text at once.

Is this specific to the classic pyautogen API, or does it work with the newer AgentChat framework too? The pattern — a typed function, a docstring the LLM can read, a confidence field in the return value — is framework-agnostic. AutoGen’s AgentChat FunctionTool wraps a plain Python function the same way; only the registration call changes, not the underlying pdfmux call or the confidence contract.