CrewAI + pdfmux: a PDF tool that tells your agent when it's wrong

Direct answer: CrewAI’s built-in file tools return raw text and nothing else — no signal about whether that text is any good. Wrap pdfmux in a custom BaseTool instead: extract with process(), return the Markdown plus a confidence score in the tool output, and give the agent a rule in its task description (“if confidence is below 0.6, flag the document instead of answering from it”). That one field is the difference between an agent that silently answers from a garbled OCR pass and one that knows to ask for help.

Why the default file tools aren’t enough

CrewAI ships a FileReadTool and a few PDF-adjacent community tools that call a text extractor once and hand the result to the agent. That works for a clean, digital-native PDF. It falls apart on anything else — a scanned contract, a table-heavy financial report, a multi-column academic paper — because the tool has no way to tell the agent the extraction went badly. The agent receives a string. It doesn’t know if that string is a faithful transcript or a wall of misread glyphs from a low-resolution scan.

This matters more for agents than for a one-off script. A human skimming garbled OCR output notices immediately and reaches for a different PDF. An agent in a loop treats the string as ground truth and reasons from it, because that’s what agents do with tool output. The failure doesn’t surface as an error — it surfaces three steps later as a wrong answer that looks confident.

The fix is the same one that works for LangChain loaders and MCP clients: give the tool a way to report its own uncertainty, and give the agent a rule for what to do with that number.

Setup

pip install pdfmux crewai

Building the tool

CrewAI tools subclass BaseTool from crewai.tools. You define an input schema with Pydantic and implement _run:

from crewai.tools import BaseTool
from pydantic import BaseModel, Field
from pdfmux import process


class PdfReadInput(BaseModel):
    file_path: str = Field(..., description="Path to the PDF file to read")
    quality: str = Field(
        default="standard",
        description="Extraction quality tier: fast, standard, or high",
    )


class PdfReadTool(BaseTool):
    name: str = "read_pdf"
    description: str = (
        "Extracts text and tables from a PDF file. Returns Markdown content "
        "plus a confidence score (0-1). Confidence below 0.6 means the "
        "extraction is unreliable — treat the content with suspicion and "
        "say so rather than answering from it directly."
    )
    args_schema: type[BaseModel] = PdfReadInput

    def _run(self, file_path: str, quality: str = "standard") -> str:
        result = process(file_path, quality=quality)

        output = [
            f"[extraction confidence: {result.confidence:.0%}, backend: {result.extractor_used}]",
            "",
            result.text,
        ]
        if result.warnings:
            output.insert(1, f"[warnings: {', '.join(result.warnings)}]")

        return "\n".join(output)

The confidence line at the top isn’t decoration — it’s the part of the output the agent actually reads first, and it’s what the task description below tells the agent to act on. Everything downstream of this tool call inherits whatever the agent decides to do with that number.

Wiring it into a crew

A minimal two-agent setup: one agent reads documents, one reviews the output for anything the reader flagged as low-confidence.

from crewai import Agent, Task, Crew, Process

pdf_tool = PdfReadTool()

document_analyst = Agent(
    role="Document Analyst",
    goal="Extract and summarize the key facts from PDF documents",
    backstory="You process incoming documents and pull out what matters.",
    tools=[pdf_tool],
    verbose=True,
)

review_task = Task(
    description=(
        "Read {file_path} using the read_pdf tool. Summarize the key points. "
        "If the extraction confidence is below 60%, do not present the summary "
        "as reliable — instead state explicitly that the document needs manual "
        "review and explain what content area triggered the low confidence "
        "(scanned pages, dense tables, or unusual formatting)."
    ),
    expected_output="A summary, or a flag for manual review with a reason.",
    agent=document_analyst,
)

crew = Crew(
    agents=[document_analyst],
    tasks=[review_task],
    process=Process.sequential,
)

result = crew.kickoff(inputs={"file_path": "quarterly-report.pdf"})
print(result)

The task description is where the confidence threshold actually gets enforced — CrewAI agents follow instructions written in plain language, not code, so the 60% cutoff has to live in the prompt, not just in the tool. If you want that threshold enforced in code instead of relying on the agent to follow instructions, gate it inside _run and return a different message shape entirely when confidence is low, so there’s no ambiguity for the LLM to interpret.

Escalating instead of just flagging

A tool that only reports low confidence is more useful than one that stays silent, but a tool that can retry at a higher quality tier closes the loop without a second agent turn:

class PdfReadWithEscalationTool(BaseTool):
    name: str = "read_pdf"
    description: str = (
        "Extracts text from a PDF. Automatically retries at higher quality "
        "if the first pass has low confidence. Returns the best result "
        "achieved plus how many attempts it took."
    )
    args_schema: type[BaseModel] = PdfReadInput

    def _run(self, file_path: str, quality: str = "standard") -> str:
        tiers = ["fast", "standard", "high"]
        start = tiers.index(quality) if quality in tiers else 1

        for attempt, tier in enumerate(tiers[start:], start=1):
            result = process(file_path, quality=tier)
            if result.confidence >= 0.6 or tier == tiers[-1]:
                return (
                    f"[confidence: {result.confidence:.0%}, tier: {tier}, "
                    f"attempts: {attempt}]\n\n{result.text}"
                )

        return "Extraction failed at all quality tiers."

This trades latency for reliability — high quality routes more pages through OCR and, on the hardest pages, a vision LLM backend, so it’s slower and can carry API cost. For an interactive agent answering a user’s question, that tradeoff is usually worth it. For a batch job processing thousands of documents overnight, you’re better off doing the retry decision outside the agent loop entirely, which is the next section.

Batch processing outside the agent loop

Don’t route bulk extraction through agent tool calls — an LLM reasoning about “should I call read_pdf on file 2,847” is slower and more expensive than a plain loop, and buys you nothing. Extract first, then hand the agent only the documents that need judgment:

import glob
from pdfmux import process

def preprocess_batch(pdf_dir: str, min_confidence: float = 0.6):
    clean, needs_review = [], []

    for path in glob.glob(f"{pdf_dir}/*.pdf"):
        result = process(path, quality="standard")
        entry = {"path": path, "text": result.text, "confidence": result.confidence}
        (clean if result.confidence >= min_confidence else needs_review).append(entry)

    return clean, needs_review

Feed needs_review to the crew — that’s where an agent’s judgment (does this need OCR retry, is it worth escalating, should a human see it) actually adds value. Feed clean straight into whatever the crew does next (indexing, summarization, structured extraction) without spending an agent turn on documents that already extracted fine.

Using pdfmux’s MCP server instead

If you’d rather not maintain a custom tool class, pdfmux also ships an MCP server with the same underlying pipeline. crewai-tools has an MCP adapter that can expose an MCP server’s tools to a crew directly, which saves you from re-declaring the args schema and keeps you on pdfmux’s tool surface (convert_pdf, analyze_pdf, extract_structured) as it evolves, rather than a hand-rolled wrapper you have to update yourself. The tradeoff is less control over exactly what the tool description says to the agent — worth it if you’re wiring up several MCP-compatible tools at once, less worth it if the confidence-threshold framing above is the whole point.

Comparing the options

FAQ

Does the agent actually respect the confidence threshold? Only as reliably as any other instruction in the task description — LLMs follow prompted rules probabilistically, not deterministically. For a threshold that must never be crossed (compliance, financial reporting), enforce it in the tool’s _run method instead of trusting the agent to read and obey a number in a string.

Can multiple agents share one PdfReadTool instance? Yes — BaseTool instances are stateless per call (no request-scoped state stored on self beyond configuration), so the same instance can be passed to multiple Agents in a crew without conflicts.

What if the PDF is too large for the agent’s context window? Chunk before handing content to the agent, don’t rely on the LLM to skim a 200-page extraction. See PDF chunking strategies for RAG for a heading-based chunker — the same one used in the pgvector ingestion pipeline — and have the tool return a chunk index the agent can request by section instead of the full text in one call.