### Setup Development Environment Source: https://github.com/nameetp/pdfmux/blob/main/CONTRIBUTING.md Clone the repository, create and activate a virtual environment, and install development dependencies. ```bash git clone https://github.com/NameetP/pdfmux.git cd pdfmux python3.12 -m venv .venv && source .venv/bin/activate pip install -e ".[dev,serve]" ``` -------------------------------- ### Install pdfmux with specific backends Source: https://github.com/nameetp/pdfmux/blob/main/README.md Install pdfmux with optional extras for specific document types or LLM providers. Use 'all' for every feature. ```bash pip install "pdfmux[tables]" ``` ```bash pip install "pdfmux[opendataloader]" ``` ```bash pip install "pdfmux[marker]" ``` ```bash pip install "pdfmux[llm]" ``` ```bash pip install "pdfmux[llm-claude]" ``` ```bash pip install "pdfmux[llm-openai]" ``` ```bash pip install "pdfmux[llm-ollama]" ``` ```bash pip install "pdfmux[llm-mistral]" ``` ```bash pip install "pdfmux[llm-all]" ``` ```bash pip install "pdfmux[watch]" ``` ```bash pip install "pdfmux[all]" ``` -------------------------------- ### Install pdfmux Source: https://github.com/nameetp/pdfmux/blob/main/README.md Install the base pdfmux package for digital PDFs. For scanned documents, install with the 'ocr' extra. ```bash pip install pdfmux ``` ```bash pip install "pdfmux[ocr]" ``` -------------------------------- ### Start MCP Server Source: https://github.com/nameetp/pdfmux/blob/main/README.md Start the MCP server for AI agent integration. Supports both stdio and HTTP modes. ```bash pdfmux serve ``` ```bash pdfmux serve --http 8080 ``` -------------------------------- ### Install LangChain Integration Source: https://github.com/nameetp/pdfmux/blob/main/README.md Install the necessary package to use pdfmux with LangChain. ```bash pip install langchain-pdfmux ``` -------------------------------- ### Install pdfmux with Arabic Support Source: https://github.com/nameetp/pdfmux/blob/main/README.md Install pdfmux using pip. For enhanced Arabic and RTL support, including Gemma 4 vision OCR, use the recommended installation command. ```bash # Default install — already includes python-bidi for RTL reordering pip install pdfmux # Recommended for Arabic-heavy docs — adds Gemma 4 vision OCR pip install "pdfmux[arabic,llm-gemma]" # One credential covers Gemma + Gemini (same Google endpoint) export GEMINI_API_KEY=... ``` -------------------------------- ### Clone and Set Up pdfmux Development Environment Source: https://github.com/nameetp/pdfmux/blob/main/README.md Clone the pdfmux repository, set up a virtual environment, and install development dependencies. This includes installing the package in editable mode. ```bash git clone https://github.com/NameetP/pdfmux.git cd pdfmux python3.12 -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" ``` -------------------------------- ### Install LlamaIndex Integration Source: https://github.com/nameetp/pdfmux/blob/main/README.md Install the required package for integrating pdfmux with LlamaIndex. ```bash pip install llama-index-readers-pdfmux ``` -------------------------------- ### Serve Static Site Locally Source: https://github.com/nameetp/pdfmux/blob/main/CLAUDE.md Start a local development server to preview the static HTML site. This command is useful for testing changes in a live environment before deployment. ```bash npx serve site/ -l 3456 # local dev server ``` -------------------------------- ### pdfmux serve Source: https://github.com/nameetp/pdfmux/blob/main/README.md Starts the MCP server for AI agent integration, supporting both stdio and HTTP modes. ```APIDOC ## pdfmux serve ### Description Starts the MCP server for AI agent integration, supporting both stdio and HTTP modes. ### Method CLI Command ### Endpoint pdfmux serve [options] ### Parameters #### Options - **--http PORT** (PORT) - Start HTTP mode on the specified port ``` -------------------------------- ### Python Example: Batch Extraction Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md This example demonstrates how to use the `batch_extract` function from the pdfmux library in Python to process a list of PDF file paths. ```python from pdfmux import batch_extract paths = [ "path/to/doc1.pdf", "path/to/doc2.pdf", ] # Process the batch of PDFs results = batch_extract(paths) # Process results (e.g., print extracted text or confidence scores) for path, result in zip(paths, results): print(f"Processed {path}: {result}") ``` -------------------------------- ### Run Benchmark Command Source: https://github.com/nameetp/pdfmux/blob/main/README.md Evaluate all installed extractors against ground truth data. ```bash pdfmux benchmark report.pdf ``` ```text ┌──────────────────┬────────┬────────────┬─────────────┬──────────────────────┐ │ Extractor │ Time │ Confidence │ Output │ Status │ ├──────────────────┼────────┼────────────┼─────────────┼──────────────────────┤ │ PyMuPDF │ 0.02s │ 95% │ 3,241 chars │ all pages good │ │ Multi-pass │ 0.03s │ 95% │ 3,241 chars │ all pages good │ │ RapidOCR │ 4.20s │ 88% │ 2,891 chars │ ok │ │ OpenDataLoader │ 0.12s │ 97% │ 3,310 chars │ best │ └──────────────────┴────────┴────────────┴─────────────┴──────────────────────┘ ``` -------------------------------- ### Directory PDF Analysis and Recommendations Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md Run pdfmux doctor --check on a directory to sample PDFs, classify them, and get recommendations for necessary installations like OCR. ```bash pdfmux doctor --check ``` -------------------------------- ### Run Doctor Command Source: https://github.com/nameetp/pdfmux/blob/main/README.md Display installed backends, coverage gaps, and recommendations for pdfmux. ```bash pdfmux doctor ``` ```text ┌──────────────────┬─────────────┬─────────┬──────────────────────────────────┐ │ Extractor │ Status │ Version │ Install │ ├──────────────────┼─────────────┼─────────┼──────────────────────────────────┤ │ PyMuPDF │ installed │ 1.25.3 │ │ │ OpenDataLoader │ installed │ 0.3.1 │ │ │ RapidOCR │ installed │ 3.0.6 │ │ │ Docling │ missing │ -- │ pip install pdfmux[tables] │ │ Surya │ missing │ -- │ pip install pdfmux[ocr-heavy] │ │ LLM (Gemini) │ configured │ -- │ GEMINI_API_KEY set │ └──────────────────┴─────────────┴─────────┴──────────────────────────────────┘ ``` -------------------------------- ### Self-host pdfmux-mcp Server Source: https://github.com/nameetp/pdfmux/blob/main/npm-wrapper/README.md Install and run the pdfmux-mcp server locally using pip. This is suitable for self-hosting and development environments. ```bash pip install "pdfmux[serve]" pdmux serve --http --port 8000 ``` -------------------------------- ### pdfmux benchmark Source: https://github.com/nameetp/pdfmux/blob/main/README.md Evaluates all installed extractors against ground truth data. ```APIDOC ## pdfmux benchmark ### Description Evaluates all installed extractors against ground truth data. ### Method CLI Command ### Endpoint pdfmux benchmark ``` -------------------------------- ### PDFMux Dependency Model Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Outlines the core and optional dependencies for PDFMux, categorized by installation profiles (e.g., `pdfmux[ocr]`, `pdfmux[llm]`). ```text Base (every install): pymupdf, pymupdf4llm, typer, rich, python-bidi Optional: pdfmux[ocr] → rapidocr + onnxruntime (~200MB, CPU) pdfmux[ocr-heavy] → surya-ocr (~5GB, GPU) pdfmux[tables] → docling (~500MB) pdfmux[opendataloader] → opendataloader-pdf (Java 11+) pdfmux[marker] → marker-pdf (~2GB models) pdfmux[llm] → google-genai (Gemini API key) pdfmux[llm-claude] → anthropic pdfmux[llm-openai] → openai pdfmux[llm-ollama] → ollama pdfmux[llm-mistral] → mistralai pdfmux[llm-all] → all LLM providers (Gemma 4 piggybacks on Gemini key) pdfmux[langchain] → langchain-core pdfmux[llamaindex] → llama-index-core pdfmux[serve] → mcp + uvicorn (MCP server) pdfmux[watch] → watchdog (`pdfmux watch `) pdfmux[all] → everything ``` -------------------------------- ### Pipeline Flow Visualization Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Illustrates the multi-pass extraction process, starting with classification, followed by a fast extract and audit, selective OCR on problematic pages, and finally merging and scoring. ```text PDF الخام classify() ← detect.py الخام _multipass_extract() ← pipeline.py الخام ├── Pass 1: Fast extract + audit الخام │ الخام │ ├── pymupdf4llm extracts every page (page_chunks=True) │ ├── audit.score_page() scores each page with 5 checks │ │ ├── "good": sufficient text density and quality │ │ ├── "bad": low text with images (text in images) │ │ └── "empty": near-blank page │ │ │ └── All pages good? → return fast text. Done. Zero overhead. الخام ├── Pass 2: Selective OCR on bad/empty pages الخام │ الخام │ ├── Try RapidOCR (preferred — lightweight, CPU) │ │ ├── "bad" pages: use OCR only if it got MORE text than fast │ │ └── "empty" pages: accept any OCR result >10 chars │ │ │ ├── Try Surya OCR (fallback — heavier, GPU) │ │ └── Same comparison logic for remaining pages │ │ │ └── Try Gemini LLM (last resort — API call) │ └── Same comparison logic for remaining pages الخام └── Pass 3: Merge + clean + score الخام ├── Combine good pages + OCR'd pages in page order ├── postprocess.clean_text() cleans text └── audit.compute_document_confidence() scores the merged result ``` -------------------------------- ### Check for pymupdf_layout Package Availability Source: https://github.com/nameetp/pdfmux/blob/main/ARCH-READING-ORDER.md This code snippet demonstrates how to check if the 'pymupdf_layout' package is installed and importable. This package is recommended for improved page layout analysis. ```python # Check if available try: import pymupdf_layout # May provide better column detection than our custom clustering except ImportError: pass ``` -------------------------------- ### Set Budget Cap for LLM Calls Source: https://github.com/nameetp/pdfmux/blob/main/README.md You can set a hard budget cap to limit LLM usage and control costs. This example stops LLM calls when the spend reaches $0.50 per document. ```bash --budget 0.50 ``` -------------------------------- ### Add pdfmux-mcp using Claude Code CLI Source: https://github.com/nameetp/pdfmux/blob/main/npm-wrapper/README.md Use the Claude Code command-line interface to add pdfmux-mcp as an MCP server. This command automatically handles installation. ```bash claude mcp add pdfmux -- npx -y pdfmux-mcp ``` -------------------------------- ### pdfmux doctor Source: https://github.com/nameetp/pdfmux/blob/main/README.md Diagnoses the installed backends, checks for coverage gaps, and provides recommendations. ```APIDOC ## pdfmux doctor ### Description Diagnoses the installed backends, checks for coverage gaps, and provides recommendations. ### Method CLI Command ### Endpoint pdfmux doctor ``` -------------------------------- ### Generate Fixture Set and Labels Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md Run this command to generate the labeled PDF fixtures and the ground-truth labels file. This step is idempotent and deterministic based on a seed. ```bash python eval/build_fixtures.py ``` -------------------------------- ### Configure LLM Provider in pdfmux Source: https://github.com/nameetp/pdfmux/blob/main/README.md Set up your LLM provider by specifying the model, API key, and optional base URL in the `llm.yaml` configuration file. Adjust `max_cost_per_page` to manage budget. ```yaml provider: claude # gemini | claude | openai | ollama | any OpenAI-compatible model: claude-sonnet-4-20250514 api_key: ${ANTHROPIC_API_KEY} base_url: https://api.anthropic.com # optional, for custom endpoints max_cost_per_page: 0.02 # budget cap ``` -------------------------------- ### Convert PDF with Options Source: https://github.com/nameetp/pdfmux/blob/main/README.md Convert a PDF file or directory with various options for output format, quality, schema, chunking, LLM provider, and cost control. ```bash pdfmux convert file.pdf --chunk --max-tokens 500 ``` ```bash pdfmux convert file.pdf --mode economy --budget 0.50 ``` ```bash pdfmux convert file.pdf --schema invoice ``` ```bash pdfmux convert file.pdf --profile invoices ``` ```bash pdfmux convert file.pdf --llm-provider claude ``` ```bash pdfmux convert file.pdf ``` ```bash pdfmux convert ./docs/ ``` -------------------------------- ### Generate Fixtures with Python Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md Generates 50 labeled PDFs for testing, covering various scenarios like clean digital, multi-page, tables, Arabic text, truncated files, and image-only PDFs. ```python eval/build_fixtures.py generates 50 labeled PDFs from a fixed seed (clean digital, multi-page, table-heavy, Arabic, 0-byte, HTML-as-PDF, heavily/lightly truncated, blank pages, micro-text, image-only-no-OCR). ``` -------------------------------- ### Calibration Summary Output Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md This JSON file contains a summary of the calibration results, likely including key metrics and recommended thresholds. ```json outputs/calibration_summary.json ``` -------------------------------- ### pdfmux extract_json with schema Source: https://github.com/nameetp/pdfmux/blob/main/README.md Extracts data from a PDF using a specified JSON schema for guided extraction, supporting both built-in and custom schema files. ```APIDOC ## Schema-guided extraction ### Description Extracts data from a PDF using a specified JSON schema. ### Usage ```python data = pdfmux.extract_json("invoice.pdf", schema="invoice") # Uses built-in invoice preset: extracts date, vendor, line items, totals # Also accepts a path to a custom JSON Schema file ``` ``` -------------------------------- ### Handle Low Confidence Extraction Source: https://github.com/nameetp/pdfmux/blob/main/README.md When confidence drops below 80%, pdfmux provides specific error messages and suggests solutions, such as installing OCR support for image-heavy pages. ```text Page 4: 32% confidence. 0 chars extracted from image-heavy page. -> Install pdfmux[ocr] for RapidOCR support on 6 image-heavy pages. ``` -------------------------------- ### CLI: Pre-flight directory check Source: https://github.com/nameetp/pdfmux/blob/main/README.md Analyze PDFs in a directory to determine which optional extras are required for optimal extraction. ```bash pdfmux doctor --check ./docs/ ``` -------------------------------- ### Running Tests Source: https://github.com/nameetp/pdfmux/blob/main/CONTRIBUTING.md Execute the test suite using pytest. Options include running the full suite, with coverage, or targeting a specific file. ```bash # full suite (151 tests) pytest ``` ```bash # with coverage pytest --cov=pdfmux --cov-report=term-missing ``` ```bash # single file pytest tests/test_pipeline.py -v ``` -------------------------------- ### Configure pdfmux-mcp for Claude Desktop/Cursor Source: https://github.com/nameetp/pdfmux/blob/main/npm-wrapper/README.md Add pdfmux-mcp as an MCP server in your Claude Desktop or Cursor configuration. This enables AI agents to reliably read PDFs. ```json { "mcpServers": { "pdfmux": { "command": "npx", "args": ["-y", "pdfmux-mcp"] } } } ``` -------------------------------- ### CLI: Profile-based extraction Source: https://github.com/nameetp/pdfmux/blob/main/README.md Apply a predefined profile (e.g., 'invoices', 'receipts') for extraction settings. ```bash pdfmux convert invoice.pdf --profile invoices ``` -------------------------------- ### CLI: Batch convert directory Source: https://github.com/nameetp/pdfmux/blob/main/README.md Convert all PDFs in a directory and generate a manifest.json file with per-document confidence scores. ```bash pdfmux convert ./docs/ -o ./output/ ``` -------------------------------- ### Run pdfmux-mcp Server with Docker Source: https://github.com/nameetp/pdfmux/blob/main/npm-wrapper/README.md Deploy the pdfmux-mcp server using Docker for remote environments like Smithery or Railway. This command pulls the latest image and maps the port. ```bash docker pull pdfmux/mcp-server docker run -p 8000:8000 pdfmux/mcp-server ``` -------------------------------- ### CLI: Cache management Source: https://github.com/nameetp/pdfmux/blob/main/README.md Manage the extraction cache. Re-runs are instant by default; bypass with --no-cache or clear with --clear-cache. ```bash pdfmux convert report.pdf --no-cache ``` ```bash pdfmux convert report.pdf --clear-cache ``` -------------------------------- ### CLI: Watch directory for new PDFs Source: https://github.com/nameetp/pdfmux/blob/main/README.md Automatically convert new PDF files added to a specified directory and output them to another. ```bash pdfmux watch ./inbox/ -o ./output/ ``` -------------------------------- ### Run Evaluation with Python Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md Executes pdfmux on generated fixtures and saves raw confidence scores to a CSV file. ```python eval/run_eval.py runs pdfmux on each fixture and writes outputs/raw_scores.csv. ``` -------------------------------- ### CLI: Cost-aware extraction Source: https://github.com/nameetp/pdfmux/blob/main/README.md Perform PDF extraction in 'economy' mode with a specified budget cap to control costs. ```bash pdfmux convert report.pdf --mode economy --budget 0.50 ``` -------------------------------- ### PDF Extraction Pipeline Components Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Illustrates the modular structure of the pdfmux pipeline, showing how different components like detection, extraction, and auditing interact. ```text pdfmux Python API / CLI / MCP server │ ├─ __init__.py public API: extract_text, extract_json, load_llm_context + type/error exports │ ├─ types.py frozen dataclasses + enums: Quality, OutputFormat, PageResult, DocumentResult, Chunk ├─ errors.py exception hierarchy with .user_message / .suggestion / .reproduce_cmd ├─ retry.py @with_retry — exponential backoff, Retry-After aware, skips on auth failures │ ├─ detect.py classify PDF (digital / scanned / graphical / mixed / tables / arabic / academic) │ ├─ pipeline.py route to extractor based on classification + quality │ │ (consults result_cache before running, persists on success) │ │ │ ├─ quality=fast → FastExtractor only │ ├─ quality=high → LLM → OCR → Fast fallback │ ├─ has_tables → TableExtractor → Fast fallback │ └─ standard → multi-pass pipeline (below) │ ├─ audit.py 5-check per-page confidence scoring + quality classification │ ├─ chunking.py section-aware splitting + token estimation │ ├─ result_cache.py SHA-256 file hashing → ~/.cache/pdfmux/results/, 30d TTL, 1 GB LRU ├─ streaming.py generator yielding NDJSON StreamEvents (classified/page/warning/complete) ├─ profiles.py ~/.config/pdfmux/profiles.yaml + 5 built-ins ├─ arabic.py BiDi reordering + Arabic detection + canonicalization │ ├─ extractors/ │ ├─ __init__.py Extractor protocol + @register decorator + priority-ordered registry │ ├─ fast.py PyMuPDF / pymupdf4llm — 0.01s/page, handles 90% (priority 10) │ ├─ rapid_ocr.py RapidOCR (PaddleOCR v4 + ONNX) — ~200MB, CPU (priority 20) │ ├─ mistral_ocr.py Mistral OCR API — $0.002/page, 96.6% TEDS (priority 25) │ ├─ ocr.py Surya OCR — legacy, ~5GB, GPU (priority 30) │ ├─ marker.py Marker neural extractor — academic papers (priority 35) │ ├─ tables.py Docling — 97.9% table accuracy (priority 40) │ ├─ opendataloader.py Java-backed reading-order extractor (priority 45) │ └─ llm.py BYOK vision LLM — Gemini, Gemma 4, Claude, GPT-4o, Ollama, Mistral (priority 50) │ ├─ providers/ │ ├─ base.py LLMProvider protocol + ModelInfo + CostEstimate │ ├─ _discovery.py auto-registers all built-in providers │ ├─ gemini.py Gemini 2.5 Flash / Pro │ ├─ gemma.py Gemma 4 27B IT (Gemini OpenAI-compat endpoint, reuses GEMINI_API_KEY) │ ├─ claude.py Claude Sonnet / Opus │ ├─ openai_native.py GPT-4o family │ ├─ openai_compatible.py Mistral + custom OpenAI-compatible APIs │ └─ ollama.py local models │ ├─ postprocess.py text cleanup + Arabic BiDi (markdown-aware) │ └─ formatters/ markdown, json, csv, llm output ``` -------------------------------- ### Watch Directory for PDFs Source: https://github.com/nameetp/pdfmux/blob/main/README.md Automatically convert any PDF added to a specified folder. ```bash pdfmux watch ./inbox/ ``` -------------------------------- ### Convert PDF using a Profile Source: https://github.com/nameetp/pdfmux/blob/main/README.md Utilize a saved or built-in profile to convert a PDF file. ```bash pdfmux convert file.pdf --profile invoices ``` -------------------------------- ### Extract Fixtures and Record Confidence Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md Execute this command to run pdfmux on each generated fixture and save the predicted confidence scores. Use --quality standard for OCR on image-only fixtures. The result cache is honored by default; use --no-cache for clean re-runs. ```bash python eval/run_eval.py --quality fast ``` -------------------------------- ### Run Ruff Linter and Formatter Source: https://github.com/nameetp/pdfmux/blob/main/CLAUDE.md Use ruff to check code for linting errors and to format the code according to project standards. Ensure these commands are run from the project root. ```bash ruff check src/ tests/ # lint ``` ```bash ruff format src/ tests/ # format ``` -------------------------------- ### MCP Server Tools Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Lists the tools exposed by the MCP (Multi-Cloud Platform) server for AI agent integration. These tools provide various PDF processing capabilities. ```text | Tool | |------| | `convert_pdf` | Full extraction → markdown / json / csv / llm | | `analyze_pdf` | Quick triage: classification, page count, recommended quality | | `extract_structured` | Tables and key-values via schema preset | | `extract_streaming` | NDJSON page-by-page events for very long documents | | `get_pdf_metadata` | Title, author, page count, embedded metadata | | `batch_convert` | Run a directory of PDFs through the pipeline | ``` -------------------------------- ### CLI: BYOK LLM for difficult pages Source: https://github.com/nameetp/pdfmux/blob/main/README.md Use your own LLM provider, like 'claude', for processing challenging pages in a PDF. ```bash pdfmux convert scan.pdf --llm-provider claude ``` -------------------------------- ### Current PDF Reading Order Pipeline Source: https://github.com/nameetp/pdfmux/blob/main/ARCH-READING-ORDER.md Illustrates the current, simplified pipeline where PDF is converted to markdown by pymupdf4llm, and no reordering occurs. ```text PDF → pymupdf4llm.to_markdown() → text with baked-in reading order ↓ no reordering anywhere ↓ pipeline merges pages → clean_text → output ``` -------------------------------- ### Linting Code Source: https://github.com/nameetp/pdfmux/blob/main/CONTRIBUTING.md Check code for style and formatting issues using ruff. Both checks must pass in CI. ```bash ruff check src/ tests/ ``` ```bash ruff format src/ tests/ ``` -------------------------------- ### Compare PDF Extractions Source: https://github.com/nameetp/pdfmux/blob/main/README.md Compare the extractions of two PDF files. ```bash pdfmux diff a.pdf b.pdf ``` -------------------------------- ### Run Tests and Linters for pdfmux Source: https://github.com/nameetp/pdfmux/blob/main/README.md Execute the test suite and run code quality checks using pytest and ruff. This ensures code correctness and adherence to formatting standards. ```bash pytest # 659 tests ruff check src/ tests/ ruff format src/ tests/ ``` -------------------------------- ### Python Script for Building Fixtures Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md This Python script is responsible for generating the fixture set of labeled PDFs. It creates the necessary input data for the evaluation pipeline. ```python build_fixtures.py ``` -------------------------------- ### CLI: Schema-guided extraction Source: https://github.com/nameetp/pdfmux/blob/main/README.md Extract data from a PDF using a predefined schema, such as 'invoice'. Supports 5 built-in presets. ```bash pdfmux convert invoice.pdf --schema invoice ``` -------------------------------- ### Compute ROC and Threshold Recommendations Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md Run this script to compute the Receiver Operating Characteristic (ROC) curve and generate threshold recommendations based on the extracted confidence scores. ```bash python eval/calibrate.py ``` -------------------------------- ### Strategy A: Post-Extraction Reorder Integration Source: https://github.com/nameetp/pdfmux/blob/main/ARCH-READING-ORDER.md Shows how to integrate multi-column detection and reordering into the fast.py extractor after initial text extraction. ```python # fast.py — after line 153 (pymupdf4llm extraction) for i, chunk in enumerate(chunks): text = chunk.get("text", "") # NEW: detect multi-column and reorder if needed if i < len(doc): layout = detect_layout(doc[i]) if layout.columns >= 2: text = _reorder_by_layout(doc[i], layout) # existing: heading injection, quality checks, etc. if i < len(doc): text = inject_headings(text, doc[i]) ... ``` -------------------------------- ### Manage pdfmux Profiles Source: https://github.com/nameetp/pdfmux/blob/main/README.md List, show, save, or delete custom configuration profiles stored in `~/.config/pdfmux/profiles.yaml`. Built-in profiles are available for common use cases. ```bash pdfmux profiles list ``` ```bash pdfmux profiles show invoices ``` ```bash pdfmux profiles save my-default --quality high --format llm --chunk ``` ```bash pdfmux profiles delete my-default ``` -------------------------------- ### Load Documents with LangChain PDFMuxLoader Source: https://github.com/nameetp/pdfmux/blob/main/README.md Load PDF documents using the LangChain PDFMuxLoader, specifying the quality for extraction. Returns a list of LangChain Documents with confidence metadata. ```python from langchain_pdfmux import PDFMuxLoader loader = PDFMuxLoader("report.pdf", quality="standard") docs = loader.load() # -> list[Document] with confidence metadata ``` -------------------------------- ### Python Script for Running Evaluation Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md This script executes the pdfmux extraction process on each fixture and records the predicted confidence scores. It is a key step in gathering data for calibration. ```python run_eval.py ``` -------------------------------- ### Documentation for Calibration Workflow Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md Documents the confidence calibration workflow and the results from May 2026, which influenced the default settings for version 1.7. ```markdown eval/README.md documents the workflow and the May 2026 calibration result that drives the 1.7 default. ``` -------------------------------- ### Pipeline Routing Logic Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Illustrates the routing strategy within the PDFMux pipeline based on quality presets and document classification. This determines the extraction process. ```text quality=fast → FastExtractor (skip audit) quality=high → LLM → OCR → Fast has_tables → Docling → Fast (skip if graphical) standard → _multipass_extract() ``` -------------------------------- ### Run Pytest for Testing Source: https://github.com/nameetp/pdfmux/blob/main/CLAUDE.md Execute the pytest test suite with verbose output to ensure all tests pass. This command should be run from the project root. ```bash pytest -v # test ``` -------------------------------- ### Watch Directory for PDF Conversion Source: https://github.com/nameetp/pdfmux/blob/main/README.md Automatically convert PDFs placed in a directory. This process continues until interrupted (Ctrl+C). ```bash pdfmux watch ./inbox/ -o ./output/ --profile bulk-rag ``` -------------------------------- ### Strategy B: Parallel Extraction and Comparison Source: https://github.com/nameetp/pdfmux/blob/main/ARCH-READING-ORDER.md Visualizes a more robust strategy where text is extracted two ways (pymupdf4llm and column-reordered) and compared using a self-scoring heuristic. ```text PDF page → pymupdf4llm text ──→ NID-like self-score ─┐ ↓ ├→ pick better └──→ column-reordered text ──→ NID-like self-score ┘ ``` -------------------------------- ### View Calibration Report Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md Open the generated calibration report in your web browser to review the ROC curve and recommended confidence thresholds. ```bash open eval/outputs/calibration_report.md ``` -------------------------------- ### Compare PDF Extractions Side-by-Side Source: https://github.com/nameetp/pdfmux/blob/main/README.md Perform a side-by-side comparison of PDF extraction results, focusing on quality, content, and cost. ```bash pdfmux diff a.pdf b.pdf --quality standard ``` -------------------------------- ### Manage pdfmux Smart Result Cache Source: https://github.com/nameetp/pdfmux/blob/main/README.md Control the pdfmux smart result cache using command-line arguments. Options include running a standard conversion, bypassing the cache, or clearing the cache entirely. ```bash pdfmux convert big-report.pdf # first run: 14.2s pdfmux convert big-report.pdf # cache hit: 0.05s pdfmux convert big-report.pdf --no-cache # bypass cache (still writes back) pdfmux convert big-report.pdf --clear-cache # purge and re-run ``` -------------------------------- ### Calibration Report Output Source: https://github.com/nameetp/pdfmux/blob/main/eval/README.md This markdown file provides a detailed report of the calibration process, including ROC curves and recommended thresholds. ```markdown outputs/calibration_report.md ``` -------------------------------- ### Reorder by Layout Algorithm Source: https://github.com/nameetp/pdfmux/blob/main/ARCH-READING-ORDER.md Implements the `_reorder_by_layout` function to re-extract page text in column-aware reading order using fitz blocks. ```python def _reorder_by_layout(page: fitz.Page, layout: PageLayout) -> str: """Re-extract page text in column-aware reading order.""" blocks = page.get_text("blocks") text_blocks = [(i, b) for i, b in enumerate(blocks) if b[6] == 0 and b[4].strip()] # Use the reading_order from detect_layout ordered_texts = [] for block_idx in layout.reading_order: for i, b in text_blocks: if i == block_idx: ordered_texts.append(b[4].strip()) break return "\n\n".join(ordered_texts) ``` -------------------------------- ### MCP Server Integration Source: https://github.com/nameetp/pdfmux/blob/main/README.md Configuration for integrating pdfmux with MCP Server for AI Agents, enabling tools like `convert_pdf` and `extract_structured`. ```APIDOC ## MCP Server (AI Agents) Listed on [mcpservers.org](https://mcpservers.org). One-line setup: ```json { "mcpServers": { "pdfmux": { "command": "npx", "args": ["-y", "pdfmux-mcp"] } } } ``` Or via Claude Code: ```bash claude mcp add pdfmux -- npx -y pdfmux-mcp ``` Tools exposed: `convert_pdf`, `analyze_pdf`, `extract_structured`, `extract_streaming`, `get_pdf_metadata`, `batch_convert`. ``` -------------------------------- ### CLI: Diff two PDF extractions Source: https://github.com/nameetp/pdfmux/blob/main/README.md Compare the extraction results of two PDF files side-by-side. ```bash pdfmux diff old.pdf new.pdf ``` -------------------------------- ### CLI: Convert PDF to Markdown Source: https://github.com/nameetp/pdfmux/blob/main/README.md Basic PDF to Markdown conversion. The output filename is derived from the input filename. ```bash pdfmux convert invoice.pdf ``` -------------------------------- ### LangChain Integration Source: https://github.com/nameetp/pdfmux/blob/main/README.md Integrates pdfmux with LangChain for document loading, providing confidence metadata. ```APIDOC ## LangChain ### Installation ```bash pip install langchain-pdfmux ``` ### Usage ```python from langchain_pdfmux import PDFMuxLoader loader = PDFMuxLoader("report.pdf", quality="standard") docs = loader.load() # -> list[Document] with confidence metadata ``` ``` -------------------------------- ### Lint Design System CSS Source: https://github.com/nameetp/pdfmux/blob/main/CLAUDE.md Run the design system linter on the site's CSS files to ensure adherence to visual rules. This script checks for common violations before deployment. ```bash python3 design-system/lint.py site/ # design system lint ``` -------------------------------- ### CLI: RAG-ready chunked conversion Source: https://github.com/nameetp/pdfmux/blob/main/README.md Convert PDF to Markdown with chunking for RAG, specifying a maximum token limit per chunk. ```bash pdfmux convert report.pdf --chunk --max-tokens 500 ``` -------------------------------- ### CLI: Strict batch conversion Source: https://github.com/nameetp/pdfmux/blob/main/README.md Batch convert PDFs in a directory, failing the run if any document's confidence is below a specified minimum. ```bash pdfmux convert ./docs/ -o ./output/ --strict --min-confidence 0.20 ``` -------------------------------- ### CLI: Estimate extraction cost Source: https://github.com/nameetp/pdfmux/blob/main/README.md Predict the cost of extracting a PDF before running the full process, specifying the LLM provider. ```bash pdfmux estimate big-report.pdf --llm-provider gemini ``` -------------------------------- ### Stream PDF Output Source: https://github.com/nameetp/pdfmux/blob/main/README.md Emit NDJSON events as pages complete, which is useful for very long PDFs and live UIs. ```bash pdfmux stream long.pdf --quality high ``` -------------------------------- ### Strict Mode for PDF Conversion Source: https://github.com/nameetp/pdfmux/blob/main/CHANGELOG.md Use the --strict flag with --min-confidence to ensure document quality. Exits with code 3 if confidence is below the threshold. ```bash pdfmux convert --strict --min-confidence FLOAT ``` -------------------------------- ### Public Python API Wrappers Source: https://github.com/nameetp/pdfmux/blob/main/docs/ARCHITECTURE.md Provides simplified Python functions for common PDF extraction tasks. Imports are lazy to prevent circular dependencies. ```python def extract_text(path, *, quality="standard") → str # Markdown string def extract_json(path, *, quality="standard") → dict # dict with locked schema def load_llm_context(path, *, quality="standard") → list[dict] # chunk dicts with tokens ``` -------------------------------- ### Batch PDF Extraction with pdfmux Source: https://github.com/nameetp/pdfmux/blob/main/README.md Use batch_extract for processing multiple PDFs. It yields (path, result) tuples as each PDF completes, allowing for immediate handling of results or exceptions. Check the confidence score to determine the quality of the extraction. ```python from pathlib import Path pdfs = list(Path("./inbox").glob("*.pdf")) for path, result in pdfmux.batch_extract(pdfs, quality="standard"): if isinstance(result, Exception): print(f"FAILED {path.name}: {result}") continue if result.confidence < 0.50: print(f"REVIEW {path.name} ({result.confidence:.2f})") else: print(f"OK {path.name} ({result.confidence:.2f})") ```