### Install PDFium via Homebrew Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/tap/README.md Commands to add the tap and install the pdfium formula, or install directly via a raw URL. ```bash brew tap raphaelmansuy/homebrew-pdfium-tap brew install pdfium # Or direct install brew install https://raw.githubusercontent.com/raphaelmansuy/homebrew-pdfium-tap/main/Formula/pdfium.rb ``` -------------------------------- ### Initialize Download-mode PDFium Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/crates/pdfium-auto/README.md Demonstrates how to bind PDFium in download mode, including an example of using a progress callback for the download process. ```rust use pdfium_auto::{bind_pdfium_silent, ensure_pdfium_library}; // One-shot: download if needed, then bind let pdfium = bind_pdfium_silent().expect("PDFium unavailable"); // Download with progress callback let path = ensure_pdfium_library(Some(&|downloaded, total| { match total { Some(t) => eprint!("\rDownloading PDFium: {}%", downloaded * 100 / t), None => eprint!("\rDownloading PDFium: {} bytes", downloaded), } })).expect("download failed"); ``` -------------------------------- ### Run PDF2MD with Azure and Ollama Providers Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Demonstrates how to execute the PDF2MD CLI tool using specific providers. The Azure example processes a remote URL, while the Ollama example shows pulling a local model before conversion. ```bash pdf2md --provider azure --model gpt-4o https://example.com/document.pdf ollama pull llava pdf2md --provider ollama --model llava document.pdf ``` -------------------------------- ### Install PDF2MD via Cargo Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/07-cli-design.md Instructions for installing the tool using Cargo, including options for default dynamic linking and static linking to bundle PDFium. ```bash # Standard installation cargo install edgequake-pdf2md # Static linking (recommended) cargo install edgequake-pdf2md --features pdfium-static ``` -------------------------------- ### Configure Azure OpenAI Environment Variables Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Provides examples for setting up standard and enterprise (ContentGen) environment variables for Azure OpenAI deployments. ```bash # Standard AZURE_OPENAI_* vars export AZURE_OPENAI_API_KEY="..." export AZURE_OPENAI_ENDPOINT="https://.openai.azure.com" export AZURE_OPENAI_DEPLOYMENT_NAME="gpt-4o" export AZURE_OPENAI_API_VERSION="2024-02-01" # Enterprise (ContentGen) env vars export AZURE_OPENAI_CONTENTGEN_API_KEY="..." export AZURE_OPENAI_CONTENTGEN_API_ENDPOINT="https://.openai.azure.com" export AZURE_OPENAI_CONTENTGEN_MODEL_DEPLOYMENT="gpt-4o" export AZURE_OPENAI_CONTENTGEN_API_VERSION="2024-02-01" ``` -------------------------------- ### Execute PDF2MD Conversion Commands Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/07-cli-design.md Provides various command-line examples for converting PDFs, including handling URLs, local LLMs, encrypted files, and CI/CD automation. ```bash # Simple conversion pdf2md report.pdf # Write to file with higher DPI pdf2md report.pdf -o output.md --dpi 200 # Convert from URL using Anthropic Claude ANTHROPIC_API_KEY=sk-... pdf2md --provider anthropic \ --model claude-3-5-sonnet-20241022 \ https://arxiv.org/pdf/2310.07093 # Local LLM via Ollama pdf2md --provider ollama --model llava:34b -c 1 private.pdf # Selected pages as JSON pdf2md --pages 3-7 --json report.pdf | jq .pages[].markdown # Encrypted PDF pdf2md --password s3cr3t encrypted.pdf -o decrypted.md # Maintain format pdf2md --maintain-format --separator --- novel.pdf # Inspect without converting pdf2md --inspect report.pdf # High-fidelity with LaTeX math pdf2md --fidelity tier3 --model gpt-4o math_paper.pdf # CI/CD — no progress bar, warn only PDF2MD_NO_PROGRESS=1 PDF2MD_QUIET=1 pdf2md report.pdf -o output.md ``` -------------------------------- ### Advanced CLI Usage and Options Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Examples of using advanced CLI flags for page selection, fidelity tiers, model selection, and output formatting. ```bash pdf2md --pages 1-5 document.pdf -o first_five.md pdf2md --fidelity tier3 --model gpt-4.1 --provider openai --dpi 200 paper.pdf -o paper.md pdf2md --maintain-format --separator hr book.pdf -o book.md pdf2md --json --metadata document.pdf > output.json pdf2md --provider bedrock --model amazon.nova-pro-v1:0 document.pdf ``` -------------------------------- ### Resumable and Fresh Conversions Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Examples of using the pdf2md command-line tool with checkpointing for resumable conversions and options for forcing a fresh conversion. ```APIDOC ## Resumable Conversion with Checkpoints (v0.7) ### Description Enables resumable conversion by saving progress to a specified directory. ### Command ```bash pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md ``` ## Resume After Interruption ### Description Re-run the same command to resume a previously interrupted conversion. ### Command ```bash pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md ``` ## Force Fresh Conversion ### Description Forces a new conversion, ignoring any existing checkpoints. ### Command ```bash pdf2md --checkpoint-dir ./checkpoints --no-resume big-doc.pdf -o out.md ``` ### Additional Information Run `pdf2md --help` for a full reference. See `docs/examples.md` for more usage patterns. ``` -------------------------------- ### Custom System Prompt for Specialized Conversion Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Demonstrates how to use a custom system prompt to guide the AI model for specific conversion tasks, such as focusing on legal terminology in contracts. The prompt is provided via a file. ```bash cat > my_prompt.txt << 'EOF' You are a legal document specialist. Convert this page to Markdown. Focus on: clause numbers, definitions, cross-references. Preserve exact legal language — do not paraphrase. EOF pdf2md --system-prompt my_prompt.txt contract.pdf -o contract.md ``` -------------------------------- ### Rust Library: Basic PDF to Markdown Conversion Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Provides a basic example of using the edgequake_pdf2md Rust library to convert a PDF file to Markdown programmatically. It shows how to configure conversion settings and access the output. ```rust use edgequake_pdf2md::{convert, ConversionConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::default(); let output = convert("document.pdf", &config).await?; println!("{}", output.markdown); println!("Pages: {}/{}", output.stats.processed_pages, output.stats.total_pages); println!("Tokens: {} in, {} out", output.stats.total_input_tokens, output.stats.total_output_tokens); Ok(()) } ``` -------------------------------- ### Dynamic Linking pdfium (macOS/Linux) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/05-crate-selection.md Instructions for dynamically linking the pdfium library. This involves installing pdfium via a package manager (macOS) or downloading it manually and setting an environment variable to point to the shared library. ```bash # macOS brew install pdfium # or manual download from bblanchon/pdfium-binaries # Linux wget https://github.com/bblanchon/pdfium-binaries/releases/latest/... export PDFIUM_DYNAMIC_LIB_PATH=/path/to/dir/containing/libpdfium.so cargo build ``` -------------------------------- ### Build and Test Commands for EdgeQuake PDF2MD Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Provides common development commands for the edgequake-pdf2md project, including setup, building release and debug binaries, running unit and integration tests, and performing quality checks like linting and formatting. ```bash # Setup make setup # Check pdfium + API key # Build make build # Release binary make build-dev # Debug binary # Test make test # Unit tests (no API key needed) make test-e2e # Integration tests (needs API key) make test-all # All tests # Quality make lint # Clippy make fmt # Format code make ci # format + lint + unit tests # Try it make demo # Convert sample page make inspect-all # Inspect test PDFs ``` -------------------------------- ### Enable Resumable Conversions with FileCheckpointStore in Rust Source: https://context7.com/raphaelmansuy/edgequake-pdf2md/llms.txt This Rust example shows how to use `FileCheckpointStore` to enable resumable PDF to Markdown conversions. By providing a directory for checkpoints, the library can save progress and resume interrupted conversions from the last saved state, which is particularly useful for large documents. Checkpoints are automatically managed, cleared on success, and retained for future resumes on partial failure. ```rust use edgequake_pdf2md::{convert, ConversionConfig, FileCheckpointStore}; use std::sync::Arc; #[tokio::main] async fn main() -> Result<(), Box> { // Create a file-based checkpoint store let store = Arc::new(FileCheckpointStore::new("./checkpoints")); let config = ConversionConfig::builder() .checkpoint_store(store) // .no_resume(true) // Uncomment to force fresh conversion .build()?; // First run: converts all pages, saves checkpoints // Interrupted run: partial checkpoints saved // Resume run: loads completed pages from checkpoints, processes remaining let output = convert("large-document.pdf", &config).await?; println!("Processed {} pages ({} resumed from checkpoint)", output.stats.processed_pages, output.stats.resumed_pages); // Checkpoints are automatically cleared on full success // Kept on partial failure for resume Ok(()) } ``` -------------------------------- ### Page Selection for Conversion Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Provides examples of selecting specific pages or page ranges for conversion. This includes converting a single page, a range of pages, or a list of specific pages. ```bash pdf2md --pages 1 document.pdf pdf2md --pages 3-15 document.pdf -o chapters.md pdf2md --pages 1,5,10,15 document.pdf -o selected.md ``` -------------------------------- ### Inject Custom LLM Provider for PDF to Markdown Conversion in Rust Source: https://context7.com/raphaelmansuy/edgequake-pdf2md/llms.txt This Rust code illustrates how to inject a pre-built LLM provider into the `ConversionConfig` for PDF to Markdown conversion. This allows for custom configurations, shared rate-limiting across multiple conversions, or specific testing scenarios. The example shows creating a provider from environment variables or a specific factory method, and then passing it to the conversion configuration. ```rust use edgequake_pdf2md::{convert, ConversionConfig}; use edgequake_llm::ProviderFactory; use std::sync::Arc; #[tokio::main] async fn main() -> Result<(), Box> { // Create provider from environment variables let (provider, _embedding) = ProviderFactory::from_env()?; // Or create a specific provider // let provider = ProviderFactory::create_llm_provider("openai", "gpt-4.1-nano")?; let config = ConversionConfig::builder() .provider(Arc::clone(&provider)) // Highest priority - bypasses auto-detection .concurrency(5) .build()?; let output = convert("document.pdf", &config).await?; println!("{}", output.markdown); Ok(()) } ``` -------------------------------- ### Implement LLM Provider and Multimodal Chat Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/02-algorithm.md Demonstrates how to initialize an LLM provider using environment variables and construct a multimodal chat message containing system instructions and image data for OCR-like tasks. ```rust use edgequake_llm::{ChatMessage, ImageData, CompletionOptions, ProviderFactory, ProviderType, LLMProvider}; let provider = ProviderFactory::from_env()?; let image = ImageData::new(base64_png_string, "image/png").with_detail("high"); let messages = vec![ ChatMessage::system(SYSTEM_PROMPT), ChatMessage::user_with_images(user_prompt, vec![image]), ]; let options = CompletionOptions { temperature: Some(0.1), max_tokens: Some(4096), ..Default::default() }; let response = provider.chat(&messages, Some(&options)).await?; let markdown = response.content; ``` -------------------------------- ### Configure PDFium Library Path Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/tap/README.md Commands to add the installed PDFium library path to the system environment variables for macOS and Linux. ```bash # macOS export DYLD_LIBRARY_PATH="$(brew --prefix pdfium)/lib:$DYLD_LIBRARY_PATH" # Linux export LD_LIBRARY_PATH="$(brew --prefix pdfium)/lib:$LD_LIBRARY_PATH" ``` -------------------------------- ### CLI Framework Configuration Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/05-crate-selection.md Uses clap for attribute-based argument parsing, environment variable support, and colored help text. ```toml [dependencies] clap = { version = "4", features = ["derive", "env", "color", "wrap_help"] } ``` -------------------------------- ### Build Documentation (Rust) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/CONTRIBUTING.md Generates documentation for the Rust project without dependencies and opens it in a web browser. It also shows how to build documentation with strict warnings enabled. ```rust cargo doc --no-deps --open ``` ```rust RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --all-features ``` -------------------------------- ### PDF Content Stream Operators Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/03-pdf-internals.md Example of PDF content stream operators used to position text and draw graphics on a page. ```text BT /F1 12 Tf 100 700 Td (Hello, World!) Tj ET q 200 300 100 150 re f Q /Im1 Do ``` -------------------------------- ### Configure Build-time Library Resolution Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/crates/pdfium-auto/README.md Demonstrates how to specify a local path for the PDFium library during the build process to avoid automatic downloads. ```bash PDFIUM_BUNDLE_LIB=/path/to/libpdfium.dylib cargo build --release ``` -------------------------------- ### Configure PDF Conversion using Builder Pattern Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/06-api-design.md Demonstrates how to initialize a ConversionConfig object using the builder pattern. This allows for setting specific parameters like DPI, concurrency, and the LLM model before initiating the conversion process. ```rust use edgequake_pdf2md::ConversionConfig; let config = ConversionConfig::builder() .dpi(150) .concurrency(10) .model("gpt-4o") .build() .unwrap(); ``` -------------------------------- ### GitHub Actions CI Workflow Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/09-testing-strategy.md A YAML configuration for GitHub Actions that handles dependency installation, linting, and conditional execution of golden tests. ```yaml name: CI on: [push, pull_request] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - run: cargo test --workspace golden: runs-on: ubuntu-latest if: github.event_name == 'push' && github.ref == 'refs/heads/main' env: PDF2MD_GOLDEN: "1" OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} steps: - uses: actions/checkout@v4 - run: cargo test --test golden ``` -------------------------------- ### Configuration File Support Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/05-crate-selection.md Uses toml as an optional dependency for loading external configuration files. ```toml [dependencies] toml = { version = "0.8", optional = true } ``` -------------------------------- ### Markdown Output with Page Separators Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Demonstrates how to add visual separators between pages in the Markdown output. This example uses 'hr' (horizontal rule) as the separator. ```bash pdf2md --separator hr document.pdf -o output.md ``` -------------------------------- ### Load PDF Document with Pdfium Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/02-algorithm.md Demonstrates how to initialize the Pdfium engine and load a PDF file from a given path. It handles basic document loading and metadata extraction. ```rust let pdfium = Pdfium::default(); let doc = pdfium.load_pdf_from_file(&path, password)?; let page_count = doc.pages().len(); let metadata = extract_metadata(&doc); ``` -------------------------------- ### Initialize Bundled PDFium Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/crates/pdfium-auto/README.md Shows how to bind the bundled PDFium library at runtime. The library is extracted to the local cache on the first call and reused thereafter. ```rust use pdfium_auto::bind_bundled; // Extracts lib on first call; cached on subsequent calls. let pdfium = bind_bundled().expect("PDFium unavailable"); ``` -------------------------------- ### LLM Provider Configuration Precedence Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/configuration.md Explains the order in which EdgeQuake resolves LLM provider settings, starting from explicit configuration to environment variables and auto-detection. ```APIDOC ## LLM Provider Configuration Precedence Provider resolution follows this order (first match wins): 1. `config.provider` (pre-built `Arc`) ↓ (if None) 2. `config.provider_name` + `config.model` (creates provider from name) ↓ (if None) 3. `EDGEQUAKE_LLM_PROVIDER` + `EDGEQUAKE_MODEL` (from env) ↓ (if not set) 4. `ProviderFactory::from_env()` (auto-detect from API keys) ``` -------------------------------- ### Automated pdfium Download via build.rs Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/05-crate-selection.md Describes the recommended approach for CI environments, where a `build.rs` script automatically downloads the appropriate pre-built pdfium binary for the target platform from the bblanchon/pdfium-binaries releases. This ensures reproducible builds. ```rust // build.rs script content would go here, automating the download process. // Example: // fn main() { // let target = std::env::var("TARGET").unwrap(); // let pdfium_url = match target.as_str() { // "x86_64-unknown-linux-gnu" => "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-linux-x64.tar.gz", // "x86_64-apple-darwin" => "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-mac-x64.tar.gz", // "aarch64-apple-darwin" => "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-mac-arm64.tar.gz", // "x86_64-pc-windows-msvc" => "https://github.com/bblanchon/pdfium-binaries/releases/latest/download/pdfium-windows-x64.zip", // _ => panic!("Unsupported target: {}", target) // }; // // Code to download and extract the archive, and set the PDFIUM_... environment variable. // } ``` -------------------------------- ### PDF Cross-Reference Stream Structure Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/03-pdf-internals.md An example of the internal structure of a PDF 1.5+ cross-reference stream, which uses compressed object streams instead of traditional xref tables. ```pdf 15 0 obj << /Type /XRef /W [1 3 1] /Filter /FlateDecode /Length 42 >> stream ...compressed xref data... endstream ``` -------------------------------- ### Configure Google Gemini Provider via CLI Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Sets the Gemini API key and runs the tool with a specific model. ```bash export GEMINI_API_KEY="AI..." pdf2md --provider gemini --model gemini-2.0-flash document.pdf ``` -------------------------------- ### Model Selection Guide Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Provides guidance on choosing the appropriate LLM model based on specific requirements such as cost, quality, complex data handling, offline processing, or enterprise compliance. ```APIDOC ## Choosing a Model ### Description This guide helps you select the most suitable LLM model for your `pdf2md` conversion task based on factors like cost, performance, specific features, and deployment environment. ### Model Recommendations - **Cheapest Option**: - Models: `gpt-4.1-nano` or `gemini-2.0-flash` - Benefit: Lowest cost, approximately $0.02 per 50 pages. - **Best Quality**: - Models: `gpt-4.1` or `gemini-2.5-pro` - Benefit: Highest accuracy and performance for general tasks. - **Complex Table Extraction**: - Model: `claude-sonnet-4-20250514` - Benefit: Optimized for understanding and extracting structured data from tables. - **Offline/Private Processing**: - Provider: Ollama - Model Example: `llama3.2-vision` - Benefit: Run models locally for privacy, security, or air-gapped environments. - **Enterprise Compliance**: - Provider: Azure OpenAI - Benefit: Meets enterprise-grade security, compliance, and scalability requirements. Requires an Azure OpenAI deployment. ``` -------------------------------- ### Rust Tracing Integration for Error Logging Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/08-error-handling.md Shows how tracing events are emitted for different error severities (fatal and recoverable) before returning errors. Includes an example of enabling structured JSON logs. ```rust // Fatal tracing::error!( error = %e, path = %path.display(), "pdf rasterisation failed" ); // Per-page recoverable tracing::warn!( page = page_num, attempt = attempt, error = %api_error, "llm call failed, retrying" ); ``` -------------------------------- ### Provider-Specific API Key Configuration Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Examples of setting API keys for different LLM providers (OpenAI, Anthropic, Google Gemini, Ollama) to use with pdf2md. This allows flexibility in choosing the AI model. ```bash export OPENAI_API_KEY="sk-..." pdf2md --model gpt-4.1-nano document.pdf export ANTHROPIC_API_KEY="sk-ant-" pdf2md --provider anthropic --model claude-sonnet-4-20250514 document.pdf export GEMINI_API_KEY="AI..." pdf2md --provider gemini --model gemini-2.0-flash document.pdf ollama pull llava pdf2md --provider ollama --model llava document.pdf ``` -------------------------------- ### Configure OpenAI Provider via CLI Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Demonstrates how to set the OpenAI API key and execute the pdf2md tool using default or specific models. ```bash export OPENAI_API_KEY="sk-..." pdf2md document.pdf pdf2md --model gpt-4.1 document.pdf pdf2md --model gpt-4.1-mini document.pdf ``` -------------------------------- ### Build and Run PDF to Markdown Conversion Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Commands to compile the project from source and execute the CLI to convert local PDF files or remote URLs into Markdown documents. ```bash cargo build --release ./target/release/pdf2md document.pdf -o output.md ./target/release/pdf2md https://arxiv.org/pdf/1706.03762 -o paper.md ./target/release/pdf2md --inspect-only document.pdf ``` -------------------------------- ### Create Release Steps (Bash & TOML) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/CONTRIBUTING.md A sequence of commands and configuration file modifications for creating a new release. This includes updating the version in Cargo.toml, updating the CHANGELOG.md, running pre-publish checks, committing changes, and creating a git tag. ```toml [package] version = "0.2.0" # Updated from 0.1.0 ``` ```markdown ## [0.2.0] - 2026-02-19 ### Added - New feature X ### Fixed - Bug fix Y ``` ```bash # Run pre-publish checks ./scripts/pre-publish-check.sh --version 0.2.0 ``` ```bash # Commit the changes git add Cargo.toml CHANGELOG.md git commit -m "chore: release v0.2.0" ``` ```bash # Create a git tag git tag v0.2.0 git push origin main v0.2.0 ``` -------------------------------- ### Rust CLI Error Handling with Anyhow Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/08-error-handling.md Demonstrates how the CLI binary uses the 'anyhow' crate to wrap library errors with context, providing more informative error messages to the user. ```rust // bin/pdf2md.rs let result = convert(&cli.input, &config).await .with_context(|| format!("Converting '{}'", cli.input))?; ``` -------------------------------- ### Static Linking pdfium (Self-contained Binary) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/05-crate-selection.md Instructions for statically linking the pdfium library to create a self-contained binary. This requires downloading the static pdfium library and setting an environment variable to its path before building the release version of the project. ```bash # Download static pdfium from bblanchon/pdfium-binaries (release build) export PDFIUM_STATIC_LIB_PATH=/path/to/dir/containing/libpdfium.a cargo build --release ``` -------------------------------- ### Convert PDF to Markdown Function (Rust) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/CONTRIBUTING.md Provides a Rust function to convert a PDF file or URL to Markdown format using a vision LLM. It includes documentation comments explaining arguments, return types, and usage examples. Dependencies include the edgequake_pdf2md crate and tokio for async operations. ```rust /// Converts a PDF to Markdown using a vision LLM. /// /// # Arguments /// /// * `input` - Path to PDF file or URL /// * `config` - Conversion configuration /// /// # Example /// /// ``` /// use edgequake_pdf2md::{convert, ConversionConfig}; /// /// # tokio::runtime::Runtime::new().unwrap().block_on(async { /// let md = convert("input.pdf", Config::default()).await.unwrap(); /// # }); /// ``` pub async fn convert(input: impl AsRef, config: ConversionConfig) -> Result { // implementation } ``` -------------------------------- ### Integrate Azure OpenAI via edgequake-litellm Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/providers.md Shows how to use the Python library to route completion requests to Azure OpenAI and list available providers. Requires the edgequake_litellm package. ```python import edgequake_litellm as litellm # Route to Azure OpenAI response = litellm.completion( model="azure/gpt-4o", messages=[{"role": "user", "content": "Summarise this PDF page."}] ) print(response.choices[0].message.content) # List all supported providers print(litellm.list_providers()) ``` -------------------------------- ### Rust Library: Per-Page Progress Callbacks Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Illustrates how to implement and use custom progress callbacks to monitor the conversion process page by page. ```APIDOC ## Library Usage: Per-Page Progress Callbacks (v0.2) ### Description Implement the `ConversionProgressCallback` trait to receive updates on the conversion progress, including page completion and errors. ### Rust Code Example ```rust use edgequake_pdf2md::{convert, ConversionConfig, ConversionProgressCallback}; use std::sync::Arc; struct MyProgress; impl ConversionProgressCallback for MyProgress { fn on_conversion_start(&self, total: usize) { eprintln!("Starting conversion of {total} pages"); } fn on_page_complete(&self, page: usize, total: usize, chars: usize) { eprintln!(" ✓ Page {page}/{total} — {chars} chars"); } fn on_page_error(&self, page: usize, total: usize, error: String) { eprintln!(" ✗ Page {page}/{total} failed: {error}"); } fn on_conversion_complete(&self, total: usize, success: usize) { eprintln!("Done: {success}/{total} pages converted"); } } #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::builder() .progress_callback(Arc::new(MyProgress) as Arc) .build()?; let output = convert("document.pdf", &config).await?; println!("{}", output.markdown); Ok(()) } ``` ``` -------------------------------- ### Resumable PDF to Markdown Conversion with Checkpoints Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Demonstrates how to perform a resumable PDF to Markdown conversion using the pdf2md CLI. The `--checkpoint-dir` option allows the process to be resumed after an interruption by re-running the same command. To start a fresh conversion and ignore existing checkpoints, use the `--no-resume` flag. ```bash pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md ``` ```bash pdf2md --checkpoint-dir ./checkpoints big-doc.pdf -o out.md ``` ```bash pdf2md --checkpoint-dir ./checkpoints --no-resume big-doc.pdf -o out.md ``` -------------------------------- ### Per-Page Progress Callbacks in Rust Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md This Rust code illustrates how to implement custom per-page progress callbacks during PDF to Markdown conversion. By implementing the `ConversionProgressCallback` trait, you can receive notifications for the start of conversion, completion or failure of individual pages, and the overall completion status. This allows for real-time feedback or custom logging. ```rust use edgequake_pdf2md::{convert, ConversionConfig, ConversionProgressCallback}; use std::sync::Arc; struct MyProgress; impl ConversionProgressCallback for MyProgress { fn on_conversion_start(&self, total: usize) { eprintln!("Starting conversion of {total} pages"); } fn on_page_complete(&self, page: usize, total: usize, chars: usize) { eprintln!(" ✓ Page {page}/{total} — {chars} chars"); } fn on_page_error(&self, page: usize, total: usize, error: String) { eprintln!(" ✗ Page {page}/{total} failed: {error}"); } fn on_conversion_complete(&self, total: usize, success: usize) { eprintln!("Done: {success}/{total} pages converted"); } } #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::builder() .progress_callback(Arc::new(MyProgress) as Arc) .build()?; let output = convert("document.pdf", &config).await?; println!("{}", output.markdown); Ok(()) } ``` -------------------------------- ### Convert PDF Bytes in Memory using Rust Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md This Rust example demonstrates converting PDF data directly from memory (bytes) without needing temporary files. It reads the PDF content into a byte vector and then uses the `convert_from_bytes` function from the `edgequake-pdf2md` library. This is useful for PDFs obtained from databases or network streams. ```rust use edgequake_pdf2md::{convert_from_bytes, ConversionConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let bytes = std::fs::read("document.pdf")?; let config = ConversionConfig::default(); let output = convert_from_bytes(&bytes, &config).await?; println!("{}", output.markdown); Ok(()) } ``` -------------------------------- ### Manual Publishing to Crates.io (Bash) Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/CONTRIBUTING.md Instructions for manually publishing the project to crates.io using cargo. This involves ensuring all checks pass and then executing the cargo publish command with the necessary token. ```bash # Ensure all checks pass ./scripts/pre-publish-check.sh --version 0.2.0 ``` ```bash # Publish to crates.io cargo publish --token $CARGO_REGISTRY_TOKEN ``` -------------------------------- ### VLM Inference with System Prompt Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/how-it-works.md Performs VLM inference for each page image, constructing a system prompt with 7 rules for faithful conversion. Supports parallel or sequential processing, with retries for API calls. The system prompt guides the VLM to preserve content, structure, and specific formats like tables and code. ```plaintext For each page image: │ ├── Build system prompt (7 rules for faithful conversion) │ ├── Optional: maintain_format context (previous page markdown) │ ├── Send to VLM with image attachment │ └── Retry up to --max-retries times (exponential backoff) │ └── Receive markdown text for that page ``` -------------------------------- ### Basic PDF to Markdown Conversion (CLI) Source: https://context7.com/raphaelmansuy/edgequake-pdf2md/llms.txt Demonstrates basic conversion of a PDF file to Markdown using the command-line interface. Supports output to stdout or a file, conversion from URLs, and specifying pages, fidelity, and DPI. ```bash # Set API credentials (choose one provider) export AWS_ACCESS_KEY_ID="AKIA..." export AWS_SECRET_ACCESS_KEY="..." # Or: export OPENAI_API_KEY="sk-..." # Or: export ANTHROPIC_API_KEY="sk-ant-..." # Basic conversion to stdout pdf2md document.pdf # Convert to file pdf2md document.pdf -o output.md # Convert from URL pdf2md https://arxiv.org/pdf/1706.03762 -o paper.md # Specific pages with high fidelity pdf2md --pages 1-5 --fidelity tier3 --dpi 200 paper.pdf -o paper.md # Use a specific provider and model pdf2md --provider openai --model gpt-4.1 document.pdf -o output.md # Sequential mode for consistent formatting across pages pdf2md --maintain-format --separator hr book.pdf -o book.md # JSON output with metadata pdf2md --json --metadata document.pdf > output.json # Inspect PDF metadata only (no API key needed) pdf2md --inspect-only document.pdf ``` -------------------------------- ### Implement Conversion Progress Callback in Rust Source: https://context7.com/raphaelmansuy/edgequake-pdf2md/llms.txt This Rust code demonstrates how to implement the `ConversionProgressCallback` trait to receive real-time updates during the PDF to Markdown conversion process. It includes methods for handling the start and completion of the conversion, as well as individual page processing, errors, and resumes. This is useful for displaying progress bars or logging conversion status. ```rust use edgequake_pdf2md::{convert, ConversionConfig, ConversionProgressCallback}; use std::sync::{Arc, atomic::{AtomicUsize, Ordering}}; struct MyProgressCallback { completed: AtomicUsize, failed: AtomicUsize, } impl ConversionProgressCallback for MyProgressCallback { fn on_conversion_start(&self, total_pages: usize) { eprintln!("Starting conversion of {} pages", total_pages); } fn on_page_start(&self, page_num: usize, total_pages: usize) { eprintln!(" Processing page {}/{}...", page_num, total_pages); } fn on_page_complete(&self, page_num: usize, total_pages: usize, markdown_len: usize) { self.completed.fetch_add(1, Ordering::SeqCst); eprintln!(" ✓ Page {}/{} - {} chars", page_num, total_pages, markdown_len); } fn on_page_error(&self, page_num: usize, total_pages: usize, error: String) { self.failed.fetch_add(1, Ordering::SeqCst); eprintln!(" ✗ Page {}/{} failed: {}", page_num, total_pages, error); } fn on_page_resumed(&self, page_num: usize, total_pages: usize) { self.completed.fetch_add(1, Ordering::SeqCst); eprintln!(" ↻ Page {}/{} resumed from checkpoint", page_num, total_pages); } fn on_conversion_complete(&self, total_pages: usize, success_count: usize) { eprintln!("Completed: {}/{} pages successful", success_count, total_pages); } } #[tokio::main] async fn main() -> Result<(), Box> { let callback = Arc::new(MyProgressCallback { completed: AtomicUsize::new(0), failed: AtomicUsize::new(0), }); let config = ConversionConfig::builder() .progress_callback(callback.clone() as Arc) .build()?; let output = convert("document.pdf", &config).await?; println!("\nFinal stats:"); println!(" Completed: {}", callback.completed.load(Ordering::SeqCst)); println!(" Failed: {}", callback.failed.load(Ordering::SeqCst)); Ok(()) } ``` -------------------------------- ### Strict Error Handling for Partial Failures in Rust Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md This Rust example shows how to enforce strict error handling for partial conversion failures. By default, `pdf2md` treats page-level errors as non-fatal. Calling `.into_result()?` on the conversion output will promote any page failures into a fatal `Err(PartialFailure)`, ensuring that the entire operation is considered failed if any page encounters an issue. ```rust use edgequake_pdf2md::{convert, ConversionConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::default(); let output = convert("document.pdf", &config).await?.into_result()?; println!("{}", output.markdown); Ok(()) } ``` -------------------------------- ### Perform PDF to Markdown Conversion in Rust Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/specs/06-api-design.md Demonstrates various ways to convert PDF documents to Markdown, including default settings, custom LLM providers like Anthropic or Ollama, streaming page results, and direct file output. ```rust use edgequake_pdf2md::convert; let output = convert("document.pdf", &Default::default()).await?; println!("{}", output.markdown); ``` ```rust use edgequake_pdf2md::{convert, ConversionConfig}; use edgequake_llm::{AnthropicProvider, LLMProvider}; use std::sync::Arc; let provider = AnthropicProvider::from_env()?; let config = ConversionConfig::builder().provider(Arc::new(provider)).model("claude-3-5-sonnet-20241022").build()?; let output = convert("math_paper.pdf", &config).await?; ``` ```rust use edgequake_pdf2md::{convert_stream, ConversionConfig, PageSelection}; use tokio_stream::StreamExt; let config = ConversionConfig::builder().pages(PageSelection::Range(5..=15)).build()?; let mut stream = convert_stream("large_report.pdf", &config).await?; while let Some(r) = stream.next().await { let page = r?; } ``` ```rust use edgequake_pdf2md::convert_to_file; let stats = convert_to_file("https://arxiv.org/pdf/2310.12345", "paper.md", &Default::default()).await?; ``` -------------------------------- ### Rust Library: Basic Conversion Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/README.md Demonstrates how to use the `edgequake-pdf2md` Rust library for basic PDF to Markdown conversion with custom configuration. ```APIDOC ## Library Usage: Basic Conversion ### Description Add `edgequake-pdf2md` to your `Cargo.toml` and use the `convert` function for basic PDF to Markdown conversion. ### Cargo.toml ```toml [dependencies] edgequake-pdf2md = "0.7" tokio = { version = "1", features = ["full"] } ``` ### Rust Code Example ```rust use edgequake_pdf2md::{convert, ConversionConfig}; #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::builder() .model("amazon.nova-lite-v1:0") .provider_name("bedrock") .pages(edgequake_pdf2md::PageSelection::Range(1, 5)) .build()?; let output = convert("document.pdf", &config).await?; println!("{}", output.markdown); println!("Processed {}/{} pages", output.stats.processed_pages, output.stats.total_pages); Ok(()) } ``` ``` -------------------------------- ### Configure Conversion Settings with Rust Library API Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/configuration.md Demonstrates how to programmatically define conversion settings using the ConversionConfig builder pattern in Rust. This allows fine-grained control over DPI, concurrency, model selection, and output quality tiers. ```rust use edgequake_pdf2md::{ConversionConfig, FidelityTier, PageSelection, PageSeparator}; let config = ConversionConfig::builder() .dpi(200) // Higher resolution .concurrency(5) // Fewer concurrent calls .model("gpt-4.1") // Specific model .provider_name("openai") // Specific provider .temperature(0.0) // Deterministic output .max_tokens(8192) // Longer page output .max_retries(5) // More retries .fidelity(FidelityTier::Tier3) // Highest quality .pages(PageSelection::Range(1, 10)) // Pages 1–10 .page_separator(PageSeparator::HorizontalRule) .include_metadata(true) // YAML front-matter .maintain_format(true) // Sequential processing .download_timeout_secs(300) // Longer download timeout .api_timeout_secs(120) // Longer API timeout .build() .expect("Invalid config"); ``` -------------------------------- ### Stream PDF Conversion in Rust Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/examples.md Demonstrates how to convert a PDF file to Markdown using an asynchronous stream. This approach allows for processing pages individually as they are converted, providing real-time feedback on progress and token usage. ```rust use edgequake_pdf2md::{convert_stream, ConversionConfig}; use tokio_stream::StreamExt; #[tokio::main] async fn main() -> Result<(), Box> { let config = ConversionConfig::builder() .concurrency(5) .build()?; let mut stream = convert_stream("document.pdf", &config).await?; while let Some(result) = stream.next().await { match result { Ok(page) => println!("Page {} done ({} tokens)", page.page_num, page.output_tokens), Err(e) => eprintln!("Page error: {}", e), } } Ok(()) } ``` -------------------------------- ### Recommended Settings by Use Case Source: https://github.com/raphaelmansuy/edgequake-pdf2md/blob/main/docs/configuration.md Provides a table of recommended DPI, Model, Concurrency, Fidelity, and Maintain Format settings for different document processing use cases. ```APIDOC ## Recommended Settings by Use Case | Use Case | DPI | Model | Concurrency | Fidelity | Maintain Format | |----------|-----|-------|-------------|----------|----------------| | Quick text extraction | 100 | gpt-4.1-nano | 20 | tier1 | no | | General documents | 150 | gpt-4.1-nano | 10 | tier2 | no | | Academic papers | 200 | gpt-4.1 | 5 | tier3 | no | | Books (format consistency) | 150 | gpt-4.1-mini | 1 | tier2 | yes | | Forms & tables | 200 | claude-sonnet-4-20250514 | 5 | tier3 | no | | Budget batch processing | 100 | gpt-4.1-nano | 20 | tier1 | no | ```