### Install Reducto SDK (Go) Source: https://docs.reducto.ai/quickstart Installs the Reducto Go SDK using the go get command. This allows Go programs to integrate with the Reducto API for extracting data from documents. ```bash go get github.com/reductoai/reducto-go-sdk ``` -------------------------------- ### Install Reducto SDK (Python) Source: https://docs.reducto.ai/quickstart Installs the Reducto AI Python SDK using pip. This package provides the necessary tools to interact with the Reducto API for document parsing. ```bash pip install reductoai ``` -------------------------------- ### Install Reducto SDK (Node.js) Source: https://docs.reducto.ai/quickstart Installs the Reducto AI Node.js SDK using npm. This package enables JavaScript and Node.js applications to leverage the Reducto API for document processing. ```bash npm install reductoai ``` -------------------------------- ### Extract Text, Tables, and Figures from Documents using the Reducto API (Quickstart) Source: https://docs.reducto.ai/llms.txt This quickstart guide provides a basic example of using the Reducto API to extract text, tables, and figures from documents. It serves as an entry point for new users. ```python from reducto import Reducto # Initialize Reducto client reducto = Reducto() # Example: Basic extraction from a document # Replace 'path/to/your/document.pdf' with the actual file path # extracted_content = reducto.extract( # file='path/to/your/document.pdf' # ) # print(extracted_content) # This basic extraction will return the processed content of the document. ``` -------------------------------- ### Parse Document and Get Results (Node.js) Source: https://docs.reducto.ai/quickstart Initiates document parsing with the Reducto Node.js SDK by calling `client.parse.run()`. This method takes the upload reference as input and returns a result object containing job details and processing information. Ensure the upload has completed before calling this. ```javascript // Parse the uploaded document const result = await client.parse.run({ input: upload }); console.log(`Job ID: ${result.job_id}`); ``` -------------------------------- ### Install Dataset Dependencies Source: https://docs.reducto.ai/cookbooks/batch-processing Installs the required libraries for interacting with the Hugging Face Hub in Python and JavaScript environments. ```bash pip install datasets ``` ```bash npm install @huggingface/hub ``` -------------------------------- ### Parse Document and Get Results (Python) Source: https://docs.reducto.ai/quickstart Initiates the document parsing process using the Reducto Python SDK. The `parse.run()` method takes the upload reference and returns a result object containing job details, usage information, and extracted content chunks. This process includes OCR, layout detection, and table extraction. ```python # Parse the uploaded document result = client.parse.run(input=upload) # Check what we got back print(f"Job ID: {result.job_id}") print(f"Pages processed: {result.usage.num_pages}") print(f"Credits used: {result.usage.credits}") print(f"Number of chunks: {len(result.result.chunks)}") ``` -------------------------------- ### Reducto Edit Command Examples Source: https://docs.reducto.ai/cli Provides practical examples of the 'reducto edit' command. These examples illustrate how to specify edits for filling in text, setting dates, and selecting options within a document. ```bash reducto edit contract.pdf -i "Fill in the client name as 'Acme Corporation' and set the contract date to January 15, 2024" reducto edit document.pdf -i "Fill out the form with: Name: John Doe, Email: john@example.com, Select 'Yes' for newsletter subscription" ``` -------------------------------- ### Install Dependencies and Configure Environment Source: https://docs.reducto.ai/cookbooks/web-browsing-browserbase Commands to install necessary Python packages and set required environment variables for Browserbase, Stagehand, and Reducto integration. ```bash pip install browserbase stagehand reductoai python-dotenv export BROWSERBASE_API_KEY="your-browserbase-api-key" export BROWSERBASE_PROJECT_ID="your-browserbase-project-id" export REDUCTO_API_KEY="your-reducto-api-key" export GOOGLE_API_KEY="your-google-api-key" ``` -------------------------------- ### Install Reducto SDKs Source: https://docs.reducto.ai/cookbooks/redlined-legal-contracts Installs the necessary Reducto SDKs for Python and JavaScript. The Python SDK requires the 'requests' library, while the JavaScript SDK is installed via npm. ```bash pip install reductoai requests ``` ```bash npm install reductoai ``` -------------------------------- ### Effective Description Examples for Split Sections (Python) Source: https://docs.reducto.ai/configs/split/configuration Illustrates how to write specific and effective descriptions for section classification by an LLM, contrasting vague examples with clear criteria. ```python # Vague - could match many things {"name": "Tables", "description": "Pages with tables"} # Specific - clear criteria for classification {"name": "Transaction History", "description": "Table showing individual transactions with dates, descriptions, and amounts. Usually appears after the account summary section."} ``` -------------------------------- ### Multi-Region Reducto Configuration Example Source: https://docs.reducto.ai/onprem/hybrid-vpc-deployment An example JSON structure for providing multi-region configurations to Reducto. It lists configurations for different regions, including bucket name, region identifier, and IAM role ARN for each. ```json { "regions": [ { "bucket_name": "reducto-us-a1b2c3d4", "region": "us-east-1", "role_arn": "arn:aws:iam::987654321098:role/reducto-us-access" }, { "bucket_name": "reducto-eu-e5f6g7h8", "region": "eu-central-1", "role_arn": "arn:aws:iam::123456789012:role/reducto-eu-access" } ] } ``` -------------------------------- ### Webhook Payload Example Source: https://docs.reducto.ai/cookbooks/batch-processing This is an example of the JSON payload received by your webhook handler when a job completes. It includes the job status, ID, and any associated metadata. ```json { "status": "Completed", "job_id": "abc123", "metadata": {"filename": "invoice_0001.pdf"} } ``` -------------------------------- ### Initialize Reducto Client (Go) Source: https://docs.reducto.ai/quickstart This Go snippet demonstrates how to initialize the Reducto client. It imports necessary packages and uses an environment variable for the API key. The client is configured using `option.WithAPIKey()`. ```go package main import ( "context" "fmt" "io" "os" reducto "github.com/reductoai/reducto-go-sdk" "github.com/reductoai/reducto-go-sdk/option" "github.com/reductoai/reducto-go-sdk/shared" ) func main() { // Initialize client with API key from environment client := reducto.NewClient(option.WithAPIKey(os.Getenv("REDUCTO_API_KEY"))) } ``` -------------------------------- ### Complete Document Parsing Script (Python) Source: https://docs.reducto.ai/quickstart A complete Python script demonstrating the Reducto SDK workflow for parsing a document. It covers client initialization, file upload, parsing, and iterating through the extracted content to print text and identify tables. ```python from pathlib import Path from reducto import Reducto client = Reducto() upload = client.upload(file=Path("finance-statement.pdf")) result = client.parse.run(input=upload) print(f"Processed {result.usage.num_pages} pages") for chunk in result.result.chunks: print(chunk.content) for block in chunk.blocks: if block.type == "Table": print(f"Found table on page {block.bbox.page}") ``` -------------------------------- ### Poll for Job Status in Python and JavaScript Source: https://docs.reducto.ai/cookbooks/batch-processing Demonstrates how to implement a polling mechanism to monitor the status of an asynchronous job until completion or failure. Includes examples for both Python and JavaScript using the Reducto client SDK. ```python import asyncio from reducto import AsyncReducto client = AsyncReducto() async def wait_for_job(job_id: str, timeout: int = 300): """Poll until job completes or times out.""" for _ in range(timeout): status = await client.job.get(job_id) if status.status == "Completed": return {"success": True, "status": status} elif status.status == "Failed": return {"success": False, "status": status} await asyncio.sleep(1) return {"success": False, "error": "Timeout"} # Example: Submit job and wait async def submit_and_wait(path): upload = await client.upload(file=path) job = await client.parse.run_job(input=upload.file_id) print(f"Submitted job {job.job_id}, waiting...") result = await wait_for_job(job.job_id) if result["success"]: print(f"Job completed!") else: print(f"Job failed or timed out") return result ``` ```javascript import Reducto from "reductoai"; import fs from "fs"; const client = new Reducto(); async function waitForJob(jobId, timeout = 300) { for (let i = 0; i < timeout; i++) { const status = await client.job.get(jobId); if (status.status === "Completed") { return { success: true, status }; } else if (status.status === "Failed") { return { success: false, status }; } await new Promise(r => setTimeout(r, 1000)); } return { success: false, error: "Timeout" }; } // Example: Submit job and wait async function submitAndWait(filePath) { const upload = await client.upload({ file: fs.createReadStream(filePath) }); const job = await client.parse.runJob({ input: upload.file_id }); console.log(`Submitted job ${job.job_id}, waiting...`); const result = await waitForJob(job.job_id); if (result.success) { console.log("Job completed!"); } else { console.log("Job failed or timed out"); } return result; } ``` -------------------------------- ### Go SDK: Upload and Parse Document Source: https://docs.reducto.ai/quickstart This Go code snippet demonstrates how to use the Reducto Go SDK to upload a PDF file and then parse its content. It initializes the client, uploads the file, and processes the parsing results, including iterating through chunks and identifying tables. Dependencies include the reducto-go-sdk and its shared and option packages. ```go package main import ( "context" "fmt" "io" "os" reducto "github.com/reductoai/reducto-go-sdk" "github.com/reductoai/reducto-go-sdk/option" "github.com/reductoai/reducto-go-sdk/shared" ) func main() { client := reducto.NewClient(option.WithAPIKey(os.Getenv("REDUCTO_API_KEY"))) file, _ := os.Open("finance-statement.pdf") defer file.Close() upload, _ := client.Upload(context.Background(), reducto.UploadParams{ File: reducto.F[io.Reader](file), }) result, _ := client.Parse.Run(context.Background(), reducto.ParseRunParams{ ParseConfig: reducto.ParseConfigParam{ DocumentURL: reducto.F[reducto.ParseConfigDocumentURLUnionParam]( shared.UnionString(upload.FileID), ), }, }) fmt.Printf("Processed %d pages\n", result.Usage.NumPages) if result.Result.Type == shared.ParseResponseResultTypeFull { chunks, _ := result.Result.Chunks.([]shared.ParseResponseResultFullResultChunk) for _, chunk := range chunks { fmt.Println(chunk.Content) for _, block := range chunk.Blocks { // Use SDK constants for block type comparisons if block.Type == shared.ParseResponseResultFullResultChunksBlocksTypeTable { fmt.Printf("Found table on page %d\n", block.Bbox.Page) } } } } } ``` -------------------------------- ### Perform Multi-Schema Extraction via Job Chaining Source: https://docs.reducto.ai/cookbooks/batch-processing This example demonstrates how to upload a file, parse it once, and perform multiple distinct extractions using the same job ID. It supports both Python and JavaScript SDKs to minimize API costs. ```python import asyncio from pathlib import Path from reducto import AsyncReducto client = AsyncReducto() async def extract_multiple_schemas(files: list[Path]): """Parse once, extract with multiple schemas.""" results = [] for path in files: # Parse once upload = await client.upload(file=path) parse_result = await client.parse.run(input=upload.file_id) job_id = parse_result.job_id # Extract headers (reuses parse) header_result = await client.extract.run( input=f"jobid://{job_id}", instructions={"schema": { "type": "object", "properties": { "invoice_number": {"type": "string"}, "customer_name": {"type": "string"}, "date": {"type": "string"} } }} ) # Extract line items (still reuses parse - no extra parse cost!) items_result = await client.extract.run( input=f"jobid://{job_id}", instructions={"schema": { "type": "object", "properties": { "line_items": { "type": "array", "items": { "type": "object", "properties": { "product": {"type": "string"}, "quantity": {"type": "number"}, "price": {"type": "number"} } } }, "total": {"type": "number"} } }} ) results.append({ "file": path.name, "header": header_result.result, "items": items_result.result }) return results ``` ```javascript import Reducto from "reductoai"; import fs from "fs"; const client = new Reducto(); async function extractMultipleSchemas(files) { const results = []; for (const filePath of files) { // Parse once const upload = await client.upload({ file: fs.createReadStream(filePath) }); const parseResult = await client.parse.run({ input: upload.file_id }); const jobId = parseResult.job_id; // Extract headers (reuses parse) const headerResult = await client.extract.run({ input: `jobid://${jobId}`, instructions: { schema: { type: "object", properties: { invoice_number: { type: "string" }, customer_name: { type: "string" }, date: { type: "string" } } } } }); // Extract line items (still reuses parse - no extra parse cost!) const itemsResult = await client.extract.run({ input: `jobid://${jobId}`, instructions: { schema: { type: "object", properties: { line_items: { type: "array", items: { type: "object", properties: { product: { type: "string" }, quantity: { type: "number" }, price: { type: "number" } } } }, total: { type: "number" } } } } }); results.push({ file: filePath, header: headerResult.result[0], items: itemsResult.result[0] }); } return results; } ``` -------------------------------- ### Initialize Reducto Client (Python) Source: https://docs.reducto.ai/quickstart Initializes the Reducto client in Python. The client automatically reads the API key from the REDUCTO_API_KEY environment variable if not explicitly provided. It requires the 'reducto' and 'pathlib' libraries. ```python from pathlib import Path from reducto import Reducto # The client reads REDUCTO_API_KEY from your environment client = Reducto() ``` -------------------------------- ### Extract Structured Data from Invoices using Reducto API Source: https://docs.reducto.ai/cookbooks/batch-processing Demonstrates how to define a JSON schema for invoice fields and use the Reducto client to extract data from PDF files. The examples include batch processing with concurrency limits to handle multiple files efficiently. ```python import asyncio import json from pathlib import Path from reducto import AsyncReducto from tqdm.asyncio import tqdm client = AsyncReducto() invoice_schema = { "type": "object", "properties": { "order_id": {"type": "string", "description": "Order ID at top of invoice"}, "customer_name": {"type": "string", "description": "Ship To customer name"}, "order_date": {"type": "string", "description": "Order date"}, "shipped_date": {"type": "string", "description": "Shipped date"}, "ship_address": {"type": "string", "description": "Full shipping address"}, "line_items": { "type": "array", "description": "Products ordered", "items": { "type": "object", "properties": { "product": {"type": "string"}, "quantity": {"type": "number"}, "unit_price": {"type": "number"}, "discount": {"type": "number"}, "extended_price": {"type": "number"} } } }, "subtotal": {"type": "number"}, "freight": {"type": "number"}, "total": {"type": "number"} } } async def extract_invoices(input_dir: Path, output_dir: Path, max_concurrency: int = 30): output_dir.mkdir(exist_ok=True) files = list(input_dir.glob("*.pdf"))[:100] sem = asyncio.Semaphore(max_concurrency) async def extract(path: Path): async with sem: upload = await client.upload(file=path) result = await client.extract.run(input=upload.file_id, instructions={"schema": invoice_schema}) output_path = output_dir / f"{path.stem}.json" output_path.write_text(json.dumps(result.result, indent=2)) return {"file": path.name, "success": True, "data": result.result} tasks = [extract(f) for f in files] return await tqdm.gather(*tasks, desc="Extracting") ``` ```javascript import Reducto from "reductoai"; import fs from "fs"; import path from "path"; import pLimit from "p-limit"; const client = new Reducto(); const invoiceSchema = { type: "object", properties: { order_id: { type: "string", description: "Order ID at top of invoice" }, customer_name: { type: "string", description: "Ship To customer name" }, order_date: { type: "string", description: "Order date" }, shipped_date: { type: "string", description: "Shipped date" }, ship_address: { type: "string", description: "Full shipping address" }, line_items: { type: "array", description: "Products ordered", items: { type: "object", properties: { product: { type: "string" }, quantity: { type: "number" }, unit_price: { type: "number" }, discount: { type: "number" }, extended_price: { type: "number" } } } }, subtotal: { type: "number" }, freight: { type: "number" }, total: { type: "number" } } }; async function extractInvoices(inputDir, outputDir, maxConcurrency = 30) { fs.mkdirSync(outputDir, { recursive: true }); const files = fs.readdirSync(inputDir).filter(f => f.endsWith(".pdf")).slice(0, 100); const limit = pLimit(maxConcurrency); async function extractFile(fileName) { const filePath = path.join(inputDir, fileName); const upload = await client.upload({ file: fs.createReadStream(filePath) }); const result = await client.extract.run({ input: upload.file_id, instructions: { schema: invoiceSchema } }); fs.writeFileSync(path.join(outputDir, `${fileName}.json`), JSON.stringify(result, null, 2)); } await Promise.all(files.map(f => limit(() => extractFile(f)))); } ``` -------------------------------- ### Parse Document with Reducto (Go) Source: https://docs.reducto.ai/quickstart This Go snippet demonstrates how to initiate document parsing using the Reducto API. It calls `client.Parse.Run()` with the `FileID` obtained from the upload step. Note the specific type wrapping required for `DocumentURL` using `shared.UnionString()` and `reducto.F[...]()`. ```go result, err := client.Parse.Run(context.Background(), reducto.ParseRunParams{ ParseConfig: reducto.ParseConfigParam{ // The file ID must be wrapped in shared.UnionString() and reducto.F[...]() DocumentURL: reducto.F[reducto.ParseConfigDocumentURLUnionParam]( shared.UnionString(upload.FileID), ), }, }) if err != nil { fmt.Printf("Parse error: %v\n", err) return } fmt.Printf("Job ID: %s\n", result.JobID) fmt.Printf("Pages: %d\n", result.Usage.NumPages) // Note: To view in Studio, construct the URL: https://studio.reducto.ai/job/{job_id} ``` -------------------------------- ### Initialize Reducto Client (Node.js) Source: https://docs.reducto.ai/quickstart Initializes the Reducto client in Node.js. It imports the necessary 'reductoai' module and the 'fs' module for file system operations. The client automatically uses the REDUCTO_API_KEY environment variable for authentication. ```javascript import Reducto from 'reductoai'; import fs from 'fs'; // The client reads REDUCTO_API_KEY from your environment const client = new Reducto(); ``` -------------------------------- ### Download and Save Dataset PDFs Source: https://docs.reducto.ai/cookbooks/batch-processing Downloads the Northwind invoices dataset from Hugging Face and saves the embedded PDF files to a local directory. It handles file system operations and stream processing for both Python and Node.js. ```python from datasets import load_dataset from pathlib import Path # Load the dataset dataset = load_dataset("AyoubChLin/northwind_invocies") # Create output directory output_dir = Path("./northwind_invoices") output_dir.mkdir(exist_ok=True) # Save PDFs to disk for i, sample in enumerate(dataset["train"]): pdf_path = output_dir / f"invoice_{i:04d}.pdf" pdf = sample["pdf"] pdf.stream.seek(0) with open(pdf_path, "wb") as f: f.write(pdf.stream.read()) print(f"Downloaded {len(dataset['train'])} invoices to {output_dir}") ``` ```javascript import { listFiles, downloadFile } from "@huggingface/hub"; import fs from "fs"; import path from "path"; // Load the dataset const repo = { type: "dataset", name: "AyoubChLin/northwind_invocies" }; // Create output directory const outputDir = "./northwind_invoices"; fs.mkdirSync(outputDir, { recursive: true }); // List and download PDF files let count = 0; for await (const fileInfo of listFiles({ repo })) { if (fileInfo.path.endsWith(".pdf")) { const response = await downloadFile({ repo, path: fileInfo.path }); const buffer = Buffer.from(await response.arrayBuffer()); const outputPath = path.join(outputDir, `invoice_${String(count).padStart(4, "0")}.pdf`); fs.writeFileSync(outputPath, buffer); count++; if (count % 100 === 0) console.log(`Downloaded ${count} invoices...`); } } console.log(`Downloaded ${count} invoices to ${outputDir}`); ``` -------------------------------- ### Configure document parsing options across languages Source: https://docs.reducto.ai/quickstart Demonstrates how to customize document parsing behavior, including AI-driven OCR cleanup, figure summarization, table formatting, and page range selection. Supports Python, Node.js, Go, and cURL implementations. ```python result = client.parse.run( input=upload, enhance={ # Use AI to clean up OCR errors in scanned documents "agentic": [{"scope": "text"}], # Generate descriptions for charts and images "summarize_figures": True }, formatting={ # Get tables as HTML, markdown, json, or csv "table_output_format": "markdown" }, settings={ # Only process pages 1-5 "page_range": {"start": 1, "end": 5} } ) ``` ```javascript const result = await client.parse.run({ input: upload, enhance: { agentic: [{scope: "text"}], summarize_figures: true }, formatting: { table_output_format: "markdown" }, settings: { page_range: {start: 1, end: 5} } }); ``` ```go result, err := client.Parse.Run(context.Background(), reducto.ParseRunParams{ ParseConfig: reducto.ParseConfigParam{ DocumentURL: reducto.F[reducto.ParseConfigDocumentURLUnionParam]( shared.UnionString(upload.FileID), ), // Output formatting - use SDK constants for table format AdvancedOptions: reducto.F(shared.AdvancedProcessingOptionsParam{ TableOutputFormat: reducto.F(shared.AdvancedProcessingOptionsTableOutputFormatMd), }), // Chunking options Options: reducto.F(shared.BaseProcessingOptionsParam{ Chunking: reducto.F(shared.BaseProcessingOptionsChunkingParam{ ChunkMode: reducto.F(shared.BaseProcessingOptionsChunkingChunkModeVariable), }), }), }, }) ``` ```bash curl -X POST "https://platform.reducto.ai/parse" \ -H "Authorization: Bearer $REDUCTO_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "input": "reducto://abc123def456.pdf", "enhance": { "agentic": [{"scope": "text"}], "summarize_figures": true }, "formatting": { "table_output_format": "markdown" }, "settings": { "page_range": {"start": 1, "end": 5} } }' ``` -------------------------------- ### IAM Role Configuration for Multi-Region Reducto Source: https://docs.reducto.ai/onprem/hybrid-vpc-deployment Example HCL configuration for `terraform.tfvars` in the EU central region. It sets the name prefix and defines Reducto principal ARNs and an external ID, crucial for IAM role setup in a multi-region deployment. ```hcl # environments/eu-central-1/terraform.tfvars name_prefix = "reducto-eu" reducto_principal_arns = [ "arn:aws:iam::731106932034:role/reducto-prod-eu", "arn:aws:iam::731106932034:user/modal-prod-eu" ] reducto_external_id = "" ``` -------------------------------- ### Process and Aggregate Invoice Data in Node.js Source: https://docs.reducto.ai/cookbooks/batch-processing This function iterates through a list of files, extracts data using a provided limit function, and saves the output as JSON. It then filters successful extractions to calculate a grand total across all processed invoices. ```javascript async function extractInvoices(inputDir, outputDir) { const files = fs.readdirSync(inputDir); async function extractFile(fileName) { try { const outputPath = path.join(outputDir, `${path.basename(fileName, ".pdf")}.json`); fs.writeFileSync(outputPath, JSON.stringify(result.result, null, 2)); return { file: fileName, success: true, data: result.result[0] }; } catch (e) { return { file: fileName, success: false, error: e.message }; } } const results = await Promise.all(files.map(f => limit(() => extractFile(f)))); const successes = results.filter(r => r.success); const grandTotal = successes.reduce((sum, r) => sum + (r.data?.total || 0), 0); console.log(`Grand total across invoices: $${grandTotal.toFixed(2)}`); return results; } ``` -------------------------------- ### Upload Document to Reducto (Go) Source: https://docs.reducto.ai/quickstart This Go code snippet shows how to upload a local PDF file to Reducto. It opens the file, uses `client.Upload()` with `reducto.UploadParams`, and handles potential errors. The returned `FileID` is used for subsequent parsing. ```go file, err := os.Open("finance-statement.pdf") if err != nil { fmt.Printf("Error opening file: %v\n", err) return } defer file.Close() upload, err := client.Upload(context.Background(), reducto.UploadParams{ File: reducto.F[io.Reader](file), }) if err != nil { fmt.Printf("Upload error: %v\n", err) return } fmt.Printf("Uploaded: %s\n", upload.FileID) ``` -------------------------------- ### Initialize S3 Client Source: https://docs.reducto.ai/cookbooks/multimodal-rag-image-results Sets up an S3 client using the AWS SDK. Requires the 'boto3' library for Python or '@aws-sdk/client-s3' for JavaScript. It also retrieves the S3 bucket name from environment variables. ```python import boto3 s3 = boto3.client("s3") bucket_name = os.environ["S3_BUCKET_NAME"] ``` ```javascript import { S3Client, PutObjectCommand, GetBucketLocationCommand } from "@aws-sdk/client-s3"; const s3 = new S3Client({ region: "us-east-1" }); // Will be updated after getting bucket region const bucketName = process.env.S3_BUCKET_NAME; ``` -------------------------------- ### Batch Invoice Processing in JavaScript Source: https://docs.reducto.ai/cookbooks/batch-processing Processes PDF invoices concurrently using JavaScript, Promise.all, and the p-limit package. It uploads each PDF, parses its content, and saves the results as JSON files. Concurrency is managed by p-limit. Dependencies include reductoai, fs, path, and p-limit. ```javascript import Reducto from "reductoai"; import fs from "fs"; import path from "path"; import pLimit from "p-limit"; // npm install p-limit const client = new Reducto(); async function processInvoices(inputDir, outputDir, maxConcurrency = 50) { fs.mkdirSync(outputDir, { recursive: true }); const files = fs.readdirSync(inputDir) .filter(f => f.endsWith(".pdf")) .map(f => path.join(inputDir, f)); console.log(`Found ${files.length} invoices to process`); // p-limit controls concurrency (like Python's Semaphore) const limit = pLimit(maxConcurrency); async function processFile(filePath) { try { const upload = await client.upload({ file: fs.createReadStream(filePath) }); const result = await client.parse.run({ input: upload.file_id }); // Save result immediately const outputPath = path.join(outputDir, `${path.basename(filePath, ".pdf")}.json`); fs.writeFileSync(outputPath, JSON.stringify({ source: path.basename(filePath), pages: result.usage.num_pages, content: result.result.chunks.map(c => c.content) }, null, 2)); return { file: path.basename(filePath), success: true, pages: result.usage.num_pages }; } catch (e) { return { file: path.basename(filePath), success: false, error: e.message }; } } // Process all files with concurrency limit const results = await Promise.all(files.map(f => limit(() => processFile(f)))); // Summary const successes = results.filter(r => r.success); const failures = results.filter(r => !r.success); const totalPages = successes.reduce((sum, r) => sum + r.pages, 0); console.log(`\nCompleted: ${successes.length} succeeded, ${failures.length} failed`); console.log(`Total pages processed: ${totalPages}`); if (failures.length > 0) { console.log("\nFailed files:"); failures.slice(0, 5).forEach(f => console.log(` - ${f.file}: ${f.error}`)); } return results; } // Run the batch const results = await processInvoices("./northwind_invoices", "./parsed_invoices"); ``` -------------------------------- ### Synchronous Document Processing (Python) Source: https://docs.reducto.ai/cookbooks/batch-processing Provides a synchronous alternative for processing PDF files using Python's threading capabilities. It uses a ThreadPoolExecutor to manage concurrent processing of files, suitable for environments where async is not available. Note that async is generally more efficient for I/O-bound tasks. ```python from concurrent.futures import ThreadPoolExecutor, as_completed from pathlib import Path from reducto import Reducto client = Reducto() def process_sync(input_dir: Path, max_workers: int = 10): files = list(input_dir.glob("*.pdf")) def process(path: Path): upload = client.upload(file=path) result = client.parse.run(input=upload.file_id) return {"file": path.name, "pages": result.usage.num_pages} results = [] with ThreadPoolExecutor(max_workers=max_workers) as executor: futures = {executor.submit(process, f): f for f in files} for future in as_completed(futures): try: results.append(future.result()) except Exception as e: results.append({"file": futures[future].name, "error": str(e)}) return results ``` -------------------------------- ### Upload Document for Parsing (Python) Source: https://docs.reducto.ai/quickstart Uploads a local PDF document to Reducto's servers using the Python SDK. The `upload()` method takes a file path and returns a reference necessary for the parsing step. Ensure the file exists at the specified path. ```python # Upload the PDF file to Reducto upload = client.upload(file=Path("finance-statement.pdf")) print(f"Uploaded: {upload}") ``` -------------------------------- ### Submit Jobs with Webhooks (Python) Source: https://docs.reducto.ai/cookbooks/batch-processing Submits PDF files from a directory for asynchronous processing using Reducto AI. Results are sent to a specified webhook URL upon job completion. This function uses asyncio for concurrent job submission and handles file uploads and job initiation. ```python import asyncio from pathlib import Path from reducto import AsyncReducto client = AsyncReducto() async def submit_with_webhooks( input_dir: Path, webhook_url: str, max_concurrency: int = 100 ): """Submit documents for processing with webhook notifications.""" files = list(input_dir.glob("*.pdf")) sem = asyncio.Semaphore(max_concurrency) async def submit(path: Path): async with sem: upload = await client.upload(file=path) job = await client.parse.run_job( input=upload.file_id, async_={ "webhook": {"mode": "direct", "url": webhook_url}, "metadata": {"filename": path.name} } ) return {"file": path.name, "job_id": job.job_id} tasks = [submit(f) for f in files] submissions = await asyncio.gather(*tasks) print(f"Submitted {len(submissions)} jobs - results will arrive via webhook") return submissions # Submit batch asyncio.run(submit_with_webhooks( input_dir=Path("./northwind_invoices"), webhook_url="https://your-app.com/webhook/reducto" )) ``` -------------------------------- ### Submit Jobs with Webhooks (JavaScript) Source: https://docs.reducto.ai/cookbooks/batch-processing Submits PDF files from a directory for asynchronous processing using Reducto AI. Results are sent to a specified webhook URL upon job completion. This function uses Promises and p-limit for concurrent job submission, handling file uploads and job initiation. ```javascript import Reducto from "reductoai"; import fs from "fs"; import path from "path"; import pLimit from "p-limit"; const client = new Reducto(); async function submitWithWebhooks(inputDir, webhookUrl, maxConcurrency = 100) { const files = fs.readdirSync(inputDir) .filter(f => f.endsWith(".pdf")) .map(f => path.join(inputDir, f)); const limit = pLimit(maxConcurrency); async function submitFile(filePath) { const upload = await client.upload({ file: fs.createReadStream(filePath) }); const job = await client.parse.runJob({ input: upload.file_id, async_: { webhook: { mode: "direct", url: webhookUrl }, metadata: { filename: path.basename(filePath) } } }); return { file: path.basename(filePath), job_id: job.job_id }; } const submissions = await Promise.all(files.map(f => limit(() => submitFile(f)))); console.log(`Submitted ${submissions.length} jobs - results will arrive via webhook`); return submissions; } // Submit batch await submitWithWebhooks("./northwind_invoices", "https://your-app.com/webhook/reducto"); ``` -------------------------------- ### POST /upload Source: https://docs.reducto.ai/quickstart Uploads a document to the Reducto platform to obtain a file reference ID for subsequent processing. ```APIDOC ## POST /upload ### Description Uploads a file to the Reducto platform. This endpoint returns a `file_id` which is required for the parsing process. ### Method POST ### Endpoint https://platform.reducto.ai/upload ### Request Body - **file** (binary) - Required - The document file to be uploaded. ### Request Example curl -X POST "https://platform.reducto.ai/upload" \ -H "Authorization: Bearer $REDUCTO_API_KEY" \ -F "file=@document.pdf" ### Response #### Success Response (200) - **file_id** (string) - A unique identifier for the uploaded file. #### Response Example { "file_id": "reducto://abc123def456.pdf" } ``` -------------------------------- ### POST /parse Source: https://docs.reducto.ai/quickstart Parses a document using specified enhancement, formatting, and page range settings. ```APIDOC ## POST /parse ### Description Parses a document from a provided URL or file reference. Supports AI-powered OCR cleanup, figure summarization, and custom table output formats. ### Method POST ### Endpoint https://platform.reducto.ai/parse ### Request Body - **input** (string) - Required - The document URI (e.g., "reducto://file_id") - **enhance** (object) - Optional - Configuration for AI enhancements - **agentic** (array) - Optional - List of scopes for AI cleanup (e.g., [{"scope": "text"}]) - **summarize_figures** (boolean) - Optional - Whether to generate descriptions for charts and images - **formatting** (object) - Optional - Output formatting options - **table_output_format** (string) - Optional - Format for tables: "html", "markdown", "json", "csv", or "ai_json" - **settings** (object) - Optional - Processing settings - **page_range** (object) - Optional - Range of pages to process (e.g., {"start": 1, "end": 5}) ### Request Example { "input": "reducto://abc123def456.pdf", "enhance": { "agentic": [{"scope": "text"}], "summarize_figures": true }, "formatting": { "table_output_format": "markdown" }, "settings": { "page_range": {"start": 1, "end": 5} } } ### Response #### Success Response (200) - **result** (object) - The parsed document content and metadata #### Response Example { "status": "success", "data": { "text": "...", "tables": [...] } } ``` -------------------------------- ### Batch Invoice Processing in Python Source: https://docs.reducto.ai/cookbooks/batch-processing Processes PDF invoices concurrently using Python's asyncio and AsyncReducto. It uploads each PDF, parses its content, and saves the results as JSON files. Concurrency is managed with asyncio.Semaphore, and progress is tracked using tqdm.asyncio. Dependencies include asyncio, json, pathlib, reducto, and tqdm. ```python import asyncio import json from pathlib import Path from reducto import AsyncReducto from tqdm.asyncio import tqdm client = AsyncReducto() async def process_invoices( input_dir: Path, output_dir: Path, max_concurrency: int = 50 ): """Process all invoices concurrently with progress tracking.""" output_dir.mkdir(exist_ok=True) files = list(input_dir.glob("*.pdf")) print(f"Found {len(files)} invoices to process") # Semaphore limits concurrent requests sem = asyncio.Semaphore(max_concurrency) async def process(path: Path): async with sem: try: upload = await client.upload(file=path) result = await client.parse.run(input=upload.file_id) # Save result immediately output_path = output_dir / f"{path.stem}.json" output_path.write_text(json.dumps({ "source": path.name, "pages": result.usage.num_pages, "content": [c.content for c in result.result.chunks] }, indent=2)) return {"file": path.name, "success": True, "pages": result.usage.num_pages} except Exception as e: return {"file": path.name, "success": False, "error": str(e)} # Process all files with progress bar tasks = [process(f) for f in files] results = await tqdm.gather(*tasks, desc="Processing invoices") # Summary successes = [r for r in results if r["success"]] failures = [r for r in results if not r["success"]] total_pages = sum(r["pages"] for r in successes) print(f"\nCompleted: {len(successes)} succeeded, {len(failures)} failed") print(f"Total pages processed: {total_pages}") if failures: print("\nFailed files:") for f in failures[:5]: # Show first 5 failures print(f" - {f['file']}: {f['error']}") return results # Run the batch results = asyncio.run(process_invoices( input_dir=Path("./northwind_invoices"), output_dir=Path("./parsed_invoices") )) ``` -------------------------------- ### POST /parse Source: https://docs.reducto.ai/quickstart Parses a document using a file reference ID or a public URL to extract structured content. ```APIDOC ## POST /parse ### Description Parses the content of a document. You can provide either a `file_id` obtained from the upload endpoint or a publicly accessible URL. ### Method POST ### Endpoint https://platform.reducto.ai/parse ### Request Body - **input** (string) - Required - The `file_id` or public URL of the document. ### Request Example { "input": "reducto://abc123def456.pdf" } ### Response #### Success Response (200) - **result** (object) - The parsed document content, including chunks and metadata. #### Response Example { "result": { "type": "full", "chunks": [...] } } ``` -------------------------------- ### Complete Reducto AI Workflow: Upload and Parse Source: https://docs.reducto.ai/upload/large-files This example demonstrates the complete workflow for processing a document with Reducto AI, starting from uploading a file to parsing its content. It covers obtaining a presigned URL, uploading the file, and then processing it using the file ID. This script is useful for understanding the end-to-end process. ```python import os import requests from reducto import Reducto # Step 1: Get presigned URL response = requests.post( "https://platform.reducto.ai/upload", headers={\"Authorization\": f\"Bearer {os.environ.get('REDUCTO_API_KEY')}\"} ) data = response.json() file_id = data["file_id"] presigned_url = data["presigned_url"] # Step 2: Upload to presigned URL with open("large_document.pdf", "rb") as f: requests.put(presigned_url, data=f) # Step 3: Process with Reducto client = Reducto() result = client.parse.run(input=file_id) print(f"Successfully processed {result.usage.num_pages} pages") for chunk in result.result.chunks: print(chunk.content[:200]) ``` ```javascript import Reducto from 'reductoai'; import fs from 'fs'; // Step 1: Get presigned URL const uploadResponse = await fetch('https://platform.reducto.ai/upload', { method: 'POST', headers: { 'Authorization': `Bearer ${process.env.REDUCTO_API_KEY}` }, }); const { file_id: fileId, presigned_url: presignedUrl } = await uploadResponse.json(); // Step 2: Upload to presigned URL await fetch(presignedUrl, { method: 'PUT', body: fs.readFileSync('large_document.pdf'), }); // Step 3: Process with Reducto const client = new Reducto(); const result = await client.parse.run({ input: fileId }); console.log(`Successfully processed ${result.usage.num_pages} pages`); ``` ```bash #!/bin/bash # Step 1: Get presigned URL UPLOAD_RESPONSE=$(curl -s -X POST https://platform.reducto.ai/upload \ -H "Authorization: Bearer $REDUCTO_API_KEY") FILE_ID=$(echo $UPLOAD_RESPONSE | jq -r '.file_id') PRESIGNED_URL=$(echo $UPLOAD_RESPONSE | jq -r '.presigned_url') # Step 2: Upload to presigned URL curl -X PUT "$PRESIGNED_URL" -T large_document.pdf # Step 3: Process with Reducto curl -X POST https://platform.reducto.ai/parse \ -H "Authorization: Bearer $REDUCTO_API_KEY" \ -H "Content-Type: application/json" \ -d "{\"input\": \"$FILE_ID\"}" ```