### Navigate to Basic Node.js Example Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Navigate to the nodejs-basic directory and run the example script. ```bash cd nodejs-basic node example.js ``` -------------------------------- ### Install project dependencies Source: https://github.com/sam247/openredaction/blob/main/CONTRIBUTING.md Run this command from the repository root to install all necessary dependencies. ```bash npm install ``` -------------------------------- ### Run React Development Server Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Start the React development server. ```bash npm run dev ``` -------------------------------- ### Install OpenRedaction Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/changelog.mdx Commands to install the library via npm. ```bash npm install openredaction@1.1.2 ``` ```bash npm install openredaction ``` -------------------------------- ### Install React Integration Source: https://github.com/sam247/openredaction/blob/main/packages/core/README.md If you are using React, install the `openredaction` and `react` packages. This is an optional peer dependency. ```bash npm install openredaction react ``` -------------------------------- ### Initialize and Use OpenRedaction Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/self-hosting.mdx Example of importing the library, configuring an instance, and performing a redaction operation. ```javascript import { OpenRedaction } from 'openredaction'; const redactor = new OpenRedaction({ preset: 'gdpr', redactionMode: 'placeholder', }); const result = await redactor.detect(yourText); console.log(result.redacted); ``` -------------------------------- ### Navigate to React Form Directory Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Navigate to the react-form directory and install dependencies. ```bash cd react-form npm install react react-dom openredaction ``` -------------------------------- ### Navigate to Express API Directory Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Navigate to the express-api directory and install dependencies. ```bash cd express-api npm install express openredaction ``` -------------------------------- ### Run Express API Server Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Start the Express API server. ```bash node server.js ``` -------------------------------- ### Install OpenRedaction Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/getting-started.mdx Commands to install the OpenRedaction package using common Node.js package managers. ```bash npm install openredaction ``` ```bash yarn add openredaction ``` ```bash pnpm add openredaction ``` -------------------------------- ### Configure and Start REST API Server Source: https://context7.com/sam247/openredaction/llms.txt Initializes the API server with security, rate limiting, and default redaction options. ```typescript import { APIServer, createAPIServer } from 'openredaction/server'; // Create and configure API server const server = createAPIServer({ port: 3000, host: '0.0.0.0', apiKey: 'your-secret-api-key', enableCors: true, corsOrigin: ['https://app.example.com'], enableRateLimit: true, rateLimit: 100, // requests per minute bodyLimit: '10mb', enableLogging: true, defaultOptions: { preset: 'gdpr', enableContextAnalysis: true } }); // Start server await server.start(); // Output: [APIServer] Server started on http://0.0.0.0:3000 // Stop server await server.stop(); ``` -------------------------------- ### Install OpenRedaction via NPM Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/index.mdx Use this command to install the OpenRedaction package in your Node.js project. ```bash npm install openredaction ``` -------------------------------- ### Install OpenRedaction Library Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/self-hosting.mdx Commands to add the OpenRedaction package to your Node.js project using npm or yarn. ```bash npm install openredaction ``` ```bash yarn add openredaction ``` -------------------------------- ### Python Batch Processing with Concurrency Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/tutorials.mdx This Python example demonstrates batch processing of text using OpenRedaction with a thread pool for concurrency. It includes options for batch size, concurrency, and redaction settings, along with error handling for each text item. ```python from openredaction import redact from concurrent.futures import ThreadPoolExecutor, as_completed import math def process_batch(texts, options=None): if options is None: options = {} batch_size = options.get('batch_size', 10) max_workers = options.get('concurrency', 5) preset = options.get('preset', 'gdpr') redaction_mode = options.get('redaction_mode', 'placeholder') use_ai = options.get('use_ai', False) def process_text(text): try: result = redact( text, preset=preset, redaction_mode=redaction_mode, use_ai=use_ai ) return { 'success': True, 'original': text, 'redacted': result['redacted_text'], 'detections': result['detections'], } except Exception as e: return { 'success': False, 'original': text, 'error': str(e), } # Process in parallel batches results = [] num_batches = math.ceil(len(texts) / batch_size) with ThreadPoolExecutor(max_workers=max_workers) as executor: futures = [] for i in range(0, len(texts), batch_size): batch = texts[i:i + batch_size] future = executor.submit( lambda batch: [process_text(text) for text in batch], batch ) futures.append(future) for i, future in enumerate(as_completed(futures)): batch_results = future.result() results.extend(batch_results) print(f'Processed {i + 1}/{num_batches} batches') return results # Usage texts = [ 'Contact john@example.com', 'Call 555-123-4567', 'Email sarah@example.com', # ... thousands more ] results = process_batch(texts, { 'batch_size': 100, 'concurrency': 5, 'use_ai': True, }) # Filter successful results successful = [r for r in results if r['success']] failed = [r for r in results if not r['success']] print(f'Processed {len(successful)} successfully, {len(failed)} failed') ``` -------------------------------- ### Quick Start: Detect PII Source: https://github.com/sam247/openredaction/blob/main/packages/core/README.md Use the `detect()` method to find and redact PII in text. Remember that `detect()` is asynchronous and requires `await`. ```typescript import { OpenRedaction } from 'openredaction'; const shield = new OpenRedaction(); const result = await shield.detect("Email john@example.com or call 07700900123"); console.log(result.redacted); // "Email [EMAIL_9619] or call [PHONE_UK_MOBILE_9478]" ``` -------------------------------- ### Node.js Batch Processing with Concurrency Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/tutorials.mdx Use this Node.js example to process text data in batches with a configurable concurrency limit. It handles individual redaction successes and failures, logging progress along the way. ```javascript import { redact } from 'openredaction'; import { chunk } from 'lodash'; async function processBatch(texts, options = {}) { const { batchSize = 10, concurrency = 5, preset = 'gdpr', redactionMode = 'placeholder', useAI = false, } = options; // Split into batches const batches = chunk(texts, batchSize); const results = []; // Process batches with concurrency limit for (let i = 0; i < batches.length; i += concurrency) { const batchGroup = batches.slice(i, i + concurrency); const batchResults = await Promise.allSettled( batchGroup.map(async (batch) => { return Promise.all( batch.map(async (text) => { try { const result = await redact(text, { preset, redactionMode, useAI, }); return { success: true, original: text, redacted: result.redacted_text, detections: result.detections, }; } catch (error) { return { success: false, original: text, error: error.message, }; } }) ); }) ); results.push(...batchResults); // Log progress console.log(`Processed ${Math.min(i + concurrency, batches.length)}/${batches.length} batches`); } return results.flat(); } // Usage const texts = [ 'Contact john@example.com', 'Call 555-123-4567', 'Email sarah@example.com', // ... thousands more ]; const results = await processBatch(texts, { batchSize: 100, concurrency: 5, useAI: true, }); // Filter successful results const successful = results.filter(r => r.success); const failed = results.filter(r => !r.success); console.log(`Processed ${successful.length} successfully, ${failed.length} failed`); ``` -------------------------------- ### Using Compliance Presets with OpenRedaction Source: https://context7.com/sam247/openredaction/llms.txt Shows how to utilize pre-configured compliance presets like GDPR, HIPAA, and PCI-DSS. Also demonstrates fetching preset configurations directly and applying them with specific redaction settings. ```typescript import { OpenRedaction, getPreset } from 'openredaction'; // GDPR preset for European data protection const gdprRedactor = new OpenRedaction({ preset: 'gdpr', redactionMode: 'placeholder' }); // HIPAA preset for US healthcare data const hipaaRedactor = new OpenRedaction({ preset: 'hipaa', redactionMode: 'mask-all' }); // PCI-DSS preset for payment card data const pciRedactor = new OpenRedaction({ preset: 'pci-dss', enableContextAnalysis: true }); // Finance preset combining multiple financial patterns const financeRedactor = new OpenRedaction({ preset: 'finance', enableFalsePositiveFilter: true, falsePositiveThreshold: 0.7 }); // Get preset configuration directly const gdprConfig = getPreset('gdpr'); console.log(gdprConfig.patterns); // Output: ['EMAIL', 'NAME', 'PHONE_UK', 'IPV4', 'POSTCODE_UK', 'NATIONAL_INSURANCE_UK', 'NHS_NUMBER', ...] // Example usage const medicalText = "Patient John Smith, SSN 123-45-6789, prescribed Lisinopril"; const result = await hipaaRedactor.detect(medicalText); console.log(result.redacted); // Output: "Patient **********, SSN ***********, prescribed Lisinopril" ``` -------------------------------- ### Run tests and linting Source: https://github.com/sam247/openredaction/blob/main/CONTRIBUTING.md Execute the test suite and linting checks from the repository root. ```bash npm test # Run the test suite (uses Turbo + Vitest in workspaces) npm run lint ``` -------------------------------- ### Create and push a git tag Source: https://github.com/sam247/openredaction/blob/main/docs/PUBLISHING.md Use these commands to trigger the automated release workflow after bumping the package version. ```bash git tag -a v1.1.2 -m "Release 1.1.2" git push origin v1.1.2 ``` -------------------------------- ### Integrate OpenRedaction with Express Middleware Source: https://context7.com/sam247/openredaction/llms.txt Configure global or route-specific middleware to detect and redact PII in Express requests. Includes examples for direct detection, report generation, and custom analysis endpoints. ```typescript import express from 'express'; import { openredactionMiddleware, detectPII, generateReport, OpenRedactionRequest } from 'openredaction'; const app = express(); app.use(express.json()); // Global middleware with auto-redaction app.use('/api/secure', openredactionMiddleware({ autoRedact: true, enableContextAnalysis: true, addHeaders: true, // Adds X-PII-Detected and X-PII-Count headers onDetection: (req, result) => { console.log(`PII detected in ${req.path}: ${result.detections.length} items`); } })); // Strict middleware that rejects requests with PII app.use('/api/strict', openredactionMiddleware({ failOnPII: true, fields: ['message', 'content'], // Only check specific fields skipRoutes: [/^\/api\/strict\/public/] })); // Protected endpoint - receives redacted body app.post('/api/secure/submit', (req: OpenRedactionRequest, res) => { console.log('Received (redacted):', req.body); console.log('PII detected:', req.pii?.detected); console.log('PII count:', req.pii?.count); res.json({ success: true, piiDetected: req.pii?.detected || false, piiCount: req.pii?.count || 0 }); }); // Direct detection endpoint app.post('/api/detect', detectPII({ enableContextAnalysis: true, preset: 'gdpr' })); // Report generation endpoint app.post('/api/report', generateReport({ enableContextAnalysis: true })); // Custom analysis endpoint app.post('/api/analyze', async (req, res) => { const { text, options = {} } = req.body; const detector = new OpenRedaction({ ...options, enableContextAnalysis: true }); const result = await detector.detect(text); res.json({ hasPII: result.detections.length > 0, count: result.detections.length, redacted: result.redacted, breakdown: result.detections.reduce((acc, d) => { acc[d.type] = (acc[d.type] || 0) + 1; return acc; }, {} as Record) }); }); app.listen(3000); ``` -------------------------------- ### Basic PII Detection and Redaction Source: https://context7.com/sam247/openredaction/llms.txt Demonstrates basic PII detection using the default settings and placeholder redaction mode. Shows how to retrieve redacted text, detected PII details, and restore original text. ```typescript import { OpenRedaction } from 'openredaction'; // Basic detection with default settings const redactor = new OpenRedaction({ redactionMode: 'placeholder' }); const result = await redactor.detect('My name is John Smith and my email is john@example.com'); console.log(result.redacted); // Output: "My name is [NAME_XXXX] and my email is [EMAIL_XXXX]" console.log(result.detections); // Output: [ // { type: 'NAME', value: 'John Smith', placeholder: '[NAME_XXXX]', position: [11, 21], severity: 'medium' }, // { type: 'EMAIL', value: 'john@example.com', placeholder: '[EMAIL_XXXX]', position: [39, 55], severity: 'high' } // ] console.log(result.redactionMap); // Output: { '[NAME_XXXX]': 'John Smith', '[EMAIL_XXXX]': 'john@example.com' } // Restore original text from redacted version const restored = redactor.restore(result.redacted, result.redactionMap); console.log(restored); // Output: "My name is John Smith and my email is john@example.com" ``` -------------------------------- ### Integrate OpenRedaction with Fluent Bit Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/tutorials.mdx Use a Lua filter in Fluent Bit to forward log messages to an OpenRedaction API endpoint for PII redaction. This example assumes an OpenRedaction service is running locally on port 3000. ```lua -- fluent-bit-lua-filter.lua function redact_pii(tag, timestamp, record) local http = require("socket.http") local ltn12 = require("ltn12") local response_body = {} local res, code = http.request{ url = "http://localhost:3000/redact", method = "POST", headers = { ["Content-Type"] = "application/json" }, source = ltn12.source.string(json.encode({text = record["message"]})), sink = ltn12.sink.table(response_body) } if code == 200 then local result = json.decode(table.concat(response_body)) record["message"] = result.redacted_text record["pii_detected"] = #result.detections > 0 end return 1, timestamp, record end ``` -------------------------------- ### Simple Redaction with Specific PII Types Source: https://github.com/sam247/openredaction/blob/main/README.md Configure OpenRedaction to include specific PII types like names, emails, and phone numbers. This example uses the 'mask-middle' redaction mode to obscure parts of the detected PII. ```typescript import { OpenRedaction } from 'openredaction'; const redactor = new OpenRedaction({ includeNames: true, includeEmails: true, includePhones: true, redactionMode: 'mask-middle' }); const input = "Contact Sarah Jones at sarah@example.com or call +1 202-555-0110"; const { redacted } = await redactor.detect(input); console.log(redacted); // "Contact S***h J***s at s***@example.com or call +1 ***-***-0110" ``` -------------------------------- ### Process logs and batches Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/examples.mdx Utilities for handling multiple entries or structured log data. ```javascript import { redact } from 'openredaction'; async function processLog(logEntry) { const redacted = await redact(logEntry.message); return { ...logEntry, message: redacted.redacted_text, piiDetected: redacted.detections.length > 0, }; } ``` ```javascript const texts = [ "Email: user1@example.com", "Phone: 555-111-2222", "Name: Jane Smith", ]; const results = await Promise.all( texts.map(text => redact(text)) ); results.forEach((result, index) => { console.log(`Text ${index + 1}:`, result.redacted_text); }); ``` -------------------------------- ### Configure Redaction Presets and Modes Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/getting-started.mdx Initialize the OpenRedaction instance with specific compliance presets and redaction modes. ```javascript import { OpenRedaction } from 'openredaction'; const redactor = new OpenRedaction({ preset: 'gdpr', redactionMode: 'placeholder', }); const result = await redactor.detect(text); ``` -------------------------------- ### Redact PII in Node.js Express.js Middleware Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/tutorials.mdx Use this middleware to redact PII from request bodies and response payloads in an Express.js application. It requires `express` and `openredaction` to be installed. Ensure JSON parsing is handled correctly after redaction. ```javascript import express from 'express'; import { redact } from 'openredaction'; const app = express(); app.use(express.json()); // Middleware to redact PII from request body async function redactRequestBody(req, res, next) { if (req.body && typeof req.body === 'object') { const bodyString = JSON.stringify(req.body); const result = await redact(bodyString, { preset: 'gdpr', redactionMode: 'placeholder', }); // Parse redacted JSON back to object try { req.body = JSON.parse(result.redacted_text); } catch (e) { // If parsing fails, use original body console.error('Failed to parse redacted body:', e); } req.piiDetected = result.detections.length > 0; } next(); } // Middleware to redact PII from response async function redactResponseBody(req, res, next) { const originalJson = res.json; res.json = function(data) { // Redact PII from response return redact(JSON.stringify(data), { preset: 'gdpr', redactionMode: 'placeholder', }).then((result) => { const redactedData = JSON.parse(result.redacted_text); return originalJson.call(this, redactedData); }); }; next(); } // Apply middleware to all routes app.use(redactRequestBody); app.use(redactResponseBody); // Example route app.post('/api/users', async (req, res) => { // Request body is already redacted const user = await createUser(req.body); res.json(user); // Response will be redacted }); ``` -------------------------------- ### Run tests and linting in packages/core Source: https://github.com/sam247/openredaction/blob/main/CONTRIBUTING.md Execute tests and linting specifically within the packages/core workspace. ```bash cd packages/core npm run test npm run lint ``` -------------------------------- ### Manage Multi-Tenant Configurations Source: https://context7.com/sam247/openredaction/llms.txt Registers tenants with specific quotas and presets, and performs tenant-isolated operations. ```typescript import { TenantManager, createTenantManager, DEFAULT_TIER_QUOTAS } from 'openredaction'; const tenantManager = createTenantManager(); // Register tenants with different tiers tenantManager.registerTenant({ tenantId: 'tenant-free-001', name: 'Free Tier Customer', status: 'active', quotas: DEFAULT_TIER_QUOTAS.free, // 1000 req/month, 10 req/min options: { preset: 'gdpr' } }); tenantManager.registerTenant({ tenantId: 'tenant-enterprise-001', name: 'Enterprise Customer', status: 'active', apiKey: 'ent-api-key-xxx', quotas: DEFAULT_TIER_QUOTAS.enterprise, // Unlimited options: { preset: 'hipaa', enableContextAnalysis: true, enableMultiPass: true }, customPatterns: [ { type: 'INTERNAL_ID', regex: /INT-\d{8}/g, priority: 10, placeholder: '[INT_ID_{n}]', severity: 'low' } ] }); // Detect with tenant isolation const result = await tenantManager.detect('tenant-free-001', 'Email: john@example.com'); console.log(result.redacted); // Authenticate by API key const tenant = tenantManager.authenticateByApiKey('ent-api-key-xxx'); console.log(tenant?.name); // 'Enterprise Customer' // Get usage statistics const usage = tenantManager.getTenantUsage('tenant-free-001'); console.log(usage); // Output: { requestsThisMonth: 1, textProcessedThisMonth: 25, piiDetectedThisMonth: 1, ... } // Manage tenant status tenantManager.suspendTenant('tenant-free-001'); tenantManager.activateTenant('tenant-free-001'); // Get aggregate stats const stats = tenantManager.getAggregateStats(); console.log(stats); // Output: { totalTenants: 2, activeTenants: 2, totalRequestsThisMonth: 150, ... } ``` -------------------------------- ### Create Reusable Configuration Preset Source: https://context7.com/sam247/openredaction/llms.txt Define and save a reusable configuration preset from the current detector's settings. This allows for easy application of specific configurations across different instances. ```typescript import { OpenRedaction, createConfigPreset } from 'openredaction'; const detector = new OpenRedaction({ preset: 'hipaa', customPatterns: [ { type: 'MRN', regex: /MRN-\d{8}/g, priority: 15, placeholder: '[MRN_{n}]', severity: 'high' } ] }); // Create reusable preset from current configuration const preset = createConfigPreset(detector, 'healthcare-v2'); ``` -------------------------------- ### Load Detector Configuration from File Source: https://context7.com/sam247/openredaction/llms.txt Initialize a new OpenRedaction detector by loading its configuration from a JSON file, typically one exported for version control. ```typescript import { OpenRedaction } from 'openredaction'; // Load configuration from file const loadedDetector = await OpenRedaction.fromConfig('.openredaction.json'); ``` -------------------------------- ### Perform Simple Redaction Source: https://github.com/sam247/openredaction/blob/main/site/docs/pages/getting-started.mdx Initialize the OpenRedaction instance and use the detect method to identify and redact PII in a string. ```javascript import { OpenRedaction } from 'openredaction'; const redactor = new OpenRedaction(); const text = "My name is John Doe and my email is john@example.com"; const result = await redactor.detect(text); console.log(result.redacted); // e.g. placeholders or masked text depending on redactionMode console.log(result.detections); // [{ type: 'NAME', value: 'John Doe', ... }, { type: 'EMAIL', value: 'john@example.com', ... }] ``` -------------------------------- ### Basic Node.js PII Detection Source: https://github.com/sam247/openredaction/blob/main/examples/README.md Demonstrates core detection features in a pure Node.js environment. Use for simple PII detection, batch processing, and HTML report generation. ```javascript const { OpenRedaction } = require('openredaction'); async function main() { const detector = new OpenRedaction(); const result = await detector.detect('Contact john@example.com'); console.log(result.redacted); console.log(result.detections); } main(); ``` -------------------------------- ### Configure OpenRedaction instance Source: https://github.com/sam247/openredaction/blob/main/README.md Initialize the OpenRedaction constructor with custom categories, patterns, and compliance settings to tailor detection behavior. ```typescript const redactor = new OpenRedaction({ // Toggle built-in categories includeNames: true, includeAddresses: false, includeEmails: true, // Filter by category or specific patterns categories: ['financial'], patterns: ['EMAIL', 'SSN'], // Add custom patterns customPatterns: [ { type: 'EMPLOYEE_ID', regex: /EMP-\d{4}/g, priority: 10, placeholder: '[EMPLOYEE_ID_{n}]', severity: 'medium', }, ], // Whitelist approved terms whitelist: ['ACME Corp'], // Redaction modes redactionMode: 'mask-all', // placeholder | mask-middle | mask-all | format-preserving | token-replace // Compliance presets preset: 'hipaa', // gdpr | hipaa | ccpa | pci-dss | soc2 | finance | education | transportation // Advanced options deterministic: true, // Stable placeholders for same value enableContextAnalysis: true, // Context-aware filtering confidenceThreshold: 0.5, enableCache: true, }); ``` -------------------------------- ### Import Server Modules Source: https://github.com/sam247/openredaction/blob/main/packages/core/README.md Import server-related classes like `APIServer` and `createPrometheusServer` from the `openredaction/server` entry point. These are not exported from the main package. ```typescript import { APIServer, createPrometheusServer } from 'openredaction/server'; ``` -------------------------------- ### Export Detector Configuration as JSON Source: https://context7.com/sam247/openredaction/llms.txt Export the current detector configuration, including presets, custom patterns, and other options, into a JSON format suitable for documentation or backup. ```typescript import { OpenRedaction, ConfigExporter, createConfigPreset, exportForVersionControl } from 'openredaction'; const detector = new OpenRedaction({ preset: 'hipaa', customPatterns: [ { type: 'MRN', regex: /MRN-\d{8}/g, priority: 15, placeholder: '[MRN_{n}]', severity: 'high' } ], whitelist: ['Hospital Name', 'Dr. Public'], enableContextAnalysis: true, confidenceThreshold: 0.7 }); // Export configuration as JSON const configJson = detector.exportConfig({ description: 'HIPAA-compliant configuration for healthcare', author: 'compliance-team', tags: ['hipaa', 'healthcare', 'production'] }); console.log(configJson); ``` -------------------------------- ### Import React Hooks Source: https://github.com/sam247/openredaction/blob/main/packages/core/README.md Import the necessary hooks from the `openredaction/react` entry point when using the React integration. ```typescript import { useOpenRedaction, usePIIDetector } from 'openredaction/react'; ```