### URL Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Shows examples of URLs that can be detected, including different schemes, ports, and paths. ```text https://example.com http://sub.example.com:8080/path?query=1 ftp://files.company.internal ``` -------------------------------- ### Domain Name Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Provides examples of domain names that can be detected. ```text example.com sub.example.co.uk mail.google.com ``` -------------------------------- ### IP Address Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Shows examples of both IPv4 and IPv6 addresses that can be detected. ```text 192.168.1.1 10.0.0.0 2001:0db8:85a3:0000:0000:8a2e:0370:7334 ::1 ``` -------------------------------- ### Install Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Install the Guardrails PII package using pip or the Guardrails hub. ```bash pip install guardrails-ai gliner presidio-analyzer presidio-anonymizer ``` ```bash guardrails hub install hub://guardrails/guardrails_pii ``` -------------------------------- ### Date/Time Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Provides examples of various date and time formats that can be detected, including full names, numeric, ISO, and time formats. ```text "January 15, 1990" (date with full name) "15/1/1990" (numeric date) "2023-01-15" (ISO format) "3:30 PM" (time) "born 5 February 1985" (date of birth) ``` -------------------------------- ### Performing a Self Install Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/CONTRIBUTING.md After setting up the validator package, perform a self-installation to include all necessary development dependencies. ```bash make dev ``` ```bash pip install -e ".[dev]" ``` -------------------------------- ### Basic PII Detection and Checking Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Demonstrates the minimal setup for PII detection and checking if any PII is found. Useful for simple validation tasks. ```python from guardrails_pii.guardrails_pii import GuardrailsPII pii_detector = GuardrailsPII() result = pii_detector.anonymize("My name is John Doe and my email is john.doe@example.com.") if result.has_pii: print("PII detected!") else: print("No PII detected.") ``` -------------------------------- ### Cryptocurrency Address Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of cryptocurrency addresses for Bitcoin and Ethereum. ```text 1A1z7agoat2QJVA5QksV5qi8ax2Y4d647 (Bitcoin) 0x71C7656EC7ab88b098defB751B7401B5f6d8976F (Ethereum) ``` -------------------------------- ### Location Examples (Context-Dependent) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Demonstrates location names detected when they appear in a relevant context. Requires GLiNER for best results. ```text "London" (in context: "traveled to London") "San Francisco" (in context: "living in San Francisco") "New York City" (in context: "based in New York City") ``` -------------------------------- ### Ambiguous Name Examples (Not Detected) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Shows examples of names that are not detected without sufficient context due to ambiguity or commonality. ```text "John" alone (too common) "Smith" alone (too ambiguous) "Michael" alone (too common) ``` -------------------------------- ### Install Guardrails PII Validator Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/README.md Install the Guardrails PII validator using the Guardrails hub. Ensure you have guardrails-ai>=0.4.0, gliner, presidio-analyzer, and presidio-anonymizer installed. ```bash $ guardrails hub install hub://guardrails/guardrails_pii ``` -------------------------------- ### Using AnalyzerEngine to detect PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Example demonstrating how to use AnalyzerEngine to detect specific PII entities like email addresses and phone numbers from text. Ensure the RecognizerRegistry is loaded with predefined recognizers and the AnalyzerEngine is initialized with supported languages. ```python from validator.analyzer_engine import AnalyzerEngine from presidio_analyzer import RecognizerRegistry registry = RecognizerRegistry() registry.load_predefined_recognizers() analyzer = AnalyzerEngine(registry=registry, supported_languages=["en"]) results = analyzer.analyze( text="Contact john@example.com or call 555-1234", language="en", entities=["EMAIL_ADDRESS", "PHONE_NUMBER"], deduplicate=True ) for result in results: print(f"{result.entity_type}: {result.start}-{result.end}") ``` -------------------------------- ### Email Address Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Demonstrates valid email addresses that are detected. Requires a perfect match (threshold 1.0). ```text john.doe@example.com jane+filter@company.co.uk support@sub.domain.org ``` -------------------------------- ### Person Name Examples (Context-Dependent) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Illustrates person names detected when they appear in a relevant context. Detection is context-aware and requires GLiNER. ```text "John Smith" (in context: "John Smith works at...") "Mary Johnson" (in context: "Mary Johnson, CEO") "Michael Chen" (in context: "Michael Chen discovered...") ``` -------------------------------- ### Python Client Library Example for Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Illustrates using the Guardrails Python client library to create a PII validator that calls the remote API. ```python from guardrails.hub import GuardrailsPII # Create validator that uses remote endpoint validator = GuardrailsPII( entities=["EMAIL_ADDRESS", "PHONE_NUMBER"], use_local=False # Use remote API ) result = validator._validate("John@example.com or 555-0123") ``` -------------------------------- ### Minimal PII Detection Example Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md This snippet shows the most basic way to initialize Guardrails PII to detect general PII entities and anonymize them on failure. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII # Create a guard with PII validator guard = Guard().use( GuardrailsPII(entities="pii", on_fail="fix") ) # Validate text text = "Contact me at john@example.com or 555-1234" result = guard.validate(text) print(result) # PassResult or FailResult with anonymized text ``` -------------------------------- ### Example Usage of Default Thresholds Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Shows how to access and use the DEFAULT_THRESHOLDS dictionary to check the confidence threshold for an entity type and to filter a list of detections. ```python from validator.constants import DEFAULT_THRESHOLDS # Check threshold for an entity threshold = DEFAULT_THRESHOLDS.get("EMAIL_ADDRESS", 0.0) print(f"EMAIL_ADDRESS threshold: {threshold}") # 1.0 # Filter detections by threshold min_score = DEFAULT_THRESHOLDS.get("PERSON", 0.5) detections = [ d for d in detections if d.score >= min_score ] ``` -------------------------------- ### Initialization Flow for GuardrailsPII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/architecture.md Describes the step-by-step initialization process for the `GuardrailsPII` class, including parameter processing, configuration storage, and the setup of Presidio and GLiNER recognizers, emphasizing lazy loading. ```text GuardrailsPII.__init__( entities, model_name, get_entity_threshold, on_fail, use_local ) │ ├─ super().__init__(...) │ └─ Initialize Validator base class │ ├─ Process entities parameter: │ ├─ If string: validate and expand using PII_ENTITIES_MAP │ └─ If list: use directly │ ├─ Store configuration: │ ├─ self.entities = entities (list) │ ├─ self.model_name = model_name │ └─ self.get_entity_threshold = get_entity_threshold │ └─ If use_local=True: ├─ Create GLiNERRecognizer │ └─ Does NOT load model yet (lazy loading) │ ├─ Create Presidio RecognizerRegistry │ └─ Load predefined recognizers │ └─ Add GLiNERRecognizer to registry │ ├─ Create custom AnalyzerEngine │ └─ Pass registry with GLiNER │ └─ Create AnonymizerEngine (from Presidio) ``` -------------------------------- ### Guardrails Hub Inference Example Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Demonstrates the lifecycle of the InferenceSpec in a Guardrails Hub context, showing lazy model loading and subsequent inference calls with cached models. ```python # During server initialization spec = InferenceSpec() spec.load() # Creates GuardrailsPII with all entities # No models downloaded yet # On first request result = spec.infer("john@example.com", ["EMAIL_ADDRESS"]) # GLiNER model downloads and loads here (~2-5s) # On subsequent requests result = spec.infer("another@example.com", ["EMAIL_ADDRESS"]) # Model already cached (~0.5-1s) ``` -------------------------------- ### Example PII Inference Input Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Instantiate InferenceInput with the text to be processed and a list of entity types to identify, such as EMAIL_ADDRESS. ```python from validator.main import InferenceInput input_data = InferenceInput( text="My email is user@example.com", entities=["EMAIL_ADDRESS"] ) ``` -------------------------------- ### Invalid Email Address Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Shows examples of email addresses not detected due to not meeting the threshold (e.g., spaces, missing TLD, missing local part). ```text john @example.com (space) john@example (no TLD) @example.com (no local) ``` -------------------------------- ### InferenceSpec Example: PII Detection Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Demonstrates how to use the `infer` method to detect PII entities like email addresses and phone numbers in a given text. It shows how to load the spec and process the results. ```python spec = InferenceSpec() spec.load() result = spec.infer( text="Contact: john@example.com or 555-1234", entities=["EMAIL_ADDRESS", "PHONE_NUMBER"] ) print(result.anonymized_text) # "Contact: or " for r in result.results: print(f"{r.entity_type} at {r.start}:{r.end}, score: {r.score}") ``` -------------------------------- ### Curl Example for Guardrails PII API Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Demonstrates how to call the Guardrails PII API using curl, sending a POST request with JSON payload. ```bash curl -X POST http://localhost:8000/guardrails_pii \ -H "Content-Type: application/json" \ -d '{ "text": "Contact john@example.com", "entities": ["EMAIL_ADDRESS"] }' ``` -------------------------------- ### Inference Spec Request Body Example Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md An example of a JSON request body for the /guardrails_pii endpoint, including text and a list of entities to find. ```json { "text": "My email is john@example.com", "entities": ["EMAIL_ADDRESS"] } ``` -------------------------------- ### Phone Number Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Illustrates various formats of phone numbers that can be detected, including local, US, and international formats. ```text 555-1234 (US local) (555) 123-4567 (US with area code) +1-555-123-4567 (E.164) +44 20 7946 0958 (UK) ``` -------------------------------- ### Example PII Inference Output Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Construct an InferenceOutput object with detected PII results and the anonymized text, showing how PII entities are replaced. ```python from validator.main import InferenceOutput, InferenceOutputResult output = InferenceOutput( results=[ InferenceOutputResult( entity_type="EMAIL_ADDRESS", start=15, end=32, score=0.99 ) ], anonymized_text="My email is " ) ``` -------------------------------- ### Example PII Detection Result Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Create an InferenceOutputResult instance to represent a specific PII detection, such as a US_SSN, and print its details. ```python from validator.main import InferenceOutputResult result = InferenceOutputResult( entity_type="US_SSN", start=10, end=23, score=0.98 ) print(f"Detected {result.entity_type} from {result.start}:{result.end}") ``` -------------------------------- ### Initialize Guardrails PII Validator in Python Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/README.md Set up the Guardrails PII validator for string output validation. This example shows how to import the necessary classes and configure the validator to detect and fix DATE_TIME entities. ```python # Import Guard and Validator from guardrails.hub import GuardrailsPII from guardrails import Guard # Setup Guard guard = Guard().use( GuardrailsPII(entities=["DATE_TIME"], on_fail="fix") ) ``` -------------------------------- ### Python Requests Example for Guardrails PII API Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Shows how to use the Python 'requests' library to interact with the Guardrails PII API, including sending JSON data and printing the response. ```python import requests response = requests.post( "http://guardrails-hub.api/guardrails_pii", json={ "text": "My SSN is 123-45-6789", "entities": ["US_SSN"] } ) print(response.json()) # { # "results": [{ # "entity_type": "US_SSN", # "start": 11, # "end": 23, # "score": 0.95 # }], # "anonymized_text": "My SSN is " # } ``` -------------------------------- ### Remote PII Inference Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md This example shows how to use Guardrails PII with remote inference via the Guardrails Hub API. API configuration is required. ```python from guardrails.hub import GuardrailsPII # Send to Guardrails Hub API validator = GuardrailsPII( entities="pii", use_local=False ) result = validator._validate("John@example.com") # Uses remote endpoint (requires API configuration) ``` -------------------------------- ### Testing PII Detection API Specification Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Provides an example of how to write unit tests for the Guardrails PII API specification. Useful for ensuring the API behaves as expected. ```python from guardrails_pii.app_inference_spec import InferenceSpec # Assuming you have a testing framework like pytest def test_inference_spec_basic(): inference_spec = InferenceSpec() request_data = { "text": "Test with PII: Alice.", "entities": ["name"] } response = inference_spec.process_request(request_data) assert response["has_pii"] is True assert len(response["spans"]) == 1 assert response["spans"][0]["entity_type"] == "name" def test_inference_spec_no_pii(): inference_spec = InferenceSpec() request_data = { "text": "This text has no PII.", "entities": ["name"] } response = inference_spec.process_request(request_data) assert response["has_pii"] is False ``` -------------------------------- ### Unit Test Example for Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md Demonstrates how to unit test the Guardrails PII validator with pytest. It covers detecting emails, phone numbers, passing clean text, and anonymizing content. ```python import pytest from guardrails.hub import GuardrailsPII @pytest.fixture def validator(): return GuardrailsPII( entities=["EMAIL_ADDRESS", "PHONE_NUMBER"], on_fail="fix" ) def test_detects_email(validator): _, result, *_ = validator._validate("Contact: user@example.com") assert "EMAIL_ADDRESS" in result or "" in result def test_detects_phone(validator): _, result, *_ = validator._validate("Call: 555-1234") assert "PHONE_NUMBER" in result or "" in result def test_passes_clean_text(validator): result = validator._validate("Hello world") assert "PASS" in str(result) or result == "PASS" def test_anonymizes_content(validator): _, result, *_ = validator._validate("Email: john@example.com") assert "john@example.com" not in result assert "" in result ``` -------------------------------- ### Making HTTP API Calls (curl) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Provides an example of how to call the Guardrails PII inference API using curl. Useful for testing the API from the command line. ```bash curl -X POST http://localhost:8000/infer \ -H "Content-Type: application/json" \ -d '{ "text": "My name is John Doe and my email is john.doe@example.com.", "entities": ["name", "email"] }' ``` -------------------------------- ### InferenceSpec Example: Valid and Invalid Requests Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Demonstrates how to use the `process_request` method with both valid and invalid `InferenceInput` objects. It shows successful processing and error handling for empty input. ```python from validator.main import InferenceInput # Valid request request = InferenceInput( text="john@example.com", entities=["EMAIL_ADDRESS"] ) args, kwargs = spec.process_request(request) # args = ("john@example.com", ["EMAIL_ADDRESS"]) # kwargs = {} # Invalid request (empty text) request = InferenceInput( text="", entities=["EMAIL_ADDRESS"] ) try: spec.process_request(request) except HTTPException as e: print(f"Error: {e.status_code}") # 400 print(f"Detail: {e.detail}") # "Invalid input" ``` -------------------------------- ### Per-Entity Confidence Tuning for PII Detection Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/configuration.md Tune confidence thresholds individually for different PII entity types to balance recall and precision. This example sets strict thresholds for emails and more lenient ones for names, with a fallback for unmapped entities. ```python from guardrails.hub import GuardrailsPII def tuned_threshold(entity: str) -> float: """Different thresholds per entity type""" return { "PERSON": 0.6, # More lenient for names "EMAIL_ADDRESS": 0.99, # Very strict for emails "PHONE_NUMBER": 0.7, # Moderate for phones "LOCATION": 0.5, # Lenient for places }.get(entity, 0.5) validator = GuardrailsPII( entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "LOCATION"], get_entity_threshold=tuned_threshold, on_fail="fix" ) ``` -------------------------------- ### General Web Logging Sanitization with Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md An example of using Guardrails PII within a general Guard object to sanitize log entries, removing personally identifiable information before storage. It utilizes a specific model for PII detection. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII log_sanitizer = Guard().use( GuardrailsPII( entities="pii", model_name="urchade/gliner_small-v2.1", on_fail="fix" ) ) log_entry = "User john@example.com logged in from 192.168.1.1" _, safe_log, *_ = log_sanitizer.validate(log_entry) # Log only safe_log to persistent storage ``` -------------------------------- ### US Passport Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of US Passport numbers, which typically range from 6 to 9 digits. ```text 123456789 987654321 ``` -------------------------------- ### US Social Security Number Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of valid US Social Security Numbers in both XXX-XX-XXXX and XXXXXXXXX formats. ```text 123-45-6789 123456789 ``` -------------------------------- ### IBAN Code Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of International Bank Account Numbers (IBAN) with check digit validation, shown for Germany and the UK. ```text DE89370400440532013000 (Germany) GB82 WEST 1234 5698 7654 32 (UK) ``` -------------------------------- ### Select GuardrailsPII Model (Small) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Initialize GuardrailsPII using the 'urchade/gliner_small-v2.1' model for fast performance. ```python GuardrailsPII(model_name="urchade/gliner_small-v2.1") # Fast ``` -------------------------------- ### US Driver License Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of US Driver License numbers, including California format and general state formats. ```text D1234567 (CA format) 12345678 (various state formats) ``` -------------------------------- ### GuardrailsPII Initialization with PII Entities Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Demonstrates how to initialize GuardrailsPII using shorthand entity categories ('pii' or 'spi') or by providing an explicit list of entity types. ```python from guardrails.hub import GuardrailsPII # Using shorthand validator_general = GuardrailsPII(entities="pii") print(validator_general.entities) # ['EMAIL_ADDRESS', 'PHONE_NUMBER', 'DOMAIN_NAME', ...] # Using shorthand for sensitive information validator_sensitive = GuardrailsPII(entities="spi") print(validator_sensitive.entities) # ['CREDIT_CARD', 'CRYPTO', 'IBAN_CODE', ...] # Equivalent explicit form validator_explicit = GuardrailsPII(entities=["EMAIL_ADDRESS", "PHONE_NUMBER"]) ``` -------------------------------- ### Scaling Recommendation for 100 Requests/Second Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Provides recommendations for scaling to handle approximately 100 requests per second using small model instances. ```text For 100 requests/second: - Use 5-10 small model instances (round-robin load balance) - Total memory: 10-30 GB - GPU optional (CPU sufficient) ``` -------------------------------- ### Invalid Credit Card Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of credit card numbers that fail Luhn algorithm validation, demonstrating incorrect formats. ```text 4532123456789013 (fails Luhn check) 1234567890123456 (fails Luhn check) ``` -------------------------------- ### US Individual Taxpayer Identification Number Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of US Individual Taxpayer Identification Numbers (ITIN) in both XXX-XX-XXXX and XXXXXXXXX formats. ```text 911-01-1234 911011234 ``` -------------------------------- ### Selecting PII Detection Models Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Illustrates how to choose between different models based on speed and accuracy trade-offs. Useful for optimizing performance for specific needs. ```python from guardrails_pii.guardrails_pii import GuardrailsPII # Use a faster, less accurate model pii_detector_fast = GuardrailsPII(model_name='small') result_fast = pii_detector_fast.anonymize("This is a test sentence.") # Use a slower, more accurate model pii_detector_accurate = GuardrailsPII(model_name='large') result_accurate = pii_detector_accurate.anonymize("This is another test sentence.") ``` -------------------------------- ### Scaling Recommendation for 1000 Requests/Second Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/inference-spec.md Offers scaling strategies for achieving approximately 1000 requests per second, including considerations for Kubernetes and GPUs. ```text For 1000 requests/second: - Use 50-100 small model instances - Kubernetes auto-scale recommended - GPU beneficial for batch processing ``` -------------------------------- ### Select GuardrailsPII Model (Base) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Initialize GuardrailsPII using the 'urchade/gliner_base-v2.1' model for a balanced speed and accuracy trade-off. ```python GuardrailsPII(model_name="urchade/gliner_base-v2.1") # Balanced ``` -------------------------------- ### Medical License Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of medical license numbers, including formats for US states like Maryland (MD) and New York (NY). ```text MD1234567 NY1234567 ``` -------------------------------- ### Create and Use InferenceOutput Instance Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Demonstrates how to create an instance of InferenceOutput with PII detection results and print the anonymized text and detected entities. ```python from validator.main import InferenceOutput, InferenceOutputResult output = InferenceOutput( results=[ InferenceOutputResult( entity_type="EMAIL_ADDRESS", start=10, end=27, score=0.99 ), InferenceOutputResult( entity_type="PHONE_NUMBER", start=35, end=48, score=0.95 ) ], anonymized_text="Contact me at or call " ) print(output.anonymized_text) # Output: "Contact me at or call " for result in output.results: print(f"{result.entity_type} detected with score {result.score}") ``` -------------------------------- ### Invalid US Social Security Number Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of invalid US Social Security Numbers, including incorrect formats and reserved numbers. ```text 123-45-678 (wrong format) 000-00-0000 (invalid SSN) 000-00-0001 (admin assigned) ``` -------------------------------- ### Credit Card Examples Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md Examples of valid credit card numbers that pass Luhn algorithm validation, including Visa, Mastercard, and American Express formats. ```text 4532123456789012 (Visa format, passes Luhn) 5412345678901234 (Mastercard format, passes Luhn) 378282246310005 (American Express) ``` -------------------------------- ### Initialize GuardrailsPII with General PII Entities Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Instantiate GuardrailsPII using the 'pii' shortcut to detect 8 general PII types. ```python GuardrailsPII(entities="pii") # 8 general PII types ``` -------------------------------- ### Initialize GuardrailsPII with Custom Entity List Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Instantiate GuardrailsPII by providing a specific list of entity types to detect. ```python GuardrailsPII(entities=["EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN"]) ``` -------------------------------- ### Initialize GuardrailsPII with a base GLiNER model Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/configuration.md Specify a different HuggingFace model using `model_name` for a balance of speed and accuracy. ```python validator = GuardrailsPII( entities=["PERSON", "LOCATION"], model_name="urchade/gliner_base-v2.1" ) ``` -------------------------------- ### Create ErrorSpan Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Used to mark validation errors or detected PII. Specify the start and end character positions and the reason for the error. ```python from guardrails.validator_base import ErrorSpan span = ErrorSpan( start=10, end=27, reason="EMAIL_ADDRESS" ) ``` -------------------------------- ### Select GuardrailsPII Model (Large) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Initialize GuardrailsPII using the 'urchade/gliner_large-v2.1' model for high accuracy. ```python GuardrailsPII(model_name="urchade/gliner_large-v2.1") # Accurate ``` -------------------------------- ### Configuring Confidence Thresholds for PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Demonstrates how to set confidence thresholds for PII detection, including strict, lenient, and entity-specific configurations. Useful for fine-tuning detection sensitivity. ```python from guardrails_pii.guardrails_pii import GuardrailsPII # Strict threshold for all entities pii_detector_strict = GuardrailsPII(threshold=0.95) result_strict = pii_detector_strict.anonymize("Potential PII: 123-456-7890") # Lenient threshold for all entities pii_detector_lenient = GuardrailsPII(threshold=0.5) result_lenient = pii_detector_lenient.anonymize("Potential PII: 123-456-7890") # Entity-specific threshold (e.g., higher for email) pii_detector_custom_threshold = GuardrailsPII(entity_threshold={'email': 0.99, 'name': 0.85}) result_custom_threshold = pii_detector_custom_threshold.anonymize("Email: test@example.com, Name: Test User") ``` -------------------------------- ### Configuring Entities for PII Detection Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Shows how to configure specific entities for detection using shorthand groups or custom entity selections. Useful for focusing detection on relevant PII types. ```python from guardrails_pii.guardrails_pii import GuardrailsPII # Detect only names and emails pii_detector = GuardrailsPII(entities=['name', 'email']) result = pii_detector.anonymize("My name is Jane Smith and my email is jane.smith@example.com.") # Detect using shorthand groups (e.g., 'all_financial') pii_detector_financial = GuardrailsPII(entities=['all_financial']) result_financial = pii_detector_financial.anonymize("Account number: 1234567890, Routing: 0987654321") ``` -------------------------------- ### Guardrails PII Local Inference Configuration Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/configuration.md Configures the Guardrails PII validator to use local models for inference. Requires `gliner`, `presidio-analyzer`, and `presidio-anonymizer` to be installed. ```python from guardrails.hub import GuardrailsPII # Local inference (default) validator = GuardrailsPII( entities="pii", use_local=True # Download and run models locally ) ``` -------------------------------- ### Integrating with API Endpoints (Local) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Demonstrates how to integrate Guardrails PII with a local API endpoint using the InferenceSpec. Useful for setting up a local PII detection service. ```python from guardrails_pii.app_inference_spec import InferenceSpec # Assuming InferenceSpec is set up to run locally inference_spec = InferenceSpec() # Example request payload request_data = { "text": "This is a test with PII: John Doe.", "entities": ["name", "email"] } # Process the request response = inference_spec.process_request(request_data) print(response) ``` -------------------------------- ### Automatic PII Fixing Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md Use 'fix' as the on_fail strategy to automatically anonymize detected PII. This example shows how GuardrailsPII can be configured to automatically fix PII. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII guard = Guard().use( GuardrailsPII(entities="pii", on_fail="fix") ) original = "John Smith (john@example.com) lives in NYC" _, anonymized, *rest = guard.validate(original) print(anonymized) # "John Smith (john@example.com) lives in NYC" # John Smith: not detected (no GLiNER in this example) # NYC: may or may not be detected (depends on model confidence) ``` -------------------------------- ### Loading InferenceSpec Specification Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Demonstrates how to load the InferenceSpec, which defines the API for PII detection. Useful for understanding and interacting with the PII detection service. ```python from guardrails_pii.app_inference_spec import InferenceSpec # Load the InferenceSpec with default entities (all available) inference_spec = InferenceSpec() # Load with a specific list of entities specific_entities = ['name', 'email', 'phone'] inference_spec_specific = InferenceSpec(entity_list=specific_entities) # Example of lazy loading (if supported by the underlying implementation) # inference_spec_lazy = InferenceSpec(lazy_load=True) ``` -------------------------------- ### Create InferenceOutputResult Instance Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Instantiate InferenceOutputResult to record a detected PII entity. This includes the entity type, its location (start and end indices), and the confidence score. ```python from validator.main import InferenceOutputResult result = InferenceOutputResult( entity_type="EMAIL_ADDRESS", start=10, end=27, score=0.99 ) print(f"Found {result.entity_type} at position {result.start}:{result.end}") print(f"Confidence: {result.score * 100:.1f}%") ``` -------------------------------- ### Custom PII Detection Threshold Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Define a custom function to set a specific confidence threshold for PII detection. This example requires 95% confidence for any entity to be flagged. ```python def strict_threshold(entity: str) -> float: return 0.95 # 95% confidence required validator = GuardrailsPII( entities="pii", get_entity_threshold=strict_threshold, on_fail="fix" ) ``` -------------------------------- ### Protect LLM Output with Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md Anonymize raw text generated by an LLM before logging or storage. This example uses the OpenAI client and integrates GuardrailsPII for validation. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII from openai import OpenAI client = OpenAI() guard = Guard().use( GuardrailsPII(entities="pii", on_fail="fix") ) prompt = "Generate a customer support response addressing their concern about privacy" response = client.chat.completions.create( model="gpt-4", messages=[{"role": "user", "content": prompt}] ) raw_text = response.choices[0].message.content # Anonymize before logging/storage _, safe_text, *rest = guard.validate(raw_text) print("Original (not logged):", raw_text) print("Safe version logged:", safe_text) ``` -------------------------------- ### Configuring PII Detection with Shorthand Groups Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md Shows how to use shorthand group names like 'pii' for general PII or 'spi' for sensitive PII to quickly configure the validator. ```python from guardrails.hub import GuardrailsPII # Detect all general PII (8 types) general_pii = GuardrailsPII(entities="pii") # EMAIL_ADDRESS, PHONE_NUMBER, DOMAIN_NAME, IP_ADDRESS, DATE_TIME, LOCATION, PERSON, URL # Detect sensitive information (10 types) sensitive_pii = GuardrailsPII(entities="spi") # CREDIT_CARD, CRYPTO, IBAN_CODE, NRP, MEDICAL_LICENSE, US_BANK_NUMBER, US_DRIVER_LICENSE, US_ITIN, US_PASSPORT, US_SSN ``` -------------------------------- ### Initialize GLiNERRecognizer Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Instantiate the GLiNERRecognizer with supported entity types and a HuggingFace model name. The model is not loaded immediately. ```python from validator.gliner_recognizer import GLiNERRecognizer recognizer = GLiNERRecognizer( supported_entities=["PERSON", "LOCATION", "EMAIL_ADDRESS"], model_name="urchade/gliner_small-v2.1" ) ``` -------------------------------- ### Protecting LLM Output with Multi-Validator Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Illustrates an integration pattern for protecting LLM outputs using multiple validators, including PII detection. Useful for ensuring the safety and compliance of generated content. ```python from guardrails_pii.guardrails_pii import GuardrailsPII from guardrails.classes.llm.llm_output import LLMOutput from guardrails.classes.validation.validation_output import ValidationOutput # Assume llm_output is an LLMOutput object from a Guardrails run # Assume pii_detector is an initialized GuardrailsPII instance def validate_pii(llm_output: LLMOutput) -> ValidationOutput: pii_detector = GuardrailsPII(entities=['email', 'phone']) result = pii_detector.anonymize(llm_output.output) if result.has_pii: # Handle PII detection failure, e.g., by returning an invalid output return ValidationOutput(passed=False, error_message="PII detected in LLM output.") else: return ValidationOutput(passed=True) # This function would be registered as a validator in Guardrails # Example usage within a Guardrails Rail: # from guardrails import Rail # rail = Rail( # validators=[ # validate_pii # ] # ) # validated_output = rail.validate(llm_output.output) ``` -------------------------------- ### Guardrails PII Custom Failure Handling Function Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/configuration.md Provides a custom function to handle PII validation failures. This example demonstrates returning the anonymized text in uppercase. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII # Custom fix function def custom_fix(validation_result): return validation_result.fix_value.upper() # Return anonymized text in uppercase guard = Guard().use( GuardrailsPII(entities="pii", on_fail=custom_fix) ) ``` -------------------------------- ### Initialize GuardrailsPII with Sensitive PII Entities Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/README.md Instantiate GuardrailsPII using the 'spi' shortcut to detect 10 sensitive PII types. ```python GuardrailsPII(entities="spi") # 10 sensitive types ``` -------------------------------- ### Balanced Entity Selection (Default) Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/entity-reference.md This is the recommended strategy for most use cases, balancing precision and recall by using both pattern matching and neural detection for a wide range of PII types. ```python entities="pii" # Recommended for most use cases # EMAIL_ADDRESS, PHONE_NUMBER, PERSON, LOCATION, DATE_TIME, etc. ``` -------------------------------- ### Define PII Entity Result Structure Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Use InferenceOutputResult to represent a single detected PII entity. It includes the entity type, its start and end positions in the text, and a confidence score. ```python class InferenceOutputResult(BaseModel): entity_type: str start: int end: int score: float ``` -------------------------------- ### AnalyzerEngine Class Reference Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/CONTENTS.txt Reference for the AnalyzerEngine class, focusing on its analyze method. ```APIDOC ## AnalyzerEngine Class ### Description Provides capabilities for analyzing text for PII entities. ### Methods #### `analyze()` method `analyze(text: str, entities: Union[str, List[str]] = 'pii', model_name: str = 'trifluoromethyl', get_entity_threshold: Optional[Callable] = None, metadata: Optional[Dict[str, Any]] = None)` ##### Description Analyzes the input text to detect specified PII entities. ##### Parameters - **text** (str) - The input text to analyze. - **entities** (Union[str, List[str]]) - Required/Optional - Specifies the entity types to detect. Can be a shorthand string like 'pii' or 'spi', or an explicit list of entity names. - **model_name** (str) - Required/Optional - The name of the model to use for inference. Defaults to 'trifluoromethyl'. - **get_entity_threshold** (Optional[Callable]) - Required/Optional - A callable function to determine the confidence threshold for each entity. - **metadata** (Optional[Dict[str, Any]]) - Required/Optional - Optional metadata to be included with the analysis. ##### Returns - (InferenceOutput) - The result of the analysis, containing detected entities and their details. ``` -------------------------------- ### Using the Presidio to GLiNER Mapping Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/types.md Demonstrates how to retrieve GLiNER labels for a given Presidio entity type and how to check for GLiNER support for a specific Presidio entity. Requires importing PRESIDIO_TO_GLINER from validator.constants. ```python from validator.constants import PRESIDIO_TO_GLINER # Get GLiNER labels for a Presidio entity type gliner_labels = PRESIDIO_TO_GLINER["LOCATION"] print(gliner_labels) # ["location", "place", "address"] # Check if a Presidio entity has GLiNER support if "CREDIT_CARD" in PRESIDIO_TO_GLINER: print("CREDIT_CARD has GLiNER support") ``` -------------------------------- ### Healthcare Context (HIPAA) with Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md An example of using Guardrails PII to anonymize sensitive patient information in a healthcare context, adhering to HIPAA guidelines. It identifies entities like names, dates, and SSNs. ```python from guardrails.hub import GuardrailsPII hipaa_validator = GuardrailsPII( entities=[ "PERSON", # Patient name "DATE_TIME", # Birth date, treatment dates "PHONE_NUMBER", # Contact information "EMAIL_ADDRESS", # Contact information "MEDICAL_LICENSE", # Healthcare provider ID "US_SSN", # Social security ], on_fail="fix" ) patient_note = "Patient John Smith (DOB: 1/15/1990) with SSN 123-45-6789 called at 555-0100" anonymized = hipaa_validator._validate(patient_note) ``` -------------------------------- ### Guardrails PII Validation Flow Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/architecture.md Illustrates the sequence of operations for validating and anonymizing data within the Guardrails PII module, starting from the public validate method and proceeding through internal validation, anonymization, and inference steps. ```mermaid graph TD A[GuardrailsPII.validate(value, metadata)] --> B(GuardrailsPII._validate(value, metadata)) B -- Extract entities from metadata or use self.entities --> C(GuardrailsPII.anonymize(text, entities)) C -- Create InferenceInput(text, entities) --> D(Call _inference(input_request)) D -- if use_local=True --> E[_inference_local (Presidio+GLiNER)] D -- if use_local=False --> F[_inference_remote (API endpoint)] E --> G(InferenceOutput with results) F --> G G --> H(anonymize() continued: Convert InferenceOutputResult to ErrorSpan objects) H --> I(Return (anonymized_text, error_spans)) I --> J(_validate() continued: If error_spans is empty → return PassResult()) J -- Else --> K(FailResult(message, fix_value, error_spans)) ``` -------------------------------- ### Dynamic Entity Selection in Guardrails PII Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/integration-examples.md Demonstrates how to dynamically select entities for PII detection during validation. You can specify entities during Guard initialization or override them for specific validation calls using metadata. ```python from guardrails import Guard from guardrails.hub import GuardrailsPII # Create validator with default entities guard = Guard().use( GuardrailsPII(entities=["EMAIL_ADDRESS"], on_fail="fix") ) # Use default entities text1 = "john@example.com" result1 = guard.validate(text1) # Checks EMAIL_ADDRESS # Override for specific call text2 = "Born on January 15, 1990" result2 = guard.validate( text2, metadata={"entities": ["DATE_TIME"]} ) # Checks DATE_TIME only ``` -------------------------------- ### Get Entity Threshold Source: https://github.com/guardrails-ai/guardrails_pii/blob/main/_autodocs/api-reference.md Retrieves the confidence threshold for a given Presidio entity type. Use this to understand the minimum score required for an entity to be considered valid. If the entity is not explicitly configured or mapped, it defaults to 0.0. ```python from validator.main import get_entity_threshold threshold = get_entity_threshold("EMAIL_ADDRESS") # 1.0 threshold = get_entity_threshold("LOCATION") # 0.5 threshold = get_entity_threshold("UNKNOWN") # 0.0 ```