### Install Besshouka Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Set up a virtual environment and install dependencies using pip. ```bash python -m venv env_besshouka source env_besshouka/bin/activate pip install -r requirements.txt ``` -------------------------------- ### Install Besshouka Source: https://github.com/go-akhi/besshouka/blob/main/README.md Install the Besshouka package using pip. This is the first step for both command-line and programmatic usage. ```bash pip install besshouka ``` -------------------------------- ### Set up Development Environment Source: https://github.com/go-akhi/besshouka/blob/main/CONTRIBUTING.md Clone the repository, create and activate a virtual environment, and install development dependencies. Requires Python 3.11 or higher. ```bash git clone https://github.com/akhi/besshouka.git cd besshouka python -m venv env_besshouka source env_besshouka/bin/activate pip install -e ".[dev]" ``` -------------------------------- ### Custom Recognizer Implementation Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/analyzer/recognizers/README.md Example of creating a custom recognizer by subclassing BaseRecognizer. Implement the name, source properties and the recognize method for your detection logic. ```python from besshouka.analyzer.recognizers.base import BaseRecognizer from besshouka.models.recognizer_result import RecognizerResult class MyRecognizer(BaseRecognizer): @property def name(self) -> str: return "my_recognizer" @property def source(self) -> str: return "custom" def recognize(self, text: str) -> list[RecognizerResult]: results = [] # Your detection logic here # For each match, append: # RecognizerResult(start=..., end=..., entity_type="...", # score=..., source=self.source, text=...) return results ``` -------------------------------- ### Lint Code with Ruff and Bandit Source: https://github.com/go-akhi/besshouka/blob/main/CONTRIBUTING.md Install linting tools and check the codebase for style issues and potential security vulnerabilities. Ruff checks formatting and style, while Bandit scans for security issues. ```bash pip install ruff bandit ruff check besshouka/ ruff format --check besshouka/ bandit -r besshouka/ -c pyproject.toml ``` -------------------------------- ### Analyze Text with Explanation (CLI) Source: https://context7.com/go-akhi/besshouka/llms.txt Use the `analyze --explain` command to get detailed analysis including confidence scores and source information for each detected entity. This helps in understanding the detection reasoning. ```bash # Analysis with score and source reasoning besshouka analyze --explain "田中太郎の電話番号は090-1234-5678です" # Output includes Score and Source columns: # │ PERSON │ 田中太郎 │ 0 │ 4 │ 0.85 │ ginza_ner │ # │ PHONE_NUMBER │ 090-1234-5678 │ 10 │ 23 │ 1.00 │ regex_registry│ ``` -------------------------------- ### Custom Operator YAML Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/anonymizer/operators/README.md Example of how to configure a custom operator in YAML. It specifies the method as 'custom' and points to the Python function to be executed. ```yaml EMPLOYEE_ID: method: custom function: "my_company.transforms.redact_employee_id" preserve_prefix: true ``` -------------------------------- ### Use Custom Operator in YAML Config Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/anonymizer/operators/README.md Example of how to use a custom operator in the YAML configuration file after it has been registered. ```yaml ENTITY_TYPE: method: my_method my_param: value ``` -------------------------------- ### Clone Besshouka Repository Source: https://github.com/go-akhi/besshouka/blob/main/README.md Clone the Besshouka GitHub repository to contribute or develop locally. Install the package in editable mode with development dependencies. ```bash git clone https://github.com/akhi/besshouka.git cd besshouka pip install -e ".[dev]" ``` -------------------------------- ### YAML Operator Configuration Example Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/anonymizer/README.md This YAML configuration maps entity types to specific anonymization operators like 'replace' or 'mask'. If an entity type is not specified, the 'keep' operator is used by default. ```yaml operators: PERSON: method: replace value: "<氏名>" PHONE_NUMBER: method: mask char: "*" from_end: 4 ``` -------------------------------- ### Analyze Text for Detected Entities (CLI) Source: https://context7.com/go-akhi/besshouka/llms.txt Employ the `analyze` command for PII detection without anonymization. It displays detected entities in a table format. Basic analysis shows Entity Type, Text, Start, and End positions. ```bash # Basic analysis showing detected entities besshouka analyze "田中太郎のメールはtanaka@example.comです" # Output: # ┌─────────────┬─────────────────────┬───────┬─────┐ # │ Entity Type │ Text │ Start │ End │ # ├─────────────┼─────────────────────┼───────┼─────┤ # │ PERSON │ 田中太郎 │ 0 │ 4 │ # │ EMAIL │ tanaka@example.com │ 9 │ 27 │ # └─────────────┴─────────────────────┴───────┴─────┘ ``` -------------------------------- ### Anonymize Text with Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/README.md Use the Besshouka command-line interface to anonymize Japanese text. It replaces detected PII with placeholders. For example, names are replaced with '<氏名>' and phone numbers are partially masked. ```bash besshouka anonymize "田中太郎の電話番号は090-1234-5678です" # Output: <氏名>の電話番号は090-1234-****です ``` -------------------------------- ### Run Full Test Suite Source: https://github.com/go-akhi/besshouka/blob/main/CONTRIBUTING.md Execute the complete test suite, including tests that require the GiNZA model. Use the -v flag for verbose output. ```bash # Full suite (requires GiNZA + ja_ginza model) pytest tests/ -v ``` -------------------------------- ### Programmatic Pipeline Usage Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/orchestrator/README.md Demonstrates how to load configurations, run the pipeline programmatically, and access anonymized text and audit trails. Preserves original text. ```python from besshouka.config.loader import load_recognizer_config, load_operator_config from besshouka.orchestrator.pipeline import run rec_config = load_recognizer_config("recognizers.yaml") op_config = load_operator_config("operators.yaml") ctx = run("田中太郎の電話は090-1234-5678です", rec_config, op_config) # Anonymized text print(ctx.engine_result.text) # Audit trail for item in ctx.engine_result.items: print(f"{item.entity_type}: '{item.original_text}' → [{item.operator}]") # Original text is always preserved print(ctx.original_text) ``` -------------------------------- ### Run Pipeline Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/orchestrator/README.md Initiates the Besshouka pipeline with the provided text, recognizer configuration, and operator configuration. ```python ctx = run(text, recognizer_config, operator_config) ``` -------------------------------- ### Analyze Text with Explanation using Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Use the CLI to analyze Japanese text and provide score/source reasoning for detections. ```bash python -m besshouka.cli analyze --explain "田中太郎の電話番号は090-1234-5678です" ``` -------------------------------- ### Run Fast Tests Source: https://github.com/go-akhi/besshouka/blob/main/CONTRIBUTING.md Execute tests excluding those that require loading the GiNZA model. Use this for quick feedback during development. ```bash # Fast tests (excludes GiNZA model loading) pytest tests/ -m "not slow" ``` -------------------------------- ### Import RecognizerResult Model Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/models/README.md Import the RecognizerResult class to define detected PII spans. ```python from besshouka.models.recognizer_result import RecognizerResult ``` -------------------------------- ### Analyze from File with Custom Recognizers (CLI) Source: https://context7.com/go-akhi/besshouka/llms.txt Perform analysis on file input using custom recognizers by specifying the input file and the recognizer configuration file with the `analyze` command. ```bash # Analyze from file with custom recognizers besshouka analyze --input document.txt --recognizers custom_patterns.yaml --explain ``` -------------------------------- ### Create and Access RecognizerResult Source: https://context7.com/go-akhi/besshouka/llms.txt Instantiate a RecognizerResult object to represent a PII detection. Access its attributes like entity type, text, position, confidence score, and source. ```python from besshouka.models.recognizer_result import RecognizerResult # Create a detection result result = RecognizerResult( start=0, end=4, entity_type="PERSON", score=0.85, source="ginza_ner", text="田中太郎", recognition_metadata={"model_version": "5.2.0"} # Optional extra info ) # Access fields print(f"Entity: {result.entity_type}") # PERSON print(f"Text: {result.text}") # 田中太郎 print(f"Position: [{result.start}:{result.end}]") # [0:4] print(f"Confidence: {result.score}") # 0.85 print(f"Source: {result.source}") # ginza_ner print(f"Metadata: {result.recognition_metadata}") # {"model_version": "5.2.0"} ``` -------------------------------- ### Load Recognizer and Operator Config Programmatically Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/config/README.md Load recognizer and operator configuration files programmatically using the provided functions. Ensure the YAML files are correctly formatted and contain the required fields. ```python from besshouka.config.loader import load_recognizer_config, load_operator_config rec_config = load_recognizer_config("my_patterns.yaml") op_config = load_operator_config("my_operators.yaml") ``` -------------------------------- ### Create Custom Operator - Subclassing BaseOperator Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/anonymizer/operators/README.md Demonstrates how to create a new custom operator by subclassing `BaseOperator` and implementing the `operate` method. ```python from besshouka.anonymizer.operators.base import BaseOperator class MyOperator(BaseOperator): def operate(self, text: str, params: dict) -> str: # Your transformation logic here return transformed_text ``` -------------------------------- ### Load Recognizer Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/config/README.md Loads and validates a recognizer registry YAML file. Each entry requires name, entity_type, pattern, and score. ```APIDOC ## Load Recognizer Configuration ### Description Loads a recognizer registry YAML file. Each entry must have a `name`, `entity_type`, `pattern`, and `score`. ### Method `load_recognizer_config(path: str) -> dict` ### Parameters #### Path Parameters - **path** (str) - Required - The file path to the recognizer YAML configuration. ### Request Body - **name** (str) - Required - Unique recognizer name. - **entity_type** (str) - Required - Standardized entity type. - **pattern** (str) - Required - Regex pattern string. - **score** (float) - Required - Confidence score (typically `1.0`). - **source** (str) - Optional - Defaults to `regex_registry`. ### Response #### Success Response (200) - **dict** - A dictionary representing the loaded recognizer configuration. #### Error Handling - `FileNotFoundError`: If the specified YAML file does not exist. - `ValueError`: If the YAML is malformed or entries are missing required fields. ``` -------------------------------- ### Run Tests with Coverage Source: https://github.com/go-akhi/besshouka/blob/main/CONTRIBUTING.md Execute tests and generate a code coverage report, highlighting missing coverage. Displays missing coverage in the terminal. ```bash # With coverage pytest tests/ --cov=besshouka --cov-report=term-missing ``` -------------------------------- ### Override Operator Rules Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/defaults/README.md Copy the default operators.yaml file to customize it and then pass it to the CLI using the --rules flag. ```bash cp besshouka/defaults/operators.yaml my_operators.yaml # Edit my_operators.yaml... python -m besshouka.cli anonymize --rules my_operators.yaml "text" ``` -------------------------------- ### Override Recognizers Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/defaults/README.md Copy the default recognizers.yaml file to customize it and then pass it to the CLI using the --recognizers flag. ```bash cp besshouka/defaults/recognizers.yaml my_recognizers.yaml # Edit my_recognizers.yaml... python -m besshouka.cli anonymize --recognizers my_recognizers.yaml "text" ``` -------------------------------- ### Run Full Anonymization Pipeline (Python API) Source: https://context7.com/go-akhi/besshouka/llms.txt Execute the complete Besshouka pipeline using the `run` function. It takes raw text and configuration dictionaries, returning a `ProcessingContext` with anonymized text and an audit trail. Ensure configurations are loaded using `load_recognizer_config` and `load_operator_config`. ```python from besshouka.config.loader import load_recognizer_config, load_operator_config from besshouka.orchestrator.pipeline import run # Load configurations rec_config = load_recognizer_config("recognizers.yaml") op_config = load_operator_config("operators.yaml") # Run the pipeline text = "田中太郎の電話番号は090-1234-5678で、メールはtanaka@example.comです" ctx = run(text, rec_config, op_config) # Access anonymized text print(ctx.engine_result.text) # Output: <氏名>の電話番号は090-1234-****で、メールは<メール>です # Access audit trail for item in ctx.engine_result.items: print(f"{item.entity_type}: '{item.original_text}' → [{item.operator}]") # Output: # PERSON: '田中太郎' → [replace] # PHONE_NUMBER: '090-1234-5678' → [mask] # EMAIL: 'tanaka@example.com' → [replace] # Original text is always preserved print(ctx.original_text) # Output: 田中太郎の電話番号は090-1234-5678で、メールはtanaka@example.comです # Access raw detection results for r in ctx.recognizer_results: print(f"{r.entity_type}: '{r.text}' at [{r.start}:{r.end}] (score={r.score}, source={r.source})") ``` -------------------------------- ### Analyze Text with Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/README.md Use the Besshouka command-line interface to analyze Japanese text, detecting PII without performing anonymization. The --explain flag provides details about the detected entities. ```bash besshouka analyze --explain "田中太郎の電話番号は090-1234-5678です" ``` -------------------------------- ### Register Custom Operator in Engine Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/anonymizer/operators/README.md Shows how to register a newly created custom operator within the `engine.py` file by adding it to the `_OPERATORS` dictionary. ```python from besshouka.anonymizer.operators.my_operator import MyOperator _OPERATORS["my_method"] = MyOperator() ``` -------------------------------- ### Load Operator Configuration with load_operator_config Source: https://context7.com/go-akhi/besshouka/llms.txt Load operator rules from a YAML file. Supports default and custom configurations. Ensure the YAML file contains a valid structure with 'method' fields for each entity type. ```python from besshouka.config.loader import load_operator_config # Load default operators config = load_operator_config("besshouka/defaults/operators.yaml") # Load custom operators config = load_operator_config("my_operators.yaml") ``` -------------------------------- ### CLI - Analyze Text (Detection Only) Source: https://context7.com/go-akhi/besshouka/llms.txt The `analyze` command performs only the PII detection phase, outputting detected entities in a table format. Use `--explain` for detailed confidence scores and source information. ```APIDOC ## CLI - Analyze Text (Detection Only) ### Description Performs only the PII detection phase without anonymization, displaying detected entities in a formatted table. Use `--explain` for detailed scores and source information. ### Method CLI Command ### Endpoint `besshouka analyze` ### Parameters #### Command Line Arguments - `--input` (string) - Optional - Path to the input file. - `--recognizers` (string) - Optional - Path to a custom recognizer configuration file (YAML). - `--explain` (boolean) - Optional - Include confidence scores and source information in the output. ### Request Example ```bash # Basic analysis showing detected entities besshouka analyze "田中太郎のメールはtanaka@example.comです" # Analysis with score and source reasoning besshouka analyze --explain "田中太郎の電話番号は090-1234-5678です" # Analyze from file with custom recognizers besshouka analyze --input document.txt --recognizers custom_patterns.yaml --explain ``` ### Response Example ``` # Basic analysis output: ┌─────────────┬─────────────────────┬───────┬─────┐ │ Entity Type │ Text │ Start │ End │ ├─────────────┼─────────────────────┼───────┼─────┤ │ PERSON │ 田中太郎 │ 0 │ 4 │ │ EMAIL │ tanaka@example.com │ 9 │ 27 │ └─────────────┴─────────────────────┴───────┴─────┘ # Output with --explain: │ PERSON │ 田中太郎 │ 0 │ 4 │ 0.85 │ ginza_ner │ │ PHONE_NUMBER │ 090-1234-5678 │ 10 │ 23 │ 1.00 │ regex_registry│ ``` ``` -------------------------------- ### Load Operator Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/config/README.md Loads and validates an operator rules YAML file. Each entity type entry must have a `method` field. ```APIDOC ## Load Operator Configuration ### Description Loads an operator rules YAML file. Each entity type entry must have a `method` field. ### Method `load_operator_config(path: str) -> dict` ### Parameters #### Path Parameters - **path** (str) - Required - The file path to the operator YAML configuration. ### Request Body - **method** (str) - Required - The operator method to apply for the entity type. ### Response #### Success Response (200) - **dict** - A dictionary representing the loaded operator configuration. #### Error Handling - `FileNotFoundError`: If the specified YAML file does not exist. - `ValueError`: If the YAML is malformed or the `method` field is missing. ``` -------------------------------- ### Analyze Text Only with Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Use the CLI to analyze Japanese text and display only the detected PII. ```bash python -m besshouka.cli analyze "田中太郎の電話番号は090-1234-5678です" ``` -------------------------------- ### GinzaRecognizer - NLP-Based Named Entity Recognition Source: https://context7.com/go-akhi/besshouka/llms.txt Wraps GiNZA/spaCy for Japanese NER, detecting person names, organizations, locations, dates, and more. The model is lazy-loaded on first use for faster startup. ```APIDOC ## GinzaRecognizer ### Description Leverages GiNZA, a Japanese NLP library built on spaCy, to perform Named Entity Recognition (NER). It can identify various entities such as persons, organizations, and locations. The underlying NLP model is loaded on demand to optimize startup performance. ### Method `GinzaRecognizer()` ### Parameters None ### Request Example ```python from besshouka.analyzer.recognizers.ginza_recognizer import GinzaRecognizer # Create the GiNZA recognizer (model loads lazily) recognizer = GinzaRecognizer() # Run recognition on Japanese text text = "田中太郎は株式会社ABC商事の東京本社で働いています" results = recognizer.recognize(text) ``` ### Response #### Success Response (200) - **results** (list[RecognizerResult]) - A list of `RecognizerResult` objects, each representing a detected named entity. Each object contains: - **start** (int) - The starting index of the detected entity in the text. - **end** (int) - The ending index of the detected entity in the text. - **entity_type** (str) - The standardized type of the detected entity (e.g., 'PERSON', 'ORGANIZATION', 'LOCATION'). - **score** (float) - The confidence score of the detection (typically 0.85 for GiNZA). - **source** (str) - The source of the detection (always 'ginza_ner' for this recognizer). - **text** (str) - The actual text span identified as the entity. #### Response Example ```json [ {"entity_type": "PERSON", "text": "田中太郎", "score": 0.85, "source": "ginza_ner"}, {"entity_type": "ORGANIZATION", "text": "株式会社ABC商事", "score": 0.85, "source": "ginza_ner"}, {"entity_type": "LOCATION", "text": "東京", "score": 0.85, "source": "ginza_ner"} ] ``` ### Error Handling None explicitly mentioned. Issues might arise if the GiNZA model cannot be loaded or if there are internal spaCy errors. ``` -------------------------------- ### Create and Use GinzaRecognizer for NLP-Based NER Source: https://context7.com/go-akhi/besshouka/llms.txt Wrap GiNZA/spaCy for Japanese Named Entity Recognition (NER). Detects entities like persons, organizations, and locations. The model is lazy-loaded on first use for efficiency. Standardized entity types are used for output. ```python from besshouka.analyzer.recognizers.ginza_recognizer import GinzaRecognizer # Create the GiNZA recognizer (model loads lazily) recognizer = GinzaRecognizer() # Run recognition on Japanese text text = "田中太郎は株式会社ABC商事の東京本社で働いています" results = recognizer.recognize(text) for r in results: print(f"{r.entity_type}: '{r.text}' (score={r.score}, source={r.source})") # Output: # PERSON: '田中太郎' (score=0.85, source=ginza_ner) # ORGANIZATION: '株式会社ABC商事' (score=0.85, source=ginza_ner) # LOCATION: '東京' (score=0.85, source=ginza_ner) # GiNZA labels are mapped to standardized entity types: # Person/PERSON → PERSON # Location/LOC/GPE/FAC → LOCATION # Organization/ORG/Company → ORGANIZATION # Date/DATE → DATE # Time/TIME → TIME ``` -------------------------------- ### Run Besshouka Tests Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Execute tests using pytest. Options are available to exclude slow tests or include coverage reporting. ```bash # All tests (excluding slow GiNZA tests) pytest tests/ -m "not slow" # All tests including GiNZA pytest tests/ # With coverage pytest tests/ --cov=besshouka --cov-report=term-missing ``` -------------------------------- ### Create and Use RegexRecognizer for Pattern-Based Detection Source: https://context7.com/go-akhi/besshouka/llms.txt Instantiate a RegexRecognizer to detect entities based on a provided regular expression pattern. Useful for structured data like phone numbers or employee IDs. The 'recognize' method returns a list of RecognizerResult objects. ```python from besshouka.analyzer.recognizers.regex_recognizer import RegexRecognizer # Create a recognizer for Japanese mobile phone numbers mobile_recognizer = RegexRecognizer( name="mobile_phone", entity_type="PHONE_NUMBER", pattern=r"0[789]0-?\d{4}-?\d{4}", score=1.0, source="regex_registry" ) # Run recognition text = "連絡先は090-1234-5678または080-9876-5432です" results = mobile_recognizer.recognize(text) for r in results: print(f"Found: '{r.text}' at [{r.start}:{r.end}]") # Output: # Found: '090-1234-5678' at [4:17] # Found: '080-9876-5432' at [21:34] # Create a custom recognizer for employee IDs employee_recognizer = RegexRecognizer( name="employee_id", entity_type="EMPLOYEE_ID", pattern=r"EMP-[A-Z]{2}\d{6}", score=1.0, source="custom" ) results = employee_recognizer.recognize("社員番号: EMP-AB123456") print(f"Found employee ID: {results[0].text}") # Output: Found employee ID: EMP-AB123456 ``` -------------------------------- ### Import EngineResult Model Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/models/README.md Import the EngineResult class to represent the complete output of the anonymization engine. ```python from besshouka.models.engine_result import EngineResult ``` -------------------------------- ### Load Recognizer Configuration (Python API) Source: https://context7.com/go-akhi/besshouka/llms.txt Use `load_recognizer_config` to load and validate YAML files containing PII detection patterns. This function can load default recognizers or custom patterns from a specified file path. ```python from besshouka.config.loader import load_recognizer_config # Load default recognizers config = load_recognizer_config("besshouka/defaults/recognizers.yaml") # Load custom recognizers config = load_recognizer_config("my_patterns.yaml") # Structure of returned config: # { # "recognizers": [ # { # "name": "mobile_phone", # "entity_type": "PHONE_NUMBER", # "pattern": "0[789]0-?\d{4}-?\d{4}", # "score": 1.0, # "source": "regex_registry" # }, # ... # ] # } ``` -------------------------------- ### Python API - run() Source: https://context7.com/go-akhi/besshouka/llms.txt The `run()` function in the orchestrator pipeline executes the full anonymization process, taking raw text and configuration objects, and returning a ProcessingContext with results and audit trails. ```APIDOC ## Python API - run() ### Description Executes the full anonymization pipeline, combining PII detection and transformation. It takes raw text and configuration dictionaries, returning a `ProcessingContext` object containing the anonymized text and an audit trail. ### Method `run(text: str, recognizer_config: dict, operator_config: dict) -> ProcessingContext` ### Endpoint `besshouka.orchestrator.pipeline.run` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body - **text** (string) - Required - The raw input text to process. - **recognizer_config** (dict) - Required - Configuration dictionary for PII recognizers (e.g., loaded by `load_recognizer_config`). - **operator_config** (dict) - Required - Configuration dictionary for PII transformation operators (e.g., loaded by `load_operator_config`). ### Request Example ```python from besshouka.config.loader import load_recognizer_config, load_operator_config from besshouka.orchestrator.pipeline import run # Load configurations rec_config = load_recognizer_config("recognizers.yaml") op_config = load_operator_config("operators.yaml") # Run the pipeline text = "田中太郎の電話番号は090-1234-5678で、メールはtanaka@example.comです" c_tx = run(text, rec_config, op_config) # Access anonymized text print(ctx.engine_result.text) # Access audit trail for item in ctx.engine_result.items: print(f"{item.entity_type}: '{item.original_text}' → [{item.operator}]") # Access raw detection results for r in ctx.recognizer_results: print(f"{r.entity_type}: '{r.text}' at [{r.start}:{r.end}] (score={r.score}, source={r.source})") ``` ### Response #### Success Response (200) - **ProcessingContext** (object) - Contains the results of the anonymization process. - **original_text** (string) - The original, unmodified input text. - **engine_result** (object) - **text** (string) - The anonymized text. - **items** (list) - A list of detected and transformed entities, including their type, original text, and applied operator. - **recognizer_results** (list) - A list of raw detection results from the recognizers, including entity type, text, start/end positions, score, and source. #### Response Example ```json { "original_text": "田中太郎の電話番号は090-1234-5678で、メールはtanaka@example.comです", "engine_result": { "text": "<氏名>の電話番号は090-1234-****で、メールは<メール>です", "items": [ { "entity_type": "PERSON", "original_text": "田中太郎", "operator": "replace" }, { "entity_type": "PHONE_NUMBER", "original_text": "090-1234-5678", "operator": "mask" }, { "entity_type": "EMAIL", "original_text": "tanaka@example.com", "operator": "replace" } ] }, "recognizer_results": [ { "entity_type": "PERSON", "text": "田中太郎", "start": 0, "end": 4, "score": 0.85, "source": "ginza_ner" }, { "entity_type": "PHONE_NUMBER", "text": "090-1234-5678", "start": 10, "end": 23, "score": 1.0, "source": "regex_registry" }, { "entity_type": "EMAIL", "text": "tanaka@example.com", "start": 25, "end": 43, "score": 1.0, "source": "regex_registry" } ] } ``` ``` -------------------------------- ### Anonymize Text from File with Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Use the CLI to anonymize text from an input file and save the output to another file. ```bash python -m besshouka.cli anonymize --input document.txt --output anonymized.txt ``` -------------------------------- ### Create and Use ProcessingContext Source: https://context7.com/go-akhi/besshouka/llms.txt Initialize a ProcessingContext to manage state throughout the anonymization pipeline. It holds original text, working text, detection results, engine output, and metadata. ```python from besshouka.orchestrator.context import ProcessingContext from besshouka.models.recognizer_result import RecognizerResult from besshouka.models.engine_result import EngineResult # Create a context (typically done by the pipeline) ctx = ProcessingContext( original_text="田中太郎の電話番号は090-1234-5678です", metadata={"filename": "document.txt", "timestamp": "2024-01-15"} ) # After normalization ctx.working_text = "田中太郎の電話番号は090-1234-5678です" # After recognition and conflict resolution ctx.recognizer_results = [ RecognizerResult(start=0, end=4, entity_type="PERSON", score=0.85, source="ginza_ner", text="田中太郎"), RecognizerResult(start=10, end=23, entity_type="PHONE_NUMBER", score=1.0, source="regex_registry", text="090-1234-5678"), ] # After anonymization ctx.engine_result = EngineResult(text="<氏名>の電話番号は090-1234-****です", items=[]) # Access all pipeline data print(f"Original: {ctx.original_text}") print(f"Anonymized: {ctx.engine_result.text}") print(f"Detections: {len(ctx.recognizer_results)}") print(f"Metadata: {ctx.metadata}") ``` -------------------------------- ### Extend Recognizers Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/defaults/README.md Add new recognizer entries to the YAML file. Ensure existing entries are preserved. ```yaml recognizers: # Keep all existing entries, then add: - name: employee_id entity_type: EMPLOYEE_ID pattern: 'EMP-[A-Z]{2}\d{6}' score: 1.0 source: custom ``` -------------------------------- ### Python API - load_recognizer_config() Source: https://context7.com/go-akhi/besshouka/llms.txt Loads and validates a YAML file containing regex pattern definitions for PII detection. This function is used to configure custom recognizers for the anonymization pipeline. ```APIDOC ## Python API - load_recognizer_config() ### Description Loads and validates a YAML file containing regex pattern definitions for PII detection. This function is used to configure custom recognizers for the anonymization pipeline. ### Method `load_recognizer_config(file_path: str) -> dict` ### Endpoint `besshouka.config.loader.load_recognizer_config` ### Parameters #### Path Parameters - **file_path** (string) - Required - The path to the YAML configuration file. #### Query Parameters None #### Request Body None ### Request Example ```python from besshouka.config.loader import load_recognizer_config # Load default recognizers config = load_recognizer_config("besshouka/defaults/recognizers.yaml") # Load custom recognizers config = load_recognizer_config("my_patterns.yaml") ``` ### Response #### Success Response (200) - **config** (dict) - A dictionary representing the loaded recognizer configuration. - **recognizers** (list) - A list of recognizer definitions. - **name** (string) - The name of the recognizer. - **entity_type** (string) - The type of PII entity to detect (e.g., "PHONE_NUMBER"). - **pattern** (string) - The regex pattern to match. - **score** (float) - The confidence score for the detection. - **source** (string) - The source of the pattern (e.g., "regex_registry"). #### Response Example ```json { "recognizers": [ { "name": "mobile_phone", "entity_type": "PHONE_NUMBER", "pattern": "0[789]0-?\d{4}-?\d{4}", "score": 1.0, "source": "regex_registry" }, { "name": "japanese_postal_code", "entity_type": "POSTAL_CODE", "pattern": "\d{3}-?\d{4}", "score": 1.0, "source": "regex_registry" } ] } ``` ``` -------------------------------- ### Anonymize with Custom Rules using Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Anonymize text using custom rule and pattern files specified via CLI arguments. ```bash python -m besshouka.cli anonymize --rules my_rules.yaml --recognizers my_patterns.yaml "text" ``` -------------------------------- ### Anonymize Text with Custom Config via CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/config/README.md Use the CLI to anonymize text, specifying custom recognizer and operator rule files. ```bash python -m besshouka.cli anonymize --recognizers my_patterns.yaml --rules my_operators.yaml "text" ``` -------------------------------- ### Import OperatorResult Model Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/models/README.md Import the OperatorResult class for logging individual anonymization operations. ```python from besshouka.models.engine_result import OperatorResult ``` -------------------------------- ### Handle FileNotFoundError in load_recognizer_config Source: https://context7.com/go-akhi/besshouka/llms.txt Catch FileNotFoundError when the configuration file for a recognizer is not found. This ensures graceful handling of missing configuration files. ```python try: config = load_recognizer_config("nonexistent.yaml") except FileNotFoundError as e: print(f"Config file not found: {e}") ``` -------------------------------- ### Define Custom Recognizer with YAML Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/analyzer/README.md Register a custom recognizer by defining its name, entity type, pattern, score, and source in a YAML file. This method allows adding new recognizers without modifying the core code. ```yaml recognizers: - name: employee_id entity_type: EMPLOYEE_ID pattern: 'EMP-[A-Z]{2}\d{6}' score: 1.0 source: custom ``` -------------------------------- ### Run Besshouka Tests Source: https://github.com/go-akhi/besshouka/blob/main/README.md Execute tests for Besshouka using pytest. Options include running all tests, excluding slow GiNZA model tests, or running with code coverage. ```bash # All tests (excluding slow GiNZA model tests) pytest tests/ -m "not slow" # All tests including GiNZA pytest tests/ # With coverage pytest tests/ --cov=besshouka --cov-report=term-missing ``` -------------------------------- ### Programmatic Text Anonymization with Besshouka Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Load custom recognizer and operator configurations and run the anonymization pipeline programmatically. The anonymized text and audit trail are accessible from the context object. ```python from besshouka.config.loader import load_recognizer_config, load_operator_config from besshouka.orchestrator.pipeline import run rec_config = load_recognizer_config("path/to/recognizers.yaml") op_config = load_operator_config("path/to/operators.yaml") ctx = run("田中太郎の電話番号は090-1234-5678です", rec_config, op_config) print(ctx.engine_result.text) # anonymized text print(ctx.engine_result.items) # audit trail ``` -------------------------------- ### load_operator_config() - Load Operator Rules Source: https://context7.com/go-akhi/besshouka/llms.txt Loads and validates a YAML file mapping entity types to anonymization operators. Each entity type entry must have a method field specifying which operator to use. ```APIDOC ## load_operator_config() ### Description Loads and validates a YAML file mapping entity types to anonymization operators. Each entity type entry must have a method field specifying which operator to use. ### Method `load_operator_config` ### Parameters #### Path Parameters - **filepath** (str) - Required - Path to the YAML configuration file. ### Request Example ```python from besshouka.config.loader import load_operator_config # Load default operators config = load_operator_config("besshouka/defaults/operators.yaml") # Load custom operators config = load_operator_config("my_operators.yaml") ``` ### Response #### Success Response (200) - **operators** (dict) - A dictionary where keys are entity types and values are dictionaries specifying the anonymization operator and its parameters. ### Response Example ```json { "operators": { "PERSON": {"method": "replace", "value": "<氏名>"}, "PHONE_NUMBER": {"method": "mask", "char": "*", "from_end": 4}, "MY_NUMBER": {"method": "redact"}, "CREDIT_CARD": {"method": "hash", "salt": "secret"} } } ``` ### Error Handling - `FileNotFoundError`: If the specified configuration file does not exist. - `ValueError`: If the configuration file is invalid or missing required fields (e.g., 'method' for an operator). ``` -------------------------------- ### Anonymize with Custom Rules (CLI) Source: https://context7.com/go-akhi/besshouka/llms.txt Customize the anonymization process by specifying custom recognizer patterns and operator rules using `--recognizers` and `--rules` flags with the `anonymize` command. ```bash # Anonymize with custom recognizer patterns and operator rules besshouka anonymize \ --recognizers my_patterns.yaml \ --rules my_operators.yaml \ --input sensitive_data.txt \ --output sanitized_data.txt ``` -------------------------------- ### Encrypt PII with Fernet using EncryptOperator Source: https://context7.com/go-akhi/besshouka/llms.txt Leverage EncryptOperator for reversible PII anonymization using Fernet symmetric encryption. The output is base64-encoded ciphertext, decryptable with the same key. ```python # YAML configuration # operators: # PERSON: # method: encrypt # key: "your-fernet-key-here" # Generate with: Fernet.generate_key() from cryptography.fernet import Fernet from besshouka.anonymizer.operators.encrypt import EncryptOperator # Generate a Fernet key (do this once and store securely) key = Fernet.generate_key().decode() print(f"Generated key: {key}") # Output: Generated key: ZmDfcTF7_60GrrY3ItVQ9i3XfpSgVqRvqHcJ7G8K8Kk= operator = EncryptOperator() # Encrypt a name encrypted = operator.operate("田中太郎", {"key": key}) print(encrypted) # Output: gAAAAABl... (base64-encoded ciphertext) # Decrypt later using the same key f = Fernet(key.encode()) decrypted = f.decrypt(encrypted.encode()).decode() print(decrypted) # Output: 田中太郎 ``` -------------------------------- ### BaseRecognizer Contract Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/analyzer/recognizers/README.md Defines the interface for all recognizer plugins. Implementations must provide name, source, and a recognize method. ```python class BaseRecognizer(ABC): @property def name(self) -> str: ... @property def source(self) -> str: ... def recognize(self, text: str) -> list[RecognizerResult]: ... ``` -------------------------------- ### Anonymize Inline Text with Besshouka CLI Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/README.md Use the CLI to anonymize a given string of Japanese text. ```bash python -m besshouka.cli anonymize "田中太郎の電話番号は090-1234-5678です" ``` -------------------------------- ### Implement Custom PII Transformations Source: https://context7.com/go-akhi/besshouka/llms.txt Use the custom operator method to apply user-defined Python functions for PII transformation. The function must accept text and parameters, returning a transformed string. ```python # Create a custom transformation function (in my_transforms.py) # def reverse_text(text: str, params: dict) -> str: # """Reverse the text characters.""" # return text[::-1] # # def prefix_transform(text: str, params: dict) -> str: # """Add a prefix to the text.""" # prefix = params.get("prefix", "ANON_") # return f"{prefix}{text}" # YAML configuration # operators: # EMPLOYEE_ID: # method: custom # function: "my_transforms.reverse_text" # INTERNAL_CODE: ``` -------------------------------- ### YAML Schema for PII Recognizer Registry Source: https://context7.com/go-akhi/besshouka/llms.txt Define custom regular expressions for PII detection in YAML format. Each entry includes a name, entity type, pattern, confidence score, and source. ```yaml # my_recognizers.yaml recognizers: # Japanese mobile phone numbers - name: mobile_phone entity_type: PHONE_NUMBER pattern: '0[789]0-?\d{4}-?\d{4}' score: 1.0 source: regex_registry # Email addresses - name: email entity_type: EMAIL pattern: '[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}' score: 1.0 source: regex_registry # My Number (マイナンバー) - 12 digits with optional spaces - name: my_number entity_type: MY_NUMBER pattern: '\d{4}\s?\d{4}\s?\d{4}' score: 1.0 source: regex_registry # Custom: Employee ID - name: employee_id entity_type: EMPLOYEE_ID pattern: 'EMP-[A-Z]{2}\d{6}' score: 1.0 source: custom # Custom: Internal project codes - name: project_code entity_type: PROJECT_CODE pattern: 'PRJ-\d{4}-[A-Z]{3}' score: 1.0 source: custom ``` -------------------------------- ### Handle ValueError in load_recognizer_config Source: https://context7.com/go-akhi/besshouka/llms.txt Catch ValueError when a recognizer configuration file is invalid, such as missing required fields. This helps in debugging configuration issues. ```python try: config = load_recognizer_config("invalid.yaml") # Missing required fields except ValueError as e: print(f"Invalid config: {e}") ``` -------------------------------- ### Anonymize Text from File to File (CLI) Source: https://context7.com/go-akhi/besshouka/llms.txt Utilize the `anonymize` command with `--input` and `--output` flags to process text from a file and save the anonymized result to another file. ```bash # Anonymize from file to file besshouka anonymize --input document.txt --output anonymized.txt ``` -------------------------------- ### Configure PII Operators in YAML Source: https://context7.com/go-akhi/besshouka/llms.txt Define PII anonymization operators using a YAML configuration. Supports methods like replace, mask, redact, hash, encrypt, and custom functions. ```yaml operators: # Replace with placeholder text PERSON: method: replace value: "<氏名>" ORGANIZATION: method: replace value: "<組織名>" LOCATION: method: replace value: "<住所>" EMAIL: method: replace value: "<メール>" # Partial masking PHONE_NUMBER: method: mask char: "*" from_end: 4 CREDIT_CARD: method: mask char: "X" from_end: 12 BANK_ACCOUNT: method: mask char: "*" from_end: 3 # Complete removal MY_NUMBER: method: redact PASSPORT: method: redact DRIVERS_LICENSE: method: redact # Hashing for pseudonymization CUSTOMER_ID: method: hash salt: "production-salt-2024" # Reversible encryption MEDICAL_RECORD: method: encrypt key: "ZmDfcTF7_60GrrY3ItVQ9i3XfpSgVqRvqHcJ7G8K8Kk=" # Custom function EMPLOYEE_ID: method: custom function: "my_company.transforms.mask_employee_id" preserve_prefix: true ``` -------------------------------- ### Extend Operators Configuration Source: https://github.com/go-akhi/besshouka/blob/main/besshouka/defaults/README.md Add new operator assignments for entity types to the YAML file. Ensure existing entries are preserved. ```yaml operators: # Keep all existing entries, then add: EMPLOYEE_ID: method: replace value: "<社員番号>" ```