### Install SilverSpeak Core Project Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

Executes Poetry to install all primary dependencies required for SilverSpeak's basic functionality. This command sets up the necessary environment for running the core application.

```bash
poetry install
```

--------------------------------

### Install All Optional SilverSpeak Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

This command installs all available optional dependencies for SilverSpeak, combining both spell checking and advanced contextual spell checking packages. It provides the most comprehensive set of features.

```bash
poetry install --with spell-check --with contextual-spell-check
```

--------------------------------

### Install Poetry Dependency Manager

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

This command installs Poetry, a Python dependency management and packaging tool, using pip. Poetry is essential for managing SilverSpeak's project dependencies and virtual environments.

```bash
pip install poetry
```

--------------------------------

### Clone SilverSpeak GitHub Repository

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

This command clones the SilverSpeak project from its GitHub repository into the current directory. Following the clone, it changes the working directory into the newly created 'silverspeak' folder.

```bash
git clone https://github.com/ACMCMC/silverspeak.git
cd silverspeak
```

--------------------------------

### Minimal Silverspeak Installation

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

Instructions for a basic installation of the Silverspeak library. This provides a simplified implementation without the full NLTK support, suitable for environments where NLTK is not required or desired. Installation can be done via Poetry or pip.

```bash
# Basic installation (simplified implementation only)
poetry install
pip install silverspeak
```

--------------------------------

### Install Optional Spell Checking Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

Installs additional dependencies specifically for SilverSpeak's spell checking normalization strategy. This includes packages like symspellpy and pyspellchecker.

```bash
poetry install --with spell-check
```

--------------------------------

### Install Optional Advanced Contextual Spell Checking Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/installation.rst

Installs optional dependencies for advanced contextual spell checking within SilverSpeak. This command specifically adds the neuspell package for neural spell checking capabilities.

```bash
poetry install --with contextual-spell-check
```

--------------------------------

### Install SilverSpeak with Graph-based Analysis Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs SilverSpeak with dependencies for graph-based analysis strategies.

```Python
pip install "silverspeak[graph-analysis]"
```

--------------------------------

### Install System-Level OCR Dependencies (Bash)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

These Bash commands provide instructions for installing system-level dependencies necessary for Pillow, a core image processing library. It covers installation steps for Ubuntu/Debian using `apt-get`, macOS using Homebrew, and notes that Windows typically includes Pillow.

```bash
# Ubuntu/Debian
sudo apt-get update
sudo apt-get install python3-pil python3-pil.imagetk

# macOS (with Homebrew)
brew install pillow

# Windows
# Pillow is typically included with Python installations
```

--------------------------------

### Install Silverspeak with Graph Support

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This snippet demonstrates how to install the `silverspeak` library with its optional graph analysis dependencies using `pip` or `poetry`. Installing with graph support enables advanced features like graph-based homoglyph normalization.

```bash
# Install with NetworkX support for optimal performance
pip install silverspeak[graph]

# Or using poetry
poetry install --with graph-analysis
```

--------------------------------

### Verify Silverspeak Graph-Based Normalization Installation

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This Python function verifies the successful installation and functionality of `silverspeak`'s graph-based homoglyph normalization strategy. It tests basic graph construction, NetworkX integration, and the end-to-end normalization pipeline, providing feedback on the setup.

```python
# Verify graph-based strategy installation
from silverspeak.homoglyphs.normalization.graph_based import CharacterGraph, GraphNormalizer

def verify_graph_installation():
    """Verify graph-based normalization capabilities."""
    
    try:
        # Test basic graph construction
        graph = CharacterGraph()
        graph.add_edge('a', 'b', 1.0)
        
        print("✓ Graph-based strategy installed")
        
        # Test NetworkX availability
        if graph.use_networkx:
            print("✓ NetworkX enhancement available")
            
            # Test advanced algorithms
            neighbors = graph.get_neighbors('a')
            if neighbors:
                print("✓ Graph algorithms working")
        else:
            print("⚠ Using simplified implementation (NetworkX not available)")
        
        # Test normalization pipeline
        from silverspeak.homoglyphs import get_normalization_map
        mapping = get_normalization_map()
        
        test_graph = CharacterGraph.build_from_normalization_map(mapping)
        standard_chars = extract_standard_characters(mapping)
        normalizer = GraphNormalizer(test_graph, standard_chars)
        
        result = normalizer.normalize("Test")
        print(f"✓ Normalization pipeline working: '{result}'")
        
        return True
        
    except Exception as e:
        print(f"✗ Graph installation error: {e}")
        return False
```

--------------------------------

### Install SilverSpeak with N-gram Analysis Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs SilverSpeak with dependencies for N-gram based analysis strategies.

```Python
pip install "silverspeak[ngram-analysis]"
```

--------------------------------

### Install Silverspeak with NLTK Support (Recommended)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

Instructions for performing a full installation of the Silverspeak library, including necessary NLTK dependencies for optimal n-gram analysis performance. This can be done using either Poetry or pip.

```bash
# Install with NLTK support for optimal performance
poetry install --with ngram-analysis

# Or using pip
pip install silverspeak[ngram]
```

--------------------------------

### Install SilverSpeak with Full OCR Support (Bash)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

These Bash commands demonstrate how to install the `silverspeak` library, including all its OCR-related dependencies. Options are provided for both `pip` (standard Python package manager) and `poetry` (a dependency management tool).

```bash
# Install with complete OCR support
pip install silverspeak[ocr]

# Or using poetry
poetry install --with ocr-analysis
```

--------------------------------

### Install SilverSpeak with OCR-based Analysis Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs `pytesseract` and `pillow`, which are required for OCR-based analysis functionalities within SilverSpeak.

```Python
pip install pytesseract pillow
```

--------------------------------

### Install SilverSpeak Python Package

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

This command installs the core SilverSpeak library from PyPI, providing basic homoglyph attack functionalities.

```Python
pip install silverspeak
```

--------------------------------

### Install All Optional SilverSpeak Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs SilverSpeak with all available optional dependencies for comprehensive analysis capabilities, including spell-check, contextual spell-check, n-gram, and graph-based analysis, along with OCR dependencies.

```Python
pip install "silverspeak[spell-check,contextual-spell-check,ngram-analysis,graph-analysis]"
pip install pytesseract pillow
```

--------------------------------

### Verify Silverspeak N-gram Installation in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

A Python script to verify the successful installation and capabilities of the Silverspeak N-gram analyzer. It checks if the NLTK enhancement is available and provides clear feedback on the current setup, including any import errors.

```python
# Verify installation and capabilities
from silverspeak.homoglyphs.normalization.ngram import CharNgramAnalyzer

try:
    analyzer = CharNgramAnalyzer()
    print("✓ N-gram strategy installed successfully")
    
    if analyzer.use_nltk:
        print("✓ NLTK enhancement available")
    else:
        print("⚠ Using simplified implementation (NLTK not available)")
        
except ImportError as e:
    print(f"✗ Installation error: {e}")
```

--------------------------------

### Install SilverSpeak with Contextual Spell Checking Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs SilverSpeak including dependencies for advanced contextual spell-checking capabilities.

```Python
pip install "silverspeak[contextual-spell-check]"
```

--------------------------------

### Install SilverSpeak with Spell Checking Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/README.md

Installs SilverSpeak along with additional dependencies required for enhanced spell-checking normalization strategies.

```Python
pip install "silverspeak[spell-check]"
```

--------------------------------

### Install Optional High-Performance Graph Libraries

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This snippet provides commands to install optional high-performance graph libraries that can further enhance `silverspeak`'s capabilities. These alternatives, such as `python-igraph`, `graph-tool`, and `networkit`, offer specialized features for large-scale graph analysis.

```bash
# Performance enhancements
pip install python-igraph      # Alternative graph library
pip install graph-tool         # High-performance graph analysis
pip install networkit          # Large-scale graph processing
```

--------------------------------

### Recommended Tokenizer Selection by Application

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Offers a dictionary of recommended HuggingFace tokenizers based on specific application types, such as general purpose, multilingual, conversation, and code generation. This guides users in choosing an appropriate tokenizer.

```python
# Choose tokenizers based on target application
tokenizer_recommendations = {
    "general_purpose": "bert-base-uncased",
    "multilingual": "bert-base-multilingual-cased", 
    "conversation": "microsoft/DialoGPT-medium",
    "code_generation": "microsoft/CodeBERT-base"
}
```

--------------------------------

### Verify OCR Capabilities Installation (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This Python function verifies the successful installation and operational status of OCR normalization capabilities within the `silverspeak` library. It checks for the `OCRConfidenceAnalyzer` and confirms the availability of the DocTR OCR model, providing diagnostic output.

```python
# Verify OCR capabilities
from silverspeak.homoglyphs.normalization.ocr_confidence import OCRConfidenceAnalyzer

def verify_ocr_installation():
    """Verify OCR normalization capabilities."""
    
    try:
        analyzer = OCRConfidenceAnalyzer()
        
        print("✓ OCR Confidence strategy installed")
        
        # Test DocTR availability
        if hasattr(analyzer, 'ocr_model') and analyzer.ocr_model is not None:
            print("✓ DocTR OCR model available")
        else:
            print("⚠ DocTR not available, using confusion matrix mode")
        
        # Test basic functionality
```

--------------------------------

### Normalize Text Using LLM Prompt-based Strategy (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/usage.rst

This example demonstrates normalizing text with homoglyphs by utilizing an LLM prompt-based strategy. This method leverages large language models for sophisticated homoglyph detection and correction, requiring a compatible transformers model. The `normalize_text` function, when used with `NormalizationStrategies.LLM_PROMPT`, takes the text and an optional model name, returning the LLM-corrected string.

```python
from silverspeak.homoglyphs import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Text with homoglyphs
text = "Tһis іs а tеst with ѕome һomoglурhs."

# Normalize using LLM prompting (requires transformers with a suitable model)
normalized_text = normalize_text(
    text,
    strategy=NormalizationStrategies.LLM_PROMPT,
    model_name="google/gemma-2-1b-it"  # Optional: specify a different model
)
print(normalized_text)
```

--------------------------------

### Install Core Python OCR Dependencies (Bash)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

These Bash commands install essential Python libraries required for OCR functionality. This includes `pillow` for image processing, `doctr` for document text recognition, and `torch` and `torchvision` which are deep learning frameworks used by DocTR models.

```bash
# Essential OCR dependencies
pip install pillow>=8.0.0
pip install doctr>=0.6.0
pip install torch>=1.8.0  # For DocTR models
pip install torchvision>=0.9.0
```

--------------------------------

### Simplified Homoglyph Normalization with normalize_text (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This example demonstrates a simpler approach to homoglyph normalization using the 'normalize_text' function, which handles automatic model loading. It shows how to specify the normalization strategy, model name, confidence threshold, and batch size for processing.

```python
from silverspeak.homoglyphs import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Automatic model loading (recommended for simplicity)
text = "Mathеmatical ехprеssion: ∫f(х)dx = ln|х| + C"
normalized_text = normalize_text(
    text, 
    strategy=NormalizationStrategies.LANGUAGE_MODEL,
    model_name="distilbert-base-multilingual-cased",  # Faster alternative
    min_confidence=0.5,  # Adjust confidence threshold
    batch_size=4         # Process multiple masks per batch
)
print(normalized_text)
```

--------------------------------

### Apply Tokenizer Strategy to Text (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

This example illustrates the basic usage of the `apply_tokenizer_strategy` function from `silverspeak.homoglyphs.normalization`. It shows how to pass a string containing homoglyphs (e.g., mixed Cyrillic characters) to the function for normalization. Note that the provided snippet is incomplete, showing only the import and the start of the text and normalization map definition.

```python
from silverspeak.homoglyphs.normalization import apply_tokenizer_strategy

text = "Тhis іs а samрle with homoglуphs."  # Mixed Cyrillic homoglyphs
normalization_map = {
```

--------------------------------

### Normalize Text Using Spell Check Strategy (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/usage.rst

This example demonstrates normalizing text by leveraging a spell-checking strategy to identify and correct homoglyphs. This method requires spell-check dependencies and can optionally specify a language for more accurate normalization. The `normalize_text` function with `NormalizationStrategies.SPELL_CHECK` takes text and an optional language, returning the normalized string.

```python
from silverspeak.homoglyphs import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Text with homoglyphs (Cyrillic 'а', 'е', 'р')
text = "This is а tеst with homoglурhs."

# Normalize using spell checking (requires spell-check dependencies)
normalized_text = normalize_text(
    text, 
    strategy=NormalizationStrategies.SPELL_CHECK,
    language="en"  # Optional: specify language (default is English)
)
print(normalized_text)  # "This is a test with homoglyphs."
```

--------------------------------

### Load and Use Custom OCR Confusion Matrices

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This example demonstrates how to load a custom OCR confusion matrix from a JSON file using `load_confusion_matrix`. It also illustrates the expected structure of such a confusion matrix, showing how characters are mapped to their common confusions with associated probabilities. Finally, it shows how to initialize the `OCRConfidenceAnalyzer` with this custom confusion data.

```python
from silverspeak.homoglyphs.normalization.ocr_confidence import load_confusion_matrix

# Load custom confusion matrix
custom_matrix = load_confusion_matrix("path/to/custom_matrix.json")

# Example confusion matrix structure
confusion_data = {
    "O": {"0": 0.6, "o": 0.4},      # 'O' confused with '0' (60%) and 'o' (40%)
    "l": {"I": 0.5, "1": 0.3, "|": 0.2},  # 'l' confused with multiple chars
    "rn": {"m": 0.8}                # Character sequence confusion
}

analyzer = OCRConfidenceAnalyzer(confusion_matrix=confusion_data)
```

--------------------------------

### Normalize Text Using Dominant Script and Block Strategy via normalize_text

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/dominant_script_and_block.rst

This example shows an alternative way to normalize text using the `normalize_text` function, specifying `NormalizationStrategies.DOMINANT_SCRIPT_AND_BLOCK`. It provides a simpler interface for applying the same dual-criteria strategy, making the code more concise.

```python
from silverspeak.homoglyphs import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

text = "Examрle tеxt with symbоls like ∑ and α."
normalized_text = normalize_text(
    text, 
    strategy=NormalizationStrategies.DOMINANT_SCRIPT_AND_BLOCK
)
print(normalized_text)
```

--------------------------------

### Basic Homoglyph Normalization with OCR Confidence

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This example shows how to perform simple homoglyph normalization using the `normalize_text` function with the `OCR_CONFIDENCE` strategy. It imports necessary modules and demonstrates normalizing a suspicious string, printing both original and normalized versions to illustrate the effect.

```python
from silverspeak.homoglyphs.normalize import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Simple OCR-based normalization
suspicious_text = "Tһis dоcument contаins ѕuѕpicious chаracters"
normalized = normalize_text(
    suspicious_text,
    strategy=NormalizationStrategies.OCR_CONFIDENCE
)

print(f"Original:   {suspicious_text}")
print(f"Normalized: {normalized}")
# Output: "This document contains suspicious characters"
```

--------------------------------

### Normalize Text by Replacing Homoglyphs (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/usage.rst

This example illustrates the use of the `normalize_text` function to convert homoglyphs in a string back to their canonical equivalents. This process is crucial for ensuring text consistency, especially when dealing with mixed-script content. The function takes a string containing homoglyphs and returns a string with normalized characters.

```python
from silverspeak.homoglyphs import normalize_text

# Define the input text containing homoglyphs
text = "exаmple"  # Note: homoglyph 'а' (Cyrillic) is used instead of 'a' (Latin)

# Normalize the text by replacing homoglyphs with their canonical equivalents
normalized_text = normalize_text(text)

# Print the normalized text
print(normalized_text)
```

--------------------------------

### Direct Application of OCR Confidence Normalization Strategy

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This example shows how to directly apply the `apply_ocr_confidence_strategy` function for homoglyph normalization. It demonstrates obtaining a normalization map and then using it with a specified confidence threshold, offering fine-grained control over the normalization process.

```python
from silverspeak.homoglyphs.normalization.ocr_confidence import apply_ocr_confidence_strategy
from silverspeak.homoglyphs import get_normalization_map

# Direct strategy usage with full control
mapping = get_normalization_map()

normalized = apply_ocr_confidence_strategy(
    text=suspicious_text,
    mapping=mapping,
    confidence_threshold=0.8
)
```

--------------------------------

### Calculate Sliding Window Context in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/local_context.rst

This Python snippet calculates the start and end indices for a sliding window around a character in a text, ensuring the window is optimally sized even at text boundaries. It dynamically adjusts to provide sufficient context for character analysis.

```python
start = max(0, i - N // 2)
end = min(len(text), i + N // 2 + 1)
context_window = text[start:end]

# Ensure we have a sufficiently sized window
if len(context_window) < min(N, len(text)):
    if start == 0:
        context_window = text[: min(N, len(text))]
    elif end == len(text):
        context_window = text[-min(N, len(text)) :]
```

--------------------------------

### Install Core Graph Processing Dependencies

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This snippet lists the essential Python packages required for graph processing within `silverspeak`, including NetworkX for graph structures, SciPy for advanced algorithms, and NumPy for numerical operations. These dependencies ensure optimal performance for graph-based features.

```bash
# Essential graph processing dependencies
pip install networkx>=2.6.0
pip install scipy>=1.7.0      # For advanced algorithms
pip install numpy>=1.21.0     # Numerical operations
```

--------------------------------

### Apply Local Context Strategy for Text Normalization in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/local_context.rst

This Python example demonstrates how to use the `apply_local_context_strategy` function to normalize text containing homoglyphs. It shows how to define a `normalization_map` to specify character replacements, leveraging the strategy's advanced scoring for contextual consistency.

```python
from silverspeak.homoglyphs.normalization import apply_local_context_strategy

text = "Exаmple tеxt with һomoglуphs."  # Contains Cyrillic homoglyphs
normalization_map = {
    "а": ["a"],  # Cyrillic 'а' to Latin 'a'
    "е": ["e"]  # Cyrillic 'е' to Latin 'e'
}
```

--------------------------------

### Applying Language Model Strategy for Homoglyph Normalization (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This example illustrates the basic usage of the 'apply_language_model_strategy' function from SilverSpeak. It loads a pre-trained BERT model and tokenizer, defines a homoglyph mapping, and then normalizes a text containing mixed Cyrillic homoglyphs.

```python
from silverspeak.homoglyphs.normalization import apply_language_model_strategy
from transformers import AutoTokenizer, AutoModelForMaskedLM

# Load model components
model_name = "bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

text = "Тhе quісk brоwn fох јumps оvеr thе lаzу dоg."  # Mixed Cyrillic homoglyphs
normalization_map = {
    "Т": ["T"], "е": ["e"], "і": ["i"], "с": ["c"], "k": ["k"],
    "о": ["o"], "w": ["w"], "n": ["n"], "х": ["x"], "ј": ["j"], 
    "m": ["m"], "р": ["p"], "s": ["s"], "v": ["v"], "r": ["r"],
    "h": ["h"], "l": ["l"], "а": ["a"], "z": ["z"], "у": ["y"], "g": ["g"]
}

normalized_text = apply_language_model_strategy(
    text=text,
    mapping=normalization_map,
    language_model=model,
    tokenizer=tokenizer
)
print(f"Original:   {text}")
print(f"Normalized: {normalized_text}")
```

--------------------------------

### Perform Context-Aware Token Matching (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

This code demonstrates how the strategy filters potential tokens to ensure they align with the existing normalized text. It iterates through possible token starts for each character and retains only those where the normalized text ends with the token's prefix. This context-aware matching ensures that character selections seamlessly integrate into the broader tokenization context of the surrounding text.

```python
# Filter tokens to match existing context
for char_key in possible_token_starts.keys():
    possible_token_starts[char_key] = [
        token_tuple for token_tuple in possible_token_starts[char_key]
        if normalized_text.endswith(token_tuple[0])  # Prefix matching
    ]
```

--------------------------------

### Apply Advanced Graph Algorithms for Character Normalization in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This example illustrates the application of advanced graph algorithms within the `CharacterGraph` for normalization tasks. It demonstrates finding optimal paths between characters, identifying the most central character among ambiguous candidates, and retrieving all similar characters with their associated weights.

```python
# Find optimal normalization paths
path = graph.find_path('һ', 'h')        # Shortest path between characters

# Identify most central character in ambiguous cases
candidates = ['0', 'O', 'o']
central_char = graph.find_most_central_character(candidates)

# Get similarity neighborhoods
neighbors = graph.get_neighbors('а')     # All similar characters with weights
```

--------------------------------

### Automatic DocTR Integration with OCRConfidenceAnalyzer

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This example illustrates the automatic integration of DocTR models within the `OCRConfidenceAnalyzer`. When initialized without specific parameters, the analyzer automatically leverages DocTR for text rendering, OCR processing, and extraction of word/character-level confidence scores, identifying low-confidence regions for further analysis.

```python
# DocTR model initialization (automatic when available)
analyzer = OCRConfidenceAnalyzer()

# The analyzer automatically:
# 1. Renders text using multiple font variations
# 2. Processes images through DocTR's OCR pipeline
# 3. Extracts word-level and character-level confidence scores
# 4. Identifies low-confidence regions for further analysis
```

--------------------------------

### Replace Characters with Homoglyphs Using HomoglyphReplacer (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/usage.rst

This snippet shows how to utilize the `HomoglyphReplacer` class to substitute characters in a given text with homoglyphs. This functionality can be employed for text obfuscation or artistic purposes. The `replace` method of an initialized `HomoglyphReplacer` instance takes a string and returns a new string with characters replaced by homoglyphs based on the replacer's configuration.

```python
from silverspeak.homoglyphs import HomoglyphReplacer

# Initialize a HomoglyphReplacer instance
replacer = HomoglyphReplacer()

# Define the input text to be modified
text = "example"

# Replace characters in the text with homoglyphs
replaced_text = replacer.replace(text)

# Print the modified text
print(replaced_text)
```

--------------------------------

### Create Optimal Analyzer for Available Resources

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This snippet outlines a function `create_optimal_analyzer` designed to initialize the `OCRConfidenceAnalyzer` optimally based on available computational resources. It suggests attempting a full DocTR initialization and implies a fallback mechanism if resources are limited.

```python
# Automatic fallback mechanism
def create_optimal_analyzer():
    """Create analyzer optimized for available resources."""
    
    try:
        # Attempt full DocTR initialization
```

--------------------------------

### Initialize and Validate Hugging Face Masked Language Model

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This snippet demonstrates loading a pre-trained Hugging Face transformer model and its tokenizer for masked language modeling (MLM). It includes a crucial validation step to ensure the loaded model supports MLM capabilities, raising a ValueError if not. The implementation also implicitly handles optimal device placement, such as GPU detection, for performance.

```python
from silverspeak.homoglyphs.normalization import apply_language_model_strategy
from transformers import AutoTokenizer, AutoModelForMaskedLM
import torch

# Load model and tokenizer
model_name = "bert-base-multilingual-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForMaskedLM.from_pretrained(model_name)

# Verify MLM capability
if not hasattr(model, 'get_output_embeddings'):
    raise ValueError("Model does not support masked language modeling")
```

--------------------------------

### Optimize Tokenizer Strategy by Pre-loading

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Illustrates an optimization strategy for `apply_tokenizer_strategy` by pre-loading the tokenizer using `AutoTokenizer.from_pretrained`. This reduces overhead when processing multiple texts with the same tokenizer.

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

# Process multiple texts efficiently
texts = ["Text 1", "Text 2", "Text 3"]
for text in texts:
    result = apply_tokenizer_strategy(
        text=text,
        mapping=normalization_map,
        tokenizer_name="bert-base-uncased"  # Reuses loaded tokenizer
    )
```

--------------------------------

### Example: Normalize Cyrillic Homoglyphs in Mixed Text (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/dominant_script.rst

This example demonstrates the practical application of the Dominant Script Strategy. It initializes a `HomoglyphReplacer` and then uses `apply_dominant_script_strategy` to normalize a string containing Cyrillic homoglyphs mixed with Latin characters. The output shows how the strategy correctly identifies Latin as the dominant script and converts the Cyrillic homoglyphs to their Latin equivalents, effectively resolving the homoglyph issue.

```Python
from silverspeak.homoglyphs.normalization import apply_dominant_script_strategy
from silverspeak.homoglyphs import HomoglyphReplacer

# Text containing Cyrillic homoglyphs mixed with Latin
text = "Examрle tеxt with sоme homoglурhs."  # Contains Cyrillic 'р', 'е', 'о', 'у'

# Initialize the replacer
replacer = HomoglyphReplacer()

# Apply dominant script normalization
normalized_text = apply_dominant_script_strategy(replacer, text)
print(normalized_text)  # Output: "Example text with some homoglyphs."
```

--------------------------------

### Optimize PyTorch Model Performance with Configuration

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This snippet demonstrates how to configure a PyTorch model for performance optimization. It sets parameters like batch size, sequence length, device, and data type, and shows how to enable mixed precision training using `torch.cuda.amp.autocast` for faster execution and reduced memory usage.

```python
# Performance optimization techniques
optimized_config = {
    "batch_size": 8,        # Process multiple positions simultaneously
    "max_length": 256,      # Reduce sequence length for speed
    "device": "cuda",       # Use GPU acceleration
    "torch_dtype": torch.float16  # Use half precision for memory efficiency
}

# Enable mixed precision for faster training
with torch.cuda.amp.autocast():
    outputs = model(**inputs)
```

--------------------------------

### Basic Text Normalization using NGRAM Strategy

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

Shows a simple example of normalizing text containing homoglyphs using the normalize_text function with the NGRAM strategy. It demonstrates the default behavior for correcting suspicious characters.

```python
from silverspeak.homoglyphs.normalize import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Simple normalization with default parameters
suspicious_text = "Tһis іs а tеst with ѕome һomoglурhs"
normalized = normalize_text(
    suspicious_text,
    strategy=NormalizationStrategies.NGRAM
)
    
print(f"Original: {suspicious_text}")
print(f"Normalized: {normalized}")
# Output: "This is a test with some homoglyphs"
```

--------------------------------

### Multi-Language NGRAM Normalization

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

Demonstrates the normalize_text function's capability to handle different languages by specifying the language parameter. It shows an example of normalizing Spanish text with specific n-gram values and a threshold, highlighting the language-specific modeling.

```python
# Language-specific normalization
spanish_text = "Estе tеxto contіеnе caractеrеs sospеchosos"
    
normalized_spanish = normalize_text(
    spanish_text,
    strategy=NormalizationStrategies.NGRAM,
    language="spanish",
    n_values=[2, 3, 4],
    threshold=0.01
)
```

--------------------------------

### Normalize Text with Tokenization Strategy

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Illustrates an alternative method for text normalization using `normalize_text` with the `TOKENIZATION` strategy. It demonstrates specifying a different tokenizer for the process.

```python
from silverspeak.homoglyphs import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

text = "Mathеmatical ехprеssion: f(х) = 2х + 1"  # Mixed scripts
normalized_text = normalize_text(
    text,
    strategy=NormalizationStrategies.TOKENIZATION,
    tokenizer_name="microsoft/DialoGPT-medium"  # Different tokenizer
)
print(normalized_text)
```

--------------------------------

### OCR Confidence Analyzer Initialization and Fallback

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This snippet demonstrates the initialization of an `OCRConfidenceAnalyzer`. It attempts to use a full DocTR OCR analysis model and falls back to a confusion matrix analysis if DocTR is unavailable or fails, printing status messages to indicate the active analysis mode.

```python
            analyzer = OCRConfidenceAnalyzer(confidence_threshold=0.7)
            if analyzer.ocr_model is not None:
                print("✓ Using full DocTR OCR analysis")
                return analyzer
        except Exception as e:
            print(f"DocTR unavailable: {e}")
        
        # Fall back to confusion matrix analysis
        print("⚠ Using confusion matrix analysis only")
        return OCRConfidenceAnalyzer(confidence_threshold=0.6)
```

--------------------------------

### Initialize and Use OCRConfidenceAnalyzer

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ocr_confidence.rst

This snippet demonstrates how to initialize the `OCRConfidenceAnalyzer` with custom parameters like confidence threshold, fonts, and font size. It then shows how to analyze a text string for suspicious characters and iterate through the results, which include the position, confidence score, and candidate replacements for each suspicious character.

```python
from silverspeak.homoglyphs.normalization.ocr_confidence import OCRConfidenceAnalyzer

# Initialize analyzer with custom parameters
analyzer = OCRConfidenceAnalyzer(
    confidence_threshold=0.7,    # OCR confidence threshold
    fonts=["Arial", "Times"],    # Fonts for rendering
    font_size=24                 # Rendering size
)

# Analyze text for suspicious characters
suspicious_chars = analyzer.analyze_text("Tһis contаins һomoglyphs")

# Results: [(position, confidence_score, [candidate_replacements])]
for pos, confidence, candidates in suspicious_chars:
    print(f"Position {pos}: confidence {confidence:.3f}, candidates: {candidates}")
```

--------------------------------

### Load Tokenizer and Analyze Vocabulary (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

This snippet demonstrates how the tokenizer strategy loads a pre-trained tokenizer, such as 'google/gemma-3-1b-pt', and processes its vocabulary. The vocabulary keys are extracted and then sorted by length in descending order, which prioritizes longer, more specific tokens during subsequent analysis. This step is crucial for preparing the tokenizer's knowledge base for homoglyph analysis.

```python
from silverspeak.homoglyphs.normalization import apply_tokenizer_strategy
from transformers import AutoTokenizer

# The strategy loads the tokenizer internally
tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-pt")
vocab = list(tokenizer.get_vocab().keys())
vocab = sorted(vocab, key=len, reverse=True)
```

--------------------------------

### Basic Text Normalization with apply_tokenizer_strategy

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Demonstrates the fundamental usage of `apply_tokenizer_strategy` for homoglyph normalization. It defines a sample `normalization_map` and applies a specified tokenizer to transform text, then prints the original and normalized versions.

```python
text = "Тіару" # Example text for demonstration
normalization_map = {
    "Т": ["T"],  # Cyrillic 'Т' to Latin 'T'
    "і": ["i"],  # Cyrillic 'і' to Latin 'i'
    "а": ["a"],  # Cyrillic 'а' to Latin 'a'
    "р": ["p"],  # Cyrillic 'р' to Latin 'p'
    "у": ["u"]
}

# Assume apply_tokenizer_strategy is imported or defined
# from silverspeak.homoglyphs.strategies import apply_tokenizer_strategy # Example import

normalized_text = apply_tokenizer_strategy(
    text=text,
    mapping=normalization_map,
    tokenizer_name="google/gemma-3-1b-pt"
)
print(f"Original:   {text}")
print(f"Normalized: {normalized_text}")
```

--------------------------------

### Integrate Tokenizer and Local Context Strategies in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Illustrates how to sequentially apply different homoglyph normalization strategies for enhanced accuracy. It shows a two-pass approach: first, applying tokenizer-based normalization, and then refining the result with local context normalization, leveraging `NormalizationStrategies`.

```python
# Sequential application for enhanced accuracy
from silverspeak.homoglyphs.utils import NormalizationStrategies

# First pass: tokenizer-based normalization
intermediate = normalize_text(text, strategy=NormalizationStrategies.TOKENIZATION)

# Second pass: local context refinement
final_result = normalize_text(intermediate, strategy=NormalizationStrategies.LOCAL_CONTEXT)
```

--------------------------------

### Compare Homoglyph Normalization Strategies Performance in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/tokenizer.rst

Provides a dictionary outlining the performance characteristics and trade-offs of various homoglyph normalization strategies, including tokenizer, local context, dominant script, and language model approaches. This helps in selecting the appropriate strategy based on accuracy, speed, and resource requirements.

```python
# Performance comparison example
strategies_comparison = {
    "tokenizer": "High accuracy for tokenized workflows, higher memory usage",
    "local_context": "Fast, context-aware, moderate accuracy",
    "dominant_script": "Very fast, script-based, lower accuracy for mixed scripts",
    "language_model": "Highest accuracy, very high computational cost"
}
```

--------------------------------

### Normalize Text Using SilverSpeak Homoglyph Strategies

Source: https://github.com/acmcmc/silverspeak/blob/main/silverspeak/homoglyphs/normalization/README.md

This Python code demonstrates how to use the `normalize_text` function from the `silverspeak.homoglyphs.normalize` module. It shows examples of normalizing text using the default local context strategy, a specified `DOMINANT_SCRIPT` strategy, and a `LANGUAGE_MODEL` strategy with a custom model name.

```python
from silverspeak.homoglyphs.normalize import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Normalize using local context (default strategy)
normalized_text = normalize_text("Hеllo wоrld")  # Contains Cyrillic 'е' and 'о'

# Normalize using a specific strategy
normalized_text = normalize_text(
    "Hеllo wоrld", 
    strategy=NormalizationStrategies.DOMINANT_SCRIPT
)

# Use language model strategy with custom model name
normalized_text = normalize_text(
    "Hеllo wоrld",
    strategy=NormalizationStrategies.LANGUAGE_MODEL,
    model_name="bert-base-uncased"
)
```

--------------------------------

### Using Domain-Specific Models for Homoglyph Normalization (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This example demonstrates how to leverage domain-specific language models to improve the accuracy of homoglyph normalization for specialized texts. It defines a dictionary of models for scientific, clinical, legal, and financial domains, then applies normalization using a scientific model.

```python
# Using domain-specific models for better accuracy
domain_models = {
    "scientific": "allenai/scibert_scivocab_uncased",
    "clinical": "emilyalsentzer/Bio_ClinicalBERT", 
    "legal": "nlpaueb/legal-bert-base-uncased",
    "financial": "ProsusAI/finbert"
}

scientific_text = "Thе protеin structurе shows ехcеllеnt stаbility."
result = normalize_text(
    scientific_text,
    strategy=NormalizationStrategies.LANGUAGE_MODEL,
    model_name=domain_models["scientific"]
)
```

--------------------------------

### Applying Local Context Strategy with Custom Map

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/local_context.rst

Demonstrates direct usage of `apply_local_context_strategy` with a custom `normalization_map` and a specified context window size `N`. It shows how to normalize text containing homoglyphs based on surrounding character properties.

```python
"һ": ["h"],  # Cyrillic 'һ' to Latin 'h'\n       "у": ["y"],  # Cyrillic 'у' to Latin 'y'\n   }\n   \n   normalized_text = apply_local_context_strategy(\n       text, \n       normalization_map, \n       N=10  # Context window size\n   )\n   print(normalized_text)  # Output: "Example text with homoglyphs."
```

--------------------------------

### Perform Basic Graph-Based Homoglyph Normalization (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/graph_based.rst

This example demonstrates how to use the `normalize_text` function for simple graph-based homoglyph normalization. It takes a suspicious string containing homoglyphs and applies the `GRAPH_BASED` strategy to convert them to their standard ASCII equivalents, printing both the original and normalized text.

```Python
from silverspeak.homoglyphs.normalize import normalize_text
from silverspeak.homoglyphs.utils import NormalizationStrategies

# Simple graph-based normalization
suspicious_text = "Tһis grаph аnаlysis detects сomplеx һomoglyph pаtterns"
normalized = normalize_text(
    suspicious_text,
    strategy=NormalizationStrategies.GRAPH_BASED
)

print(f"Original:   {suspicious_text}")
print(f"Normalized: {normalized}")
# Output: "This graph analysis detects complex homoglyph patterns"
```

--------------------------------

### Download NLTK Data for Silverspeak in Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/ngram.rst

Demonstrates how NLTK data, such as 'words', 'punkt', 'brown', and 'gutenberg', is typically handled automatically by Silverspeak. Manual download commands are provided for cases where explicit control over NLTK data is needed.

```python
# Automatic NLTK data download (handled internally)
import nltk

# Manual download if needed
nltk.download('words')
nltk.download('punkt')
nltk.download('brown')
nltk.download('gutenberg')
```

--------------------------------

### Extracting Confidence Scores for Token Predictions (Python)

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This snippet demonstrates how to extract prediction confidence scores from a language model's output using PyTorch. It shows how to apply softmax to logits to get probability distributions and then filter top tokens based on a minimum confidence threshold.

```python
# Extract predictions with confidence scores
with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits
    
# Apply softmax for probability distribution
probs = torch.softmax(logits, dim=-1)
top_values, top_indices = torch.topk(probs, k=10)

# Filter by confidence threshold
candidates = [
    (token, confidence) for token, confidence in zip(top_tokens, top_values)
    if confidence >= min_confidence
]
```

--------------------------------

### Normalizing Text using normalize_text with Local Context Strategy

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/local_context.rst

Illustrates an alternative usage of the local context strategy via the higher-level `normalize_text` function. It shows how to specify the strategy and an optional context window size for homoglyph normalization.

```python
from silverspeak.homoglyphs import normalize_text\nfrom silverspeak.homoglyphs.utils import NormalizationStrategies\n\ntext = "Exаmple tеxt with һomoglуphs."\nnormalized_text = normalize_text(\n    text, \n    strategy=NormalizationStrategies.LOCAL_CONTEXT,\n    N=10  # Optional: specify context window size\n)\nprint(normalized_text)
```

--------------------------------

### Centralized Language Model Configuration for Different Environments

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This Python dictionary provides a centralized configuration for language models, tailored for 'production', 'research', and 'development' environments. It specifies parameters such as model name, batch size, minimum confidence, maximum sequence length, and device, allowing for easy management and switching of model settings.

```python
   # Centralized configuration for different use cases
   LANGUAGE_MODEL_CONFIGS = {
       "production": {
           "model_name": "distilbert-base-multilingual-cased",
           "batch_size": 4,
           "min_confidence": 0.8,
           "max_length": 256,
           "device": "auto"
       },
       "research": {
           "model_name": "microsoft/mdeberta-v3-base",
           "batch_size": 1,
           "min_confidence": 0.5,
           "max_length": 512,
           "device": "cuda"
       },
       "development": {
           "model_name": "bert-base-multilingual-cased",
           "batch_size": 2,
           "min_confidence": 0.6,
           "max_length": 256,
           "device": "auto"
       }
   }
```

--------------------------------

### Implement Resource-Aware Processing with Python

Source: https://github.com/acmcmc/silverspeak/blob/main/docs/source/normalization_strategies/language_model.rst

This function demonstrates how to apply resource constraints, specifically memory and time limits, to a processing task. It uses `psutil` to check available memory and `signal` to set a timeout, raising errors if limits are exceeded to prevent resource exhaustion.

```python
def resource_aware_normalization(text, max_memory_gb=4, timeout_seconds=300):
    """Apply language model strategy with resource constraints."""
    import psutil
    import signal
    
    # Check available memory
    available_memory = psutil.virtual_memory().available / (1024**3)
    if available_memory < max_memory_gb:
        raise RuntimeError(f"Insufficient memory: {available_memory:.1f}GB < {max_memory_gb}GB")
    
    # Set timeout for processing
    def timeout_handler(signum, frame):
        raise TimeoutError("Processing exceeded time limit")
    
    signal.signal(signal.SIGALRM, timeout_handler)
    signal.alarm(timeout_seconds)
    
    try:
        result = apply_language_model_strategy(text, mapping)
        return result
    finally:
        signal.alarm(0)  # Disable timeout
```