### CharBoundary CLI Help and Basic Commands (Bash)

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Provides examples for using the CharBoundary command-line interface (CLI) to access help documentation and perform basic operations. It shows how to get general help and help for specific commands like 'analyze', 'train', and 'best-model'.

```bash
# Get help for all commands
charboundary --help

# Get help for a specific command
charboundary analyze --help
charboundary train --help
charboundary best-model --help
```

--------------------------------

### Install Charboundary CLI

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Provides instructions for installing the charboundary command-line interface (CLI) globally as an application using pipx or within a project using pip.

```bash
pipx install charboundary
pip install charboundary
```

--------------------------------

### Install CharBoundary with Optional Dependencies

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Installs the CharBoundary library with optional dependencies for enhanced functionality. The `numpy` extra enables faster processing, while the `onnx` extra allows for ONNX model conversion and faster inference. Installing both extras includes all optional dependencies.

```bash
pip install charboundary

```

```bash
# With NumPy support for faster processing
pip install charboundary[numpy]

```

```bash
# With ONNX support for model conversion and faster inference
pip install charboundary[onnx]

```

```bash
# With all optional dependencies
pip install charboundary[numpy,onnx]

```

--------------------------------

### CharBoundary Threshold Calibration Example (Bash)

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates the effect of the 'threshold' parameter in the 'charboundary analyze' command on text segmentation. It shows how different threshold values (low, default, high) impact the identification of sentence boundaries, particularly in complex legal text.

```bash
# Create a test file
echo "The plaintiff, Mr. Brown vs. Johnson Corp., argued that patent no. 12345 was infringed. Dr. Smith provided expert testimony on Feb. 2nd." > legal_text.txt

# Low threshold (0.2) - High recall, more boundaries detected
charboundary analyze --model charboundary/resources/small_model.skops.xz --input legal_text.txt --format sentences --threshold 0.2

# Default threshold (0.5) - Balanced approach
charboundary analyze --model charboundary/resources/small_model.skops.xz --input legal_text.txt --format sentences --threshold 0.5

# High threshold (0.8) - High precision, only confident boundaries
charboundary analyze --model charboundary/resources/small_model.skops.xz --input legal_text.txt --format sentences --threshold 0.8
```

--------------------------------

### Train Custom Model with CharBoundary CLI (Bash)

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Shows how to train a custom text segmentation model using the CharBoundary command-line interface. The example demonstrates the basic usage of the 'train' command, specifying the training data file and the output model file.

```bash
# Train with default parameters
charboundary train --data training_data.txt --output model.skops
```

--------------------------------

### Configuring TextSegmenter with ONNX Optimization

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Shows how to customize a TextSegmenter using SegmenterConfig, including enabling ONNX inference and specifying the desired optimization level. This example details various parameters for model, window, domain knowledge, and performance settings.

```python
from charboundary.segmenters import TextSegmenter, SegmenterConfig

config = SegmenterConfig(
    # Model configuration
    model_type="random_forest",  # Type of model to use
    model_params={
        "n_estimators": 100,
        "max_depth": 16,
        "class_weight": "balanced"
    },
    threshold=0.5,               # Classification threshold (0.0-1.0)
                                 # Lower=more sentences (recall), Higher=fewer sentences (precision)
    
    # Window configuration
    left_window=3,               # Size of left context window
    right_window=3,              # Size of right context window
    
    # Domain knowledge
    abbreviations=["Dr.", "Mr.", "Mrs.", "Ms."],  # Custom abbreviations
    
    # Performance settings
    use_numpy=True,              # NumPy for faster processing
    cache_size=1024,             # Cache size for character encoding
    num_workers=4,               # Number of worker processes
    
    # ONNX acceleration (2-5x faster inference)
    use_onnx=True,               # Enable ONNX inference if available
    onnx_optimization_level=2    # Optimization level (recommended=2):
                                 #   0=None/debug, 1=Basic, 2=Extended, 3=Maximum
)

segmenter = TextSegmenter(config=config)
```

--------------------------------

### ONNX Model Conversion and Benchmarking Utility

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Provides command-line examples for using the `onnx_utils.py` script to manage ONNX models. This includes converting built-in or custom models to ONNX, benchmarking model performance, and testing ONNX model functionality. It offers flexibility in controlling optimization levels and processing models.

```bash
# Convert all built-in models to ONNX with optimal optimization levels
python scripts/onnx_utils.py convert --all-models

# Convert a specific built-in model to ONNX with a custom optimization level
python scripts/onnx_utils.py convert --model-name small --optimization-level 2

# Convert a custom model file to ONNX
python scripts/onnx_utils.py convert --input-file path/to/model.skops.xz --output-file path/to/model.onnx

# Benchmark all built-in models with optimal optimization levels
python scripts/onnx_utils.py benchmark --all-models

# Benchmark a specific built-in model with all optimization levels
python scripts/onnx_utils.py benchmark --model-name medium

# Test ONNX model functionality and accuracy
python scripts/onnx_utils.py test --all-models
```

--------------------------------

### Command-Line Text Analysis with Charboundary

Source: https://context7.com/alea-institute/charboundary/llms.txt

Provides examples of using the charboundary command-line interface to analyze text files. Covers basic analysis with default output, specifying output formats (sentences, paragraphs), saving segmented text and metrics, and adjusting segmentation sensitivity using the threshold parameter.

```bash
# Basic analysis with default annotated output
charboundary analyze --model charboundary/resources/small_model.skops.xz --input input.txt

# Output sentences (one per line)
charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input input.txt --format sentences

# Output paragraphs
charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input input.txt --format paragraphs

# Save output and generate metrics JSON
charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input input.txt --output segmented.txt --metrics metrics.json

# Adjust segmentation sensitivity with threshold
charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input input.txt --threshold 0.3  # Higher recall (more boundaries)

charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input input.txt --threshold 0.8  # Higher precision (fewer boundaries)

# Example with test file
echo "Dr. Smith testified. The case was Brown v. Board, 347 U.S. 483." > legal.txt

charboundary analyze --model charboundary/resources/small_model.skops.xz \
  --input legal.txt --format sentences --threshold 0.5

# Output:
# Dr. Smith testified.

```

--------------------------------

### ONNX Model Conversion and Usage in Python

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to integrate ONNX models for faster text segmentation. This includes getting pre-configured ONNX segmenters, creating models with ONNX support, converting existing models to ONNX, and loading/running inference with ONNX models. It requires the 'onnx' optional dependency.

```python
from charboundary import get_default_segmenter, get_medium_onnx_segmenter
from charboundary.models import create_model
from charboundary.segmenters import SegmenterConfig

# Option 1: Get a segmenter with ONNX already enabled (downloads model if needed)
segmenter = get_medium_onnx_segmenter()
sentences = segmenter.segment_to_sentences(text)

# Option 2: Create a model with ONNX support enabled and optimization level
model = create_model(
    model_type="random_forest", 
    use_onnx=True,
    onnx_optimization_level=2  # Use extended optimizations (level 0-3)
)

# Option 3: Create a segmenter with ONNX configuration
segmenter = TextSegmenter(
    config=SegmenterConfig(
        use_onnx=True,
        onnx_optimization_level=2  # Extended optimizations
    )
)

# Option 4: Convert an existing model to ONNX
segmenter = get_default_segmenter()
segmenter.model.to_onnx()  # Converts the model to ONNX format

# Save the ONNX model to a file (now with XZ compression by default)
segmenter.model.save_onnx("model.onnx")  # Creates model.onnx.xz by default
segmenter.model.save_onnx("model.onnx", compress=False)  # No compression

# Load an ONNX model with specified optimization level (handles compressed files automatically)
new_segmenter = get_default_segmenter()
new_segmenter.model.load_onnx("model.onnx")  # Works with both model.onnx or model.onnx.xz
new_segmenter.model.enable_onnx(True, optimization_level=2)  # Enable ONNX with extended optimizations

# Run inference with the ONNX model
sentences = new_segmenter.segment_to_sentences(text)
```

--------------------------------

### Get Sentences with Character Spans

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to retrieve sentences along with their exact start and end character positions within the original text. It also shows how to extract only the spans and verifies that the spans cover the entire text.

```python
from charboundary import get_default_segmenter

segmenter = get_default_segmenter()
text = "This is a sample text. It has multiple sentences. Here's the third one."

# Get sentences with their character spans
sentences_with_spans = segmenter.segment_to_sentences_with_spans(text)
for sentence, (start, end) in sentences_with_spans:
    print(f"Span ({start}-{end}): {sentence}")
    # Verify the span points to the correct text
    assert text[start:end].strip() == sentence.strip()

# Get only the spans without the text
spans = segmenter.get_sentence_spans(text)
print(f"Spans: {spans}")

# The spans cover EVERY character in the input
assert sum(end - start for start, end in spans) == len(text)

# Do the same for paragraphs
paragraph_spans = segmenter.get_paragraph_spans(text)
```

--------------------------------

### Downloading ONNX Models Automatically

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Shows how to retrieve ONNX models, which are automatically downloaded from GitHub if not found locally. This includes functions for getting segmenters with different ONNX model sizes and explicitly downloading a specific ONNX model. This functionality is useful for managing model availability without manual intervention.

```python
# These functions download models from GitHub if not found locally
from charboundary import (
    get_small_onnx_segmenter,    # Small model with ONNX (~5MB, included in package)
    get_medium_onnx_segmenter,   # Medium model with ONNX (~33MB, downloaded on demand)
    get_large_onnx_segmenter     # Large model with ONNX (~188MB, downloaded on demand)
)

# Explicitly download an ONNX model
from charboundary import download_onnx_model
download_onnx_model("large", force=True)  # Force redownload even if exists
```

--------------------------------

### Segment Text into Paragraphs with Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Demonstrates how to segment multi-paragraph text into individual paragraphs using the default segmenter. It shows how to get a list of paragraphs, paragraphs with their character spans, and just the character spans of paragraphs.

```python
from charboundary import get_default_segmenter

segmenter = get_default_segmenter()

text = """The court in Brown v. Board of Education, 347 U.S. 483 (1954), declared that
racial segregation in public schools was unconstitutional. This landmark decision was
delivered by Chief Justice Earl Warren.

After the decision, implementation was delegated to district courts with orders to
desegregate \"with all deliberate speed.\" The case was argued by NAACP attorney
Thurgood Marshall."""

# Get list of paragraphs
paragraphs = segmenter.segment_to_paragraphs(text)
print(f"Found {len(paragraphs)} paragraphs:")
for i, paragraph in enumerate(paragraphs, 1):
    print(f"\n{i}. {paragraph}")

# Get paragraphs with character spans
paragraphs_with_spans = segmenter.segment_to_paragraphs_with_spans(text)
for paragraph, (start, end) in paragraphs_with_spans:
    print(f"Paragraph at ({start}-{end}), length={end-start} chars")

# Get only paragraph spans
paragraph_spans = segmenter.get_paragraph_spans(text)
for i, (start, end) in enumerate(paragraph_spans, 1):
    excerpt = text[start:min(start+50, end)] + "..."
    print(f"{i}. Characters {start}-{end}: {excerpt}")

# Output:
# Found 2 paragraphs:
# 
# 1. The court in Brown v. Board of Education, 347 U.S. 483 (1954)...
# 
# 2. After the decision, implementation was delegated to district courts...

```

--------------------------------

### Get Character Spans for Sentences in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Explains how to obtain precise character start and end indices for each detected sentence within a text using CharBoundary. The `segment_to_sentences_with_spans` method returns both the sentence text and its corresponding span, while `get_sentence_spans` returns only the list of spans. This functionality is crucial for precise text manipulation and analysis based on sentence boundaries.

```python
from charboundary import get_default_segmenter

segmenter = get_default_segmenter()

text = "This is a sample text. It has multiple sentences. Here's the third one."

# Get sentences with their character spans
sentences_with_spans = segmenter.segment_to_sentences_with_spans(text)
for sentence, (start, end) in sentences_with_spans:
    print(f"Span ({start}-{end}): {sentence}")
    # Verify the span points to the correct text
    assert text[start:end].strip() == sentence.strip()

# Get only the spans without the text
spans = segmenter.get_sentence_spans(text)
print(f"\nSpans: {spans}")

# The spans cover EVERY character in the input
total_coverage = sum(end - start for start, end in spans)
assert total_coverage == len(text)

# Extract text using spans
for i, (start, end) in enumerate(spans, 1):
    print(f"Sentence {i}: {text[start:end]}")
```

--------------------------------

### Run Charboundary Benchmark Tests

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

These commands execute benchmark tests for the small, medium, and large models of the charboundary library, allowing users to verify performance independently.

```bash
python scripts/test/test_small_model.py
python scripts/test/test_medium_model.py
python scripts/test/test_large_model.py
```

--------------------------------

### Configure and Manage Text Segmenter in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Demonstrates how to create a custom configuration for the TextSegmenter, including model settings, context windows, domain knowledge like abbreviations, and performance optimizations. It also shows how to dynamically manage abbreviations (add, remove, set) and use the configured segmenter for text processing.

```python
config = SegmenterConfig(
    # Model settings
    model_type="random_forest",
    model_params={
        "n_estimators": 100,
        "max_depth": 16,
        "class_weight": "balanced"
    },
    threshold=0.5,                    # Classification threshold (0.0-1.0)

    # Context window sizes
    left_window=3,                    # Characters before potential boundary
    right_window=3,                   # Characters after potential boundary

    # Domain knowledge
    abbreviations=[
        "Dr.", "Mr.", "Mrs.", "Ms.",
        "Prof.", "Ph.D.", "Inc.", "Corp."
    ],

    # Performance settings
    use_numpy=True,                   # Use NumPy for faster processing
    cache_size=1024,                  # Character encoding cache size
    num_workers=4,                    # Parallel processing workers

    # ONNX acceleration (requires pip install charboundary[onnx])
    use_onnx=True,                    # Enable ONNX inference
    onnx_optimization_level=2         # Optimization: 0=None, 1=Basic, 2=Extended, 3=Max
)

# Create segmenter with custom config
segmenter = TextSegmenter(config=config)

# Manage abbreviations dynamically
current_abbrevs = segmenter.get_abbreviations()
print(f"Current abbreviations: {len(current_abbrevs)}")

# Add new abbreviations
segmenter.add_abbreviation("Esq.")
segmenter.add_abbreviation("Jr.")

# Remove an abbreviation
segmenter.remove_abbreviation("Ms.")

# Replace entire abbreviation list
legal_abbreviations = ["v.", "vs.", "et al.", "et seq.", "U.S.C.", "F.3d", "S.Ct."]
segmenter.set_abbreviations(legal_abbreviations)

# Use the configured segmenter
text = "Smith v. Jones, 123 F.3d 456 (2020). The court held et al. that..."
sentences = segmenter.segment_to_sentences(text)
```

--------------------------------

### Find Best Model Configuration with CharBoundary CLI

Source: https://context7.com/alea-institute/charboundary/llms.txt

This command searches the parameter space to find the optimal model configuration for text segmentation. It allows customization of parameter ranges for search (e.g., window sizes, estimators, depth, threshold) and can utilize validation data for more robust model selection. The output indicates the best parameters found and the corresponding performance metric, saving the optimized model.

```bash
# Find best model with default parameter ranges
charboundary best-model --data training_data.txt --output best_model.skops

# Customize parameter search space
charboundary best-model --data training_data.txt --output best_model.skops \
  --left-window-values 3 5 7 \
  --right-window-values 3 5 7 \
  --n-estimators-values 50 100 200 \
  --max-depth-values 8 16 24 \
  --threshold-values 0.3 0.5 0.7 \
  --sample-rate 0.1 --max-samples 10000

# Use validation data for model selection
charboundary best-model --data training_data.txt --output best_model.skops \
  --validation validation_data.txt --metrics-file best_metrics.json

# Example: Run parameter search
cat > train_small.txt << 'EOF'
Sentence one.<|sentence|> Sentence two.<|sentence|><|paragraph|>
More text here.<|sentence|> And another.<|sentence|><|paragraph|>
EOF

charboundary best-model --data train_small.txt --output optimized.skops \
  --n-estimators-values 32 64 128 \
  --max-depth-values 8 16 \
  --metrics-file results.json

# Output:
# Testing 6 parameter combinations...
# Best parameters: n_estimators=128, max_depth=16
# Best F1-score: 0.9234
# Model saved to: optimized.skops
```

--------------------------------

### Download ONNX Model (Python)

Source: https://github.com/alea-institute/charboundary/blob/main/charboundary/resources/onnx/README.md

Downloads a specific ONNX model (small, medium, or large) for the charboundary library. The `force` parameter can be used to re-download the model even if it already exists.

```python
from charboundary import download_onnx_model

# Download a specific model (small, medium, or large)
download_onnx_model("large", force=True)  # Force re-download
```

--------------------------------

### Using ONNX Optimization Levels with CharBoundary Models

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to enable ONNX inference and set optimization levels when creating or configuring CharBoundary models and segmenters. This includes methods for model creation, segmenter initialization, and applying settings to pre-built segmenters.

```python
from charboundary.models import create_model

model = create_model(
    model_type="random_forest", 
    use_onnx=True,
    onnx_optimization_level=2  # Extended optimizations recommended
)

from charboundary import TextSegmenter
from charboundary.segmenters import SegmenterConfig

segmenter = TextSegmenter(
    config=SegmenterConfig(
        use_onnx=True,
        onnx_optimization_level=2  # Extended optimizations recommended
    )
)

from charboundary import get_medium_onnx_segmenter

segmenter = get_medium_onnx_segmenter()
segmenter.model.enable_onnx(True, optimization_level=2)

segmenter.model.enable_onnx(True, optimization_level=2)
```

--------------------------------

### Load a Model using Skops

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Shows how to load a model previously saved with `skops`. It includes options for loading with default security checks (rejecting custom types) and with `trust_model=True` for loading models from trusted sources, which enables trusted types.

```python
# Load a model with security checks (default)
# This will reject loading custom types for security
segmenter = TextSegmenter.load("model.skops", use_skops=True)

# Load a model with trusted types enabled 
# Only use this with models from trusted sources
segmenter = TextSegmenter.load("model.skops", use_skops=True, trust_model=True)
```

--------------------------------

### Train Custom Models with CharBoundary CLI

Source: https://context7.com/alea-institute/charboundary/llms.txt

This command trains custom sentence and paragraph segmentation models using annotated data files. It supports basic training with default parameters, custom parameter tuning (e.g., window sizes, estimators, depth, sample rate), and automatic feature selection for optimized models. The output includes a trained model file and optionally a metrics file detailing performance.

```bash
# Basic training with default parameters
charboundary train --data training_data.txt --output model.skops

# Train with custom parameters
charboundary train --data training_data.txt --output custom_model.skops \
  --left-window 4 --right-window 6 \
  --n-estimators 100 --max-depth 16 \
  --sample-rate 0.1 --max-samples 10000 \
  --threshold 0.5 --metrics-file train_metrics.json

# Train with automatic feature selection
charboundary train --data training_data.txt --output optimized_model.skops \
  --use-feature-selection \
  --feature-selection-threshold 0.01 \
  --max-features 50

# Example: Create training data file
cat > train.txt << 'EOF'
This is a sentence.<|sentence|> This is another.<|sentence|><|paragraph|>
New paragraph here.<|sentence|> Multiple sentences work.<|sentence|><|paragraph|>
Dr. Jones arrived.<|sentence|> He met colleagues.<|sentence|><|paragraph|>
EOF

# Train model
charboundary train --data train.txt --output my_model.skops \
  --n-estimators 64 --max-depth 12 --sample-rate 0.05

# View training output
cat train_metrics.json

# Output JSON:
# {
#   "accuracy": 0.9823,
#   "precision": 0.9145,
#   "recall": 0.8876,
#   "f1_score": 0.9008,
#   "boundary_accuracy": 0.9145
# }
```

--------------------------------

### Find Best Charboundary Model Parameters

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

This command finds the best model parameters by training multiple models. It can use default parameter ranges or allow customization of search spaces for various parameters like window sizes, estimators, depth, and thresholds. Validation data can also be provided.

```bash
charboundary best-model --data training_data.txt --output best_model.skops
charboundary best-model --data training_data.txt --output best_model.skops \
  --left-window-values 3 5 7 --right-window-values 3 5 7 \
  --n-estimators-values 50 100 200 --max-depth-values 8 16 24 \
  --threshold-values 0.3 0.5 0.7 --sample-rate 0.1 --max-samples 10000
charboundary best-model --data training_data.txt --output best_model.skops \
  --validation validation_data.txt --metrics-file best_metrics.json
```

--------------------------------

### Train Charboundary Model with Custom Parameters

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

This command trains a charboundary model using specified data and custom parameters for the training process. It allows fine-tuning of window sizes, estimators, depth, sampling rates, and metrics.

```bash
charboundary train --data training_data.txt --output model.skops \
  --left-window 4 --right-window 6 --n-estimators 100 --max-depth 16 \
  --sample-rate 0.1 --max-samples 10000 --threshold 0.5 --metrics-file train_metrics.json
```

--------------------------------

### Convert Models to ONNX (Bash)

Source: https://github.com/alea-institute/charboundary/blob/main/charboundary/resources/onnx/README.md

Scripts to convert scikit-learn models to ONNX format. The first command converts all built-in models, while the second converts a specific model specified by input and output file paths.

```bash
# Convert all built-in models
python scripts/convert_models_to_onnx.py
```

```bash
# Convert a specific model
python scripts/convert_model_to_onnx.py --input model.skops.xz --output model.onnx
```

--------------------------------

### Save a Model using Skops

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to save a trained `TextSegmenter` model using the `skops` library. This method is preferred over `pickle` for its enhanced security features when sharing or loading models.

```python
# Train a model
segmenter = TextSegmenter()
segmenter.train(data=training_data)

# Save the model with skops
segmenter.save("model.skops", format="skops")
```

--------------------------------

### ONNX Acceleration for Faster Text Segmentation in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Illustrates how to use ONNX models for significantly faster text segmentation with no accuracy loss. It shows obtaining medium and large ONNX segmenters and processing text to extract sentences.

```python
from charboundary import get_medium_onnx_segmenter, get_large_onnx_segmenter

# Get ONNX-accelerated medium model (automatically downloads if needed)
# Provides ~1.29x speedup over scikit-learn
segmenter = get_medium_onnx_segmenter()

# Get ONNX-accelerated large model (automatically downloads if needed)
# Provides ~2.09x speedup over scikit-learn
large_segmenter = get_large_onnx_segmenter()

text = """The plaintiff, Mr. Brown vs. Johnson Corp., argued that patent no. 12345 was
infringed. Dr. Smith provided expert testimony on Feb. 2nd."""

# Process text at maximum speed with identical results
sentences = segmenter.segment_to_sentences(text)
print(f"Processed {len(text)} characters")
print(f"Found {len(sentences)} sentences:")
for i, sentence in enumerate(sentences, 1):
    print(f"  {i}. {sentence}")

# Output:
# Processed 124 characters
# Found 2 sentences:
#   1. The plaintiff, Mr. Brown vs. Johnson Corp., argued that patent no. 12345 was infringed.
#   2. Dr. Smith provided expert testimony on Feb. 2nd.

```

--------------------------------

### Train a Custom Text Segmenter

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Illustrates how to train a custom `TextSegmenter` model using provided training data. The training data uses special tokens (`<|sentence|>` and `<|paragraph|>`) to mark sentence and paragraph boundaries. After training, it shows how to segment new text into sentences and paragraphs.

```python
from charboundary import TextSegmenter

# Create a segmenter (will be initialized with default parameters)
segmenter = TextSegmenter()

# Train the model on sample data
training_data = [
    "This is a sentence.<|sentence|> This is another sentence.<|sentence|><|paragraph|>",
    "This is a new paragraph.<|sentence|> It has multiple sentences.<|sentence|><|paragraph|>"
]
segmenter.train(data=training_data)

# Segment text into sentences and paragraphs
text = "Hello, world! This is a test. This is another sentence."
segmented_text = segmenter.segment_text(text)
print(segmented_text)

# Get list of sentences
sentences = segmenter.segment_to_sentences(text)
print(sentences)
```

--------------------------------

### ONNX Model Conversion, Optimization, and Loading in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

This section details how to convert trained charboundary models to the ONNX format, apply various optimization levels, save and load these ONNX models, and enable ONNX inference directly during model creation or segmenter configuration. It also includes a benchmark for comparing performance across different optimization levels.

```python
from charboundary import get_default_segmenter
from charboundary.models import create_model
from charboundary.segmenters import SegmenterConfig, TextSegmenter

# Option 1: Convert existing model to ONNX
segmenter = get_default_segmenter()
segmenter.model.to_onnx()  # Convert to ONNX format

# Save with XZ compression (default, 60% smaller)
segmenter.model.save_onnx("model.onnx")  # Creates model.onnx.xz

# Save without compression
segmenter.model.save_onnx("model.onnx", compress=False)

# Load ONNX model (handles both compressed and uncompressed)
new_segmenter = get_default_segmenter()
new_segmenter.model.load_onnx("model.onnx")  # Works with .onnx or .onnx.xz

# Enable ONNX with extended optimizations (recommended)
new_segmenter.model.enable_onnx(True, optimization_level=2)

# Option 2: Create model with ONNX from the start
model = create_model(
    model_type="random_forest",
    use_onnx=True,
    onnx_optimization_level=2  # 0=None, 1=Basic, 2=Extended, 3=Maximum
)

# Option 3: Create segmenter with ONNX configuration
segmenter = TextSegmenter(
    config=SegmenterConfig(
        use_onnx=True,
        onnx_optimization_level=2
    )
)

# Benchmark different optimization levels
import time

text = "Sample text. " * 1000  # Create large test text

for level in [0, 1, 2, 3]:
    segmenter.model.enable_onnx(True, optimization_level=level)
    start = time.time()
    sentences = segmenter.segment_to_sentences(text)
    elapsed = time.time() - start
    chars_per_sec = len(text) / elapsed
    print(f"Level {level}: {chars_per_sec:.0f} chars/sec")

# Output:
# Level 0: 180,000 chars/sec
# Level 1: 301,500 chars/sec
# Level 2: 379,700 chars/sec
# Level 3: 376,900 chars/sec
```

--------------------------------

### Segment Text into Sentences with Configurable Threshold in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Shows how to segment a given text into individual sentences using the CharBoundary library. It illustrates the use of the `segment_to_sentences` method with different `threshold` values (0.2 for high recall, 0.5 default, 0.8 for high precision) to control the sensitivity of boundary detection. This is useful for adapting the segmentation to specific text characteristics and desired output granularity.

```python
from charboundary import get_default_segmenter

segmenter = get_default_segmenter()

text = """Dr. Smith visited Washington D.C. last week. He met with Prof. Johnson at 2:30 p.m.
The court held in Brown v. Board of Education, 347 U.S. 483 (1954) that racial segregation
was unconstitutional."""

# Default threshold (0.5) - balanced precision/recall
sentences = segmenter.segment_to_sentences(text)
print(f"Found {len(sentences)} sentences:")
for sentence in sentences:
    print(f"  - {sentence}")

# Lower threshold (0.2) - higher recall, more boundaries detected
high_recall = segmenter.segment_to_sentences(text, threshold=0.2)
print(f"\nHigh recall: {len(high_recall)} sentences")

# Higher threshold (0.8) - higher precision, only confident boundaries
high_precision = segmenter.segment_to_sentences(text, threshold=0.8)
print(f"High precision: {len(high_precision)} sentences")
```

--------------------------------

### Perform Text Segmentation with Default Segmenter

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates basic text segmentation into sentences and paragraphs using the default pre-trained medium-sized segmenter. The segmenter is retrieved using `get_default_segmenter()`, and then applied to a provided multi-line text string.

```python
from charboundary import get_default_segmenter

# Get the pre-trained medium-sized segmenter (default)
segmenter = get_default_segmenter()

# Segment text into sentences and paragraphs
text = """
Employee also specifically and forever releases the Acme Inc. (Company) and the Company Parties (except where and 
to the extent that such a release is expressly prohibited or made void by law) from any claims based on unlawful 
employment discrimination or harassment, including, but not limited to, the Federal Age Discrimination in 
Employment Act (29 U.S.C. § 621 et. seq.). This release does not include Employee’s right to indemnification, 
and related insurance coverage, under Sec. 7.1.4 or Ex. 1-1 of the Employment Agreement, his right to equity awards,
or continued exercise, pursuant to the terms of any specific equity award (or similar) agreement between 
Employee and the Company nor to Employee’s right to benefits under any Company plan or program in which
Employee participated and is due a benefit in accordance with the terms of the plan or program as of the Effective
Date and ending at 11:59 p.m. Eastern Time on Sep. 15, 2013.
"
sentences = segmenter.segment_to_sentences(text)
paragraphs = segmenter.segment_to_paragraphs(text)

```

--------------------------------

### Load Pre-trained Segmentation Models in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Demonstrates how to load different pre-trained segmentation models (default, small, large) provided by the CharBoundary library. These models are used for detecting sentence boundaries in text, handling complex cases like legal documents with abbreviations and citations. The loading functions include `get_default_segmenter`, `get_small_segmenter`, and `get_large_segmenter`.

```python
from charboundary import get_default_segmenter, get_small_segmenter, get_large_segmenter

# Load the medium model (default, best balance of speed and accuracy)
# Processes ~280,000 characters/second
segmenter = get_default_segmenter()

# Load the small model (faster, smaller memory footprint)
# Processes ~85,000 characters/second
small_segmenter = get_small_segmenter()

# Load the large model (highest accuracy)
# Processes ~175,000 characters/second
large_segmenter = get_large_segmenter()

# Segment complex legal text with abbreviations and citations
legal_text = """Employee releases the Acme Inc. (Company) from claims under the Federal Age
Discrimination in Employment Act (29 U.S.C. § 621 et. seq.). This release does not include
Employee's right to indemnification under Sec. 7.1.4 or Ex. 1-1 of the Employment Agreement."""

sentences = segmenter.segment_to_sentences(legal_text)
for i, sentence in enumerate(sentences, 1):
    print(f"{i}. {sentence}")
```

--------------------------------

### Load Different Model Sizes

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Shows how to load segmenters with different model sizes. `get_small_segmenter` provides faster processing with a smaller memory footprint, suitable for resource-constrained environments. `get_large_segmenter` offers the highest accuracy at the cost of a larger memory footprint.

```python
from charboundary import get_small_segmenter, get_large_segmenter

# For faster processing with smaller memory footprint
small_segmenter = get_small_segmenter()

# For highest accuracy (but larger memory footprint)
large_segmenter = get_large_segmenter()
```

--------------------------------

### Manage Abbreviations with CharBoundary (Python)

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to manage abbreviations within the CharBoundary library using Python. This includes retrieving all current abbreviations, adding new ones, removing existing abbreviations, and setting a completely new list of abbreviations for the segmenter.

```python
# Get current abbreviations
abbrevs = segmenter.get_abbreviations()

# Add new abbreviations
segmenter.add_abbreviation("Ph.D")

# Remove abbreviations
segmenter.remove_abbreviation("Dr.")

# Set a new list of abbreviations
segmenter.set_abbreviations(["Dr.", "Mr.", "Prof.", "Ph.D."])
```

--------------------------------

### Profile Charboundary Performance

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

This script profiles the performance of the charboundary library for various operations including training, inference, and model loading. It allows specifying the mode of operation, number of samples, iterations, and output file.

```bash
python scripts/benchmark/profile_model.py --mode all
python scripts/benchmark/profile_model.py --mode train --samples 500
python scripts/benchmark/profile_model.py --mode inference --iterations 200
python scripts/benchmark/profile_model.py --mode load --model charboundary/resources/medium_model.skops.xz
python scripts/benchmark/profile_model.py --output profile_results.txt
```

--------------------------------

### Train Custom Text Segmentation Models in Python

Source: https://context7.com/alea-institute/charboundary/llms.txt

Details the process of training custom text segmentation models using domain-specific annotated data. It covers preparing training data, training with custom parameters, displaying metrics, and saving/loading the trained model.

```python
from charboundary import TextSegmenter

# Create a segmenter with default configuration
segmenter = TextSegmenter()

# Prepare training data with sentence and paragraph markers
training_data = [
    "This is a sentence.<|sentence|> This is another sentence.<|sentence|><|paragraph|>",
    "New paragraph here.<|sentence|> It has multiple sentences.<|sentence|><|paragraph|>",
    "Dr. Smith arrived.<|sentence|> He met with Prof. Johnson.<|sentence|><|paragraph|>"
]

# Train with custom parameters
metrics = segmenter.train(
    data=training_data,
    model_params={
        "n_estimators": 128,      # Number of trees
        "max_depth": 20,          # Maximum tree depth
        "class_weight": "balanced"
    },
    sample_rate=0.1,              # Sample 10% of non-boundary positions
    max_samples=50000,            # Maximum training samples
    left_window=5,                # Left context window size
    right_window=5                # Right context window size
)

# Display training metrics
print(f"Training Results:")
print(f"  Overall accuracy:    {metrics['accuracy']:.4f}")
print(f"  Boundary precision:  {metrics['precision']:.4f}")
print(f"  Boundary recall:     {metrics['recall']:.4f}")
print(f"  Boundary F1-score:   {metrics['f1_score']:.4f}")

# Save the trained model
segmenter.save("my_custom_model.skops", format="skops")

# Load and use the model later
loaded_segmenter = TextSegmenter.load("my_custom_model.skops", trust_model=True)
sentences = loaded_segmenter.segment_to_sentences("Test text. Another sentence.")

# Output:
# Training Results:
#   Overall accuracy:    0.9856
#   Boundary precision:  0.9234
#   Boundary recall:     0.8967
#   Boundary F1-score:   0.9098

```

--------------------------------

### Train Charboundary Model with Feature Selection

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

This command trains a charboundary model with automatic feature selection enabled to enhance performance. It supports setting a threshold for feature importance and limiting the maximum number of features used.

```bash
charboundary train --data training_data.txt --output model.skops \
  --use-feature-selection --feature-selection-threshold 0.01 --max-features 50
```

--------------------------------

### Control Sentence Segmentation Sensitivity

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Demonstrates how to control the sensitivity of sentence segmentation using the 'threshold' parameter. A lower threshold leads to more aggressive segmentation (higher recall), while a higher threshold results in more conservative segmentation (higher precision). The lengths of the resulting sentence lists are printed.

```python
# Control segmentation sensitivity with threshold parameter

# Lower threshold = more aggressive segmentation (higher recall)
high_recall_sentences = segmenter.segment_to_sentences(text, threshold=0.1)

# Higher threshold = conservative segmentation (higher precision)
high_precision_sentences = segmenter.segment_to_sentences(text, threshold=0.9)

print(len(high_recall_sentences), len(high_precision_sentences))
```

--------------------------------

### Perform Text Segmentation with ONNX Acceleration

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Utilizes the ONNX-accelerated large segmenter for maximum text processing speed. This code snippet demonstrates how to obtain the optimized model (which downloads automatically if not present) and then segment input text into sentences.

```python
# Install ONNX support
pip install charboundary[onnx]

# Use ONNX-accelerated model with recommended optimization level
from charboundary import get_large_onnx_segmenter

# Get optimized model (downloads automatically if needed)
segmenter = get_large_onnx_segmenter()

# Process text at maximum speed
sentences = segmenter.segment_to_sentences(your_text)

```

--------------------------------

### Segment Text into Sentences (Default Threshold)

Source: https://github.com/alea-institute/charboundary/blob/main/README.md

Segments a given text into a list of sentences using the default sensitivity threshold. The output is then joined by newline characters for display. This function is useful for basic sentence extraction.

```python
from charboundary import get_default_segmenter

text = "This is a sample text. It has multiple sentences. Here's the third one."
segmenter = get_default_segmenter()
sentences = segmenter.segment_to_sentences(text)
print("\n-\n".join(sentences))
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.