### Clone and Setup FreqProb Repository (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Clones the FreqProb repository from GitHub and navigates into the project directory. This is the initial step for setting up the local development environment. Requires Git to be installed.

```bash
git clone https://github.com/tresoldi/freqprob.git
cd freqprob
```

--------------------------------

### FreqProb Quick Start Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Demonstrates basic usage of the freqprob library, including creating a frequency distribution, applying Maximum Likelihood Estimation (MLE), Laplace smoothing, and comparing models.

```python
import freqprob

# Create a frequency distribution
freqdist = {'the': 1000, 'cat': 50, 'dog': 30, 'bird': 10}

# Basic Maximum Likelihood Estimation
mle = freqprob.MLE(freqdist, logprob=False)
print(f"P(cat) = {mle('cat')}")  # Expected output: P(cat) = 0.045454545454545456

# Laplace Smoothing
laplace = freqprob.Laplace(freqdist, logprob=False)
print(f"P(unknown) = {laplace('unknown')}")  # Expected output: P(unknown) = 1.0000000000000002e-05 (or similar small value)

# Model Comparison
models = {'mle': mle, 'laplace': laplace}
test_data = ['the', 'cat', 'unknown']
comparison = freqprob.model_comparison(models, test_data)
print(comparison)
```

--------------------------------

### Python Docstring Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Illustrates the Google style docstring conventions used for Python functions within the project. It details the structure for brief descriptions, longer explanations, arguments, return values, raised exceptions, and usage examples.

```python
def example_function(param1: str, param2: int = 5) -> bool:
    """Brief description of the function.

    Longer description explaining the function's purpose,
    behavior, and any important details.

    Args:
        param1: Description of the first parameter.
        param2: Description of the second parameter. Defaults to 5.

    Returns:
        Description of the return value.

    Raises:
        ValueError: When param1 is empty.

    Example:
        >>> result = example_function("test", 10)
        >>> print(result)
        True
    """
    pass
```

--------------------------------

### Install FreqProb Library

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Installs the freqprob library using pip. This is the primary method for getting the library into your Python environment.

```bash
pip install freqprob
```

--------------------------------

### Development Setup and Testing

Source: https://github.com/tresoldi/freqprob/blob/main/README.md

Illustrates the bash commands required for setting up the FreqProb development environment. This includes cloning the repository, installing Hatch, and running the test suite.

```bash
git clone https://github.com/tresoldi/freqprob.git
cd freqprob
pip install hatch
hatch run test  # Run test suite
```

--------------------------------

### Documentation Commands (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Details Hatch commands for building and serving project documentation. This includes starting a live-reloading server for development, building static documentation files, and testing Jupyter notebooks.

```bash
# Start documentation server (auto-reload)
hatch run docs:serve

# Build documentation
hatch run docs:build

# Test Jupyter notebooks
hatch run docs:test-notebooks

# Launch Jupyter for editing notebooks
hatch run docs:notebooks
```

--------------------------------

### Essential Development Commands (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Lists key Hatch commands for common development tasks, including local installation, running tests, linting, formatting, and cleaning build artifacts. These commands streamline the development process.

```bash
# Install project locally in development mode
hatch run install-dev

# Run all tests
hatch run test

# Run tests with coverage
hatch run test-cov

# Run linting and formatting
hatch run lint:all

# Clean build artifacts
hatch run clean
```

--------------------------------

### Troubleshooting: Install Optional Dependencies

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Command to install optional dependencies required for comprehensive testing, including hypothesis, nltk, scipy, matplotlib, and seaborn.

```bash
# Install optional dependencies for comprehensive testing
pip install hypothesis nltk scipy matplotlib seaborn
```

--------------------------------

### Manage Hatch Environments and Commands (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Provides essential commands for interacting with Hatch environments, which organize development tasks. Users can view available environments or run specific commands within default or custom environments. Requires Hatch to be installed.

```bash
# Show available environments
hatch env show

# Run commands in default environment
hatch run <command>

# Run commands in specific environment
hatch run lint:<command>
hatch run docs:<command>
hatch run ci:<command>
```

--------------------------------

### Install FreqProb

Source: https://github.com/tresoldi/freqprob/blob/main/README.md

Installs the FreqProb library using pip. An optional 'all' extra installs all optional dependencies for extended features.

```bash
pip install freqprob

pip install freqprob[all]  # All optional dependencies
```

--------------------------------

### Create Compressed Distribution

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Illustrates the creation of a compressed distribution for large vocabularies, balancing memory usage and accuracy. It shows how to configure quantization levels and enable compression.

```python
# Use compressed representations
compressed_dist = create_compressed_distribution(
    large_freqdist,
    quantization_levels=2048,  # Balance memory vs accuracy
    use_compression=True
)
```

--------------------------------

### Setup and Import Libraries (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_1_basic_smoothing.ipynb

Imports essential Python libraries for data analysis and visualization, including `collections.Counter`, `matplotlib`, `numpy`, `seaborn`, and the `freqprob` library. It also configures plotting styles and displays the installed `freqprob` version.

```python
from collections import Counter

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

import freqprob

# Set style for better plots
plt.style.use("default")
sns.set_palette("husl")

print(f"FreqProb version: {freqprob.__version__ if hasattr(freqprob, '__version__') else 'dev'}")
```

--------------------------------

### GitHub Actions CI Workflow

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Configures a GitHub Actions workflow to run on push/pull requests, setting up Python, installing dependencies, running the validation suite, and uploading results.

```yaml
# .github/workflows/validation.yml
name: Validation

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          pip install -e .[dev]
          pip install hypothesis nltk scipy

      - name: Run validation suite
        run: |
          python scripts/validation_report.py --quick --output-dir validation_results

      - name: Upload validation results
        uses: actions/upload-artifact@v3
        with:
          name: validation-results
          path: validation_results/
```

--------------------------------

### StreamingFrequencyDistribution Example (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Demonstrates the use of StreamingFrequencyDistribution for processing data streams with bounded memory. It automatically manages vocabulary size and minimum count thresholds.

```python
from freqprob import StreamingFrequencyDistribution

# Assuming data_stream is an iterable of tokens
# stream_dist = StreamingFrequencyDistribution(
#     max_vocabulary_size=10000,
#     min_count_threshold=2,
#     decay_factor=0.99  # Exponential forgetting
# )

# Process streaming data
# for token in data_stream:
#     stream_dist.update(token)

# Automatically maintains vocabulary size limit
# print(f"Vocabulary size: {stream_dist.get_vocabulary_size()}")
```

--------------------------------

### Resolve test failures with Hatch

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Troubleshoot test failures by checking for missing dependencies or recreating the development environment. `hatch run pip list` shows installed packages, and `hatch env create --force` rebuilds the environment.

```bash
hatch run pip list  # Show installed packages
hatch env create --force  # Recreate environment
```

--------------------------------

### Pre-commit Hook Management (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Provides commands for managing pre-commit hooks, which automate code formatting, linting, and type checking before commits. This includes installing, running manually, running on staged files, and updating hook versions.

```bash
# Install pre-commit hooks
hatch run precommit-install

# Run pre-commit on all files manually
hatch run precommit

# Run pre-commit on staged files only
hatch run precommit-staged

# Update pre-commit hook versions
hatch run precommit-update
```

--------------------------------

### Compressed Distribution Creation (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Shows how to create memory-efficient representations of frequency distributions using compression techniques, including quantization for further memory savings.

```python
from freqprob import create_compressed_distribution

# Assuming large_freqdist is a large dictionary of frequencies
# large_freqdist = {f'word_{i}': max(1, 10000-i) for i in range(100000)}

# With quantization for additional compression
# compressed = create_compressed_distribution(
#     large_freqdist,
#     quantization_levels=1024,  # Trade-off: memory vs precision
#     use_compression=True
# )
```

--------------------------------

### Run Performance Benchmarks and Validation Reports

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Commands to execute performance benchmarks and generate comprehensive validation reports, including quick checks for CI/CD pipelines.

```bash
# Run performance benchmarks
python docs/benchmarks.py --quick --output perf_results

# Run validation report (includes performance)
python scripts/validation_report.py --output-dir validation_results
```

--------------------------------

### Install Hatch and Pre-commit Hooks (Bash)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Installs Hatch, a modern Python project manager, and sets up pre-commit hooks for automated code quality checks. This ensures code consistency and quality before commits. Requires pip to be installed.

```bash
pip install hatch
hatch run precommit-install
```

--------------------------------

### Fix type checking errors with Hatch

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Correct type checking errors by installing missing type stubs. The `hatch run lint:typing` command can automatically install necessary stubs.

```bash
hatch run lint:typing  # Will install missing stubs
```

--------------------------------

### Streaming Updates and Real-time Learning Initialization

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Demonstrates initializing and using streaming models for real-time applications. Includes generating streaming data with evolving vocabulary and simulating updates.

```python
import random

# Assume StreamingMLE and StreamingLaplace are imported

print("STREAMING UPDATES AND REAL-TIME LEARNING")
print("=" * 42)
print()

# Initialize streaming models
initial_data = {f"word_{i}": max(1, 100 - i) for i in range(50)}  # Small initial vocabulary

streaming_mle = StreamingMLE(initial_data, max_vocabulary_size=100, logprob=False)
streaming_laplace = StreamingLaplace(initial_data, max_vocabulary_size=100, logprob=False)

print(f"Initial vocabulary size: {len(initial_data)}")
print("Maximum vocabulary size: 100")
print()

# Simulate streaming data

def generate_streaming_data(n_updates=500):
    """Generate streaming data with evolving vocabulary."""
    stream = []

    # Mix of existing and new words
    existing_words = list(initial_data.keys())

    for i in range(n_updates):
        word = random.choice(existing_words) if random.random() < 0.7 else f"new_word_{i}"
        count = random.randint(1, 5)
        stream.append((word, count))

    return stream


streaming_data = generate_streaming_data(500)
print(f"Generated {len(streaming_data)} streaming updates")
```

--------------------------------

### Troubleshooting: Performance Test Failures

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Example of adjusting performance thresholds for different hardware by increasing max_duration_seconds and max_memory_mb when validating performance regressions.

```python
# Adjust performance thresholds for different hardware
validator.validate_performance_regression(
    method_class, test_dist,
    max_duration_seconds=20.0,  # Increased threshold
    max_memory_mb=2000.0        # Increased memory limit
)
```

--------------------------------

### Benchmark Execution with Hatch

Source: https://github.com/tresoldi/freqprob/blob/main/docs/performance_comparison.md

Demonstrates how to run the full benchmark suite for FreqProb using the Hatch build tool. Includes options for quick runs and plot generation, highlighting the recommended method for benchmark execution.

```bash
hatch run bench-all
hatch run bench-all --quick --plots
```

--------------------------------

### Troubleshooting: Hypothesis Test Failures

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Commands to increase example count for more thorough hypothesis testing and to enable verbose output for debugging hypothesis failures.

```bash
# Increase example count for more thorough testing
pytest tests/test_property_based.py --hypothesis-max-examples=200

# Debug hypothesis failures
pytest tests/test_property_based.py --hypothesis-verbosity=verbose
```

--------------------------------

### Performance Scaling Analysis Setup

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Prepares data for performance scaling analysis by extracting key metrics (MLE creation time, Lazy MLE creation time, vectorization speedup) for each dataset size. This data is intended for visualization.

```python
# Performance scaling analysis
print("\n\nPerformance Scaling Analysis:")
print("=" * 31)

dataset_sizes = [len(datasets["small"]),
                 len(datasets["medium"]),
                 len(datasets["large"]) ]
mle_creation_times = []
lazy_creation_times = []
vectorization_speedups = []

for dataset_name in ["Small (1K)", "Medium (10K)", "Large (50K)"]:
    results = benchmark_results[dataset_name]
    mle_creation_times.append(results["MLE"]["creation_time"])
    lazy_creation_times.append(results["Lazy MLE"]["creation_time"])
    vectorization_speedups.append(results["Vectorization"]["speedup"])

# Visualize scaling
fig, axes = plt.subplots(2, 2, figsize=(15, 10))
```

--------------------------------

### Demonstrating Search Results

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_4_real_world_applications.ipynb

This snippet illustrates how to perform searches using an initialized retrieval system. It defines sample queries, retrieves the top-k results for each, and prints the document ID, score, and a truncated version of the document text for display.

```python
# Define test queries and relevance judgments
test_queries = {
    "artificial intelligence machine learning": [
        "technology_000",
        "technology_001",
        "technology_004",
    ],
    "basketball championship games": ["sports_001", "sports_005"],
    "government policy healthcare": ["politics_000", "politics_004"],
    "smartphone technology apple": ["technology_000", "technology_007"],
    "olympic athletes competition": ["sports_000", "sports_003"],
}

print("Search Results Examples:")
print("=" * 25)
print()

# Demonstrate search results
sample_queries = ["artificial intelligence", "basketball championship", "government policy"]
best_system = next(iter(retrieval_systems.values()))  # Use first system for examples

for query in sample_queries:
    print(f"Query: '{query}'")
    results = best_system.search(query, top_k=3)

    for i, (doc_id, score, doc_text) in enumerate(results, 1):
        # Truncate document text for display
        display_text = doc_text[:80] + "..." if len(doc_text) > 80 else doc_text
        print(f"  {i}. {doc_id} (score: {score:.2f})")
        print(f"     {display_text}")
    print()
```

--------------------------------

### Memory profiling with memory_profiler

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Install the memory_profiler package and use it to profile the memory usage of a Python script. This is useful for identifying memory leaks or high memory consumption.

```bash
pip install memory_profiler
python -m memory_profiler your_script.py
```

--------------------------------

### Retrieval System Initialization and Indexing

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_4_real_world_applications.ipynb

This section demonstrates initializing different retrieval system models, including Laplace and MLE with varying interpolation lambdas. It then iterates through these systems to index the previously created document collection, preparing them for search operations.

```python
# Initialize and train retrieval system
retrieval_systems = {
    "Laplace (λ=0.8)": LanguageModelRetrieval("laplace", 0.8),
    "Laplace (λ=0.5)": LanguageModelRetrieval("laplace", 0.5),
    "MLE (λ=0.8)": LanguageModelRetrieval("mle", 0.8),
}

for name, system in retrieval_systems.items():
    print(f"Indexing documents for {name}...")
    system.index_documents(document_collection)
    print()
```

--------------------------------

### Run Property-Based Tests

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Commands to execute property-based tests using pytest and Hypothesis, allowing for customization of test thoroughness via example counts and stateful testing.

```bash
# Run with default settings
pytest tests/test_property_based.py -v

# Run with more examples for thorough testing
pytest tests/test_property_based.py -v --hypothesis-max-examples=100

# Run stateful testing
pytest tests/test_property_based.py::TestFreqProbStateMachine -v
```

--------------------------------

### Setup and Data Preparation for NLP Applications

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_4_real_world_applications.ipynb

This Python snippet sets up the environment for FreqProb NLP tutorials by importing necessary libraries like numpy, matplotlib, and freqprob. It also prepares sample datasets for news articles and movie reviews, categorizing them for later use in tasks like text classification and sentiment analysis.

```python
import math
import random
import re
from collections import Counter, defaultdict

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns

import freqprob

# Set up plotting and random seeds
plt.style.use("default")
sns.set_palette("husl")
random.seed(42)
np.random.seed(42)

print("FreqProb Real-World Applications Tutorial")
print("=" * 40)
print()

# Sample datasets for demonstration
# In practice, you would load these from files or APIs

# 1. News articles dataset
news_articles = {
    "technology": [
        "Apple announces new iPhone with advanced AI capabilities and improved camera system",
        "Google launches innovative machine learning platform for developers worldwide",
        "Microsoft introduces cloud computing solutions for enterprise customers",
        "Tesla unveils autonomous driving technology with neural network improvements",
        "Amazon develops new artificial intelligence algorithms for recommendation systems",
        "Facebook invests heavily in virtual reality and metaverse technologies",
        "Intel releases next generation processors with enhanced performance capabilities",
        "Samsung showcases foldable smartphone technology at tech conference",
    ],
    "sports": [
        "Olympic athletes compete in swimming championships with record breaking performances",
        "Basketball team wins championship after intense playoff games throughout season",
        "Soccer world cup features exciting matches between international teams",
        "Tennis tournament showcases incredible athletic talent and competitive spirit",
        "Baseball season concludes with thrilling world series games",
        "Hockey players demonstrate exceptional skills during championship playoffs",
        "Marathon runners compete in challenging race through city streets",
        "Golf tournament attracts professional players from around the world",
    ],
    "politics": [
        "Government announces new policies regarding healthcare and education reform",
        "Presidential election campaign focuses on economic issues and foreign policy",
        "Congress debates legislation concerning environmental protection and climate change",
        "International diplomacy efforts aim to resolve conflicts through peaceful negotiations",
        "Local elections determine representatives for state and municipal governments",
        "Political parties present platforms addressing social justice and equality",
        "Supreme court decisions impact constitutional law and civil rights",
        "Trade agreements between nations affect global economic relationships",
    ],
}

# 2. Movie reviews dataset (sentiment analysis)
movie_reviews = {
    "positive": [
        "This movie is absolutely fantastic with amazing performances and brilliant storytelling",
        "Incredible cinematography and outstanding acting make this film truly exceptional",
        "Wonderful story with excellent character development and beautiful visuals",
        "Superb direction and phenomenal performances create an unforgettable experience",
        "Magnificent film with incredible depth and emotional resonance throughout",
        "Brilliant screenplay and outstanding cast deliver an amazing cinematic experience",
        "Excellent movie with fantastic performances and compelling narrative structure",
        "Remarkable film featuring wonderful acting and beautiful cinematographic work",
    ],
    "negative": [
        "This movie is terrible with awful performances and boring storyline throughout",
        "Horrible cinematography and poor acting make this film completely unwatchable",
        "Terrible story with weak character development and disappointing visual effects",
        "Poor direction and bad performances create a frustrating viewing experience",
        "Awful film with no depth and completely uninteresting plot development",
        "Bad screenplay and terrible cast deliver a disappointing cinematic disaster",
        "Horrible movie with weak performances and confusing narrative structure",
        "Disappointing film featuring poor acting and uninspiring cinematographic choices",
    ],
}

print(f"Loaded {sum(len(articles) for articles in news_articles.values())} news articles")
print(f"Loaded {sum(len(reviews) for reviews in movie_reviews.values())} movie reviews")
print()

```

--------------------------------

### Debug Utilities: Profiler Analysis

Source: https://github.com/tresoldi/freqprob/blob/main/docs/validation_guide.md

Code snippet demonstrating how to use a PerformanceProfiler to analyze test results, get summary statistics, and export profiling data for external analysis.

```python
# Analyze profiler results
profiler = PerformanceProfiler()
# ... run tests ...

summary = profiler.get_summary_statistics()
print(json.dumps(summary, indent=2))

# Export for external analysis
profiler.export_results('debug_profile.json', format='json')
```

--------------------------------

### Bayesian Smoothing Example (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Illustrates Bayesian Smoothing using a Dirichlet prior distribution for principled probability estimates. The alpha parameter controls the degree of smoothing, from MLE to uniform prior.

```python
from freqprob import BayesianSmoothing

# Assuming freqdist is a frequency distribution object
# freqdist = {...}

bayesian = BayesianSmoothing(freqdist, alpha=1.0, logprob=False)
```

--------------------------------

### Interpolated Smoothing Example (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Demonstrates how to combine probability estimates from multiple models using weighted linear interpolation. This technique is useful for blending different n-gram orders or domain-specific models.

```python
from freqprob import InterpolatedSmoothing

# Combine trigram and bigram models
trigrams = {('the', 'big', 'cat'): 3, ('a', 'big', 'dog'): 2}
bigrams = {('big', 'cat'): 5, ('big', 'dog'): 3}

interpolated = InterpolatedSmoothing(
    trigrams, bigrams, lambda_weight=0.7, logprob=False
)
```

--------------------------------

### Build Background and Document Language Models

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Demonstrates building a background frequency distribution from all documents and then creating individual language models for each document using the background model. This is a foundational step for many NLP tasks within the library.

```python
# Build background model from all documents
all_text = " ".join(documents)
background_freqdist = word_frequency(all_text.split())
background_model = freqprob.MLE(background_freqdist, logprob=True)

# Build document models
doc_models = []
for doc in documents:
    model = build_document_language_model(doc, background_model)
    doc_models.append(model)

# Score query
query = "machine learning classification"
scores = score_query(query, doc_models)

# Rank documents
ranked_docs = sorted(enumerate(scores), key=lambda x: x[1], reverse=True)
for rank, (doc_idx, score) in enumerate(ranked_docs, 1):
    print(f"Rank {rank}: Document {doc_idx+1} (score: {score:.2f})")
```

--------------------------------

### Display Performance Benchmarks and Recommendations

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Prints a header for performance benchmarks and best practices, followed by a list of memory optimization recommendations. These recommendations cover using compressed or sparse representations, monitoring long-running processes, employing streaming models, profiling operations, and implementing garbage collection strategies.

```python
print("PERFORMANCE BENCHMARKS AND BEST PRACTICES")
print("=" * 44)
print()

print("\nMemory Optimization Recommendations:")
print("=" * 38)
print("1. Use compressed representations for large vocabularies (50%+ savings)")
print("2. Consider sparse representations for distributions with many zeros")
print("3. Monitor memory usage during long-running processes")
print("4. Use streaming models for real-time applications with bounded memory")
print("5. Profile operations to identify memory bottlenecks")
print("6. Implement garbage collection strategies for batch processing")

```

--------------------------------

### Python Test Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

A Python unit test function demonstrating how to test Laplace smoothing functionality. It includes assertions for known and unknown word probabilities, using a frequency distribution and Laplace smoothing parameters.

```python
def test_laplace_smoothing_basic():
    """Test basic Laplace smoothing functionality."""
    freqdist = {'cat': 3, 'dog': 2}
    laplace = freqprob.Laplace(freqdist, bins=100, logprob=False)

    # Test known word probability
    expected_cat = (3 + 1) / (5 + 100)
    assert abs(laplace('cat') - expected_cat) < 1e-10

    # Test unknown word probability  
    expected_unknown = 1 / (5 + 100)
    assert abs(laplace('bird') - expected_unknown) < 1e-10
```

--------------------------------

### Exercise: Text Corpus Analysis and Smoothing Comparison

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_1_basic_smoothing.ipynb

Provides exercises for users to practice creating their own text corpora, comparing different smoothing methods, finding optimal gamma for Lidstone smoothing, and analyzing the effect of vocabulary size estimates.

```python
# EXERCISE 1: Create your own text corpus and compare smoothing methods
# TODO: Replace this with your own text data
your_corpus = [
    "machine learning is fascinating",
    "deep learning models are powerful",
    "natural language processing uses machine learning",
    "artificial intelligence and machine learning overlap",
    "learning algorithms improve with data",
]

# Create frequency distribution
your_words = []
for sentence in your_corpus:
    your_words.extend(sentence.split())

your_freqdist = Counter(your_words)
print("Your frequency distribution:")
print(your_freqdist)

# TODO: Create different smoothing models and compare them
# Hint: Use the code patterns from above

# EXERCISE 2: Find the optimal gamma for Lidstone smoothing
# TODO: Create a validation set and test different gamma values
# Hint: Use perplexity to evaluate performance

# EXERCISE 3: Analyze the effect of different vocabulary size estimates
# TODO: Try bins=[100, 500, 1000, 5000] and see how it affects smoothing

print("\nComplete the exercises above to deepen your understanding!")
print("Experiment with different:")
print("- Text corpora (different domains, sizes)")
print("- Smoothing parameters (gamma values)")
print("- Vocabulary size estimates")
print("- Evaluation metrics")
```

--------------------------------

### Practical Recommendations and Insights

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_1_basic_smoothing.ipynb

Prints key insights derived from analyzing smoothing methods and provides practical recommendations for using them in real-world applications. It emphasizes avoiding pure MLE and suggests Laplace or Lidstone smoothing with parameter tuning.

```python
print("Key Insights:")
print("- MLE gives zero probability to unseen words (problematic)")
print("- Smoothing methods allocate probability mass to unseen events")
print("- Higher smoothing → more probability reserved for unseen words")
print("- Trade-off: fitting training data vs. handling unseen data")

print("PRACTICAL RECOMMENDATIONS:")
print("=" * 50)
print()

print("1. NEVER use pure MLE for real applications")
print("   → Zero probabilities break many algorithms")
print()

print("2. Laplace smoothing (add-one) is a good baseline")
print("   → Simple, robust, works well for small datasets")
print()

print("3. Use ELE (gamma=0.5) for theoretically motivated smoothing")
print("   → Good balance between smoothing and data fidelity")
print()

print("4. Tune Lidstone gamma parameter using validation data")
print("   → Cross-validation to find optimal smoothing strength")
print()

print("5. Consider vocabulary size when setting 'bins' parameter")
print("   → Underestimating leads to over-smoothing")
print("   → Overestimating leads to under-smoothing")
print()
```

--------------------------------

### Validation & Testing: Correctness and Benchmarking

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Details how to validate implementation correctness and benchmark performance using the library's testing utilities. This includes quick validation, comprehensive suite runs, and performance profiling of specific methods.

```python
# Validate implementation correctness
is_valid = freqprob.quick_validate_method(
    freqprob.Laplace, test_distribution, bins=1000
)

# Comprehensive validation suite
validator = freqprob.ValidationSuite()
results = validator.run_comprehensive_validation(
    method_classes=[freqprob.MLE, freqprob.Laplace, freqprob.ELE],
    test_distributions=[test_dist1, test_dist2]
)

# Performance benchmarking
performance_metrics = freqprob.profile_method_performance(
    freqprob.KneserNey, test_distribution
)
print(f"Creation time: {performance_metrics.duration_seconds:.4f}s")
```

--------------------------------

### Print Best Practices Summary (Python)

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Outputs a structured summary of best practices for optimizing FreqProb, covering performance, memory management, trade-offs, and an implementation checklist. This section provides guidance for production deployment.

```Python
print("\nBEST PRACTICES SUMMARY")
print("=" * 23)
print()

print("🚀 PERFORMANCE OPTIMIZATION:")
print("  • Use vectorized operations for batch processing (2-10x speedup)")
print("  • Choose lazy evaluation for sparse access patterns")
print("  • Implement caching for expensive computations (SGT, etc.)")
print("  • Profile operations to identify bottlenecks")
print()

print("💾 MEMORY MANAGEMENT:")
print("  • Use compressed representations for large vocabularies (50%+ savings)")
print("  • Consider sparse representations for distributions with many zeros")
print("  • Implement streaming for real-time applications")
print("  • Monitor memory usage and set appropriate limits")
print()

print("⚖️ TRADE-OFFS TO CONSIDER:")
print("  • Accuracy vs. Memory: Higher compression = lower accuracy")
print("  • Speed vs. Memory: Eager computation = faster queries, more memory")
print("  • Complexity vs. Performance: Advanced methods may not always be better")
print("  • Real-time vs. Batch: Streaming good for real-time, batch good for throughput")
print()

print("🎯 CHOOSING THE RIGHT APPROACH:")
print("  • Small datasets (<10K): Standard implementations sufficient")
print("  • Medium datasets (10K-100K): Consider vectorization and compression")
print("  • Large datasets (>100K): Use streaming, compression, and lazy evaluation")
print("  • Real-time applications: Streaming models with bounded memory")
print("  • Batch processing: Vectorized operations with memory monitoring")
print()

print("🔧 IMPLEMENTATION CHECKLIST:")
print("  ☐ Profile your specific use case")
print("  ☐ Choose appropriate data representations")
print("  ☐ Implement memory monitoring")
print("  ☐ Test with realistic data sizes")
print("  ☐ Validate accuracy vs. efficiency trade-offs")
print("  ☐ Plan for scaling and growth")
```

--------------------------------

### Running GitHub Actions Locally with Hatch

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Shell commands to simulate GitHub Actions workflows locally using Hatch. This includes running the full CI pipeline, quick checks, and individual components like linting, testing, and documentation builds.

```bash
# Run full CI pipeline (equivalent to GitHub Actions)
hatch run ci:ci-full

# Run quick CI checks
hatch run ci:ci-quick

# Run individual components
hatch run lint:all        # Linting and formatting
hatch run test-cov        # Tests with coverage
hatch run docs:build      # Documentation build
hatch run bench --quick   # Quick benchmarks
```

--------------------------------

### Demonstrate SGT Caching with freqprob

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Illustrates the caching mechanism of freqprob's SimpleGoodTuring (SGT) model. It compares the performance of creating models from a cold cache versus a hot cache, demonstrating speedup. It also shows how to retrieve cache statistics and clear the cache.

```Python
import time
import numpy as np
from functools import lru_cache

# Assuming freqprob and datasets are available in the environment
# from freqprob import SimpleGoodTuring, get_cache_stats, clear_all_caches
# from datasets import load_dataset # Placeholder for dataset loading

# Mock objects for demonstration if actual library is not available
class MockSimpleGoodTuring:
    def __init__(self, dataset, logprob=False):
        print("MockSimpleGoodTuring initialized")
        self.dataset = dataset
        self.logprob = logprob
        # Simulate some processing time
        time.sleep(0.1)

    def __call__(self, word):
        # Simulate scoring
        return len(word) # Dummy score

class MockFreqprob:
    def __init__(self):
        self._cache = {}
        self.SimpleGoodTuring = MockSimpleGoodTuring

    def get_cache_stats(self):
        return {"total_entries": len(self._cache), "hits": 0, "misses": 0}

    def clear_all_caches(self):
        print("Mock clearing all caches")
        self._cache = {}

freqprob = MockFreqprob()

# Mock dataset
small_dataset = {"apple": 10, "banana": 5, "cherry": 2, "date": 1, "elderberry": 1}
large_dataset = {"word" + str(i): i for i in range(10000)}

datasets = {"small": small_dataset, "large": large_dataset}

print("Simple Good-Turing Caching Demo:")
print("-" * 35)

try:
    # First creation (cold cache)
    print("Creating first SGT model (cold cache)...")
    start_time = time.time()
    sgt1 = freqprob.SimpleGoodTuring(small_dataset, logprob=False)
    first_time = time.time() - start_time
    print(f"Time: {first_time:.4f} seconds")

    # Second creation (hot cache)
    print("\nCreating second SGT model (hot cache)...")
    start_time = time.time()
    sgt2 = freqprob.SimpleGoodTuring(small_dataset, logprob=False)
    second_time = time.time() - start_time
    print(f"Time: {second_time:.4f} seconds")

    print(f"\nSpeedup from caching: {first_time / second_time:.1f}x")

    # Verify cache is working by checking results are identical
    test_words = list(small_dataset.keys())[:10]
    scores1 = [sgt1(word) for word in test_words]
    scores2 = [sgt2(word) for word in test_words]

    max_diff = max(abs(s1 - s2) for s1, s2 in zip(scores1, scores2, strict=False))
    print(f"Maximum score difference: {max_diff:.2e} (should be 0)")

    # Cache statistics
    cache_stats = freqprob.get_cache_stats()
    print("\nCache Statistics:")
    for key, value in cache_stats.items():
        print(f"  {key}: {value}")

    # Clear cache and show effect
    print("\nClearing all caches...")
    freqprob.clear_all_caches()

    print("Creating third SGT model (cache cleared)...")
    start_time = time.time()
    sgt3 = freqprob.SimpleGoodTuring(small_dataset, logprob=False)
    third_time = time.time() - start_time
    print(f"Time: {third_time:.4f} seconds (should be similar to first time)")

except Exception as e:
    print(f"SGT caching demo failed: {e}")
    print("This can happen with certain frequency distributions")

```

--------------------------------

### StreamingMLE Basic Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/api_reference.md

Demonstrates the basic instantiation and usage of the StreamingMLE class for updating and querying probabilities.

```python
streaming_mle = freqprob.StreamingMLE(max_vocabulary_size=10000, logprob=False)
streaming_mle.update_single('word1', 5)
streaming_mle.update_batch(['word2', 'word3'])
prob = streaming_mle('word1')
```

--------------------------------

### Basic Smoothing Usage Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/api_reference.md

Demonstrates the basic usage of MLE and Laplace smoothing classes with a predefined frequency distribution.

```python
import freqprob

# Create frequency distribution
freqdist = {'the': 100, 'cat': 50, 'dog': 30, 'bird': 10}

# Basic smoothing
mle = freqprob.MLE(freqdist, logprob=False)
laplace = freqprob.Laplace(freqdist, bins=1000, logprob=False)

print(f"MLE P(cat) = {mle('cat'):.4f}")
print(f"Laplace P(cat) = {laplace('cat'):.4f}")
print(f"Laplace P(unseen) = {laplace('elephant'):.6f}")
```

--------------------------------

### Performance and Validation Benchmarking

Source: https://github.com/tresoldi/freqprob/blob/main/docs/development.md

Shell commands to execute performance benchmarks and validation tests using the Hatch build tool. Includes options for quick and comprehensive benchmarks, as well as validation checks.

```bash
# Run quick benchmarks
hatch run bench

# Run comprehensive benchmarks
hatch run bench-all

# Run validation tests
hatch run validate

# Run quick validation
hatch run validate-quick
```

--------------------------------

### Advanced Smoothing and Evaluation Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/api_reference.md

Shows advanced usage including N-gram modeling with Kneser-Ney smoothing, perplexity calculation, and vectorized scoring.

```python
# N-gram language modeling
bigrams = {('the', 'cat'): 5, ('the', 'dog'): 3, ('a', 'cat'): 2}
kn = freqprob.KneserNey(bigrams, discount=0.75, logprob=True)

# Model evaluation
test_bigrams = [('the', 'cat'), ('a', 'dog')]
pp = freqprob.perplexity(kn, test_bigrams)
print(f"Perplexity: {pp:.2f}")

# Efficiency features
vectorized = freqprob.VectorizedScorer(laplace)
batch_scores = vectorized.score_batch(['cat', 'dog', 'bird'])
```

--------------------------------

### FreqProb Setup and Imports

Source: https://github.com/tresoldi/freqprob/blob/main/docs/tutorial_3_efficiency_memory.ipynb

Imports necessary libraries like gc, random, time, matplotlib, numpy, psutil, seaborn, and freqprob modules for efficiency and memory analysis. Sets up plotting styles and random seeds.

```python
import gc
import random
import time

import matplotlib.pyplot as plt
import numpy as np
import psutil
import seaborn as sns

import freqprob

# Import efficiency features
from freqprob import (
    BatchScorer,
    DistributionMemoryAnalyzer,
    MemoryProfiler,
    StreamingLaplace,
    StreamingMLE,
    VectorizedScorer,
    create_compressed_distribution,
    create_lazy_mle,
    create_sparse_distribution,
)

# Set up plotting
plt.style.use("default")
sns.set_palette("husl")
random.seed(42)
np.random.seed(42)

print("FreqProb Efficiency and Memory Management Tutorial")
print("=" * 50)
print(f"NumPy version: {np.__version__}")
print(f"Current memory usage: {psutil.Process().memory_info().rss / 1024 / 1024:.1f} MB")
```

--------------------------------

### Legacy Benchmark Script Usage

Source: https://github.com/tresoldi/freqprob/blob/main/docs/performance_comparison.md

Details the usage of older benchmark scripts located in the 'docs' directory for custom or detailed analysis. It shows how to specify output formats and run quick benchmarks.

```bash
cd docs/
python benchmarks.py --output benchmark_results --format all
python benchmarks.py --quick --output quick_results --format json
```

--------------------------------

### Streaming Data Processing Example

Source: https://github.com/tresoldi/freqprob/blob/main/docs/api_reference.md

Illustrates real-time learning using StreamingMLE, including updating the model with single observations and monitoring progress.

```python
# Real-time learning
streaming = freqprob.StreamingMLE(max_vocabulary_size=10000, logprob=False)

# Process incoming data
for word in data_stream:
    streaming.update_single(word)
    if streaming.get_update_count() % 1000 == 0:
        print(f"Processed {streaming.get_update_count()} updates")

# Get current probability
current_prob = streaming('word')
```

--------------------------------

### Create and Query Sparse Distribution

Source: https://github.com/tresoldi/freqprob/blob/main/docs/user_guide.md

Demonstrates the creation and usage of sparse distributions, optimized for data with many zero counts. It shows how to initialize a sparse distribution and perform efficient queries for top elements and elements within a frequency range.

```python
from freqprob import create_sparse_distribution

# Optimized for distributions with many zeros
sparse_freqdist = {'rare_word': 1, 'common_word': 10000}
sparse = create_sparse_distribution(sparse_freqdist)

# Efficient queries
top_10 = sparse.get_top_k(10)
mid_freq_words = sparse.get_elements_with_count_range(10, 100)
```