### Install and Build Documentation

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Commands to install documentation dependencies and build or serve the documentation locally using MkDocs.

```bash
# Install docs dependencies
pip install ".[docs]"

# Build documentation (if using MkDocs)
mkdocs build

# Serve locally
mkdocs serve
```

--------------------------------

### Install Chunkana

Source: https://context7.com/asukhodko/chunkana/llms.txt

Install the Chunkana library using pip. The `docs` extra can be installed for documentation tools.

```bash
pip install chunkana

# Optional: documentation tools extras
pip install "chunkana[docs]"
```

--------------------------------

### Quick Start Dify Integration

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/dify.md

Basic setup for using Chunkana with Dify workflows. Imports necessary functions for chunking markdown text and rendering it in Dify's style.

```python
# Using Chunkana with Dify
from chunkana import chunk_markdown
from chunkana.renderers import render_dify_style

chunks = chunk_markdown(text)
result = render_dify_style(chunks)
```

--------------------------------

### Set Up Development Environment

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Install dependencies and set up a virtual environment for development.

```bash
# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e ".[dev]"
```

--------------------------------

### Example Bug Report with Reproducible Example

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

A template for reporting bugs, including environment details and a minimal Python code example that reproduces the issue.

```markdown
## Bug Description
Chunking fails on documents with nested code blocks

## Environment
- Python: 3.12.1
- Chunkana: 0.1.5
- OS: Ubuntu 22.04

## Minimal Example
```python
from chunkana import chunk_markdown

text = """
# Code Example
```python
def outer():
    ```
    nested code
    ```
```
"

chunks = chunk_markdown(text)  # Raises ValueError
```

## Expected Behavior
Should create one chunk with the code block intact

## Actual Behavior
ValueError: Unmatched code fence
```

--------------------------------

### Example Docstring with Type Hints

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Illustrates a Python function docstring following Google style, including type hints, arguments, return values, and a usage example.

```python
def chunk_markdown(text: str, config: ChunkConfig | None = None) -> list[Chunk]:
    """Chunk Markdown text into semantic units.    
    Args:
        text: The Markdown text to chunk
        config: Optional configuration for chunking behavior        
    Returns:
        List of chunks with metadata        
    Example:
        >>> chunks = chunk_markdown("# Title\nContent here")
        >>> len(chunks)
        1
        >>> chunks[0].metadata["header_path"]
        "/Title"
    """
```

--------------------------------

### Example Output of Quicksort

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/code_context.md

Shows the expected output when using the quicksort function with a sample input array.

```text
>>> quicksort([3, 6, 8, 10, 1, 2, 1])
[1, 1, 2, 3, 6, 8, 10]
```

--------------------------------

### Canonical Output Schema Example (JSONL)

Source: https://github.com/asukhodko/chunkana/blob/main/docs/testing/fixtures.md

Example of a JSONL file containing canonical chunks with detailed metadata. Used for baseline testing.

```json
{
  "chunk_index": 0,
  "content": "# Introduction\n\nThis is the introduction...",
  "start_line": 1,
  "end_line": 10,
  "metadata": {
    "chunk_id": "abc12345",
    "strategy": "structural",
    "header_path": "/Introduction",
    "content_type": "section",
    "previous_content": null,
    "next_content": "This is the next section..."
  }
}
```

--------------------------------

### Unit Test Example

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Example of a unit test for basic chunk creation functionality. Asserts properties of the created chunk object.

```python
def test_chunk_creation():
    """Test basic chunk creation functionality."""
    chunk = Chunk(
        content="# Test\nContent here",
        start_line=1,
        end_line=2,
        metadata={"content_type": "section"}
    )
    
    assert chunk.size == 18
    assert chunk.metadata["content_type"] == "section"
```

--------------------------------

### Verify Setup with Tests and Linting

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Run the test suite and check code style to ensure the development environment is set up correctly.

```bash
# Run tests
pytest

# Check code style
ruff check src/chunkana
mypy src/chunkana
```

--------------------------------

### View-Level Output Schema Example (JSONL)

Source: https://github.com/asukhodko/chunkana/blob/main/docs/testing/fixtures.md

Example of a JSONL file containing rendered output, including metadata embedded within a text block. Used for view-level testing.

```json
{
  "chunk_index": 0,
  "text": "<metadata>\n{\"chunk_index\": 0, \"strategy\": \"structural\"}\n</metadata>\n\n# Introduction\n\nThis is the introduction..."
}
```

--------------------------------

### Database Configuration in YAML

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/code_context.md

Example YAML configuration for a database connection, specifying host, port, and database name.

```yaml
database:
  host: localhost
  port: 5432
  name: myapp
```

--------------------------------

### Robust Processing with Error Recovery

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples.md

Illustrates the setup for robust processing with error recovery using chunkana. This example imports necessary components for handling potential exceptions during chunking.

```python
from chunkana import chunk_markdown, ChunkConfig
from chunkana.exceptions import ChunkanaError
import logging
```

--------------------------------

### Test with Minimal Example

Source: https://github.com/asukhodko/chunkana/blob/main/docs/errors.md

Execute chunking on a small, controlled text example to verify basic functionality and isolate issues. If this test passes, the problem is likely specific to the user's document.

```python
def test_minimal_case():
    """Test with minimal example to isolate issues."""
    
    minimal_text = """
# Test Header

Some content here.

```python
def test():
    return True
```

More content.
"""
    
    try:
        chunks = chunk_markdown(minimal_text)
        print(f"Minimal test passed: {len(chunks)} chunks")
        return True
    except Exception as e:
        print(f"Minimal test failed: {e}")
        return False

if test_minimal_case():
    print("Basic functionality works, issue is with your specific document")
else:
    print("Basic functionality broken, check installation")
```

--------------------------------

### Dify Workflow Example

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/dify.md

Example of a Dify Code node function that chunks text and formats it for Dify. Returns the formatted chunks and their count.

```python
# In Dify Code node
from chunkana import chunk_markdown
from chunkana.renderers import render_dify_style

def main(text: str) -> dict:
    chunks = chunk_markdown(text)
    formatted = render_dify_style(chunks)

    return {
        "chunks": formatted,
        "count": len(formatted),
    }
```

--------------------------------

### Install Chunkana in n8n Python Environment

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/n8n.md

Install the Chunkana library using pip within your n8n Python environment.

```bash
pip install chunkana
```

--------------------------------

### Search for Broken Documentation Links

Source: https://github.com/asukhodko/chunkana/blob/main/rollback_procedures.md

Search documentation files for broken references to migration guide or baseline files. This helps identify issues with restored documentation.

```bash
grep -r "MIGRATION_GUIDE.md" docs/
grep -r "BASELINE.md" docs/
```

--------------------------------

### Create a Custom XML Renderer

Source: https://github.com/asukhodko/chunkana/blob/main/docs/renderers.md

Implement a custom renderer to convert chunks into an XML format. This example shows how to structure the XML with elements for content and metadata, including chunk index and other attributes.

```python
def render_xml(chunks):
    """Custom XML renderer."""
    import xml.etree.ElementTree as ET
    
    root = ET.Element('document')
    root.set('chunk_count', str(len(chunks)))
    
    for chunk in chunks:
        chunk_elem = ET.SubElement(root, 'chunk')
        chunk_elem.set('index', str(chunk.metadata['chunk_index']))
        
        # Content
        content_elem = ET.SubElement(chunk_elem, 'content')
        content_elem.text = chunk.content
        
        # Metadata
        meta_elem = ET.SubElement(chunk_elem, 'metadata')
        for key, value in chunk.metadata.items():
            if isinstance(value, (str, int, float, bool)):
                attr_elem = ET.SubElement(meta_elem, key)
                attr_elem.text = str(value)
    
    return ET.tostring(root, encoding='unicode')

# Usage
chunks = chunk_markdown(text)
xml_output = render_xml(chunks)
```

--------------------------------

### Table Grouping Examples

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples/advanced-usage.md

Implement strict or loose table grouping by adjusting `max_distance_lines`, `require_same_section`, and `max_group_size` parameters in `TableGroupingConfig`.

```python
# Strict table grouping
table_config = TableGroupingConfig(
    max_distance_lines=5,     # Tables must be close
    require_same_section=True,
    max_group_size=4096,
)

# Loose table grouping
table_config = TableGroupingConfig(
    max_distance_lines=20,    # Allow distant tables
    require_same_section=False,
    max_group_size=12288,     # Allow larger groups
)
```

--------------------------------

### LaTeX Handling Examples

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples/advanced-usage.md

Control LaTeX block handling with `latex_max_size` and `latex_break_strategy`. Choose between strict preservation, no size limit, or breaking large blocks at equation boundaries.

```python
# Strict LaTeX preservation
config = ChunkerConfig(
    preserve_latex_blocks=True,
    preserve_atomic_blocks=True,
    # Don't break LaTeX blocks even if they're large
    latex_max_size=None,  # No size limit
)

# Balanced LaTeX handling
config = ChunkerConfig(
    preserve_latex_blocks=True,
    latex_max_size=2048,  # Break very large LaTeX blocks
    latex_break_strategy="equation",  # Break at equation boundaries
)
```

--------------------------------

### Create a Custom CSV Renderer

Source: https://github.com/asukhodko/chunkana/blob/main/docs/renderers.md

Implement a custom renderer to convert chunks into a CSV format. This example defines the CSV header and writes each chunk's data, including metadata fields and content, escaping newlines within the content.

```python
def render_csv(chunks):
    """Custom CSV renderer."""
    import csv
    import io
    
    output = io.StringIO()
    writer = csv.writer(output)
    
    # Header
    writer.writerow([
        'chunk_index', 'content', 'header_path', 
        'content_type', 'start_line', 'end_line', 'size'
    ])
    
    # Data rows
    for chunk in chunks:
        writer.writerow([
            chunk.metadata['chunk_index'],
            chunk.content.replace('\n', '\\n'),  # Escape newlines
            chunk.metadata['header_path'],
            chunk.metadata['content_type'],
            chunk.start_line,
            chunk.end_line,
            chunk.size,
        ])
    
    return output.getvalue()

# Usage
chunks = chunk_markdown(text)
csv_output = render_csv(chunks)
```

--------------------------------

### Chunk Technical Documentation with Code Context

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples.md

Configure Chunkana for technical documents with code examples using `ChunkConfig`. Set `code_threshold` and enable `code_context_binding` to improve code chunking and context preservation.

```python
from chunkana import chunk_markdown, ChunkConfig

# Configuration optimized for code-heavy documents
code_config = ChunkConfig(
    max_chunk_size=3000,
    min_chunk_size=500,
    overlap_size=100,
    code_threshold=0.2,  # Lower threshold for code-aware strategy
    enable_code_context_binding=True,
    max_context_chars_before=400,
    max_context_chars_after=200,
)

technical_doc = """
# API Documentation

""" # Assuming technical_doc content would follow here
```

--------------------------------

### Chunk Markdown with Quality Metrics

Source: https://context7.com/asukhodko/chunkana/llms.txt

Use `chunk_with_metrics` to get both the resulting chunks and detailed quality statistics, such as chunk size distribution and counts of undersized/oversized chunks. This is valuable for monitoring chunking pipelines and tuning `ChunkConfig` parameters.

```python
from chunkana import chunk_with_metrics, ChunkConfig

config = ChunkConfig(max_chunk_size=2048, min_chunk_size=256)
chunks, metrics = chunk_with_metrics(text, config)

print(f"Total chunks:    {metrics.total_chunks}")
print(f"Average size:    {metrics.avg_chunk_size:.0f} chars")
print(f"Std deviation:   {metrics.std_dev_size:.0f}")
print(f"Min size:        {metrics.min_size} chars")
print(f"Max size:        {metrics.max_size} chars")
print(f"Undersize (<{config.min_chunk_size}): {metrics.undersize_count}")
print(f"Oversize  (>{config.max_chunk_size}): {metrics.oversize_count}")

# Monitor quality in production
if metrics.undersize_count > metrics.total_chunks * 0.1:
    print("Warning: >10% of chunks are undersized — consider lowering min_chunk_size")
if metrics.oversize_count > 0:
    print(f"Warning: {metrics.oversize_count} oversized chunks (large atomic blocks)")
```

--------------------------------

### Adjusting Chunk Size and Preserving Blocks

Source: https://github.com/asukhodko/chunkana/blob/main/docs/errors.md

Configure Chunkana to manage chunk sizes and ensure atomic blocks like code are preserved. This example sets minimum and maximum chunk sizes, and forces a code-aware strategy.

```python
# Increase minimum chunk size
config = ChunkConfig(min_chunk_size=200)

# Check for empty chunks
empty_chunks = [i for i, c in enumerate(chunks) if len(c.content.strip()) < 10]
if empty_chunks:
    print(f"Empty chunks at indices: {empty_chunks}")
```

```python
# Reduce maximum chunk size
config = ChunkConfig(max_chunk_size=1500)

# Force specific strategy
config = ChunkConfig(strategy_override="code_aware")

# Check chunk sizes
large_chunks = [i for i, c in enumerate(chunks) if c.size > 2000]
if large_chunks:
    print(f"Large chunks at indices: {large_chunks}")
```

```python
# Ensure atomic blocks are preserved
config = ChunkConfig(
    preserve_atomic_blocks=True,
    max_chunk_size=4096,  # Increase if too small
    strategy_override="code_aware"  # Force code-aware strategy
)
```

--------------------------------

### Basic Chunkana Configuration

Source: https://github.com/asukhodko/chunkana/blob/main/docs/api/parameter-mapping.md

Sets up default Chunkana parameters equivalent to Dify plugin defaults. Ensure necessary imports are present.

```python
from chunkana import ChunkerConfig

# Equivalent to plugin defaults
config = ChunkerConfig(
    max_chunk_size=4096,
    min_chunk_size=512,
    overlap_size=200,
    preserve_atomic_blocks=True,
    extract_preamble=True,
)
```

--------------------------------

### Set up API Authentication Headers

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples.md

Configure the necessary headers for making authenticated API requests. Replace 'YOUR_API_KEY' with your actual API key.

```python
import requests

# Set up authentication
headers = {
    'Authorization': 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json'
}
```

--------------------------------

### Property-Based Test Example

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Example of a property-based test using Hypothesis to ensure chunking preserves content. It generates random text and verifies reconstruction.

```python
from hypothesis import given, strategies as st

@given(st.text(min_size=1, max_size=1000))
def test_chunking_preserves_content(text):
    """Test that chunking preserves all content."""
    chunks = chunk_markdown(text)
    reconstructed = "".join(chunk.content for chunk in chunks)
    
    # Content should be preserved (allowing for overlap)
    assert len(reconstructed) >= len(text)
```

--------------------------------

### Validate Line Number Consistency

Source: https://github.com/asukhodko/chunkana/blob/main/docs/errors.md

Verify that the start and end line numbers for each chunk are within the valid range of the original document and that the start line does not exceed the end line.

```python
# Validate line number consistency
def validate_line_numbers(chunks, original_text):
    lines = original_text.split('\n')
    total_lines = len(lines)
    
    for i, chunk in enumerate(chunks):
        if chunk.start_line < 1 or chunk.end_line > total_lines:
            print(f"Chunk {i} has invalid line range: {chunk.start_line}-{chunk.end_line}")
        
        if chunk.start_line > chunk.end_line:
            print(f"Chunk {i} has inverted line range")

validate_line_numbers(chunks, original_text)
```

--------------------------------

### GET Endpoints

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/table_grouping.md

Retrieve data for users, posts, and comments.

```APIDOC
## GET /users

### Description
List all users.

### Method
GET

### Endpoint
/users

### Response
#### Success Response (200)
- **users** (array) - List of user objects.

## GET /users/:id

### Description
Get a specific user by their ID.

### Method
GET

### Endpoint
/users/:id

### Parameters
#### Path Parameters
- **id** (integer) - Required - The unique identifier of the user.

### Response
#### Success Response (200)
- **user** (object) - The user object.

## GET /posts

### Description
List all posts.

### Method
GET

### Endpoint
/posts

### Response
#### Success Response (200)
- **posts** (array) - List of post objects.

## GET /posts/:id

### Description
Get a specific post by its ID.

### Method
GET

### Endpoint
/posts/:id

### Parameters
#### Path Parameters
- **id** (integer) - Required - The unique identifier of the post.

### Response
#### Success Response (200)
- **post** (object) - The post object.
```

--------------------------------

### Performance Optimization Configurations

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples/advanced-usage.md

Set up chunker configurations optimized for memory usage, speed, or output quality.

```python
# Memory-optimized configuration
memory_config = ChunkerConfig(
    max_chunk_size=1024,       # Smaller chunks
    overlap_size=50,           # Minimal overlap
    validate_invariants=False, # Skip validation
    preserve_metadata=False,   # Minimal metadata
)

# Speed-optimized configuration
speed_config = ChunkerConfig(
    strategy_override="fallback",  # Fastest strategy
    preserve_atomic_blocks=False,  # Skip complex analysis
    enable_code_context_binding=False,  # Skip context binding
    validate_invariants=False,     # Skip validation
)

# Quality-optimized configuration
quality_config = ChunkerConfig(
    use_adaptive_sizing=True,      # Better chunk boundaries
    group_related_tables=True,     # Better table handling
    preserve_latex_blocks=True,    # Better LaTeX handling
    validate_invariants=True,      # Ensure quality
)
```

--------------------------------

### Advanced Chunkana Configuration

Source: https://github.com/asukhodko/chunkana/blob/main/docs/api/parameter-mapping.md

Demonstrates advanced Chunkana configuration with adaptive sizing and table grouping. Requires `AdaptiveSizeConfig` and potentially `TableGroupingConfig` to be defined.

```python
config = ChunkerConfig(
    max_chunk_size=2048,
    strategy_override="code_aware",
    use_adaptive_sizing=True,
    adaptive_config=AdaptiveSizeConfig(
        base_size=1500,
        code_weight=0.4,
    ),
    group_related_tables=True,
)
```

--------------------------------

### JavaScript Express Route

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/adaptive_sizing.md

Defines a GET route for an Express.js application that uses the previously defined handler.

```javascript
app.get('/api/:id', handler);
```

--------------------------------

### Basic Markdown Chunking with Chunkana

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/windmill.md

Demonstrates basic markdown chunking using `chunk_markdown` and rendering the output as JSON. Ensure 'chunkana' is in your requirements.

```python
#requirements:
#chunkana

from chunkana import chunk_markdown
from chunkana.renderers import render_json

def main(text: str) -> dict:
    chunks = chunk_markdown(text)

    return {
        "chunks": render_json(chunks),
        "count": len(chunks),
    }
```

--------------------------------

### Define a Python Greeting Function

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/mixed_content.md

A simple Python function to greet a user by name. No special setup is required.

```python
def greet(name):
    return f"Hello, {name}!"
```

--------------------------------

### Run Baseline Compatibility Tests

Source: https://github.com/asukhodko/chunkana/blob/main/docs/api/compatibility.md

Execute these commands to verify canonical chunk and view-level output compatibility. Property-based tests are also available for comprehensive validation.

```bash
# Canonical chunk compatibility
pytest tests/baseline/test_canonical.py -v

# View-level output compatibility  
pytest tests/baseline/test_view_level.py -v

# Property-based tests
pytest tests/property/ -v
```

--------------------------------

### Configure Adaptive Sizing with AdaptiveSizeConfig

Source: https://github.com/asukhodko/chunkana/blob/main/docs/config.md

Define custom parameters for adaptive chunk sizing, including base size, code weight, and min/max sizes. Requires importing AdaptiveSizeConfig.

```python
from chunkana.adaptive_sizing import AdaptiveSizeConfig

adaptive_config = AdaptiveSizeConfig(
    base_size=1500,       # Base chunk size
    code_weight=0.4,      # Weight for code content
    min_size=500,         # Minimum adaptive size
    max_size=8000,        # Maximum adaptive size
)
```

--------------------------------

### Compare Chunking Configurations

Source: https://github.com/asukhodko/chunkana/blob/main/docs/performance.md

Benchmark and compare the performance of various ChunkConfig settings. Useful for identifying the optimal configuration for specific text and hardware.

```python
def compare_configurations(text):
    """Compare performance of different configurations."""
    
    configs = {
        'default': ChunkConfig(),
        'fast': ChunkConfig(
            validate_invariants=False,
            overlap_size=0,
            strategy_override="fallback"
        ),
        'quality': ChunkConfig(
            validate_invariants=True,
            overlap_size=200,
            preserve_atomic_blocks=True
        ),
        'memory_efficient': ChunkConfig(
            max_chunk_size=1024,
            min_chunk_size=256,
            overlap_size=50
        ),
    }
    
    results = {}
    for name, config in configs.items():
        print(f"\nTesting {name} configuration...")
        results[name] = benchmark_chunking(text, config)
    
    # Print comparison
    print("\nPerformance Comparison:")
    print(f"{'Config':<15} {'Time (ms)':<10} {'Rate (K/s)':<12} {'Memory (MB)':<12} {'Chunks':<8}")
    print("-" * 60)
    
    for name, result in results.items():
        print(f"{name:<15} {result['processing_time']*1000:<10.1f} "
              f"{result['chars_per_second']/1000:<12.0f} "
              f"{result['peak_memory_mb']:<12.1f} {result['chunks']:<8}")
    
    return results

# Usage
results = compare_configurations(text)
```

--------------------------------

### Get Strategy Used for Chunking

Source: https://github.com/asukhodko/chunkana/blob/main/docs/faq.md

Retrieve the chunking strategy automatically selected by Chunkana by accessing the 'strategy' key in the chunk's metadata.

```python
# Check which strategy was used
chunks = chunk_markdown(text)
print(f"Strategy used: {chunks[0].metadata['strategy']}")
```

--------------------------------

### Profile Memory Usage During Chunking

Source: https://github.com/asukhodko/chunkana/blob/main/docs/performance.md

Profile memory usage of chunking operations with different configurations using `tracemalloc` and `memory_profiler`. Requires `memory-profiler` to be installed.

```python
import tracemalloc
from memory_profiler import profile

@profile
def memory_intensive_chunking(text):
    """Profile memory usage during chunking."""
    
    # Start detailed tracing
    tracemalloc.start()
    
    # Chunk with different configurations
    configs = [
        ChunkConfig(overlap_size=0),
        ChunkConfig(overlap_size=200),
        ChunkConfig(validate_invariants=True),
    ]
    
    results = []
    for i, config in enumerate(configs):
        snapshot_before = tracemalloc.take_snapshot()
        
        chunks = chunk_markdown(text, config)
        
        snapshot_after = tracemalloc.take_snapshot()
        top_stats = snapshot_after.compare_to(snapshot_before, 'lineno')
        
        results.append({
            'config_index': i,
            'chunks': len(chunks),
            'memory_diff': sum(stat.size_diff for stat in top_stats[:10])
        })
    
    tracemalloc.stop()
    return results

# Usage (requires: pip install memory-profiler)
# python -m memory_profiler your_script.py
```

--------------------------------

### Run Test Suite

Source: https://github.com/asukhodko/chunkana/blob/main/rollback_procedures.md

Execute the full test suite to ensure system integrity after a rollback. Coverage can be included by using 'make test-cov'.

```bash
make test
make test-cov
```

--------------------------------

### Configure Chunk Size for Search Indexing

Source: https://github.com/asukhodko/chunkana/blob/main/docs/faq.md

Configure `max_chunk_size` to balance detail and precision for search indexing purposes.

```python
# For search indexing
config = ChunkConfig(max_chunk_size=2500)  # Balance detail vs. precision
```

--------------------------------

### Chunk Structure in JSON Output

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/n8n.md

Example of the JSON structure for a single chunk produced by `render_json()`. It includes content, line numbers, size, and metadata.

```json
{
  "content": "Chunk text content...",
  "start_line": 1,
  "end_line": 10,
  "size": 256,
  "line_count": 10,
  "metadata": {
    "chunk_index": 0,
    "strategy": "structural",
    "header_path": "/Section 1",
    "content_type": "section"
  }
}
```

--------------------------------

### Migrate from dify-markdown-chunker to Chunkana

Source: https://github.com/asukhodko/chunkana/blob/main/docs/faq.md

Demonstrates how to update import paths and use renderers for dify-style output when migrating from dify-markdown-chunker.

```python
# Old way (dify-markdown-chunker)
from dify_markdown_chunker import chunk_markdown as old_chunk

# New way (Chunkana)
from chunkana import chunk_markdown
from chunkana.renderers import render_dify_style

# Same parameters work
chunks = chunk_markdown(text, max_chunk_size=4096)
dify_format = render_dify_style(chunks)
```

--------------------------------

### Content Analysis and Metrics

Source: https://github.com/asukhodko/chunkana/blob/main/README.md

Analyze Markdown content for metrics like code ratio using `analyze_markdown` or get chunking metrics along with chunks using `chunk_with_metrics`.

```python
from chunkana import analyze_markdown, chunk_with_metrics

analysis = analyze_markdown(text)
print(f"Code ratio: {analysis.code_ratio}")

chunks, metrics = chunk_with_metrics(text)
print(f"Average chunk size: {metrics.avg_chunk_size}")
```

--------------------------------

### Generate and Run Baseline Tests

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Generate new baseline data using `dify-markdown-chunker` and then run the baseline tests to ensure compatibility.

```bash
# Generate new baseline data (requires dify-markdown-chunker)
python scripts/generate_baseline.py

# Run baseline tests
pytest tests/baseline/
```

--------------------------------

### Basic Chunking with MarkdownChunker

Source: https://github.com/asukhodko/chunkana/blob/main/docs/debug_mode.md

Demonstrates basic usage of MarkdownChunker for standard document chunking with a specified maximum chunk size.

```python
from chunkana import MarkdownChunker, ChunkConfig

# Standard chunking
config = ChunkConfig(max_chunk_size=1000)
chunker = MarkdownChunker(config)
chunks = chunker.chunk(document)
```

--------------------------------

### Make API Requests with Chunkana

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples.md

A function to make GET or POST requests to a specified API endpoint using predefined headers. Handles JSON data for POST requests.

```python
def make_api_request(endpoint, data=None):
    url = f"https://api.example.com/{endpoint}"
    
    if data:
        response = requests.post(url, json=data, headers=headers)
    else:
        response = requests.get(url, headers=headers)
    
    return response.json()

# Example usage
result = make_api_request('users/profile')
print(result)
```

--------------------------------

### Configure Max and Min Chunk Sizes

Source: https://github.com/asukhodko/chunkana/blob/main/docs/faq.md

Set `max_chunk_size` as a hard limit and `min_chunk_size` as a soft target to merge smaller chunks with neighbors.

```python
config = ChunkConfig(
    max_chunk_size=2000,  # Never exceed 2000 chars
    min_chunk_size=500,   # Try to merge chunks smaller than 500 chars
)
```

--------------------------------

### Adaptive Sizing Strategies

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples/advanced-usage.md

Define different adaptive sizing strategies by adjusting weights and size ratios. Choose conservative or aggressive settings based on content and desired chunk granularity.

```python
# Conservative adaptive sizing
adaptive_config = AdaptiveSizeConfig(
    base_size=2000,
    code_weight=0.2,      # Less aggressive for code
    min_size_ratio=0.7,   # Don't go too small
    max_size_ratio=1.5,   # Don't go too large
)

# Aggressive adaptive sizing
adaptive_config = AdaptiveSizeConfig(
    base_size=1000,
    code_weight=0.6,      # More aggressive for code
    min_size_ratio=0.3,   # Allow very small chunks
    max_size_ratio=3.0,   # Allow very large chunks
)
```

--------------------------------

### Format Chunks for Dify using render_dify_style()

Source: https://github.com/asukhodko/chunkana/blob/main/docs/renderers.md

Employ render_dify_style to format chunks with metadata blocks, ensuring compatibility with Dify workflows. This renderer outputs a list of strings, each starting with a JSON metadata block.

```python
from chunkana.renderers import render_dify_style

chunks = chunk_markdown(text)
dify_output = render_dify_style(chunks)

# Output: List[str]
print(dify_output[0])
```

--------------------------------

### Hierarchical Chunking and Navigation

Source: https://github.com/asukhodko/chunkana/blob/main/docs/debug_mode.md

Illustrates hierarchical chunking using MarkdownChunker, accessing all chunks, retrieving flat chunks, and navigating the hierarchy via chunk IDs.

```python
# Hierarchical chunking
config = ChunkConfig(max_chunk_size=1000)
chunker = MarkdownChunker(config)
result = chunker.chunk_hierarchical(document)

# Access all chunks
all_chunks = result.chunks

# Access only leaf chunks (for flat retrieval)
flat_chunks = result.get_flat_chunks()

# Navigate hierarchy
root = result.get_chunk(result.root_id)
children = result.get_children(result.root_id)
```

--------------------------------

### Get Indexable Chunks from Hierarchical Chunking

Source: https://github.com/asukhodko/chunkana/blob/main/docs/faq.md

Use `get_flat_chunks()` to extract a subset of chunks suitable for indexing in vector databases. Access all chunks, including parents and children, via the `.chunks` attribute for navigation and summarization.

```python
result = chunker.chunk_hierarchical(text)

# For vector database indexing
indexable_chunks = result.get_flat_chunks()

# For navigation and summaries
all_chunks = result.chunks
```

--------------------------------

### Push to Fork

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Push your feature branch to your fork of the repository.

```bash
git push origin feature/your-feature-name
```

--------------------------------

### Verify File Existence

Source: https://github.com/asukhodko/chunkana/blob/main/rollback_procedures.md

Check if key files and directories have been restored correctly after a rollback operation. This is a crucial verification step.

```bash
ls -la MIGRATION_GUIDE.md BASELINE.md
ls -la tests/baseline/plugin_config_keys.json
ls -la tests/baseline/golden_dify_style/
```

--------------------------------

### Create Default ChunkerConfig

Source: https://github.com/asukhodko/chunkana/blob/main/docs/config.md

Instantiate the default configuration for Chunkana. This provides a baseline for chunking parameters.

```python
from chunkana import ChunkerConfig

# Default configuration
config = ChunkerConfig.default()
```

--------------------------------

### Chunking with Basic Configuration

Source: https://github.com/asukhodko/chunkana/blob/main/README.md

Customize chunking behavior using `ChunkConfig` to set parameters like `max_chunk_size`, `min_chunk_size`, and `overlap_size`.

```python
from chunkana import chunk_markdown, ChunkConfig

config = ChunkConfig(
    max_chunk_size=2048,
    min_chunk_size=256,
    overlap_size=100,
)

chunks = chunk_markdown(text, config)
```

--------------------------------

### Clone Repository

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Clone the Chunkana repository to your local machine.

```bash
git clone https://github.com/YOUR_USERNAME/chunkana.git
cd chunkana
```

--------------------------------

### Configure ChunkerConfig for Documentation Sites

Source: https://github.com/asukhodko/chunkana/blob/main/docs/config.md

Adjust Chunkana settings for processing documentation websites, emphasizing smaller chunks and structural awareness.

```python
config = ChunkerConfig(
    max_chunk_size=2048,
    min_chunk_size=256,
    overlap_size=150,
    structure_threshold=2,
)
```

--------------------------------

### POST Endpoints

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/table_grouping.md

Create new users, posts, and comments.

```APIDOC
## POST /users

### Description
Create a new user.

### Method
POST

### Endpoint
/users

### Request Body
- **username** (string) - Required - The username for the new user.
- **email** (string) - Required - The email address for the new user.

### Response
#### Success Response (201)
- **user** (object) - The newly created user object.

## POST /posts

### Description
Create a new post. Requires authentication.

### Method
POST

### Endpoint
/posts

### Auth
Yes

### Request Body
- **user_id** (integer) - Required - The ID of the user creating the post.
- **title** (string) - Required - The title of the post.
- **content** (string) - Required - The content of the post.

### Response
#### Success Response (201)
- **post** (object) - The newly created post object.

## POST /comments

### Description
Add a new comment to a post. Requires authentication.

### Method
POST

### Endpoint
/comments

### Auth
Yes

### Request Body
- **post_id** (integer) - Required - The ID of the post to comment on.
- **user_id** (integer) - Required - The ID of the user making the comment.
- **body** (string) - Required - The content of the comment.

### Response
#### Success Response (201)
- **comment** (object) - The newly created comment object.
```

--------------------------------

### Check Code Quality with Ruff and MyPy

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Use Ruff for linting and formatting, and MyPy for static type checking to maintain code quality.

```bash
# Lint and format
ruff check src/chunkana
ruff format src/chunkana

# Type checking
mypy src/chunkana
```

--------------------------------

### Configure Streaming Options

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples/advanced-usage.md

Customize `StreamingConfig` for memory-constrained environments or specific file reading needs. Options include buffer size, overlap, encoding, and read chunk size.

```python
streaming_config = StreamingConfig(
    buffer_size=50_000,      # Smaller buffer for memory-constrained environments
    overlap_lines=10,        # Minimal overlap
    encoding='utf-8',        # File encoding
    chunk_size=8192,         # File read chunk size
)
```

--------------------------------

### Prepare Chunks for Vector Database Integration

Source: https://github.com/asukhodko/chunkana/blob/main/docs/renderers.md

Formats chunks into a structure suitable for ingestion into a vector database, including metadata mapping.

```python
from chunkana.renderers import render_json

def prepare_for_vector_db(chunks, source_name):
    """Prepare chunks for vector database ingestion."""
    
    json_chunks = render_json(chunks)
    
    vector_docs = []
    for chunk_data in json_chunks:
        doc = {
            'id': f"{source_name}_{chunk_data['metadata']['chunk_index']}",
            'content': chunk_data['content'],
            'metadata': {
                'source': source_name,
                'header_path': chunk_data['metadata']['header_path'],
                'content_type': chunk_data['metadata']['content_type'],
                'has_code': chunk_data['metadata']['has_code'],
                'size': chunk_data['size'],
                'line_range': f"{chunk_data['start_line']}-{chunk_data['end_line']}",
            }
        }
        vector_docs.append(doc)
    
    return vector_docs

# Usage
chunks = chunk_markdown(text)
vector_docs = prepare_for_vector_db(chunks, 'user_manual')
```

--------------------------------

### Prepare Document Chunks for Vector Database

Source: https://github.com/asukhodko/chunkana/blob/main/docs/examples.md

Use `prepare_for_vector_db` to chunk documents with RAG-optimized settings and render them as JSON. Enrich chunks with metadata like source file, chunk ID, and ingestion timestamp.

```python
from chunkana import chunk_markdown, ChunkConfig
from chunkana.renderers import render_json
import json

def prepare_for_vector_db(document_path: str, source_name: str):
    """Prepare document chunks for vector database ingestion."""
    
    # RAG-optimized configuration
    config = ChunkConfig(
        max_chunk_size=1500,    # Fit in embedding model context
        min_chunk_size=300,     # Ensure meaningful content
        overlap_size=150,       # Context continuity
    )
    
    # Read and chunk document
    with open(document_path, 'r', encoding='utf-8') as f:
        text = f.read()
    
    chunks = chunk_markdown(text, config)
    
    # Convert to JSON format
    json_chunks = render_json(chunks)
    
    # Enrich with source metadata
    for i, chunk_data in enumerate(json_chunks):
        chunk_data['metadata'].update({
            'source_file': source_name,
            'document_path': document_path,
            'chunk_id': f"{source_name}_{i}",
            'ingested_at': '2024-01-01T00:00:00Z',
        })
    
    return json_chunks

# Usage
chunks = prepare_for_vector_db('docs/user_guide.md', 'user_guide')

# Save for ingestion
with open('chunks_for_ingestion.json', 'w') as f:
    json.dump(chunks, f, indent=2)

print(f"Prepared {len(chunks)} chunks for vector database")
```

--------------------------------

### Run Test Suite

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Execute the test suite to ensure changes do not introduce regressions. Options include running all tests, with coverage, or specific test categories.

```bash
# Run all tests
pytest

# Run with coverage
pytest --cov=chunkana --cov-report=html

# Run specific test categories
pytest tests/unit/          # Unit tests
pytest tests/property/      # Property-based tests
pytest tests/baseline/      # Compatibility tests
```

--------------------------------

### Compatibility Verification with Pytest

Source: https://github.com/asukhodko/chunkana/blob/main/docs/integrations/dify.md

Commands to run baseline and view level tests using pytest to verify Chunkana's compatibility, particularly within a Dify context.

```bash
# Run baseline tests to verify compatibility
pytest tests/baseline/test_canonical.py -v
pytest tests/baseline/test_view_level.py -v
```

--------------------------------

### Connect to PostgreSQL Database in Python

Source: https://github.com/asukhodko/chunkana/blob/main/tests/baseline/fixtures/code_context.md

Python code snippet using psycopg2 to establish a connection to a PostgreSQL database based on provided credentials.

```python
import psycopg2

conn = psycopg2.connect(
    host="localhost",
    port=5432,
    dbname="myapp"
)
```

--------------------------------

### Restore Test Configuration Files

Source: https://github.com/asukhodko/chunkana/blob/main/rollback_procedures.md

Restore test configuration files from the backup directory to the tests/baseline directory. This ensures test configurations are available.

```bash
cp .backup/migration-cleanup/plugin_config_keys.json tests/baseline/
cp .backup/migration-cleanup/plugin_tool_params.json tests/baseline/
```

--------------------------------

### Run Specific Tests

Source: https://github.com/asukhodko/chunkana/blob/main/CONTRIBUTING.md

Execute specific tests by providing file paths or markers. This is useful for targeted testing during development.

```bash
# Specific test file
pytest tests/unit/test_chunk.py

# Performance tests (marked as slow)
pytest -m performance

# Baseline compatibility tests
pytest tests/baseline/ -v
```