### Check Chunklet CLI Version and Help

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Displays the current version of the chunklet CLI or provides a general help guide. These are fundamental commands for understanding the tool's capabilities and getting started.

```bash
chunklet --version
chunklet --help
```

--------------------------------

### Build Chunklet-py from Source

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/installation.md

For users who prefer to build from source, this method involves cloning the repository and installing the package locally. It also allows for the installation of all optional features using the '[all]' specifier.

```bash
git clone https://github.com/speedyk-005/chunklet-py.git
cd chunklet-py
pip install .[all]
```

--------------------------------

### Install Chunklet-py via Pip

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/installation.md

The primary method for installing Chunklet-py is using pip. This command installs the package and a subsequent command verifies the installation by displaying the version.

```bash
pip install chunklet-py
chunklet --version
```

--------------------------------

### Set Up Chunklet-py Development Environment

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/installation.md

This section outlines how to set up a development environment for Chunklet-py, including cloning the repository and installing the package in editable mode with various development-focused dependency sets like '[dev]', '[docs]', and '[dev-all]'.

```bash
git clone https://github.com/speedyk-005/chunklet-py.git
cd chunklet-py
pip install -e ".[dev]"
pip install -e ".[docs]"
pip install -e ".[dev-all]"
```

--------------------------------

### Install Chunklet-py with Optional Dependencies

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/installation.md

Chunklet-py supports optional dependencies to enable specific functionalities. These can be installed using pip with extra specifiers like '[document]', '[code]', or '[all]' for comprehensive installation.

```bash
pip install "chunklet-py[document]"
pip install "chunklet-py[code]"
pip install "chunklet-py[all]"
```

--------------------------------

### CLI: Chunk Text and Documents

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

Provides command-line examples for installing chunklet, checking the version, chunking raw text with specified token and sentence limits, and chunking files from a directory with various options including metadata and progress display.

```bash
# Install chunklet
pip install chunklet-py

# Check version
chunklet --version

# Chunk text directly
chunklet chunk "Your text here. Multiple sentences. More content." \
  --max-tokens 20 \
  --max-sentences 2 \
  --overlap-percent 15

# Chunk a document file
chunklet chunk \
  --source document.pdf \
  --destination chunks/ \
  --max-tokens 500 \
  --overlap-percent 20 \
  --verbose \
  --metadata

# Batch process directory
chunklet chunk \
  --source documents/ \
  --destination output/ \
  --max-sentences 10 \
  --n-jobs 8 \
  --show-progress \
  --on-errors skip

# Sentence splitting
chunklet split "First sentence. Second sentence. Third sentence." \
  --lang en
```

--------------------------------

### C++ Code Example for Chunking Output

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A C++ code snippet demonstrating basic input/output and function definitions. This is an example of content that might be processed and chunked.

```cpp
#include <iostream>
#include <string>

void say_hello(std::string name) {
    std::cout << "Hello, " << name << std::endl;
}

int calculate_sum(int a, int b) {
    if (a < 0 || b < 0) {
        return -1;
    }
    int result = a + b;
    return result;
}
```

--------------------------------

### Java Data Processor Code Example

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A Java code example defining a `DataProcessor` class with a constructor and two methods: `getPath` to retrieve the source path and `process` to simulate data processing logic. This snippet is designed for batch processing.

```java
package com.chunker.data;

public class DataProcessor {
    private String sourcePath;

    // Constructor
    public DataProcessor(String path) {
        this.sourcePath = path;
    }

    // Method 1: Getter
    public String getPath() {
        return this.sourcePath;
    }

    // Method 2: Core processing logic
    public boolean process() {
        if (this.sourcePath.isEmpty()) {
            return false;
        }
        // Assume processing logic here
        return true;
    }
}
```

--------------------------------

### Clone Chunklet Repository (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/CONTRIBUTING.md

Clones your forked repository of Chunklet and navigates into the project directory. This is the first step after forking the repository to start making changes.

```bash
git clone https://github.com/YOUR_USERNAME/chunklet.git
cd chunklet
```

--------------------------------

### Go Configuration Struct and Functions

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A Go code example defining a `Config` struct with `Timeout` and `Retries` fields. It includes a factory function `NewConfig` to create a new configuration and a method `displayInfo` to print the configuration details. This snippet is intended for batch processing.

```go
package main

import (
    "fmt"
)

// Struct definition
type Config struct {
    Timeout int
    Retries int
}

// Function 1: Factory function
func NewConfig() Config {
    return Config{
        Timeout: 5000,
        Retries: 3,
    }
}

// Function 2: Method on the struct
func (c *Config) displayInfo() {
    fmt.Printf("Timeout: %dms, Retries: %d\n", c.Timeout, c.Retries)
}
```

--------------------------------

### C++ Calculator Code Example

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A simple C++ code example containing two functions: `say_hello` for printing a greeting and `calculate_sum` for adding two integers with basic error checking. This snippet is intended to be used as an input file for batch processing.

```cpp
#include <iostream>
#include <string>

// Function 1: Simple greeting
void say_hello(std::string name) {
    std::cout << "Hello, " << name << std::endl;
}

// Function 2: Logic block
int calculate_sum(int a, int b) {
    if (a < 0 || b < 0) {
        return -1; // Error code
    }
    int result = a + b;
    return result;
}
```

--------------------------------

### Get Specific Command Help in Chunklet CLI

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Retrieves detailed help information for specific chunklet CLI commands like 'split' or 'chunk'. This is useful for understanding the available options and parameters for each command.

```bash
chunklet split --help
chunklet chunk --help
```

--------------------------------

### Install Chunklet Dependencies (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/CONTRIBUTING.md

Installs project dependencies using pip within a Python virtual environment. It includes installing development dependencies with `.[dev]` and assumes a virtual environment named `.venv` is used.

```bash
python -m venv .venv
source .venv/bin/activate  # On Windows, use `.venv\Scripts\activate`
pip install -e ".[dev]"
```

--------------------------------

### Go Configuration Struct and Methods for Chunking Output

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A Go code snippet defining a Config struct with timeout and retries, a function to create a new configuration, and a method to display configuration info. This is an example of Go code that could be chunked.

```go
package main

import (
    "fmt"
)


type Config struct {
    Timeout int
    Retries int
}


func NewConfig() Config {
    return Config{
        Timeout: 5000,
        Retries: 3,
    }
}


func (c *Config) displayInfo() {
    fmt.Printf("Timeout: %dms, Retries: %d\n", c.Timeout, c.Retries)
}
```

--------------------------------

### CodeChunker: Chunking Code by Lines (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

Demonstrates chunking a Python code string by lines using CodeChunker. It highlights parameters like `max_lines`, `include_comments`, `docstring_mode`, and `strict`. The example iterates through the generated chunks, printing their content and metadata.

```python
from chunklet.experimental.code_chunker import CodeChunker

PYTHON_CODE = '''
"""
Module docstring
"""

import os

class Calculator:
    """
    A simple calculator class.

    A calculator that Contains basic arithmetic operations for demonstration purposes.
    """

    def add(self, x, y):
        """Add two numbers and return result.

        This is a longer description that should be truncated
        in summary mode. It has multiple lines and details.
        """
        result = x + y
        return result

    def multiply(self, x, y):
        # Multiply two numbers
        return x * y

def standalone_function():
    """A standalone function."""
    return True
'''

# Helper function
def simple_token_counter(text: str) -> int:
    """Simple Token Counter For Testing."""
    return len(text.split())

chunker = CodeChunker(token_counter=simple_token_counter)

chunks = chunker.chunk(
    PYTHON_CODE,                
    max_lines=10,               # (1)!
    include_comments=True,      # (2)!
    docstring_mode="all",       # (3)!
    strict=False,               # (4)!
)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(f"Content:\n{chunk.content}")
    print("Metadata:")
    for k,v in chunk.metadata.items():
        print(f"{k}: {v}")
    print()
```

--------------------------------

### Chunklet CLI: Before vs. After (v1.4.0 vs. v2.0.0)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/migration.md

Compares the command-line syntax for chunking operations before and after the v2.0.0 update. The older version used a simpler structure, while the new version introduces the 'chunk' command and supports explicit chunker selection.

```bash
# Before v1.4.0
chunklet "Your text here." --mode sentence --max-sentences 5
```

```bash
# After v2.0.0
# Chunking a string uses PlainTextChunker
chunklet chunk "Your text here." --max-sentences 5

# Chunking a file from a path uses DocumentChunker by default
chunklet chunk --source your_text.txt --max-sentences 5
```

--------------------------------

### Lint Code with Flake8 (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/CONTRIBUTING.md

Runs the `flake8` linter on the `src/` and `tests/` directories to check for style guide violations and potential errors.

```bash
flake8 src/ tests/
```

--------------------------------

### PlainTextChunker: Batch Processing with Parallel Execution (Python)

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

Shows how to process multiple text documents in parallel using PlainTextChunker's `batch_chunk` method. This example demonstrates error handling (`on_errors='skip'`) and progress tracking (`show_progress=True`), specifying the number of parallel workers (`n_jobs`).

```python
from chunklet.plain_text_chunker import PlainTextChunker

def token_counter(text: str) -> int:
    return len(text.split())

chunker = PlainTextChunker(token_counter=token_counter, verbose=True)

# Multiple texts to process
texts = [
    "First document about artificial intelligence and machine learning.",
    "Second document covering natural language processing techniques.",
    "Third document explaining computer vision and image recognition.",
]

# Batch process with parallel execution
all_chunks = []
for chunk in chunker.batch_chunk(
    texts=texts,
    max_tokens=20,
    overlap_percent=10,
    n_jobs=4,              # Use 4 parallel workers
    show_progress=True,    # Show progress bar
    on_errors="skip"       # Skip failed items, continue processing
):
    all_chunks.append(chunk)
    print(f"Processed chunk {chunk.metadata.chunk_num} from text {chunk.metadata.text_index}")

print(f"\nTotal chunks created: {len(all_chunks)}")

```

--------------------------------

### Update Language Detection: Integrate into SentenceSplitter (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/migration.md

Illustrates the refactoring of language detection from a standalone utility to an integrated method within SentenceSplitter. This change simplifies internal processes and provides a new method for direct language detection.

```Python
from chunklet.sentence_splitter import SentenceSplitter

# Before (v1.4.0)
# from chunklet.utils.detect_text_language import detect_text_language
# lang, confidence = detect_text_language(text)

# After (v2.0.0)
splitter = SentenceSplitter()
lang, confidence = splitter.detected_top_language(text)
```

--------------------------------

### Chunk Text by Sentences and Tokens (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/plain_text_chunker.md

This Python example shows how to chunk text by limiting both the maximum number of sentences and the maximum token count within each chunk. The chunking process stops when either of these limits is reached first, ensuring both structural and length constraints are met.

```python
chunks = chunker.chunk(
    text=text,
    max_sentences=5,
    max_tokens=100
)

```

--------------------------------

### Combine line count and function count constraints for chunking (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

This example illustrates how to combine line count and function count constraints when chunking code. By setting `max_lines=10` and `max_functions=1`, the chunking process ensures that chunks do not exceed 10 lines while also respecting function boundaries, creating well-defined code segments.

```python
chunks = chunker.chunk(
    PYTHON_CODE,
    max_lines=10,
    max_functions=1
)
```

--------------------------------

### reStructuredText Table of Contents Example

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/document_chunker.md

Provides an example of a Table of Contents (TOC) generated in reStructuredText format. This TOC includes links to different chapters or sections within a book or document.

```rst
Table of Contents=====
1. [Chapter 1](chapter_1.xhtml)
2. [Chapter 2](chapter_2.xhtml)
3. [Copyright](copyright.xhtml)
```

--------------------------------

### CodeChunker: Valid Source Inputs for Single Run (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

Shows how to initialize and use CodeChunker with different valid input types for the 'source' parameter: a direct code string, a file path as a string, or a pathlib.Path object. The chunker automatically handles file reading when a path is provided.

```python
from pathlib import Path

# Assuming 'chunker' is an initialized instance of CodeChunker

# All of the following are valid:
chunks_from_string = chunker.chunk("def my_func():\n  return 1")
chunks_from_path_str = chunker.chunk("/path/to/your/code.py")
chunks_from_path_obj = chunker.chunk(Path("/path/to/your/code.py"))
```

--------------------------------

### Initialize and Batch Chunk Texts with PlainTextChunker

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/plain_text_chunker.md

Initializes PlainTextChunker and processes multiple texts in batches using the `batch_chunk` method. It demonstrates parameters like `max_sentences`, `max_tokens`, `n_jobs`, `on_errors`, and `show_progress`. The output shows the metadata and content of each generated chunk.

```python
from chunklet.plain_text_chunker import PlainTextChunker
from chunklet.token_counters import word_counter

# Assuming EN_TEXT, ES_TEXT, FR_TEXT are defined elsewhere

chunker = PlainTextChunker(token_counter=word_counter)

chunks = chunker.batch_chunk(
    texts=[EN_TEXT, ES_TEXT, FR_TEXT],
    max_sentences=5,
    max_tokens=20,
    n_jobs=2,                    # (1)!
    on_errors="raise",           # (2)!
    show_progress=True,          # (3)!
)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(f"Metadata: {chunk.metadata}")
    print(f"Content: {chunk.content}")
    print()

```

--------------------------------

### Python: Simple Token Counter Function

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

Defines a basic Python function to count tokens in a string by splitting it into words. This function serves as a simple example for customizing the tokenization process in CodeChunker.

```python
def simple_token_counter(text: str) -> int:
    """Simple Token Counter For Testing."""
    return len(text.split())
```

--------------------------------

### Code Chunking: Extracting Function Summaries

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Chunks code from a file, focusing on extracting only the summary (first line) of docstrings for each function. This is useful for generating concise API documentation or overviews.

```bash
chunklet chunk --code --source utils.py \
  --docstring-mode summary \
  --max-functions 1 \

```

--------------------------------

### JavaScript Utility and Processing Functions

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A JavaScript code example featuring a utility function `sanitizeInput` for cleaning strings and a main function `processArray` for calculating the sum of elements in an array. This snippet is suitable for batch processing.

```javascript
// Utility function
const sanitizeInput = (input) => {
    return input.trim().substring(0, 100);
};

// Main function with control flow
function processArray(data) {
    if (!data || data.length === 0) {
        return 0;
    }

    let total = 0;
    // Loop structure
    for (let i = 0; i < data.length; i++) {
        total += data[i];
    }
    return total;
}
```

--------------------------------

### Java DataProcessor Class for Chunking Output

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

A Java class designed for data processing, including a constructor, a getter for the source path, and a method to check if processing is possible. This serves as an example of Java code that might be chunked.

```java
package com.chunker.data;

public class DataProcessor {
    private String sourcePath;


    public DataProcessor(String path) {
        this.sourcePath = path;
    }


    public String getPath() {
        return this.sourcePath;
    }


    public boolean process() {
        if (this.sourcePath.isEmpty()) {
            return false;
        }

        return true;
    }
}
```

--------------------------------

### PlainTextChunker: Basic Text Chunking with Token Limits and Overlap (Python)

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

Demonstrates initializing PlainTextChunker and chunking text with specified token limits and overlap percentage for context preservation. It includes a simple token counter function and shows how to access chunk content and metadata like chunk number, token count, span, and language.

```python
from chunklet.plain_text_chunker import PlainTextChunker

# Define a simple token counter
def simple_token_counter(text: str) -> int:
    return len(text.split())

# Initialize chunker
chunker = PlainTextChunker(
    token_counter=simple_token_counter,
    verbose=False,
    continuation_marker="..."
)

# Chunk text with token limit
text = """Artificial intelligence is transforming technology.
Machine learning enables computers to learn from data.
Deep learning uses neural networks for complex patterns."""

chunks = chunker.chunk(
    text=text,
    max_tokens=15,
    overlap_percent=20,
    lang="en"
)

# Access chunk content and metadata
for i, chunk in enumerate(chunks):
    print(f"Chunk {i+1}:")
    print(f"  Content: {chunk.content}")
    print(f"  Chunk number: {chunk.metadata.chunk_num}")
    print(f"  Token count: {chunk.metadata.token_count}")
    print(f"  Span: {chunk.metadata.span}")  # (start, end) in original
    print(f"  Language: {chunk.metadata.lang}")
    print()

```

--------------------------------

### Use SentenceSplitter instead of preview_sentences in Python

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/migration.md

Replace calls to the removed 'preview_sentences' method with direct usage of the SentenceSplitter class. This refactoring separates sentence splitting logic for better modularity and flexibility in Python.

```python
from chunklet import Chunklet
chunker = Chunklet()
sentences, warnings = chunker.preview_sentences(text, lang="en")
```

```python
from chunklet import SentenceSplitter
splitter = SentenceSplitter()
sentences = splitter.split(text, lang="en")
```

--------------------------------

### Chunk Text by Sentences using PlainTextChunker (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/plain_text_chunker.md

This Python code demonstrates how to use the PlainTextChunker to divide a given text into smaller chunks, with each chunk containing a specified maximum number of sentences. It explains the initialization of the chunker, the `chunk` method parameters like `text`, `lang`, `max_sentences`, `overlap_percent`, and `offset`, and how to process the resulting chunks.

```python
from chunklet.plain_text_chunker import PlainTextChunker

chunker = PlainTextChunker()  # (1)!

chunks = chunker.chunk(
    text=text,
    lang="auto",             # (2)!
    max_sentences=2,
    overlap_percent=0,       # (3)!
    offset=0                 # (4)!
)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(f"Metadata: {chunk.metadata}")
    print(f"Content: {chunk.content}")
    print()

```

--------------------------------

### Verbatim Code Block in reStructuredText

Source: https://github.com/speedyk-005/chunklet-py/blob/main/samples/What_is_rst.rst

Demonstrates how to include a verbatim code block in reStructuredText using the '::' directive. This is useful for displaying source code snippets accurately, preserving formatting and special characters. No external dependencies are required.

```rst
.. code-block:: cpp

  int main ( int argc, char *argv[] ) {
      printf("Hello World\n");
      return 0;
  }

```

--------------------------------

### Verbose Debugging with Chunklet CLI

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Enables verbose logging for the chunklet CLI, useful for debugging a single file processing operation. It shows the detailed steps and information during the chunking process.

```bash
chunklet chunk --source problematic_file.txt \
  --max-tokens 100 \
  --verbose
```

--------------------------------

### Remove use_cache flag from PlainTextChunker in Python

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/migration.md

Adapt your Python code by removing the 'use_cache' argument from 'chunk' and 'batch_chunk' method calls. Chunklet-py v2 manages caching internally for optimized performance, simplifying the API.

```python
chunker = PlainTextChunker()
chunks = chunker.chunk(text, use_cache=False)
```

```python
chunker = PlainTextChunker()
chunks = chunker.chunk(text)
```

--------------------------------

### Python CodeChunker Initialization and Usage

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

This Python snippet shows how to import the CodeChunker class and the split_at function from more_itertools. It sets up the foundation for using these tools, likely for code processing tasks.

```python
from chunklet.code_chunker import CodeChunker
from more_itertools import split_at
```

--------------------------------

### Chunk Python Code by Lines, Tokens, and Functions

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

This Python snippet demonstrates how to chunk a Python code string using multiple constraints: maximum lines, maximum tokens, and maximum functions. This provides granular control over the output chunks. It requires the `chunker` library and a Python code string as input.

```python
from chunklet.code_chunker import CodeChunker

# Helper function
def simple_token_counter(text: str) -> int:
    """Simple Token Counter For Testing."""
    return len(text.split())

# Initialize the chunker
chunker = CodeChunker(token_counter=simple_token_counter)

PYTHON_CODE = """def example_function():
    pass

def another_function():
    print('hello')

if __name__ == "__main__":
    example_function()
    another_function()"""

chunks = chunker.chunk(
    PYTHON_CODE,
    max_lines=8,
    max_tokens=150,
    max_functions=1
)
```

--------------------------------

### Rename Chunklet class to PlainTextChunker in Python

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/migration.md

Update your Python code to import and instantiate the PlainTextChunker class instead of the previous Chunklet class. This change aims for clearer naming and paves the way for future chunker types.

```python
from chunklet import Chunklet
chunker = Chunklet()
```

```python
from chunklet import PlainTextChunker
chunker = PlainTextChunker()
```

--------------------------------

### Chunk Code Files (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Enables chunking specifically for code files using the `--code` flag. This command is designed to create semantically meaningful chunks from source code, preserving code structure and context for LLM analysis.

```bash
chunklet chunk --code --source main.py --destination code_chunks/ --max-tokens 512
```

--------------------------------

### Chunk code by maximum number of functions (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

This example shows how to chunk Python code, ensuring that each chunk contains at most one function. The CodeChunker is configured with `max_functions=1` to group related functions or code blocks logically. The output displays the segmented code and associated metadata for each chunk.

```python
chunks = chunker.chunk(
    PYTHON_CODE,
    max_functions=1,
)

for i, chunk in enumerate(chunks):
    print(f"--- Chunk {i+1} ---")
    print(f"Content:\n{chunk.content}")
    print("Metadata:")
    for k,v in chunk.metadata.items():
        print(f"{k}: {v}")
    print()
```

--------------------------------

### Chunk Document Files (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Processes various document file types (e.g., PDF, DOCX, HTML) into RAG-ready chunks using the DocumentChunker. This command utilizes the `--doc` flag and can handle multiple source files or directories, with options for metadata.

```bash
chunklet chunk --doc --source report.pdf --destination chunks/ --max-sentences 5 --metadata
```

--------------------------------

### Split Text from File to File (Bash)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/cli.md

Processes a text file, splits its content into sentences, and writes the output to a specified destination file. This command allows for batch processing of documents and specifying the language for accurate segmentation.

```bash
chunklet split --source my_novel_chapter.txt --destination sentences.txt --lang en
```

--------------------------------

### Register Custom Processor Without Decorator in Python

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/document_chunker.md

This example shows how to manually register a custom processor function using the `register()` method of `CustomProcessorRegistry`. This approach is useful for dynamic registration or when decorators are not preferred. It takes the processor function, desired file extensions, and an optional processor name.

```python
from chunklet.document_chunker import CustomProcessorRegistry

registry = CustomProcessorRegistry()

def my_other_processor(file_path: str) -> tuple[str, dict]:
    # ... your logic ...
    return "some text", {"source": file_path}

registry.register(my_other_processor, ".custom", name="MyOtherProcessor")
```

--------------------------------

### Split and Chunk Files with Chunklet CLI

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

Utilize the chunklet CLI for splitting files and chunking source code. Supports various options for controlling chunk size, content inclusion, and processing modes for single files, directories, or batches.

```bash
chunklet split \
  --source input.txt \
  --destination output.txt \
  --lang auto \
  --verbose
```

```bash
# Chunk single code file
chunklet chunk \
  --source app.py \
  --code \
  --max-tokens 500 \
  --max-lines 50 \
  --max-functions 3 \
  --destination chunks/

# Chunk with docstring control
chunklet chunk \
  --source src/ \
  --code \
  --max-tokens 400 \
  --docstring-mode summary \
  --include-comments \
  --strict \
  --destination output/

# Batch process multiple code files
chunklet chunk \
  --source file1.py \
  --source file2.js \
  --source file3.go \
  --code \
  --max-functions 2 \
  --n-jobs 4 \
  --show-progress \
  --metadata

# Process entire directory
chunklet chunk \
  --source src/ \
  --code \
  --max-lines 100 \
  --docstring-mode excluded \
  --on-errors skip \
  --verbose
```

--------------------------------

### Figure Inclusion in reStructuredText

Source: https://github.com/speedyk-005/chunklet-py/blob/main/samples/What_is_rst.rst

Shows how to include an image figure in reStructuredText documents. The '.. figure::' directive allows specifying the image file and its display properties like width. Supported formats include PNG, suitable for both HTML and LaTeX (PDF).

```rst
.. figure:: image.png
   :width: 300pt


   The magnetisation in a small ferromagnetic disk. The diametre is of the order of 120 nanometers and the material is Ni20Fe80. Png is a file format that is both acceptable for html pages as well as for (pdf)latex.

```

--------------------------------

### SentenceSplitter: Multilingual Sentence Segmentation

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

This code snippet demonstrates the SentenceSplitter's ability to automatically detect and split sentences in different languages. It shows examples for English and French, including explicit language specification and confidence scores for language detection.

```python
from chunklet.sentence_splitter import SentenceSplitter

# Initialize splitter
splitter = SentenceSplitter(verbose=True)

# Auto-detect language and split
english_text = "Hello world. How are you today? I hope you're doing well!"
sentences_en = splitter.split(text=english_text, lang="auto")

print("English sentences:")
for i, sentence in enumerate(sentences_en, 1):
    print(f"  {i}. {sentence}")

# Explicit language specification
french_text = "Bonjour! Comment allez-vous? J'espère que vous allez bien."
sentences_fr = splitter.split(text=french_text, lang="fr")

print("\nFrench sentences:")
for i, sentence in enumerate(sentences_fr, 1):
    print(f"  {i}. {sentence}")

# Language detection
detected_lang, confidence = splitter.detected_top_language(french_text)
print(f"\nDetected language: {detected_lang} (confidence: {confidence:.2%})")
```

--------------------------------

### Python and C#: Code Chunking with Separators

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/code_chunker.md

Demonstrates batch chunking of multiple source code snippets using CodeChunker. It shows how to define sources, set a maximum token limit, and use a custom separator. The output is then processed by splitting at these separators.

```python
SIMPLE_SOURCES = [
    # Python: Simple Function Definition Boundary
    '''
def greet_user(name):
    """Returns a simple greeting string."""
    message = "Welcome back, " + name
    return message
''',
    # C#: Simple Method and Class Boundary
    '''
public class Utility
{
    // C# Method
    public int Add(int x, int y)
    {
        int sum = x + y;
        return sum;
    }
}
'''
]

chunker = CodeChunker(token_counter=simple_token_counter)
custom_separator = "---END_OF_SOURCE---"

chunks_with_separators = chunker.batch_chunk(
    sources=SIMPLE_SOURCES,
    max_tokens=20,
    separator=custom_separator,
)

chunk_groups = split_at(chunks_with_separators, lambda x: x == custom_separator)
# Process the results using split_at
for i, code_chunks in enumerate(chunk_groups):
    if code_chunks: # (1)!
        print(f"--- Chunks for Document {i+1} ---")
        for chunk in code_chunks:
            print(f"Content:\n {chunk.content}\n")
            print(f"Metadata: {chunk.metadata}")
        print()
```

--------------------------------

### Batch Process Code Files with Chunking

Source: https://context7.com/speedyk-005/chunklet-py/llms.txt

Iterates through source files, performing batch chunking with specified parameters for maximum lines, tokens, docstring mode, and comment inclusion. Handles errors by skipping and shows progress.

```python
for chunk in chunker.batch_chunk(
    sources=source_files,
    max_lines=100,
    max_tokens=600,
    docstring_mode="all",
    include_comments=False,
    strict=False,
    n_jobs=4,
    show_progress=True,
    on_errors="skip"
):
    print(f"File: {chunk.metadata.source}")
    print(f"Language: {Path(chunk.metadata.source).suffix}")
    print(f"Lines: {chunk.metadata.start_line}-{chunk.metadata.end_line}")
    print(f"Tree: {chunk.metadata.tree}")
    print(f"Chunk: {chunk.content[:100]}...")
    print()
```

--------------------------------

### Batch Process Multiple Texts Efficiently (Python)

Source: https://github.com/speedyk-005/chunklet-py/blob/main/docs/getting-started/programmatic/plain_text_chunker.md

Utilizes the `batch_chunk` method for parallel processing of multiple texts, returning a generator for memory efficiency. Supports arguments similar to `chunk` and additional batching parameters. Example demonstrates processing English, Spanish, and French texts.

```python
from chunklet.plain_text_chunker import PlainTextChunker

def word_counter(text: str) -> int:
    return len(text.split())

EN_TEXT = "This is the first document. It has multiple sentences for chunking. Here is the second document. It is a bit longer to test batch processing effectively. And this is the third document. Short and sweet, but still part of the batch. The fourth document. Another one to add to the collection for testing purposes."
ES_TEXT = "Este es el primer documento. Contiene varias frases para la segmentación de texto. El segundo ejemplo es más extenso. Queremos probar el procesamiento en diferentes idiomas."
FR_TEXT = "Ceci est le premier document. Il est essentiel pour l'évaluation multilingue. Le deuxième document est court mais important. La variation est la clé."

# Example usage (assuming chunker is initialized elsewhere)
# results = chunker.batch_chunk(texts=[EN_TEXT, ES_TEXT, FR_TEXT], token_counter=word_counter)
```