Gemini Agent

https://docs.rs/shimmytok/latest/
Admin
This project implements a versatile AI agent designed to interact with different language models,...
Tokens:13,605
Snippets:91
Trust Score:9.1
Update:4 months ago
Show doc for...
Tokens:
Raw
### Quick Start: Load and Use Tokenizer

Source: https://docs.rs/shimmytok/latest/shimmytok

Demonstrates how to load a tokenizer from a GGUF file, encode text into tokens, and decode tokens back into text. Also shows streaming token-by-token decoding.

```rust
use shimmytok::Tokenizer;

// Load tokenizer from any GGUF model
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// Encode text to tokens
let tokens = tokenizer.encode("Hello, world!", true)?;

// Decode back to text
let text = tokenizer.decode(&tokens, true)?;

// Stream tokens one at a time (for LLM generation)
for token_id in &tokens {
    print!("{}", tokenizer.decode_single(*token_id, false)?);
}
```

--------------------------------

### Quick Start: Load, Encode, Decode with shimmytok

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Demonstrates the basic usage of the shimmytok library to load a tokenizer from a GGUF file, encode text into token IDs, and decode token IDs back into text. It also shows how to stream tokens individually.

```rust
use shimmytok::Tokenizer;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
// Load tokenizer from any GGUF model
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// Encode text to tokens
let tokens = tokenizer.encode("Hello, world!", true)?;

// Decode back to text
let text = tokenizer.decode(&tokens, true)?;

// Stream tokens one at a time (for LLM generation)
for token_id in &tokens {
    print!("{}", tokenizer.decode_single(*token_id, false)?) ;
}
# Ok(())
# }

```

--------------------------------

### Get or Compile Regex Patterns

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

Retrieves pre-compiled regex patterns from a cache or compiles them if not found. Handles mutex locking for thread-safe access to the cache.

```Rust
fn get_regexes(&self, pre_type: &str) -> Result<Vec<fancy_regex::Regex>, String> {
        let mut cache = self
            .regex_cache
            .lock()
            .map_err(|e| format!("Mutex lock failed: {e}"))?;

        if let Some(regexes) = cache.get(pre_type) {
            return Ok(regexes.clone());
        }

        let patterns = Self::get_patterns(pre_type);
        let mut regexes = Vec::new();
        for pattern in patterns {
            let regex = fancy_regex::Regex::new(pattern)
                .map_err(|e| format!("Failed to compile regex for '{pre_type}': {e}"))?;
            regexes.push(regex);
        }

        cache.insert(pre_type.to_string(), regexes.clone());
        Ok(regexes)
    }
```

--------------------------------

### Get Model Type

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Retrieves the model type identifier from the GGUF metadata. This is useful for identifying the underlying model architecture.

```rust
use shimmytok::Tokenizer;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
println!("Model type: {}", tokenizer.model_type());
# Ok(())
# }

```

--------------------------------

### Get UTF-8 Codepoint Length

Source: https://docs.rs/shimmytok/latest/src/shimmytok/ugm.rs.html

Determines the byte length of a UTF-8 codepoint based on its first byte. This is a utility function for correctly parsing UTF-8 encoded strings.

```rust
fn utf8_cp_len(first: u8) -> usize {
    match first {
        0x00..=0x7F => 1,
        0xC0..=0xDF => 2,
        0xE0..=0xEF => 3,
        _ => 4,
    }
}
```

--------------------------------

### Get Token Type

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Retrieves the type of a given token ID. Returns TokenType::Undefined if the ID is out of bounds.

```rust
use shimmytok::Tokenizer;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
let token_type = tokenizer.token_type(1); // BOS token
println!("Token type: {:?}", token_type);
# Ok(())
# }
```

--------------------------------

### Tokenizer Usage

Source: https://docs.rs/shimmytok/latest/shimmytok/index.html

Demonstrates the basic usage of the Tokenizer, including loading from a GGUF file, encoding text to tokens, and decoding tokens back to text.

```APIDOC
## Tokenizer

### Description

The main tokenizer interface for encoding and decoding text.

### Methods

- **`Tokenizer::from_gguf_file(path: &str) -> Result<Tokenizer, Error>`**
  Loads a tokenizer from a GGUF model file.

- **`tokenizer.encode(text: &str, add_special_tokens: bool) -> Result<Vec<TokenId>, Error>`**
  Encodes a given text string into a vector of token IDs.

- **`tokenizer.decode(tokens: &[TokenId], skip_special_tokens: bool) -> Result<String, Error>`**
  Decodes a slice of token IDs back into a string.

- **`tokenizer.decode_single(token_id: TokenId, skip_special_tokens: bool) -> Result<String, Error>`**
  Decodes a single token ID into a string.

### Type Aliases

- **`TokenId`**: Type alias for token IDs.

### Constants

- **`MAX_INPUT_SIZE`**: Maximum input text size in bytes (10MB).
- **`MAX_OUTPUT_TOKENS`**: Maximum output tokens (1M tokens max).

### Example
```rust
use shimmytok::Tokenizer;

// Load tokenizer from any GGUF model
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// Encode text to tokens
let tokens = tokenizer.encode("Hello, world!", true)?;

// Decode back to text
let text = tokenizer.decode(&tokens, true)?;

// Stream tokens one at a time (for LLM generation)
for token_id in &tokens {
    print!("{}", tokenizer.decode_single(*token_id, false)?);
}
```
```

--------------------------------

### Tokenizer Usage

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Demonstrates the basic usage of the Tokenizer, including loading from a GGUF file, encoding text to tokens, and decoding tokens back to text. It also shows how to stream tokens individually.

```APIDOC
## Tokenizer

### Description

The `Tokenizer` struct provides the main interface for interacting with the shimmytok library. It allows loading tokenizer configurations from GGUF files and performing encoding and decoding operations.

### Methods

- **`Tokenizer::from_gguf_file(path: &str) -> Result<Self, Error>`**: Loads a tokenizer from a specified GGUF model file.
- **`tokenizer.encode(text: &str, add_special_tokens: bool) -> Result<Vec<TokenId>, Error>`**: Encodes a given string into a vector of token IDs. The `add_special_tokens` flag determines if special tokens (like BOS/EOS) should be added.
- **`tokenizer.decode(tokens: &[TokenId], skip_special_tokens: bool) -> Result<String, Error>`**: Decodes a slice of token IDs back into a string. The `skip_special_tokens` flag controls whether special tokens are included in the output.
- **`tokenizer.decode_single(token_id: TokenId, include_special_text: bool) -> Result<String, Error>`**: Decodes a single token ID into its string representation. `include_special_text` determines if special token text should be emitted.

### Example

```rust
use shimmytok::Tokenizer;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
// Load tokenizer from any GGUF model
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// Encode text to tokens
let tokens = tokenizer.encode("Hello, world!", true)?;

// Decode back to text
let text = tokenizer.decode(&tokens, true)?;

// Stream tokens one at a time (for LLM generation)
for token_id in &tokens {
    print!("{}", tokenizer.decode_single(*token_id, false)?);
}
# Ok(())
# }
```
```

--------------------------------

### Batch Encoding with Rayon Parallelism

Source: https://docs.rs/shimmytok/latest/shimmytok

Shows how to efficiently encode multiple text inputs in parallel using the `encode_batch` method, leveraging Rayon for performance.

```rust
// Parallel encoding with Rayon
let texts = vec!["Hello", "World", "Rust"];
let batched = tokenizer.encode_batch(&texts, true)?;
```

--------------------------------

### Batch Encoding with Rayon Parallelism

Source: https://docs.rs/shimmytok/latest/index.html

Shows how to efficiently encode multiple text strings in parallel using the `encode_batch` method, leveraging Rayon for performance. This is useful for processing large datasets.

```rust
// Parallel encoding with Rayon
let texts = vec!["Hello", "World", "Rust"];
let batched = tokenizer.encode_batch(&texts, true?);
```

--------------------------------

### Load GGUF Metadata from File

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Opens a GGUF file and reads its magic number and version to ensure it's a valid GGUF file. Supports GGUF versions 2 and 3.

```rust
use crate::{Error, TokenType};
use std::collections::HashMap;
use std::fs::File;
use std::io::{BufReader, Read};
use std::path::Path;

pub fn load_metadata<P: AsRef<Path>>(path: P) -> Result<GGUFMetadata, Error> {
    let file = File::open(path)?;
    let mut reader = BufReader::new(file);

    // Track total string allocation to prevent OOM (Issue R2#3)
    let mut total_string_bytes: usize = 0;

    // Read magic
    let mut magic = [0u8; 4];
    reader.read_exact(&mut magic)?;

    if &magic != b"GGUF" {
        return Err(Error::InvalidMetadata("Not a GGUF file".into()));
    }

    // Read version
    let version = read_u32(&mut reader)?;
    // Support both version 2 and 3 (GPT-2 uses v3)
    if !(2..=3).contains(&version) {
        return Err(Error::InvalidMetadata(format!(
            "Unsupported GGUF version: {version} (only versions 2-3 are supported)"
        )));
    }

    // Read counts
    let _tensor_count = read_u64(&mut reader)?;
    let metadata_count = read_u64(&mut reader)?;

    // Read metadata key-value pairs
    let mut kv_pairs = HashMap::new();
    for _ in 0..metadata_count {
        let key = read_string(&mut reader, &mut total_string_bytes)?;
        let value = read_value(&mut reader, &mut total_string_bytes)?;
        kv_pairs.insert(key, value);
    }

    // Extract tokenizer metadata
    let tokens = match kv_pairs.get("tokenizer.ggml.tokens") {
        Some(Value::StringArray(arr)) => arr.clone(),
        _ => {
            return Err(Error::InvalidMetadata(
                "Missing tokenizer.ggml.tokens".into(),
            ))
        }
    };

    let scores = match kv_pairs.get("tokenizer.ggml.scores") {
        Some(Value::F32Array(arr)) => Some(arr.clone()),
        _ => None,
    };

    let token_types = match kv_pairs.get("tokenizer.ggml.token_type") {
        Some(Value::I32Array(arr)) => Some(arr.iter().map(|&t| TokenType::from(t)).collect()),
        _ => None,
    };

    let model_type = match kv_pairs.get("tokenizer.ggml.model") {
        Some(Value::String(s)) => s.clone(),
        _ => "llama".to_string(), // Default
    };

    let pre_type = match kv_pairs.get("tokenizer.ggml.pre") {
        Some(Value::String(s)) => Some(s.clone()),
        _ => None,
    };

    // Special tokens
    let bos_token_id = match kv_pairs.get("tokenizer.ggml.bos_token_id") {
        Some(Value::U32(v)) => Some(*v),
        _ => None,
    };

    let eos_token_id = match kv_pairs.get("tokenizer.ggml.eos_token_id") {
        Some(Value::U32(v)) => Some(*v),
        _ => None,
    };

    // ... (rest of the function)
    // Placeholder for the rest of the GGUF metadata extraction
    Ok(GGUFMetadata {
        tokens,
        scores,
        token_types,
        model_type,
        pre_type,
        bos_token_id,
        eos_token_id,
        unk_token_id: None, // Placeholder
        pad_token_id: None, // Placeholder
        eot_token_id: None, // Placeholder
        eog_token_id: None, // Placeholder
        sep_token_id: None, // Placeholder
        nl_token_id: None, // Placeholder
        fim_pre_token_id: None, // Placeholder
        fim_suf_token_id: None, // Placeholder
        fim_mid_token_id: None, // Placeholder
        mask_token_id: None, // Placeholder
        add_bos_token: None, // Placeholder
        add_eos_token: None, // Placeholder
        add_space_prefix: None, // Placeholder
        clean_spaces: None, // Placeholder
        remove_extra_whitespaces: None, // Placeholder
        escape_whitespaces: None, // Placeholder
        treat_whitespace_as_suffix: None, // Placeholder
        merges: None, // Placeholder
    })
}

// Dummy implementations for helper functions and types used in the snippet
struct GGUFMetadata {
    tokens: Vec<String>,
    scores: Option<Vec<f32>>,
    token_types: Option<Vec<TokenType>>,
    model_type: String,
    pre_type: Option<String>,
    bos_token_id: Option<u32>,
    eos_token_id: Option<u32>,
    unk_token_id: Option<u32>,
    pad_token_id: Option<u32>,
    eot_token_id: Option<u32>,
    eog_token_id: Option<u32>,
    sep_token_id: Option<u32>,
    nl_token_id: Option<u32>,
    fim_pre_token_id: Option<u32>,
    fim_suf_token_id: Option<u32>,
    fim_mid_token_id: Option<u32>,
    mask_token_id: Option<u32>,
    add_bos_token: Option<bool>,
    add_eos_token: Option<bool>,
    add_space_prefix: Option<bool>,
    clean_spaces: Option<bool>,
    remove_extra_whitespaces: Option<bool>,
    escape_whitespaces: Option<bool>,
    treat_whitespace_as_suffix: Option<bool>,
    merges: Option<Vec<(String, String)>>,
}

enum TokenType { Basic, Merged } // Example enum
impl From<i32> for TokenType { fn from(i: i32) -> Self { if i == 0 { TokenType::Basic } else { TokenType::Merged } } } // Example impl

enum Value { String(String), StringArray(Vec<String>), F32Array(Vec<f32>), I32Array(Vec<i32>), U32(u32) } // Example enum

fn read_u32(reader: &mut BufReader<File>) -> Result<u32, Error> { Ok(0) } // Dummy
fn read_u64(reader: &mut BufReader<File>) -> Result<u64, Error> { Ok(0) } // Dummy
fn read_string(reader: &mut BufReader<File>, total_string_bytes: &mut usize) -> Result<String, Error> { Ok("".to_string()) } // Dummy
fn read_value(reader: &mut BufReader<File>, total_string_bytes: &mut usize) -> Result<Value, Error> { Ok(Value::String("".to_string())) } // Dummy

// Dummy Error enum and impl
#[derive(Debug)]
enum Error { InvalidMetadata(String) }
impl From<std::io::Error> for Error { fn from(_: std::io::Error) -> Self { Error::InvalidMetadata("IO Error".into()) } }

```

--------------------------------

### Batch Encoding with shimmytok

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Shows how to efficiently encode multiple text strings in parallel using the `encode_batch` method, leveraging Rayon for performance.

```rust
# use shimmytok::Tokenizer;
# fn main() -> Result<(), Box<dyn std::error::Error>> {
# let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
// Parallel encoding with Rayon
let texts = vec!["Hello", "World", "Rust"];
let batched = tokenizer.encode_batch(&texts, true)?;
# Ok(())
# }

```

--------------------------------

### DeepSeek-LLM Pre-tokenization Patterns

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

A sequence of regex patterns for pre-tokenization in DeepSeek-LLM models. These patterns are applied sequentially to refine tokenization boundaries.

```Rust
r"[\r\n]"
```

```Rust
r"\s?\p{L}+"
```

```Rust
r"\s?[!-/:-~！-／：-～'-‟　-。]+"
```

```Rust
r"\s+$"
```

```Rust
r"[一-龥ࠀ-一가-퟿]+"
```

```Rust
r"\p{N}+"
```

--------------------------------

### Pre-tokenize Text with Single Regex

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

Fast path for pre-tokenizing text when only a single regex pattern is used. It directly applies the regex to find all matches.

```Rust
return Ok(regexes[0]
                .find_iter(text)
                .filter_map(std::result::Result::ok)
                .map(|m| m.as_str().to_string())
                .collect());
```

--------------------------------

### Encoding and Decoding Options

Source: https://docs.rs/shimmytok/latest/index.html

Options structs to control the behavior of encoding and decoding operations, ensuring compatibility with llama.cpp.

```APIDOC
## Structs

### EncodeOptions

Options for encoding text (llama.cpp parity)

### DecodeOptions

Options for decoding tokens (llama.cpp parity)
```

--------------------------------

### Batch Encoding

Source: https://docs.rs/shimmytok/latest/shimmytok/index.html

Illustrates how to perform parallel encoding of multiple text strings using Rayon.

```APIDOC
## Batch Encoding

### Description

Provides a method for parallel encoding of multiple text strings using Rayon for performance.

### Method

- **`tokenizer.encode_batch(texts: &[&str], add_special_tokens: bool) -> Result<Vec<Vec<TokenId>>, Error>`**
  Encodes a slice of text strings into a vector of token ID vectors in parallel.

### Example
```rust
// Parallel encoding with Rayon
let texts = vec!["Hello", "World", "Rust"];
let batched = tokenizer.encode_batch(&texts, true)?;
```
```

--------------------------------

### DeepSeek-Coder Pre-tokenization Patterns

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

A sequence of regex patterns for pre-tokenization in DeepSeek-Coder models. These patterns are applied sequentially.

```Rust
r"[\r\n]"
```

```Rust
r"\s?\p{L}+"
```

--------------------------------

### Tokenizer Interface

Source: https://docs.rs/shimmytok/latest/index.html

The main Tokenizer interface allows for loading tokenizers from GGUF files and performing encoding and decoding operations.

```APIDOC
## Structs

### Tokenizer

Main tokenizer interface for encoding and decoding text.

#### Methods

- **`from_gguf_file(path: &str) -> Result<Self, Error>`**
  Loads a tokenizer from a GGUF model file.

- **`encode(text: &str, add_special_tokens: bool) -> Result<Vec<TokenId>, Error>`**
  Encodes a given text string into a vector of token IDs.

- **`decode(tokens: &[TokenId], special_tokens: bool) -> Result<String, Error>`**
  Decodes a slice of token IDs back into a string.

- **`decode_single(token_id: TokenId, special_tokens: bool) -> Result<String, Error>`**
  Decodes a single token ID into its string representation.

- **`encode_batch(texts: &[&str], add_special_tokens: bool) -> Result<Vec<Vec<TokenId>>, Error>`**
  Encodes a batch of text strings into vectors of token IDs in parallel using Rayon.
```

--------------------------------

### DecodeOptions

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Explains the `DecodeOptions` struct for customizing the token decoding process, including skipping special tokens and handling whitespace.

```APIDOC
## DecodeOptions

### Description

`DecodeOptions` struct provides options to customize the token decoding behavior, mirroring llama.cpp's decoding functionalities.

### Fields

- **`skip_special_tokens`** (*bool*) - Required - If true, special tokens (like BOS, EOS) are omitted from the decoded output string.
- **`lstrip`** (*bool*) - Required - If true, leading whitespace is stripped from each token piece during decoding.
- **`include_special_text`** (*bool*) - Required - If false, special or control tokens are emitted as empty strings instead of their textual representation.

### Constructors

- **`DecodeOptions::with_skip_special(skip_special_tokens: bool) -> Self`**: Creates options focusing on the `skip_special_tokens` setting.
- **`DecodeOptions::new(skip_special_tokens: bool, lstrip: bool, include_special_text: bool) -> Self`**: Creates options with all decoding parameters.

### Default

Provides default options where `skip_special_tokens` is false, `lstrip` is false, and `include_special_text` is true.
```

--------------------------------

### Pre-tokenize Text with Multiple Regex Patterns

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

Refines text fragmentation using multiple regex patterns sequentially, preserving both matches and non-matching gaps. This mimics llama.cpp's offset-based approach.

```Rust
let mut offsets: Vec<(usize, usize)> = vec![(0, text.len())];

        for regex in regexes {
            let mut new_offsets = Vec::new();

            for (start, end) in offsets {
                let fragment = &text[start..end];

                // Collect all matches in this fragment
                let matches: Vec<_> = regex
                    .find_iter(fragment)
                    .filter_map(std::result::Result::ok)
                    .collect();

                if matches.is_empty() {
                    // No matches - keep the original offset unchanged
                    new_offsets.push((start, end));
                } else {
                    // Split into matched and unmatched regions
                    let mut last_pos = 0;

                    for m in matches {
                        // Add unmatched gap before this match
                        if m.start() > last_pos {
                            new_offsets.push((start + last_pos, start + m.start()));
                        }

                        // Add the match
                        new_offsets.push((start + m.start(), start + m.end()));
                        last_pos = m.end();
                    }

                    // Add final unmatched portion
                    if last_pos < fragment.len() {
                        new_offsets.push((start + last_pos, end));
                    }
                }
            }

            offsets = new_offsets;
        }

        // Convert offsets to strings
        Ok(offsets
            .iter()
            .map(|(start, end)| text[*start..*end].to_string())
            .collect())
```

--------------------------------

### BPE Tokenization Logic

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

Implements the core BPE tokenization algorithm for a given text fragment. It initializes symbols from characters, builds a work queue of potential bigram merges, and iteratively applies merges based on rank until no more merges are possible. This function is a direct port of llama.cpp's `llm_tokenizer_bpe_session::tokenize`.

```rust
fn bpe_fragment(&self, text: &str, vocab: &Vocabulary) -> Result<Vec<TokenId>, crate::Error> {
        // Text is a single word from regex pre-tokenization
        // llama.cpp initializes with UTF-8 characters as symbols

        // Build merge rank map: (left, right) -> rank
        let merge_ranks: HashMap<(String, String), usize> = vocab
            .get_merges()
            .iter()
            .enumerate()
            .map(|(rank, (l, r))| ((l.clone(), r.clone()), rank))
            .collect();

        // Split into UTF-8 characters as initial symbols
        let char_indices: Vec<(usize, char)> = text.char_indices().collect();
        let mut symbols: Vec<Symbol> = Vec::with_capacity(char_indices.len());

        for (i, (byte_pos, _ch)) in char_indices.iter().enumerate() {
            let next_byte_pos = if i + 1 < char_indices.len() {
                char_indices[i + 1].0
            } else {
                text.len()
            };

            symbols.push(Symbol {
                text_start: *byte_pos,
                text_len: next_byte_pos - byte_pos,
                prev: if i == 0 { None } else { Some(i - 1) },
                next: if i + 1 < char_indices.len() {
                    Some(i + 1)
                } else {
                    None
                },
            });
        }

        if symbols.is_empty() {
            return Ok(Vec::new());
        }

        // Build initial work queue with all adjacent bigrams
        let mut work_queue = BinaryHeap::new();
        for i in 0..symbols.len().saturating_sub(1) {
            if let Some(next) = symbols[i].next {
                self.try_add_bigram(i, next, text, &symbols, &merge_ranks, &mut work_queue);
            }
        }

        // Apply merges in priority order
        while let Some(bigram) = work_queue.pop() {
            let left = bigram.left;
            let right = bigram.right;

            // Validate bigram is still valid
            if symbols[left].text_len == 0
                || symbols[right].text_len == 0
                || symbols[left].next != Some(right)
            {
                continue;
            }

            // CRITICAL: Validate that current symbol texts match the merge rule
            // Symbols may have changed since bigram was added to queue
            let left_text =
                &text[symbols[left].text_start..symbols[left].text_start + symbols[left].text_len];
            let right_text = &text
                [symbols[right].text_start..symbols[right].text_start + symbols[right].text_len];

            if let Some(&expected_rank) = 
                merge_ranks.get(&(left_text.to_string(), right_text.to_string()))
            {
                if expected_rank == bigram.rank {
                    // Merge: extend left symbol to include right symbol
                    symbols[left].text_len += symbols[right].text_len;
                    symbols[right].text_len = 0; // Mark right as deleted

                    // Update linked list
                    symbols[left].next = symbols[right].next;
                    if let Some(next) = symbols[right].next {
                        symbols[next].prev = Some(left);
                    }

                    // Add new potential merges with neighbors
                    if let Some(prev) = symbols[left].prev {
                        self.try_add_bigram(
                            prev,
                            left,
                            text,
                            &symbols,
                            &merge_ranks,
                            &mut work_queue,
                        );
                    }
                    if let Some(next) = symbols[left].next {
                        self.try_add_bigram(
                            left,
                            next,
                            text,
                            &symbols,
                            &merge_ranks,
                            &mut work_queue,
                        );
                    }
                }
            }
        }

        // Convert final symbols to token IDs
        let mut token_ids = Vec::with_capacity(symbols.len());
        for symbol in symbols.iter().filter(|s| s.text_len > 0) {
            let symbol_text = &text[symbol.text_start..symbol.text_start + symbol.text_len];
            token_ids.push(vocab.get_token_id(symbol_text).unwrap_or_else(|| vocab.get_byte_fallback_token_id(symbol_text.as_bytes()[0])));
        }

        Ok(token_ids)
    }
```

--------------------------------

### Test Trie Insertion and Traversal

Source: https://docs.rs/shimmytok/latest/src/shimmytok/ugm.rs.html

Unit tests for the `NaiveTrie` data structure, demonstrating insertion of a key-value pair and successful traversal to retrieve the value.

```rust
#[test]
    fn test_trie_insert_and_traverse() {
        let mut trie = NaiveTrie::new();
        trie.insert("hello", Some(42));

        let node1 = trie.traverse(0, b'h').unwrap();
        let node2 = trie.traverse(node1, b'e').unwrap();
        let node3 = trie.traverse(node2, b'l').unwrap();
        let node4 = trie.traverse(node3, b'l').unwrap();
        let node5 = trie.traverse(node4, b'o').unwrap();

        assert_eq!(trie.value(node5), Some(42));
    }
```

--------------------------------

### Constants

Source: https://docs.rs/shimmytok/latest/index.html

Constants defining limits for input size and output tokens to prevent memory exhaustion.

```APIDOC
## Constants

### MAX_INPUT_SIZE

Maximum input text size in bytes (10MB) - Issue R4#2

### MAX_OUTPUT_TOKENS

Maximum output tokens (1M tokens max) - prevents memory exhaustion
```

--------------------------------

### EncodeOptions

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Details the `EncodeOptions` struct used to configure the text encoding process, including options for special tokens and parsing special token strings.

```APIDOC
## EncodeOptions

### Description

`EncodeOptions` struct allows fine-grained control over the text encoding process, ensuring compatibility with llama.cpp's behavior regarding special tokens.

### Fields

- **`add_special_tokens`** (*bool*) - Required - Determines if BOS/EOS tokens should be added based on the model's configuration.
- **`parse_special`** (*bool*) - Required - If true, special token strings within the input text (e.g., `<|eot_id|>`) are parsed and converted into their corresponding token IDs.

### Constructors

- **`EncodeOptions::with_special_tokens(add_special_tokens: bool) -> Self`**: Creates options with only the `add_special_tokens` setting.
- **`EncodeOptions::with_parse_special(add_special_tokens: bool, parse_special: bool) -> Self`**: Creates options with both `add_special_tokens` and `parse_special` settings.

### Default

Provides default options where `add_special_tokens` is false and `parse_special` is false.
```

--------------------------------

### Placeholder Normalization

Source: https://docs.rs/shimmytok/latest/src/shimmytok/ugm.rs.html

A placeholder function for text normalization. For exact parity with llama.cpp, this function should be routed through GGUF charsmap/XCDA parsing.

```rust
fn normalize_ugm(text: &str) -> String {
    text.to_string()
}
```

--------------------------------

### Apply Llama.cpp Clean Spaces Post-processing

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Applies text cleaning passes similar to llama.cpp's detokenize function, including removing spaces before punctuation and merging contractions.

```rust
fn apply_clean_spaces(text: &str) -> String {
    let mut chars: Vec<char> = text.chars().collect();
    if chars.is_empty() {
        return String::new();
    }

    // Pass 1: Remove space before punctuation ?!.,
    let mut i = 1;
    while i < chars.len() {
        // ... implementation continues ...
        i += 1;
    }
    // ... other passes would follow ...
    chars.into_iter().collect()
}
```

--------------------------------

### Test UTF-8 Codepoint Length Calculation

Source: https://docs.rs/shimmytok/latest/src/shimmytok/ugm.rs.html

Unit tests for the `utf8_cp_len` function, verifying its correctness for various byte ranges representing different UTF-8 codepoint lengths.

```rust
#[test]
    fn test_utf8_cp_len() {
        assert_eq!(utf8_cp_len(b'a'), 1);
        assert_eq!(utf8_cp_len(0xC2), 2); // Start of 2-byte sequence
        assert_eq!(utf8_cp_len(0xE0), 3); // Start of 3-byte sequence
        assert_eq!(utf8_cp_len(0xF0), 4); // Start of 4-byte sequence
    }
```

--------------------------------

### decode_with_options

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Decodes a sequence of token IDs back into text using specified decoding options. This method allows for fine-grained control over the decoding process, including skipping special tokens, stripping leading whitespace, and controlling the inclusion of special text.

```APIDOC
## decode_with_options

### Description
Decodes a sequence of token IDs back into text with full options.

### Method
`decode_with_options(&self, tokens: &[TokenId], options: &DecodeOptions) -> Result<String, Error>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* `tokens` - Slice of token IDs to decode.
* `options` - Decoding options (skip_special_tokens, lstrip, include_special_text).

### Returns
Returns the decoded text as a `String` on success, or an `Error` if decoding fails.

### Example
```no_run
use shimmytok::{Tokenizer, DecodeOptions};

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
let tokens = vec![15043, 3186];

// Decode with lstrip to remove leading whitespace from tokens
let opts = DecodeOptions::new(true, true, false);
let text = tokenizer.decode_with_options(&tokens, &opts)?;
# Ok(())
# }
```
```

--------------------------------

### Read String with Size Limits

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Reads a string from a reader, enforcing maximum string size and total string data limits. It also checks for UTF-8 validity and potential overflows.

```Rust
fn read_string<R: Read>(reader: &mut R, total_bytes: &mut usize) -> Result<String, Error> {
    const MAX_STRING_SIZE: usize = 1024 * 1024; // 1MB max per string
    const MAX_TOTAL_STRING_DATA: usize = 100 * 1024 * 1024; // 100MB total
    let len_u64 = read_u64(reader)?;

    // Prevent truncation on 32-bit systems (Issue R4#12)
    if len_u64 > usize::MAX as u64 {
        return Err(Error::InvalidMetadata(format!(
            "String length {len_u64} exceeds platform limit"
        )));
    }
    let len = len_u64 as usize;

    if len > MAX_STRING_SIZE {
        return Err(Error::InvalidMetadata(format!(
            "String too large: {len} bytes (max: {MAX_STRING_SIZE})"
        )));
    }

    // Check total allocation with overflow protection (Issue R3#2)
    *total_bytes = total_bytes
        .checked_add(len)
        .ok_or_else(|| Error::InvalidMetadata("Total string data overflow".to_string()))?;
    if *total_bytes > MAX_TOTAL_STRING_DATA {
        return Err(Error::InvalidMetadata(format!(
            "Total string data too large: {} bytes (max: {})",
            *total_bytes, MAX_TOTAL_STRING_DATA
        )));
    }

    let mut buf = vec![0u8; len];
    reader.read_exact(&mut buf)?;
    String::from_utf8(buf).map_err(|e| Error::InvalidMetadata(format!("Invalid UTF-8: {e}")))
}
```

--------------------------------

### Decode Token IDs with Options

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Decodes a sequence of token IDs into text using specified decoding options. Supports skipping special tokens, stripping leading whitespace, and controlling the inclusion of special text.

```rust
use shimmytok::{Tokenizer, DecodeOptions};

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
let tokens = vec![15043, 3186];

// Decode with lstrip to remove leading whitespace from tokens
let opts = DecodeOptions::new(true, true, false);
let text = tokenizer.decode_with_options(&tokens, &opts)?;
# Ok(())
# }

```

--------------------------------

### Llama-3 Pre-tokenization Pattern

Source: https://docs.rs/shimmytok/latest/src/shimmytok/bpe.rs.html

This regex pattern is used for pre-tokenization in Llama-3 family models. It handles contractions, words, numbers, and whitespace.

```Rust
r"(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^
\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+"
```

--------------------------------

### Decode Token IDs to Text

Source: https://docs.rs/shimmytok/latest/src/shimmytok/plamo2.rs.html

The `decode` method reconstructs the original text from a sequence of token IDs. It handles both regular vocabulary tokens and special byte tokens, mapping them back to their corresponding byte representations. The resulting bytes are then converted into a UTF-8 string.

```APIDOC
## decode

### Description
Decodes a slice of token IDs back into a `String`. This method is crucial for converting the output of the tokenizer back into human-readable text. It correctly interprets byte tokens and vocabulary tokens to reconstruct the original sequence of bytes, which is then validated as UTF-8.

### Method
`pub fn decode(&self, tokens: &[u32], vocab: &Vocabulary) -> Result<String, Error>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* `tokens`: A slice of `u32` representing the token IDs to decode.
* `vocab`: A reference to the `Vocabulary` object, used to look up text for vocabulary tokens.

### Response
#### Success Response
- `Ok(String)`: A `String` containing the decoded text if the byte sequence is valid UTF-8.

#### Error Response
- `Err(Error::InvalidUtf8)`: If the reconstructed byte sequence is not valid UTF-8.

### Request Example
```rust
// Assuming `tokenizer` is an instance of the tokenizer and `vocab` is a valid Vocabulary
let tokens = vec![101, 102, 103]; // Example token IDs
match tokenizer.decode(&tokens, &vocab) {
    Ok(text) => println!("Decoded text: {}", text),
    Err(e) => eprintln!("Decoding error: {}", e),
}
```

### Response Example
```json
{
  "success": true,
  "data": "Decoded text string"
}
```

### Error Handling
- `Error::InvalidUtf8`: Returned when the byte sequence derived from the tokens cannot be converted into a valid UTF-8 string.
```

--------------------------------

### bos_token

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Retrieves the token ID that marks the beginning of a sequence.

```APIDOC
## bos_token

### Description
Gets the Beginning-of-Sequence (BOS) token ID.

### Method
`bos_token(&self) -> TokenId`

### Parameters
None

### Returns
The token ID used to mark the beginning of a sequence.
```

--------------------------------

### vocab_size

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Retrieves the total number of tokens in the tokenizer's vocabulary.

```APIDOC
## vocab_size

### Description
Gets the vocabulary size.

### Method
`vocab_size(&self) -> usize`

### Parameters
None

### Returns
The total number of tokens in the vocabulary.
```

--------------------------------

### Parse GGUF Metadata

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Parses key-value pairs from GGUF metadata to extract tokenizer-specific information like token IDs and flags. Handles various data types including U32, Bool, and StringArray.

```Rust
let mut kv_pairs = HashMap::new();
kv_pairs.insert("tokenizer.ggml.unknown_token_id".to_string(), Value::U32(1));
kv_pairs.insert("tokenizer.ggml.padding_token_id".to_string(), Value::U32(2));
kv_pairs.insert("tokenizer.ggml.eot_token_id".to_string(), Value::U32(3));
kv_pairs.insert("tokenizer.ggml.eog_token_id".to_string(), Value::U32(4));
kv_pairs.insert("tokenizer.ggml.sep_token_id".to_string(), Value::U32(5));
kv_pairs.insert("tokenizer.ggml.nl_token_id".to_string(), Value::U32(6));
kv_pairs.insert("tokenizer.ggml.fim_pre_token_id".to_string(), Value::U32(7));
kv_pairs.insert("tokenizer.ggml.fim_suf_token_id".to_string(), Value::U32(8));
kv_pairs.insert("tokenizer.ggml.fim_mid_token_id".to_string(), Value::U32(9));
kv_pairs.insert("tokenizer.ggml.mask_token_id".to_string(), Value::U32(10));
kv_pairs.insert("tokenizer.ggml.add_bos_token".to_string(), Value::Bool(true));
kv_pairs.insert("tokenizer.ggml.add_eos_token".to_string(), Value::Bool(false));
kv_pairs.insert("tokenizer.ggml.add_space_prefix".to_string(), Value::Bool(true));
kv_pairs.insert("tokenizer.ggml.clean_spaces".to_string(), Value::Bool(false));
kv_pairs.insert("tokenizer.ggml.remove_extra_whitespaces".to_string(), Value::Bool(true));
kv_pairs.insert("tokenizer.ggml.escape_whitespaces".to_string(), Value::Bool(false));
kv_pairs.insert("tokenizer.ggml.treat_whitespace_as_suffix".to_string(), Value::Bool(true));
kv_pairs.insert("tokenizer.ggml.merges".to_string(), Value::StringArray(vec!["a b".to_string(), "c d".to_string()]));

let tokens = Some(vec![Value::String("token1".to_string()), Value::String("token2".to_string())]);
let scores = Some(vec![1.0f32, 2.0f32]);
let token_types = Some(vec![0i32, 1i32]);
let model_type = Some("llama".to_string());
let pre_type = Some("llama".to_string());
let bos_token_id = Some(0u32);
let eos_token_id = Some(1u32);

let result = parse_gguf_metadata(kv_pairs, tokens, scores, token_types, model_type, pre_type, bos_token_id, eos_token_id);

match result {
    Ok(metadata) => {
        println!("Successfully parsed GGUF metadata:");
        println!("  Unknown Token ID: {:?}", metadata.unk_token_id);
        println!("  Padding Token ID: {:?}", metadata.pad_token_id);
        println!("  EOT Token ID: {:?}", metadata.eot_token_id);
        println!("  Add BOS Token: {:?}", metadata.add_bos_token);
        println!("  Clean Spaces: {:?}", metadata.clean_spaces);
        println!("  Merges: {:?}", metadata.merges);
    }
    Err(e) => {
        eprintln!("Error parsing GGUF metadata: {}", e);
    }
}
```

--------------------------------

### Read F32 from Bytes

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Reads exactly 4 bytes from a reader and converts them into an f32 using little-endian byte order. This is a helper function for reading floating-point numbers.

```Rust
fn read_f32<R: Read>(reader: &mut R) -> Result<f32, Error> {
    let mut buf = [0u8; 4];
    reader.read_exact(&mut buf)?;
    Ok(f32::from_le_bytes(buf))
}
```

--------------------------------

### model_type

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Retrieves the model type identifier from the tokenizer's vocabulary, such as 'llama', 'mistral', etc.

```APIDOC
## model_type

### Description
Gets the tokenizer model type.

### Method
`model_type(&self) -> &str`

### Parameters
None

### Returns
The model type string, such as "llama", "mistral", "gpt2", "qwen", "qwen2", or "gemma".

### Example
```no_run
use shimmytok::Tokenizer;

# fn main() -> Result<(), Box<dyn std::error::Error>> {
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
println!("Model type: {}", tokenizer.model_type());
# Ok(())
# }
```
```

--------------------------------

### Verify Table Column Indices

Source: https://docs.rs/shimmytok/latest/src/shimmytok/plamo2.rs.html

Asserts that the defined constants for table column indices match their expected values. This test ensures correct mapping to underlying data structures.

```Rust
assert_eq!(T_PIECE_LEN, 0);
        assert_eq!(T_TOKEN_ID, 1);
        assert_eq!(T_SCORE, 2);
        assert_eq!(T_PIECE_ID, 3);
```

--------------------------------

### Decode Token IDs to Text

Source: https://docs.rs/shimmytok/latest/src/shimmytok/ugm.rs.html

Converts a slice of token IDs back into a human-readable string using a provided vocabulary. It iterates through token IDs, retrieves their corresponding text from the vocabulary, and appends it to the result string.

```rust
pub fn decode(&self, tokens: &[u32], vocab: &Vocabulary) -> Result<String, Error> {
        let mut s = String::new();
        for &t in tokens {
            if let Some(txt) = vocab.get_token_text(t) {
                s.push_str(txt);
            }
        }
        Ok(s)
    }
```

--------------------------------

### Type Aliases

Source: https://docs.rs/shimmytok/latest/index.html

Type alias for token IDs used throughout the library.

```APIDOC
## Type Aliases

### TokenId

Type alias for token IDs
```

--------------------------------

### Read U64 from Reader

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Reads a 64-bit unsigned integer from a given reader in little-endian format. This is a utility function for parsing binary data.

```Rust
fn read_u64<R: Read>(reader: &mut R) -> Result<u64, Error> {
    let mut buf = [0u8; 8];
    reader.read_exact(&mut buf)?;
    Ok(u64::from_le_bytes(buf))
}
```

--------------------------------

### Split Text on Special Tokens

Source: https://docs.rs/shimmytok/latest/src/shimmytok/lib.rs.html

Splits a given text into fragments, identifying special tokens and regular text chunks. Special tokens are prioritized based on longest match first.

```rust
use std::collections::HashMap;

// Assuming TextFragment and special_map are defined elsewhere
// enum TextFragment { Special(TokenId), Text(String) }

fn split_on_special_tokens(
    text: &str,
    special_map: &HashMap<String, TokenId>,
) -> Vec<TextFragment> {
    if special_map.is_empty() || text.is_empty() {
        return vec![TextFragment::Text(text.to_string())];
    }

    // Sort special tokens by length descending (longest match first)
    let mut special_tokens: Vec<_> = special_map.iter().collect();
    special_tokens.sort_by(|a, b| b.0.len().cmp(&a.0.len()));

    let mut result = Vec::new();
    let mut remaining = text;

    while !remaining.is_empty() {
        // Try to match any special token at the start
        let mut found = false;
        for (token_str, &token_id) in &special_tokens {
            if remaining.starts_with(token_str.as_str()) {
                result.push(TextFragment::Special(token_id));
                remaining = &remaining[token_str.len()..];
                found = true;
                break;
            }
        }

        if !found {
            // No special token at start - find the next special token
            let mut next_special_pos = remaining.len();
            for (token_str, _) in &special_tokens {
                if let Some(pos) = remaining.find(token_str.as_str()) {
                    if pos < next_special_pos {
                        next_special_pos = pos;
                    }
                }
            }

            // Add text up to the next special token (or end)
            let text_chunk = &remaining[..next_special_pos];
            if !text_chunk.is_empty() {
                result.push(TextFragment::Text(text_chunk.to_string()));
            }
            remaining = &remaining[next_special_pos..];
        }
    }

    result
}
```

--------------------------------

### Read I32 from Reader

Source: https://docs.rs/shimmytok/latest/src/shimmytok/gguf.rs.html

Reads a 32-bit signed integer from a given reader in little-endian format. This is a utility function for parsing binary data.

```Rust
fn read_i32<R: Read>(reader: &mut R) -> Result<i32, Error> {
    let mut buf = [0u8; 4];
    reader.read_exact(&mut buf)?;
    Ok(i32::from_le_bytes(buf))
}
```

--------------------------------

### Verify Score Ordering

Source: https://docs.rs/shimmytok/latest/src/shimmytok/plamo2.rs.html

Ensures that score constants are ordered correctly at compile time using const assertions. This is a compile-time check for internal consistency.

```Rust
// Verify score ordering at compile time via const assertions
        const _: () = assert!(INVALID_SCORE < UNKNOWN_SCORE);
        const _: () = assert!(UNKNOWN_SCORE < 0);
```
Tokens:
Raw