### Documenting Tokenizer Support Limitations

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_ANALYSIS.md

Example Rust documentation comment showing how to clearly state supported and unsupported tokenizer types, indicating future support for BPE.

```rust
/// Load tokenizer from GGUF file
///
/// # Supported Models
///
/// - ✅ LLaMA, Llama-2, Llama-3 (SentencePiece)
/// - ✅ Phi-3 (SentencePiece)
/// - ❌ GPT-2, GPT-3 (BPE) - Coming in v0.2.0
///
/// # Example
/// ...
```

--------------------------------

### Rust Tokenizer Quick Start

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/README.md

Load a tokenizer from a GGUF file, encode text to tokens, and decode tokens back to text. Also demonstrates streaming token decoding.

```rust
use shimmytok::Tokenizer;

// Load tokenizer from any GGUF model
let tokenizer = Tokenizer::from_gguf_file("llama-3.gguf")?;

// Encode text to tokens
let tokens = tokenizer.encode("Hello, world!", true)?;
println!("Tokens: {:?}", tokens);

// Decode back to text
let text = tokenizer.decode(&tokens, true)?;
println!("Text: {}", text);

// Stream tokens one at a time (for LLM generation)
for token_id in tokens {
    print!("{}", tokenizer.decode_single(token_id, false)?);
}
```

--------------------------------

### Verify BPE Implementation with Real Models

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HOSTILE_BPE_AUDIT.md

This example shows a bash command to search for BPE tests. The lack of results indicates that the BPE implementation has not been tested against actual models, which is a critical gap for production-ready code.

```bash
$ grep -r "test.*bpe" tests/
# NO RESULTS - ZERO BPE TESTS
```

--------------------------------

### Adding a New BPE Pattern Mapping

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_PATTERNS.md

Example of how to add a new mapping for a custom BPE pre-tokenizer pattern in the `get_pattern()` function.

```rust
fn get_pattern(pre_type: &str) -> &'static str {
    match pre_type {
        // ... existing patterns ...
        "my-model" => MY_MODEL_PATTERN,
        _ => GPT2_PATTERN,
    }
}
```

--------------------------------

### Cargo.toml Package Metadata

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PRODUCTION_AUDIT.md

Example of package metadata to include in Cargo.toml for the shimmytok library, such as description, keywords, categories, repository, license, and readme.

```toml
[package]
description = "Pure Rust GGUF tokenizer with llama.cpp compatibility"
keywords = ["tokenizer", "gguf", "llama", "sentencepiece", "llm"]
categories = ["text-processing", "encoding", "parser-implementations"]
repository = "https://github.com/yourusername/shimmytok"
license = "MIT"
readme = "README.md"

```

--------------------------------

### BPE Merge Format Example

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BETTER_PLAN.md

Illustrates the string format for Byte Pair Encoding (BPE) merges as stored in GGUF files. It shows the special 'Ġ' prefix indicating a space.

```text
Format: "Ġt he" (with special Ġ prefix)
```

--------------------------------

### Initialize Regex Safely with Result

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HOSTILE_BPE_AUDIT.md

Avoid panicking when initializing regex patterns. This example uses `.expect()`, which will cause a panic if the regex is invalid. It's recommended to return a `Result` instead to handle potential errors gracefully.

```rust
self.llama3_regex.get_or_init(|| {
    regex::Regex::new(LLAMA3_PATTERN)
        .expect("Llama-3 regex pattern is invalid")
});
```

--------------------------------

### Changelog Entry for v0.2.0

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Example changelog entry detailing the addition of the `encode_batch` method for parallel encoding, performance improvements, and internal changes like vocabulary caching and the use of rayon.

```markdown
## [0.2.0] - 2025-10-XX

### Added
- `encode_batch()` method for parallel encoding of multiple texts
- Comprehensive benchmark suite for performance tracking

### Performance
- 1.5-2x speedup on `encode()` via vocabulary caching
- 2-4x speedup on batch encoding via parallel processing
- Optimized token lookup paths

### Internal
- Added HashMap cache for piece → token ID lookups
- Added rayon for parallel processing
```

--------------------------------

### Adding a New BPE Pattern Constant

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_PATTERNS.md

Example of how to add a new BPE pre-tokenizer pattern constant in `src/bpe.rs`.

```rust
const MY_MODEL_PATTERN: &str = r"regex pattern here";
```

--------------------------------

### README Limitations Section Example

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HONEST_AUDIT_FINAL.md

This markdown snippet shows the content of the 'Limitations' section added to the README file. It clearly states that BPE validation is incomplete, only GGUF v2 and v3 are supported, and performance is not yet optimized.

```markdown
## Limitations

- **BPE validation incomplete**: Algorithm implemented but not yet tested 
  against actual GPT-2 models (no test file available)
- Supports GGUF v2 and v3 formats only
- Optimized for correctness, not yet for maximum performance
- Only 2 pre-tokenizer patterns implemented (GPT-2, Llama-3); llama.cpp has 40+
```

--------------------------------

### Shimmytok Metadata Access

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/README.md

Examples of how to retrieve metadata such as vocabulary size, special tokens, model type, and pre-tokenization pattern from a loaded tokenizer.

```rust
tokenizer.vocab_size()    // → usize
tokenizer.bos_token()     // → Option<TokenId>  
tokenizer.eos_token()     // → Option<TokenId>  
tokenizer.model_type()    // → &str ("llama", "gpt2", etc.)
tokenizer.pre_type()      // → &str (pre-tokenization pattern)
```

--------------------------------

### Build Documentation

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Build the crate's documentation, excluding dependencies, to verify rendering and content.

```bash
cargo doc --no-deps
```

--------------------------------

### Publishing Steps for Shimmytok

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/EXECUTIVE_SUMMARY.md

This bash script outlines the essential steps to initialize a Git repository, commit changes, set up a remote origin, push to GitHub, tag a version, and publish the crate to Cargo.

```bash
git init
git add -A
git commit -m "Initial release v0.1.0"
git remote add origin https://github.com/YOURUSERNAME/shimmytok.git
git push -u origin main
git tag v0.1.0
git push origin v0.1.0

```

```bash
cargo publish
```

--------------------------------

### Initialize Git Repository

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Initializes a new Git repository, stages all files, and creates the initial commit with release details.

```bash
git init
git add -A
git commit -m "Initial release v0.1.0

- Pure Rust GGUF tokenizer
- 100% llama.cpp compatible
- SentencePiece with resegment algorithm
- 819 LOC, fully tested
"
```

--------------------------------

### Special Token Patterns

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BETTER_PLAN.md

Examples of patterns that llama.cpp uses to auto-detect special tokens like End-of-Turn (EOT) and Fill-in-the-Middle (FIM) prefix.

```text
EOT: "<|eot_id|>", "<|im_end|>", "<end_of_turn>"
FIM_PREFIX: "<|fim_prefix|>", "<PRE>", etc.
```

--------------------------------

### Run Full Test Suite

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Execute all tests for the project with all features enabled. This is a crucial step for validating the implementation.

```bash
cargo test --all-features
```

--------------------------------

### Example of Ignoring Result Error

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_1.md

Demonstrates a common mistake where the Result returned by an encode function is ignored, potentially leading to unexpected behavior if an error occurs.

```rust
tokenizer.encode(text, false); // forgot .unwrap()!
```

--------------------------------

### Cargo Build and Test Commands

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_COMPLETION_SUMMARY.md

Demonstrates the commands used to build the project in release mode and run all tests. The output indicates a clean build with no warnings and all tests passing, including SentencePiece validation.

```bash
✅ cargo build --release - CLEAN (0 warnings)
✅ cargo test - ALL PASSING
   - test_basic: ok
   - test_comprehensive: ok (8/8 SentencePiece matches)
   - test_debug: ok
   - test_detailed: ok
   - test_merge_debug: ok
   - test_tokens: ok
   - 5 doctests: ok
```

--------------------------------

### Basic llama.cpp Tokenize Usage

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/TOKENIZER_VALIDATION_AUDIT.md

Demonstrates the basic command structure for using the llama-tokenize executable. This includes specifying the model file, providing text input, and using common options for output formatting.

```bash
# Basic usage
../llama.cpp/build/bin/Release/llama-tokenize.exe -m MODEL.gguf -p "text" --ids --no-bos --log-disable
```

--------------------------------

### Verify Package Contents

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Create a local package archive to inspect its contents and ensure all necessary files are included.

```bash
cargo package --allow-dirty
```

--------------------------------

### Publication Checklist and Commands

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PRODUCTION_AUDIT.md

Provides a checklist for pre-publication steps and the necessary bash commands for publishing the Rust crate to crates.io.

```bash
# 1. Verify everything works
cargo test --release
cargo doc --no-deps
cargo package --dry-run

# 2. Create git repository
git init
git add -A
git commit -m "Initial release v0.1.0"
git tag v0.1.0

# 3. Publish to crates.io
cargo publish
```

--------------------------------

### Get llama.cpp Tokenizer Output for WPM Model

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/TOKENIZER_VALIDATION_AUDIT.md

Command to obtain the tokenized output for a specific prompt from a WordPiece tokenizer model using llama-tokenize. This output serves as the expected result for validation.

```bash
../llama.cpp/build/bin/Release/llama-tokenize.exe \
  -m ../llama.cpp/models/ggml-vocab-bert-bge.gguf \
  -p "Hello, world!" --ids --no-bos --log-disable
```

--------------------------------

### Rust: Safely get token score, handling missing scores

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_4.md

This function retrieves a token's score, defaulting to 0.0 if the score is not found. It assumes scores and tokens have a one-to-one correspondence.

```rust
pub fn get_token_score(&self, id: TokenId) -> f32 {
    self.scores.get(id as usize).copied().unwrap_or(0.0)
}
```

--------------------------------

### Loading Tokenizers with Kitoken

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/RUST_TOKENIZER_LANDSCAPE.md

Demonstrates how to load tokenizers using the Kitoken crate from various file formats like SentencePiece (.model), Tiktoken (.tiktoken), and HuggingFace (.json). It highlights the absence of direct GGUF loading.

```rust
let encoder = Kitoken::from_sentencepiece_file("models/llama2.model")?;
let encoder = Kitoken::from_tiktoken_file("models/o200k_base.tiktoken")?;
let encoder = Kitoken::from_tokenizers_file("tokenizer.json")?;
```

--------------------------------

### Current BPE Implementation Key Dependencies

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/ARCHITECTURE_ANALYSIS.md

Lists the key dependencies used by the current BPE implementation, such as vocabulary lookups and merge retrieval.

```rust
- `Vocabulary::get_token_id(text)` - lookup merged text
- `Vocabulary::get_merges()` - get (token1, token2) -> rank pairs
- `Vocabulary::byte_to_token(byte)` - fallback for unknowns
- `Vocabulary::get_token_text(id)` - for decode
```

--------------------------------

### Removing Misleading Stub Implementation Comment

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HONEST_AUDIT_FINAL.md

This code example illustrates the removal of a misleading comment indicating a 'stub implementation' for loading GGUF metadata. The comment was removed because the code was actually functional, not a stub.

```rust
// This is where we'd load from GGUF
// For now, using stub implementation
let metadata = crate::gguf::load_metadata(path)?;
```

```rust
let metadata = crate::gguf::load_metadata(path)?;
```

--------------------------------

### Get Model Path for Tests

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_1.md

A helper function to determine the model path for tests, attempting to find it in common user directories or falling back to a default name. This is used to locate test fixtures.

```rust
fn get_model_path() -> String {
    std::env::var("HOME")
        .or_else(|_| std::env::var("USERPROFILE"))
        .map(|home| format!("{}/.cache/models/gguf/gpt2.Q4_K_M.gguf", home))
        .unwrap_or_else(|_| "gpt2.Q4_K_M.gguf".to_string())
}
```

--------------------------------

### Rust: Safely get token type, handling undefined types

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_4.md

This function retrieves a token's type, defaulting to TokenType::Undefined if the type is not found. It relies on prior validation of token_types length.

```rust
pub fn get_token_type(&self, id: TokenId) -> TokenType {
    self.token_types
        .get(id as usize)
        .copied()
        .unwrap_or(TokenType::Undefined)
}
```

--------------------------------

### Add Qwen Tokenizer to GGUF Loader

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Illustrates adding 'qwen' and 'qwen2' model types to the GGUF loader's match statement, suggesting a BPE tokenizer implementation for these models.

```rust
// If Qwen uses BPE with custom pattern
match vocab.model_type() {
    "llama" => Box::new(sentencepiece::SentencePieceTokenizer::new()),
    "gpt2" => Box::new(bpe::BPETokenizer::new()),
    "mistral" => Box::new(sentencepiece::SentencePieceTokenizer::new()),
    "qwen" | "qwen2" => Box::new(bpe::BPETokenizer::new()), // NEW
    model => return Err(Error::UnsupportedModel(model.to_string())),
};

```

--------------------------------

### Run All Tests

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Execute all tests in release mode to ensure code stability before publishing.

```bash
cargo test --release
```

--------------------------------

### Handle Leading Space in BPE Tokens

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_ANALYSIS.md

This C++ code snippet checks if a token fragment starts with a space and prepends the 'Ġ' character to indicate a leading space, as used in GPT-2's BPE vocabulary.

```cpp
// Check if token starts with space
if (fragment[0] == ' ') {
    // Add Ġ prefix to indicate "this token had leading space"
    token = "Ġ" + token.substr(1);
}
```

--------------------------------

### Basic shimmytok Usage

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/RUST_TOKENIZER_LANDSCAPE.md

Shows the fundamental API for shimmytok, demonstrating how to load a tokenizer directly from a GGUF file and then use it to encode text.

```rust
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
let tokens = tokenizer.encode("text", true)?;
```

--------------------------------

### Get Beginning-of-Sequence Token ID

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_1.md

Retrieves the token ID for the beginning-of-sequence (BOS) token. If the GGUF file does not define a BOS token, this function currently returns 0, which could be misinterpreted as a valid token ID.

```rust
pub fn bos_token_id(&self) -> TokenId {
    self.bos_token
}
```

--------------------------------

### Core Shimmytok API Methods

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/README.md

Demonstrates loading a tokenizer from a GGUF file, encoding text to tokens, decoding tokens to text, and streaming single token decoding.

```rust
// Load from GGUF file
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// Encode text → tokens
let tokens = tokenizer.encode("Hello", true)?;

// Decode tokens → text  
let text = tokenizer.decode(&tokens, true)?;

// Streaming decode (for LLM generation)
let piece = tokenizer.decode_single(token_id, false)?;
```

--------------------------------

### Initial Commit with DCO

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/GITHUB_ACTIONS_ASSESSMENT.md

Commit all current changes with DCO sign-off. Use '-s' flag for DCO.

```bash
git add -A
git commit -s -m "feat: add OSS governance and documentation

- Add CONTRIBUTING, DCO, CoC, SECURITY policies
- Style README with badges and sponsorship
- Add CHANGELOG, SPONSORS, AUTHORS
- Fix Cargo.toml repository URL
- Correct LOC and method count claims
"
```

--------------------------------

### Implement Tokenizer Model Type Method

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Adds a public `model_type` method to the `Tokenizer` struct. This method returns the model type identifier from the GGUF metadata, such as 'llama', 'gpt2', 'mistral', 'qwen', or 'gemma'. Includes documentation and an example of its usage.

```rust
impl Tokenizer {
    /// Get the tokenizer model type
    ///
    /// Returns the model type identifier from the GGUF metadata,
    /// such as "llama", "gpt2", "mistral", "qwen", or "gemma".
    ///
    /// # Example
    ///
    /// ```no_run
    /// # use shimmytok::Tokenizer;
    /// # let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
    /// println!("Model type: {}", tokenizer.model_type());
    /// # Ok::<(), shimmytok::Error>(())
    /// ```
    pub fn model_type(&self) -> &str {
        self.vocab.model_type()
    }
}
```

--------------------------------

### BPE Merge Algorithm Logic in llama.cpp

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_IMPLEMENTATION_PLAN.md

The core logic from llama.cpp's BPE tokenizer, illustrating the process of splitting words into symbols, finding bigrams, and applying merges using a priority queue.

```cpp
// From llama.cpp llm_tokenizer_bpe_session::tokenize()
for (each word fragment from regex) {
    // Split into UTF-8 characters
    while (offset < word.size()) {
        size_t char_len = unicode_len_utf8(word[offset]);
        symbols.push_back({text: &word[offset], len: char_len});
        offset += char_len;
    }
    
    // Find all possible merges
    for (i in 0..symbols.len()-1) {
        add_new_bigram(i, i+1);  // checks if merge exists in bpe_ranks
    }
    
    // Apply merges in rank order (priority queue)
    while (!work_queue.empty()) {
        bigram = work_queue.pop();  // Highest priority (lowest rank) 
        
        // Merge left + right
        left_symbol.len += right_symbol.len;
        right_symbol.len = 0;  // Mark as deleted
        
        // Update linked list
        left_symbol.next = right_symbol.next;
        
        // Add new potential merges
        add_new_bigram(left_symbol.prev, left);
        add_new_bigram(left, left_symbol.next);
    }
}
```

--------------------------------

### SentencePiece Prefix Handling for Spaces

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/LLAMA_CPP_PARITY_PLAN.md

Shows the current implementation in SentencePiece where a leading space is always replaced by '▁', regardless of the `add_space_prefix` flag. The required behavior is to conditionally add this prefix based on the flag and whether the text already starts with a space.

```rust
let processed_text = if text.starts_with(' ') {
    text.replace(' ', "▁")
} else {
    format!("▁{}", text.replace(' ', "▁"))  // Always adds prefix
};
```

--------------------------------

### Automatic BPE Pre-tokenizer Selection

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_PATTERNS.md

Demonstrates how the tokenizer automatically selects the appropriate BPE pre-tokenizer pattern based on the `tokenizer.ggml.pre` field in the GGUF metadata. Defaults to GPT-2 if not specified.

```rust
// Automatic selection from GGUF metadata
let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;

// The pre-tokenizer type is read from metadata and the appropriate
// regex pattern is used automatically
let tokens = tokenizer.encode("Hello world!", false)?;
```

--------------------------------

### Rust incomplete UTF-8 validation in BPE decode

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_4.md

Checks for the presence of the byte `0xEF` and the replacement character '�' within a decoded string. This check is noted as convoluted and potentially incorrect, as `0xEF` is a valid UTF-8 start byte and the check might lead to false positives on valid text.

```rust
if !decoded.is_empty() && decoded.as_bytes().iter().any(|&b| b == 0xEF && decoded.contains('�')) {
    return Err(...);
}
```

--------------------------------

### Preprocess WPM Input

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/GPT_PLANS.md

Demonstrates the basic usage of the `preprocess_wpm` function. This function is intended to be used with a GGUF-derived vocab fixture for accurate token ID assertions.

```rust
        // You should replace with a GGUF-derived vocab fixture and assert exact token IDs.
        let _ = preprocess_wpm("Hello, world!");
    }
}
```
```

--------------------------------

### Current BPE Implementation Algorithm Flow

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/ARCHITECTURE_ANALYSIS.md

Details the algorithm flow for the current BPE implementation, including pre-tokenization, initial symbol creation, merge loop, and byte fallback.

```rust
1. **Pre-tokenization**: Uses regex to split text into fragments
2. **Initial symbols**: Creates UTF-8 character symbols for each fragment
3. **Merge loop**: Uses priority queue with merge ranks from vocab
4. **Byte fallback**: Uses `vocab.byte_to_token(byte)` for unknown tokens
```

--------------------------------

### UgmTokenizer Initialization

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/GPT_PLANS.md

Initializes the UGM tokenizer by building tries for normal and user-defined tokens from the vocabulary. Calculates the score for unknown tokens.

```rust
pub struct UgmTokenizer {
    trie: NaiveTrie,
    user_defined_trie: NaiveTrie,
    unknown_token_score: f64,
}

impl UgmTokenizer {
    pub fn new(vocab: &Vocabulary) -> Self {
        let mut trie = NaiveTrie::new();
        let mut user_defined_trie = NaiveTrie::new();

        let mut min_score = f64::INFINITY;
        let mut _max_score = f64::NEG_INFINITY;

        for id in 0..(vocab.n_tokens() as u32) {
            let txt = match vocab.get_token_text(id) {
                Some(t) => t,
                None => continue,
            };
            let ttype = vocab.get_token_type(id);

            // llama.cpp: insert normals, user_defined, unused into token_matcher :contentReference[oaicite:9]{index=9}
            if matches!(ttype, TokenType::Normal | TokenType::UserDefined | TokenType::Unused) {
                trie.insert(txt, Some(id));
            }
            if matches!(ttype, TokenType::UserDefined) {
                user_defined_trie.insert(txt, Some(id));
            }

            if matches!(ttype, TokenType::Normal) {
                let sc = vocab.get_token_score(id) as f64;
                min_score = min_score.min(sc);
                _max_score = _max_score.max(sc);
            }
        }

        let unknown_token_score_penalty = 10.0f64;
        let unknown_token_score = min_score - unknown_token_score_penalty;

        Self { trie, user_defined_trie, unknown_token_score }
    }
}
```

--------------------------------

### llama.cpp Byte Fallback Approach

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/ARCHITECTURE_ANALYSIS.md

Illustrates the byte fallback logic from the llama.cpp implementation, where each byte is treated as a single-character string for vocabulary lookup.

```cpp
for (auto j = str.begin(); j != str.end(); ++j) {
    std::string byte_str(1, *j);  // Single byte as string
    auto token_multibyte = vocab.text_to_token(byte_str);
    if (token_multibyte != LLAMA_TOKEN_NULL) {
        output.push_back(token_multibyte);
    }
}
```

--------------------------------

### Define Test Inputs for Tokenization

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HOSTILE_AUDIT.md

Creates a file named test_inputs.txt containing various text strings to be used as input for the llama-tokenize tool. Includes standard text, special characters, and control sequences.

```bash
# Generate test vectors
cat > test_inputs.txt << EOF
Hello world
The quick brown fox
émoji 👍🏽 test
<|im_start|>system
EOF
```

--------------------------------

### Login to crates.io

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Logs the user into the crates.io registry, required before publishing.

```bash
cargo login
```

--------------------------------

### Publish Crate to crates.io

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PUBLISHING_CHECKLIST.md

Publishes the current version of the crate to the crates.io registry.

```bash
cargo publish
```

--------------------------------

### Proposed llama.cpp BPE Algorithm Changes

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/ARCHITECTURE_ANALYSIS.md

Highlights the differences between the current BPE algorithm and the proposed llama.cpp approach, specifically regarding initial symbols and byte fallback.

```rust
1. **Pre-tokenization**: Same (regex split)
2. **Initial symbols**: ⚠️ DIFFERENT - whole words from regex, not characters
3. **Merge loop**: Same concept, different initialization
4. **Byte fallback**: ⚠️ DIFFERENT - lookup bytes as strings in vocab directly
```

--------------------------------

### Handle BPE Pre-tokenization Failures Gracefully

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/AUDIT_ROUND_1.md

Avoid panics by returning a Result instead of using .expect() for BPE pre-tokenization. This addresses potential failures due to deeply nested or malformed input.

```rust
let fragments = self.pre_tokenize(&text_encoded, vocab)
    .expect("BPE pre-tokenization regex failed - this should never happen with hardcoded patterns");
```

--------------------------------

### Create Git Tag for v0.3.0

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Create an annotated Git tag for the v0.3.0 release, marking the 'Model support expansion'.

```bash
git tag -a v0.3.0 -m "Model support expansion"
```

--------------------------------

### Create Git Tag for v0.2.0

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/IMPLEMENTATION_PLAN.md

Create an annotated Git tag for the v0.2.0 release, marking the 'Performance optimization release'.

```bash
git tag -a v0.2.0 -m "Performance optimization release"
```

--------------------------------

### Basic Shimmytok Usage: Encode and Decode

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/PRODUCTION_AUDIT.md

Demonstrates how to load a tokenizer from a GGUF file and use it to encode text into tokens and decode tokens back into text.

```rust
use shimmytok::Tokenizer;

let tokenizer = Tokenizer::from_gguf_file("model.gguf")?;
let tokens = tokenizer.encode("Hello world", true)?;
let text = tokenizer.decode(&tokens, true)?;

```

--------------------------------

### Updating Documentation for BPE Tokenization Status

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/HONEST_AUDIT_FINAL.md

This snippet shows the correction of 'fraudulent documentation' by changing the status of BPE tokenization from 'Full support' to 'Experimental'. This reflects that the algorithm is implemented but not yet tested with real models.

```markdown
| GPT-2 / GPT-3 (BPE) | ✅ Full support |
```

```markdown
| GPT-2 / GPT-3 (BPE) | ⚠️ Experimental | Algorithm implemented, not yet tested with real models |
```

```markdown
> - ✅ GPT-2 / GPT-3 style BPE
```

```markdown
> - ⚠️ GPT-2 / GPT-3 style BPE - Algorithm implemented, not yet validated
```

--------------------------------

### BPE Tokenizer Encoding Logic

Source: https://github.com/michael-a-kuykendall/shimmytok/blob/master/docs/BPE_IMPLEMENTATION_PLAN.md

Implements the main BPE encoding process, including pre-tokenization and applying BPE to each fragment. This is the primary entry point for tokenizing text.

```rust
impl BPETokenizer {
    pub fn encode(&self, text: &str, vocab: &Vocabulary) -> Vec<TokenId> {
        // Step 1: Pre-tokenization via regex
        let fragments = self.pre_tokenize(text, vocab.pre_type());
        
        // Step 2: For each fragment, apply BPE
        let mut result = Vec::new();
        for fragment in fragments {
            let tokens = self.bpe_fragment(&fragment, vocab);
            result.extend(tokens);
        }
        
        result
    }
    
    fn pre_tokenize(&self, text: &str, pre_type: &str) -> Vec<String> {
        let pattern = match pre_type {
            "gpt2" | "gpt-2" => GPT2_PATTERN,
            "llama3" | "llama-bpe" => LLAMA3_PATTERN,
            _ => GPT2_PATTERN,  // Default
        };
        
        let re = Regex::new(pattern).unwrap();
        re.find_iter(text)
            .map(|m| m.as_str().to_string())
            .collect()
    }
    
    fn bpe_fragment(&self, text: &str, vocab: &Vocabulary) -> Vec<TokenId> {
        // Build merge rank map
        let merge_ranks: HashMap<(String, String), usize> = 
            vocab.get_merges()
                .iter()
                .enumerate()
                .map(|(rank, (l, r))| ((l.clone(), r.clone()), rank))
                .collect();
        
        // Split into UTF-8 characters
        let mut symbols = Vec::new();
        let mut offset = 0;
        for (i, ch) in text.char_indices() {
            let next_offset = text.char_indices()
                .nth(i + 1)
                .map(|(pos, _)| pos)
                .unwrap_or(text.len());
            
            symbols.push(Symbol {
                text_start: offset,
                text_len: next_offset - offset,
                prev: if i == 0 { None } else { Some(i - 1) },
                next: if next_offset == text.len() { None } else { Some(i + 1) },
            });
            offset = next_offset;
        }
        
        // Build initial work queue
        let mut work_queue = BinaryHeap::new();
        for i in 0..symbols.len().saturating_sub(1) {
            self.try_add_bigram(i, i + 1, text, &symbols, &merge_ranks, &mut work_queue);
        }
        
        // Apply merges
        while let Some(bigram) = work_queue.pop() {
            // Validate bigram is still valid
            if symbols[bigram.left].text_len == 0 || 
               symbols[bigram.right].text_len == 0 ||
               symbols[bigram.left].next != Some(bigram.right) {
                continue;
            }
            
            // Merge
            symbols[bigram.left].text_len += symbols[bigram.right].text_len;
            symbols[bigram.right].text_len = 0;
            
            // Update linked list
            symbols[bigram.left].next = symbols[bigram.right].next;
            if let Some(next) = symbols[bigram.right].next {
                symbols[next].prev = Some(bigram.left);
            }
            
            // Add new potential merges
            if let Some(prev) = symbols[bigram.left].prev {
                self.try_add_bigram(prev, bigram.left, text, &symbols, &merge_ranks, &mut work_queue);
            }
            if let Some(next) = symbols[bigram.left].next {
                self.try_add_bigram(bigram.left, next, text, &symbols, &merge_ranks, &mut work_queue);
            }
        }
        
        // Convert symbols to token IDs
        let mut result = Vec::new();
        for sym in &symbols {
            if sym.text_len > 0 {
                let token_text = &text[sym.text_start..sym.text_start + sym.text_len];
                if let Some(id) = vocab.get_token_id(token_text) {
                    result.push(id);
                } else {
                    // Byte fallback
                    for byte in token_text.bytes() {
                        result.push(vocab.byte_to_token(byte));
                    }
                }
            }
        }
        
        result
    }
    
    fn try_add_bigram(
        &self,
        left: usize,
        right: usize,
        text: &str,
        symbols: &[Symbol],
        merge_ranks: &HashMap<(String, String), usize>,
        work_queue: &mut BinaryHeap<Bigram>,
    ) {
        let left_text = &text[symbols[left].text_start..symbols[left].text_start + symbols[left].text_len];
        let right_text = &text[symbols[right].text_start..symbols[right].text_start + symbols[right].text_len];
        
        if let Some(&rank) = merge_ranks.get(&(left_text.to_string(), right_text.to_string())) {
            work_queue.push(Bigram {
                left,
                right,
                rank,
                text: format!("{}{}", left_text, right_text),
            });
        }
    }
}

```