### Setup and Run WASM Node.js Example

Source: https://github.com/crutcher/wordchipper/blob/main/examples/wasm-node/README.md

Execute these commands to set up the environment and run the Node.js example. The setup script handles building the WASM package and downloading vocabulary files.

```bash
./examples/wasm-node/setup.sh
cd examples/wasm-node
node index.mjs
```

--------------------------------

### Setup and Run WASM Browser Example

Source: https://github.com/crutcher/wordchipper/blob/main/examples/wasm-browser/README.md

Execute these commands to set up and run the interactive browser demo. The setup script builds the WASM package, copies it, and downloads vocab files. Then, serve the directory using Python's HTTP server.

```bash
./examples/wasm-browser/setup.sh
cd examples/wasm-browser
python3 -m http.server 8080
```

--------------------------------

### BPE Training Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/how-tokenizers-work.md

Illustrates the BPE training process by showing how pairs of tokens are merged iteratively to build a vocabulary. Starts with individual bytes and merges the most frequent adjacent pairs.

```text
Corpus: [a, a, b, a, a, ' ', a, a, b]

Round 1: Most frequent pair is (a, a), count=3
  Merge: (a, a) -> aa (token 3)
  Corpus becomes: [aa, b, aa, ' ', aa, b]

Round 2: Most frequent pair is (aa, b), count=2
  Merge: (aa, b) -> aab (token 4)
  Corpus becomes: [aab, aa, ' ', aab]

Round 3: Most frequent pair is (aab, aa), count=1
  (tie with others; pick first)
  ...
```

--------------------------------

### Setup WASM Interactive Demo

Source: https://github.com/crutcher/wordchipper/blob/main/book/README.md

Runs the setup script to build the WebAssembly module and download necessary vocabulary files for the interactive tokenizer demo.

```bash
./setup-wasm.sh
```

--------------------------------

### Set up Development Environment

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/python/README.md

Instructions for setting up the Python development environment using uv, installing dependencies, and building the Rust extension module.

```bash
cd bindings/python

# Set up environment and build
uv venv .venv
source .venv/bin/activate
uv pip install maturin pytest
maturin develop --features python-extension-module
```

--------------------------------

### Initialize Tokenizer and Encode/Decode Text

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/wasm/README.md

Quick start example to initialize a tokenizer with a pretrained model and perform encoding and decoding operations. Remember to free the tokenizer when done.

```javascript
import { Tokenizer } from "./js/dist/index.js";

const tok = await Tokenizer.fromPretrained("o200k_base");

const tokens = tok.encode("hello world"); // Uint32Array [24912, 2375]
const text = tok.decode(tokens); // "hello world"

tok.free();
```

--------------------------------

### JavaScript/TypeScript (WASM) Quick Start

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/bindings.md

Provides a basic example of initializing a Tokenizer from a pre-trained model and performing encode/decode operations in JavaScript/TypeScript.

```APIDOC
## JavaScript / TypeScript (WASM) Quick Start

### Initialize Tokenizer

```javascript
import { Tokenizer } from "./js/dist/index.js";

// Initialize tokenizer from a pre-trained model
const tok = await Tokenizer.fromPretrained("o200k_base");

// Encode text to tokens
const tokens = tok.encode("hello world"); // Returns Uint32Array, e.g., [24912, 2375]

// Decode tokens back to text
const text = tok.decode(tokens); // Returns string, e.g., "hello world"

// Remember to free WASM memory when done
tok.free();
```
```

--------------------------------

### Install wordchipper-cli

Source: https://github.com/crutcher/wordchipper/blob/main/crates/wordchipper-cli/README.md

Install the wordchipper-cli tool using Cargo.

```bash
cargo install wordchipper-cli
```

--------------------------------

### Pre-tokenization Example Output

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/how-tokenizers-work.md

Demonstrates how the regex pattern splits the input string 'Hello, world! 123' into a list of segments. Each segment is then independently BPE-encoded.

```text
["Hello", ",", " world", "!", " 123"]
```

--------------------------------

### Install wordchipper

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/python/README.md

Install the wordchipper library using pip.

```bash
pip install wordchipper
```

--------------------------------

### Install Development Tools

Source: https://github.com/crutcher/wordchipper/blob/main/CONTRIBUTING.md

Installs the necessary Rust toolchains and components for development, including Miri for undefined behavior detection.

```sh
rustup toolchain install stable nightly
rustup component add --toolchain nightly rustfmt miri
```

--------------------------------

### Minimal BPE Training Example in Rust

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Demonstrates the basic workflow for training a BPE tokenizer using the wordchipper-training API, including configuration, data feeding, training, and tokenizer usage.

```rust
use std::sync::Arc;

use wordchipper::{
    TokenEncoder, TokenDecoder,
    Tokenizer, TokenizerOptions, UnifiedTokenVocab,
    pretrained::openai::OA_CL100K_BASE_PATTERN,
    vocab::ByteMapVocab,
};
use wordchipper_training::BPETRainerOptions;

fn main() {
    // 1. Configure the trainer
    let options = BPETRainerOptions::new(
        OA_CL100K_BASE_PATTERN,  // regex pattern for pre-tokenization
        1000,                     // target vocabulary size
    );
    let mut trainer = options.init();

    // 2. Feed it training data
    let samples = vec![
        "the cat sat on the mat",
        "the dog sat on the log",
        "the cat and the dog",
    ];
    trainer.update_from_samples(samples.iter());

    // 3. Train
    let byte_vocab: ByteMapVocab<u32> = Default::default();
    let vocab: Arc<UnifiedTokenVocab<u32>> = trainer
        .train(byte_vocab)
        .expect("training failed")
        .into();

    // 4. Use the tokenizer
    let tokenizer = TokenizerOptions::default().build(vocab);

    let tokens = tokenizer.try_encode("the cat").unwrap();
    let decoded = tokenizer.try_decode_to_string(&tokens).unwrap().unwrap();
    assert_eq!(decoded, "the cat");

    println!("'the cat' -> {:?}", tokens);
}
```

--------------------------------

### Install mdbook

Source: https://github.com/crutcher/wordchipper/blob/main/book/README.md

Installs the mdbook toolchain using cargo. This is a prerequisite for building and serving the book.

```bash
cargo install mdbook
```

--------------------------------

### Install Benchmark Dependencies

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/python/README.md

Install necessary libraries for running performance benchmarks, including pytest-benchmark, tiktoken, tokenizers, and pyarrow.

```bash
# Install benchmark dependencies
uv pip install pytest-benchmark tiktoken tokenizers pyarrow
```

--------------------------------

### Install System Dependencies

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/report-tool/README.md

Before installing Wordchipper, ensure these system packages are installed. This command is for Debian-based systems like Ubuntu.

```bash
sudo apt install pkg-config libfreetype6-dev libfontconfig1-dev
```

--------------------------------

### BPE Merge Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/advanced-span-encoders.md

Illustrates the Byte Pair Encoding (BPE) merge process with a simple span and ranked merges, showing the step-by-step transformation of tokens.

```text
Start:   [a, b, c, d]
Step 1:  [ab, c, d]       merge (a,b) at rank 0
Step 2:  [ab, cd]          merge (c,d) at rank 1
Step 3:  [abcd]            merge (ab,cd) at rank 2
```

--------------------------------

### BPE Encoding Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/how-tokenizers-work.md

Demonstrates the BPE encoding process using a trained vocabulary and merge table. Shows how an input string is converted into a sequence of tokens by applying merges based on their rank (priority).

```text
Start:    [a, a, b]       (3 byte tokens)
Pairs:    (a,a)=rank 0    (a,b)=not in table
Merge:    [aa, b]         (merge (a,a) at rank 0)
Pairs:    (aa,b)=rank 1
Merge:    [aab]           (merge (aa,b) at rank 1)
No pairs: done.
Result:   [4]             (token 4 = "aab")
```

--------------------------------

### JSON Configuration Example

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/benches/data/multilingual.txt

A sample JSON object representing a configuration for a language model, including model name, messages, and various generation parameters.

```json
{"model": "gpt-4o", "messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "What is the capital of France?"}], "temperature": 0.7, "max_tokens": 150, "top_p": 1.0, "frequency_penalty": 0.0, "presence_penalty": 0.0}
```

--------------------------------

### Concrete BPE Merge Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Illustrates a single merge operation on a token span, showing the removal of old pairs and addition of new pairs after the merge.

```text
Before: [104, 101, 108, 108, 111]
         h    e    l    l    o

Pairs removed: (101, 108), (108, 108), (108, 111)
Pairs added:   (101, 256), (256, 111)

After:  [104, 101, 256, 111]
         h    e    ll   o
```

--------------------------------

### Build WASM Package

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/wasm/README.md

Build the WebAssembly package for web targets using wasm-pack. Ensure Rust and wasm-pack are installed.

```bash
# Build the WASM package
wasm-pack build bindings/wasm --target web
```

--------------------------------

### Build TypeScript Wrapper

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/wasm/README.md

Build the TypeScript wrapper for the WASM package. Navigate to the JS directory, install dependencies, and run the build script.

```bash
# Build the TypeScript wrapper
cd bindings/wasm/js
npm install
npm run build
```

--------------------------------

### Simple Text to Token Encoding Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/introduction.md

Illustrates how a simple string is encoded into tokens using a specific vocabulary like cl100k_base. Note that the token for ' world' includes a leading space.

```text
"hello world" -> [15339, 1917]
```

--------------------------------

### HTML Structure Example

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/benches/data/multilingual.txt

A basic HTML document structure demonstrating common elements like headings, paragraphs, lists, and entity encoding.

```html
<html><head><title>Example Page</title></head><body><div class="container"><h1>Hello, World!</h1><p>This is a <strong>test</strong> of HTML tokenization with &amp; entities &lt;tags&gt; and "quotes".</p><ul><li>Item 1</li><li>Item 2</li><li>Item 3</li></ul></div></body></html>
```

--------------------------------

### Enable wordchipper Client Feature

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/feature-flags.md

Example of how to enable the 'client' feature for the wordchipper crate in your Cargo.toml file. This feature includes everything needed to load pretrained vocabularies.

```toml
[dependencies]
wordchipper = { version = "0.8", features = ["client"] }
```

--------------------------------

### Tokenizer Training Progress Log

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Example log output during tokenizer training, showing progress at 1% intervals and the frequency of merges. Early merges are for common byte pairs, while later merges are statistically rarer.

```text
INFO Reading shards:
INFO 0: shard_00000.parquet
INFO 1: shard_00001.parquet
INFO Training Tokenizer...
INFO Starting BPE training: 50025 merges to compute
INFO Building pair index...
INFO Building heap with 16044 unique pairs
INFO Starting merge loop
INFO Progress: 1% (501/50025 merges) - Last merge: (69, 120) -> 756 (frequency: 166814)
INFO Progress: 2% (1001/50025 merges) - Last merge: (66, 77) -> 1256 (frequency: 17847)
... 
INFO Progress: 100% (50025/50025 merges) - Last merge: (302, 305) -> 50280 (frequency: 18)
INFO Finished training: 50025 merges completed
INFO Vocabulary Size: 50280
```

--------------------------------

### Build WASM Package and TypeScript Wrapper

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/bindings.md

Build the WebAssembly package using wasm-pack and then build the TypeScript wrapper by navigating to the JS directory, installing dependencies, and running the build script.

```bash
wasm-pack build bindings/wasm --target web
```

```bash
cd bindings/wasm/js
npm install
npm run build
```

--------------------------------

### Rust Documentation Comment Structure

Source: https://github.com/crutcher/wordchipper/blob/main/STYLE_GUIDE.md

Demonstrates the standard structure for Rust doc comments, including sections for summary, arguments, returns, panics, errors, and examples.

```rust
/// Short single-line summary.
///
/// Optional longer description paragraph.
///
/// ## Arguments
/// * `foo` - Description of foo.
/// * `bar` - Description of bar.
///
/// ## Returns
/// What the function returns.
///
/// ## Panics
/// When this function panics (omit if it never panics).
///
/// ## Errors
/// What errors can be returned (for functions returning `Result`).
///
/// ## Example
/// ```rust
/// // example code
/// ```
pub fn my_function(foo: u32, bar: &str) -> WCResult<String> { ... }
```

--------------------------------

### Serve Documentation Book

Source: https://github.com/crutcher/wordchipper/blob/main/CONTRIBUTING.md

Builds and serves the user-facing documentation locally using mdBook. Ensure the WASM tokenizer demo is set up first.

```sh
cd book

```

```sh
./setup-wasm.sh   # first time only

```

```sh
mdbook serve
```

--------------------------------

### Whitespace Rule Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/how-tokenizers-work.md

Illustrates the specific whitespace rule `\s+(?!\S)` which matches trailing whitespace and ensures the last space is included with the following word. This helps the model identify word starts.

```text
["  ", " world"]
```

--------------------------------

### Build Python Bindings from Source

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/bindings.md

Instructions for building the Python bindings from source using Rust, uv, and maturin. Includes debug and release builds, and running tests.

```bash
cd bindings/python
uv venv .venv && source .venv/bin/activate
uv pip install maturin pytest
maturin develop           # debug build
maturin develop --release # release build for benchmarks
pytest tests/ -v
```

--------------------------------

### Tokenizer Initialization

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/wasm/README.md

Demonstrates how to initialize the Tokenizer, either by fetching a pre-trained vocabulary from a CDN or by providing custom vocabulary data.

```APIDOC
## Loading

```js
// Fetch vocab from OpenAI CDN
const tok = await Tokenizer.fromPretrained("cl100k_base");

// Or from your own vocab bytes
const data = new Uint8Array(/* .tiktoken file bytes */);
const tok = await Tokenizer.fromVocabData("cl100k_base", data);
```
```

--------------------------------

### Configure BPETrainerOptions with Custom Regex Patterns

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Demonstrates how to initialize BPETrainerOptions with custom regex patterns for text splitting. Choose patterns based on specific domain needs or desired tokenization behavior.

```rust
# use wordchipper_training::BPETRainerOptions;
// Split only on whitespace boundaries
let options = BPETRainerOptions::new(r"\S+|\s+", 4000);
```

```rust
# use wordchipper_training::BPETRainerOptions;
// Split on individual characters (character-level BPE)
let options = BPETRainerOptions::new(r".", 1000);
```

```rust
# use wordchipper_training::BPETRainerOptions;
// Domain-specific: keep email addresses and URLs intact
let options = BPETRainerOptions::new(
    r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}|https?://\S+|\w+|\s+|.",
    8000,
);
```

--------------------------------

### Run Bench-JSON with Divan Arguments

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/README.md

Execute the bench-json binary and pass additional arguments to the underlying divan benchmarking framework. Arguments after -- are forwarded.

```bash
cargo run --release -p wordchipper-bench --bin bench-json -- --bench spanning -- --sample-count 10
```

--------------------------------

### Run Bench-JSON for All Targets to File

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/README.md

Execute the bench-json binary to run all benchmark targets and save the combined JSON output to a file. Use -o to specify the output file.

```bash
cargo run --release -p wordchipper-bench --bin bench-json -- -o results.json
```

--------------------------------

### Python Basic Usage

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/bindings.md

Demonstrates how to initialize a Tokenizer, encode text to tokens, and decode tokens back to text using the Python bindings. Also shows batch operations.

```APIDOC
## Python Bindings

### Initialize Tokenizer

```python
from wordchipper import Tokenizer

tok = Tokenizer.from_pretrained("cl100k_base")
```

### Encode and Decode Text

```python
# Encode text to tokens
tokens = tok.encode("hello world")       # Example output: [15339, 1917]

# Decode tokens back to text
text = tok.decode(tokens)                 # Example output: "hello world"
```

### Batch Operations

```python
# Encode multiple strings in parallel
results = tok.encode_batch(["hello", "world", "foo bar"])

# Decode multiple token arrays in parallel
texts = tok.decode_batch(results)
```
```

--------------------------------

### Get Vocabulary Information

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/python/README.md

Access vocabulary details and token IDs using the tokenizer object. These methods do not require network requests as vocabularies are embedded.

```python
tok.get_vocab()
# {'hello': 24912, ...}
```

```python
tok.get_vocab_size()
# 200000
```

```python
tok.token_to_id("hello")
# 24912
```

--------------------------------

### Run WordChipper Benchmarks

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/sample-timer/README.md

Execute the sample-timer benchmark with specific configurations for dataset directory and decoding enabled. This command demonstrates how to run the benchmark and displays the parsed arguments and loaded tokenizer models.

```terminaloutput
$ RAYON_NUM_THREADS=48 cargo run --release -p sample-timer -- \
    --dataset-dir $DATASET_DIR --decode
Args {
    dataset_dir: "/media/Data/nanochat/dataset",
    shards: [
        0,
        1,
    ],
    batch_size: 1024,
    model: OpenaiO200kHarmony,
    ignore_missing: true,
    tiktoken: true,
    tokenizers: true,
    decode: true,
    validate: true,
    respan_input_for_decode_check: false,
}
Loaded:
- "wordchipper::openai/o200k_harmony"
- "tiktoken-rs::o200k_harmony"
- "tokenizers::Xenova/gpt-4o"

Samples Summary:
- num batches: 104
- avg bytes/sample: 4777
- avg bytes/token: 4.8

Encoder Batch Timing:
- "wordchipper::openai/o200k_harmony"
  - batch:      37.1ms
  - sample:     36.2µs
  - bps:    125.85 MiB/s
- "tiktoken-rs::o200k_harmony"
  - batch:      37.4ms
  - sample:     36.5µs
  - bps:    124.68 MiB/s
- "tokenizers::Xenova/gpt-4o"
  - batch:     201.1ms
  - sample:    196.4µs
  - bps:    23.20 MiB/s
  
Decoder Batch Timing:
- "wordchipper::openai/o200k_harmony"
  - batch:       2.9ms
  - sample:      2.9µs
  - bps:    1.55 GiB/s
- "tiktoken-rs::o200k_harmony"
  - batch:       2.2ms
  - sample:      2.1µs
  - bps:    2.12 GiB/s
- "tokenizers::Xenova/gpt-4o"
  - batch:       9.0ms
  - sample:      8.8µs
  - bps:    518.15 MiB/s

```

--------------------------------

### Define Custom Logos Token Enum

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/custom-logos-lexers.md

Example of defining a custom token enum for a lexer using Logos. Each variant maps to a regex pattern for tokenization.

```rust
use logos::Logos;

#[derive(Logos, Debug, PartialEq, Clone)]
enum MyToken {
    #[regex(r"\p{Letter}+ জৈ")]
    Letters,

    #[regex(r"\p{Number}{1,3}")]
    Digits,

    #[regex(r" ?[^\s\p{Letter}\p{Number}]+[\r\n]*")]
    Punctuation,

    #[regex(r"\s*[\r\n]+")]
    Newline,

    #[regex(r"[^\S\r\n]+")]
    Whitespace,
}
```

--------------------------------

### Run Bench-JSON with Human Output Echo

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/README.md

Execute the bench-json binary and simultaneously echo the human-readable benchmark output to stderr while parsing it into JSON. Use --tee for this.

```bash
cargo run --release -p wordchipper-bench --bin bench-json -- --bench spanning --tee
```

--------------------------------

### Internal no_std structure

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/no-std.md

This pattern ensures the crate always starts in no_std mode, conditionally linking std when the 'std' feature is active. Collection types are sourced from 'alloc'.

```rust
#![no_std]

#[cfg(feature = "std")]
extern crate std;

extern crate alloc;
```

--------------------------------

### Complete Tokenizer Training, Saving, and Usage Workflow

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

This snippet shows the end-to-end process of training a tokenizer using BPE, saving the vocabulary, and then using the tokenizer for encoding and decoding. Ensure the vocabulary path is correctly specified for saving.

```rust
use std::sync::Arc;

use wordchipper::{
    TokenEncoder, TokenDecoder,
    Tokenizer, TokenizerOptions, UnifiedTokenVocab, VocabIndex,
    pretrained::openai::OA_CL100K_BASE_PATTERN,
    vocab::{ByteMapVocab, io::save_base64_span_map_path},
};
use wordchipper_training::BPETRainerOptions;

fn main() -> wordchipper::WCResult<()> {
    // --- Train ---
    let mut trainer = BPETRainerOptions::new(OA_CL100K_BASE_PATTERN, 2000).init();

    // In practice, feed much more data
    let corpus = vec![
        "The quick brown fox jumps over the lazy dog.",
        "Pack my box with five dozen liquor jugs.",
        "How vexingly quick daft zebras jump!",
    ];
    trainer.update_from_samples(corpus.iter());

    let vocab: Arc<UnifiedTokenVocab<u32>> = trainer
        .train(ByteMapVocab::default())
        .expect("training failed")
        .into();

    // --- Save ---
    save_base64_span_map_path(
        &vocab.span_vocab().span_map(),
        "/tmp/my_vocab.tiktoken",
    )?;

    // --- Use ---
    let tokenizer = TokenizerOptions::default().build(vocab);
    let tokens = tokenizer.try_encode("The quick brown fox")?;
    let decoded = tokenizer.try_decode_to_string(&tokens)?.unwrap();
    assert_eq!(decoded, "The quick brown fox");

    println!("Encoded {} tokens: {:?}", tokens.len(), tokens);

    Ok(())
}

```

--------------------------------

### Run Bench-JSON for Specific Target

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/README.md

Execute the bench-json binary to run a specific benchmark target and output results. Use --release for optimized builds.

```bash
cargo run --release -p wordchipper-bench --bin bench-json -- --bench spanning
```

--------------------------------

### Implement Gpt2FamilyLogos for Custom Token

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/custom-logos-lexers.md

Map custom token variants to `Gpt2FamilyTokenRole` to define their interaction with preceding whitespace. This example shows specific roles for Letters, Punctuation, Newline, and Digits.

```rust
use wordchipper::spanners::span_lexers::logos::gpt2_family::{
    Gpt2FamilyLogos,
    Gpt2FamilyTokenRole,
};

impl Gpt2FamilyLogos<'_> for MyToken {
    fn family_role(&self) -> Gpt2FamilyTokenRole {
        match self {
            // Whitespace is buffered; last char may merge into next token.
            Self::Whitespace => Gpt2FamilyTokenRole::Whitespace,

            // Letters absorb a preceding space when the token starts with
            // a letter. No contraction splitting needed for our pattern.
            Self::Letters => Gpt2FamilyTokenRole::Word {
                check_contraction: false,
                first_char_is_letter: true,
            },

            // Punctuation absorbs a preceding ASCII space (the ` ?` prefix).
            Self::Punctuation => Gpt2FamilyTokenRole::Punctuation,

            // Newlines are buffered separately.
            Self::Newline => Gpt2FamilyTokenRole::Newline,

            // Digits stand alone. They never merge with preceding whitespace.
            Self::Digits => Gpt2FamilyTokenRole::Standalone,
        }
    }
}
```

--------------------------------

### Quick Start JavaScript/TypeScript WASM Tokenizer

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/bindings.md

Load a tokenizer from a pre-trained model, encode text to tokens, and decode tokens back to text. Remember to free WASM memory when done.

```javascript
import { Tokenizer } from "./js/dist/index.js";

const tok = await Tokenizer.fromPretrained("o200k_base");

const tokens = tok.encode("hello world"); // Uint32Array [24912, 2375]
const text = tok.decode(tokens); // "hello world"

tok.free(); // release WASM memory when done
```

--------------------------------

### Span Counting Example

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Illustrates the first phase of tokenizer training: counting the occurrences of unique spans after splitting input text using a regex pattern. This builds the initial frequency table.

```text
Input: "the cat sat on the mat"

Regex splits: ["the", " cat", " sat", " on", " the", " mat"]

Span counts:
  "the"  -> 1
  " the" -> 1
  " cat" -> 1
  " sat" -> 1
  " on"  -> 1
  " mat" -> 1
```

--------------------------------

### Run wordchipper vs. tiktoken-rs Benchmark in Bash

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/performance.md

Execute the `sample-timer` tool to compare wordchipper's performance against tiktoken-rs. Specify the dataset directory, number of shards, and the vocabulary model to use for the benchmark. This command requires the `parallel` feature to be enabled.

```bash
RAYON_NUM_THREADS=48 cargo run --release -p sample-timer -- \
    --dataset-dir $DATASET_DIR --shards 0 --model openai::cl100k_base
```

--------------------------------

### Build Static HTML for Wordchipper Book

Source: https://github.com/crutcher/wordchipper/blob/main/book/README.md

Builds the static HTML version of the Wordchipper book. The output will be located in the `book/book/` directory.

```bash
mdbook build
```

--------------------------------

### Feeding Training Data in Batches

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/training.md

Shows how to incrementally feed training data to the BPE trainer by calling `update_from_samples` multiple times, allowing for processing large datasets.

```rust
# use wordchipper::pretrained::openai::OA_CL100K_BASE_PATTERN;
# use wordchipper_training::BPETRainerOptions;
# let mut trainer = BPETRainerOptions::new(OA_CL100K_BASE_PATTERN, 1000).init();
// First batch
let batch1 = vec!["hello world", "hello there"];
trainer.update_from_samples(batch1.iter());

// Second batch
let batch2 = vec!["world peace", "hello again"];
trainer.update_from_samples(batch2.iter());

// Counts from both batches are combined before training
```

--------------------------------

### Decode Tokens to String with TokenDecoder

Source: https://context7.com/crutcher/wordchipper/llms.txt

Decodes a vector of tokens into a string. Handles potential decoding errors by returning a `WCResult<DecodeResult<String>>`. The `DecodeResult` can be unwrapped to get the string or handle incomplete decodes. Also demonstrates batch decoding.

```rust
use wordchipper::{
    TokenDecoder,
    TokenizerOptions,
    WCResult,
    disk_cache::WordchipperDiskCache,
    load_vocab,
};

fn decode_example() -> WCResult<()> {
    let mut cache = WordchipperDiskCache::default();
    let labeled = load_vocab("openai:cl100k_base", &mut cache)?;
    let tokenizer = TokenizerOptions::default().build(labeled.vocab().clone());

    let tokens: Vec<u32> = vec![15339, 1917];
    let result = tokenizer.try_decode_to_string(&tokens)?;
    let text: String = result.unwrap(); // unwrap DecodeResult
    println!("{text:?}"); // "hello world"

    // Batch decode
    let batch: Vec<&[u32]> = vec![&[15339], &[1917]];
    let strings = tokenizer
        .try_decode_batch_to_strings(&batch)?
        .unwrap(); // Vec<String>
    println!("{strings:?}"); // ["hello", " world"]

    Ok(())
}
```

--------------------------------

### List Available Models with wordchipper-cli models list

Source: https://github.com/crutcher/wordchipper/blob/main/crates/wordchipper-cli/README.md

Lists all available pretrained tokenizer models that can be used with wordchipper-cli.

```bash
wordchipper-cli models list                  
"openai" - Pretrained vocabularies from OpenAI
  * "openai:gpt2"
    GPT-2 `gpt2` vocabulary
  * "openai:r50k_base"
    GPT-2 `p50k_base` vocabulary
  * "openai:p50k_base"
    GPT-2 `p50k_base` vocabulary
  * "openai:p50k_edit"
    GPT-2 `p50k_edit` vocabulary
  * "openai:cl100k_base"
    GPT-3 `cl100k_base` vocabulary
  * "openai:o200k_base"
    GPT-5 `o200k_base` vocabulary
  * "openai:o200k_harmony"
    GPT-5 `o200k_harmony` vocabulary
```

--------------------------------

### Use u16 Token Type for Smaller Vocabularies

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/getting-started.md

Illustrates how to use a `u16` token type for vocabularies that fit within 16 bits (less than 65,536 entries), saving memory. This example converts a default vocabulary to `u16` and builds a tokenizer with it.

```rust
use std::sync::Arc;
use wordchipper::{UnifiedTokenVocab, TokenizerOptions, load_vocab, disk_cache::WordchipperDiskCache};

fn main() -> wordchipper::WCResult<()> {
    let mut cache = WordchipperDiskCache::default();
    let (_desc, vocab) = load_vocab("openai:r50k_base", &mut cache)?;

    // Convert to u16 (works for r50k_base with ~50k tokens)
    let vocab_u16: Arc<UnifiedTokenVocab<u16>> = Arc::new(vocab.as_ref().to_token_type());
    let tok = TokenizerOptions::default().build(vocab_u16);

    Ok(())
}

```

--------------------------------

### Tokenizer.fromPretrained (JavaScript/WASM)

Source: https://context7.com/crutcher/wordchipper/llms.txt

Loads a tokenizer by fetching the vocab file and initializing the WASM module. Releases WASM memory with `tok.free()`.

```APIDOC
## Tokenizer.fromPretrained

### Description
Loads a tokenizer in the browser or Node.js by fetching the vocab file via `fetch()` (browser) or the network (Node.js). It initializes the WASM module and returns a `Tokenizer` instance. Remember to call `tok.free()` to release WASM memory when done.

### Method
```javascript
static async fromPretrained(modelName)
```

### Parameters
#### Path Parameters
- **modelName** (string) - Required - The name of the pre-trained model to load (e.g., "o200k_base").

### Request Example
```javascript
import { Tokenizer } from "./js/dist/index.js";

// Load from OpenAI CDN
const tok = await Tokenizer.fromPretrained("o200k_base");

// ... use tok ...

tok.free();
```

### Response
#### Success Response (Tokenizer Instance)
- **tok** (Tokenizer) - An initialized Tokenizer instance.

### Related Methods
- `Tokenizer.fromVocabData(modelName, data)`: Load from pre-fetched bytes.
- `tok.free()`: Release WASM memory.
```

--------------------------------

### Load Tokenizer Vocabulary by Name

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/pretrained-models.md

Use `special_tokens::<T>()` to get the special token list for a given tokenizer name. Use `load_vocab::<T>(loader)` to load the vocabulary with download support. The `o200k_base` tokenizer is recommended for general use due to its large vocabulary.

```rust
use wordchipper::special_tokens;
use wordchipper::load_vocab;

// Load the cl100k_base tokenizer vocabulary
let vocab = load_vocab::<cl100k_base>(None).unwrap();

// Get the special tokens for cl100k_base
let special_tokens = special_tokens::<cl100k_base>();
```

```rust
use wordchipper::special_tokens;
use wordchipper::load_vocab;

// Load the o200k_base tokenizer vocabulary
let vocab = load_vocab::<o200k_base>(None).unwrap();

// Get the special tokens for o200k_base
let special_tokens = special_tokens::<o200k_base>();
```

```rust
use wordchipper::special_tokens;
use wordchipper::load_vocab;

// Load the p50k_base tokenizer vocabulary
let vocab = load_vocab::<p50k_base>(None).unwrap();

// Get the special tokens for p50k_base
let special_tokens = special_tokens::<p50k_base>();
```

```rust
use wordchipper::special_tokens;
use wordchipper::load_vocab;

// Load the r50k_base tokenizer vocabulary
let vocab = load_vocab::<r50k_base>(None).unwrap();

// Get the special tokens for r50k_base
let special_tokens = special_tokens::<r50k_base>();
```

--------------------------------

### Build and Run Benchmarks

Source: https://github.com/crutcher/wordchipper/blob/main/bindings/python/README.md

Build the library in release mode for accurate benchmarks and run all benchmarks using pytest. Filtering options are available to target specific tests.

```bash
# Build in release mode for meaningful numbers
maturin develop --release --features python-extension-module

# Run all benchmarks
pytest benchmarks/

# Run only single-encode benchmarks
pytest benchmarks/ -k "TestSingleEncode"

# Run only batch-encode benchmarks
pytest benchmarks/ -k "TestBatchEncode"

# Run only decode benchmarks
pytest benchmarks/ -k "TestSingleDecode"

# Filter by model
pytest benchmarks/ -k "cl100k_base"
```

--------------------------------

### Initialize WASM and Tokenizer

Source: https://github.com/crutcher/wordchipper/blob/main/book/src/interactive-tokenizer.md

Imports and initializes the WebAssembly module for the tokenizer. Handles potential loading errors and provides feedback on the loading status.

```javascript
const statusEl = document.getElementById("demo-status");
const outputEl = document.getElementById("demo-output");
const modelEl = document.getElementById("demo-model");
const textEl = document.getElementById("demo-text");
const runBtn = document.getElementById("demo-run");

if (!statusEl || !runBtn) throw new Error("demo elements not found");

const VOCAB_URLS = {
    r50k_base:   "wasm/vocab/r50k_base.tiktoken",
    p50k_base:   "wasm/vocab/p50k_base.tiktoken",
    cl100k_base: "wasm/vocab/cl100k_base.tiktoken",
    o200k_base:  "wasm/vocab/o200k_base.tiktoken",
};

let init, Tokenizer;
try {
    const mod = await import("./wasm/wordchipper_wasm.js");
    init = mod.default;
    Tokenizer = mod.Tokenizer;
    await init();
} catch (e) {
    statusEl.textContent = "WASM not loaded. Run ./book/setup-wasm.sh to build.";
    throw e;
}

const cache = {};

statusEl.textContent = "WASM loaded. Select a model and click Encode & Decode.";
runBtn.disabled = false;
```

--------------------------------

### Run Specific Cargo Benchmarks

Source: https://github.com/crutcher/wordchipper/blob/main/dev-crates/wordchipper-bench/README.md

Execute individual benchmark targets within the wordchipper-bench crate. Use the --bench flag followed by the benchmark name.

```bash
cargo bench -p wordchipper-bench --bench encoding_single
```

```bash
cargo bench -p wordchipper-bench --bench encoding_parallel
```

```bash
cargo bench -p wordchipper-bench --bench decoding_single
```

```bash
cargo bench -p wordchipper-bench --bench spanning
```