### Tokenizer Usage Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Demonstrates how to get a CoreBPE instance for a specific tokenizer type and encode text. Ensure you have the necessary imports.

```rust
use tiktoken_rs::tokenizer::Tokenizer;
use tiktoken_rs::bpe_for_tokenizer;

let tokenizer = Tokenizer::O200kBase;
let bpe = bpe_for_tokenizer(tokenizer)?;
let tokens = bpe.encode_with_special_tokens("hello");
```

--------------------------------

### Generate Site Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Example of generating a site using the gen_site function. This snippet is part of a larger script and is typically run when the script is executed directly.

```python
if __name__ == "__main__":
    gen_site("basic-blog", [""], 250, paginate=True)
```

--------------------------------

### Example Usage of CoreBPE and EncodeError

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Demonstrates how to initialize a CoreBPE instance using cl100k_base() and handle potential EncodeError during the encoding process. This example shows basic tokenization and error reporting.

```rust
use tiktoken_rs::{cl100k_base, EncodeError};
use std::collections::HashSet;

let bpe = cl100k_base()?;
match bpe.encode("text", &HashSet::new()) {
    Ok((tokens, _)) => println!("Tokens: {:?}", tokens),
    Err(e: EncodeError) => eprintln!("Encoding failed: {}", e.message),
}
```

--------------------------------

### Heap Profiling Output Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Example output from DHAT when performing heap profiling. This output details memory usage and provides a filename for the saved data.

```text
dhat: Total:     1,256 bytes in 6 blocks
dhat: At t-gmax: 1,256 bytes in 6 blocks
dhat: At t-end:  1,256 bytes in 6 blocks
dhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html
```

--------------------------------

### Decode Tokens Back to Text

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

This example demonstrates how to decode a list of token IDs back into human-readable text using the `cl100k_base` tokenizer and its `decode` method. It first encodes a string to get token IDs and then decodes them.

```rust
use tiktoken_rs::cl100k_base;

fn main() -> Result<()> {
    let bpe = cl100k_base()?;
    
    // Encode some text
    let tokens = bpe.encode_with_special_tokens("Hello world");
    println!("Tokens: {:?}", tokens);
    
    // Decode back
    let text = bpe.decode(&tokens)?;
    println!("Decoded: {}", text);
    Ok(())
}
```

--------------------------------

### Prepare API Request Parameters

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-index.md

This example demonstrates how to estimate the maximum number of tokens that can be used in an API request for a given model and message list. This is useful for setting request limits.

```rust
use tiktoken_rs::{ChatCompletionRequestMessage, get_chat_completion_max_tokens};

let messages = vec![/* ... */];
let max_tokens = get_chat_completion_max_tokens("gpt-4o", &messages)?;

// Use max_tokens in API request
```

--------------------------------

### Inspect Token Breakdown

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-index.md

This example illustrates how to split a string into its constituent tokens and inspect each token individually. It uses the `cl100k_base` tokenizer.

```rust
use tiktoken_rs::cl100k_base;

let bpe = cl100k_base()?;
let tokens = bpe.split_by_token("Hello world", true)?;
for (i, token) in tokens.iter().enumerate() {
    println!("Token {}: '{}'", i, token);
}
```

--------------------------------

### DHAT Heap Profiling Output Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

This snippet shows an example of the backtrace output generated by DHAT when heap profiling is enabled. It illustrates the format and the type of information provided, including allocation details and call stacks.

```rust
#!     #11: 0x10ae8441b: std::sync::once::Once::call_once_force::{{closure}} (src/sync/once.rs:320:40)
//!     #12: 0x10aea564c: std::sync::once::Once::call_inner (src/sync/once.rs:419:21)
//!     #13: 0x10ae81b1b: std::sync::once::Once::call_once_force (src/sync/once.rs:320:9)
//!     #14: 0x10ae81b1b: std::lazy::SyncOnceCell<T>::get_or_init_pin (std/src/lazy.rs:374:9)
//!     #15: 0x10ae81b1b: std::io::stdio::stdout (src/io/stdio.rs:679:16)
//!     #16: 0x10ae81b1b: std::io::stdio::print_to (src/io/stdio.rs:1196:21)
//!     #17: 0x10ae81b1b: std::io::stdio::_print (src/io/stdio.rs:1209:5)
//!     #18: 0x10ae2fe20: dhatter::main (dhatter/src/main.rs:8:5)
//!   }
//! }
```

--------------------------------

### Install tiktoken-rs with Cargo

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md

Install the tiktoken-rs crate locally using Cargo. This is the first step before using the library in your Rust code.

```sh
cargo add tiktoken-rs
```

--------------------------------

### Example Usage of ENDOFTEXT Special Token

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Demonstrates how to use the `ENDOFTEXT` special token constant in Rust. This example shows importing and printing the token string.

```rust
use tiktoken_rs::tiktoken_ext::openai_public::ENDOFTEXT;

let special_token = ENDOFTEXT;
println!("Token: {}", special_token);  // "<|endoftext|>"
```

--------------------------------

### Create FunctionCall Instance

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Example of creating a `FunctionCall` instance with a function name and JSON arguments. Ensure arguments are validated before use.

```rust
use tiktoken_rs::FunctionCall;

let fc = FunctionCall {
    name: "get_weather".to_string(),
    arguments: r#"{"city": "Paris"}"#.to_string(),
};
```

--------------------------------

### Get Context Size for a Model

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

This example demonstrates how to retrieve the context size (maximum number of tokens) for various models using `get_context_size`. It handles cases where a model's context size might be unknown.

```rust
use tiktoken_rs::model::get_context_size;

fn main() {
    let models = vec!["gpt-4o", "gpt-4", "gpt-3.5-turbo"];
    
    for model in models {
        match get_context_size(model) {
            Some(size) => println!("{}: {} tokens", model, size),
            None => println!("{}: unknown", model),
        }
    }
}
```

--------------------------------

### Initialize Heap Profiler in Main Function

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Add this code to the start of your main function to initialize heap profiling. This should only be enabled while profiling.

```rust
let _profiler = dhat::Profiler::new_heap();
```

--------------------------------

### p50k_base_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the p50k_base tokenizer. Returns a static reference to a `CoreBPE` instance.

```APIDOC
## p50k_base_singleton()

### Description
Get or initialize a cached singleton instance of the p50k_base tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls.

### Signature
```rust
pub fn p50k_base_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.
```

--------------------------------

### DHAT Viewer Node Structure Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

An example of a node structure as displayed in the DHAT viewer. It shows memory statistics, allocation counts, average sizes, and lifetime information.

```text
PP 1.1/2 {
  Total:     1,024 bytes (98.46%, 14,422,535.21/s) in 1 blocks (50%, 14,084.51/s), avg size 1,024 bytes, avg lifetime 35 µs (49.3% of program duration)
  Max:       1,024 bytes in 1 blocks, avg size 1,024 bytes
  At t-gmax: 1,024 bytes (98.46%) in 1 blocks (50%), avg size 1,024 bytes
  At t-end:  1,024 bytes (100%) in 1 blocks (100%), avg size 1,024 bytes
  Allocated at {
    #1: 0x10ae8441b: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc/src/alloc.rs:226:9)
    #2: 0x10ae8441b: alloc::raw_vec::RawVec<T,A>::allocate_in (alloc/src/raw_vec.rs:207:45)
    #3: 0x10ae8441b: alloc::raw_vec::RawVec<T,A>::with_capacity_in (alloc/src/raw_vec.rs:146:9)
    #4: 0x10ae8441b: alloc::vec::Vec<T,A>::with_capacity_in (src/vec/mod.rs:609:20)
    #5: 0x10ae8441b: alloc::vec::Vec<T>::with_capacity (src/vec/mod.rs:470:9)
    #6: 0x10ae8441b: std::io::buffered::bufwriter::BufWriter<W>::with_capacity (io/buffered/bufwriter.rs:115:33)
    #7: 0x10ae8441b: std::io::buffered::linewriter::LineWriter<W>::with_capacity (io/buffered/linewriter.rs:109:29)
    #8: 0x10ae8441b: std::io::buffered::linewriter::LineWriter<W>::new (io/buffered/linewriter.rs:89:9)
    #9: 0x10ae8441b: std::io::stdio::stdout::{{closure}} (src/io/stdio.rs:680:58)
    #10: 0x10ae8441b: std::lazy::SyncOnceCell<T>::get_or_init_pin::{{closure}} (std/src/lazy.rs:375:25)
  }
}
```

--------------------------------

### Heap Usage Testing Configuration

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

This example demonstrates how to configure a Rust project for heap usage testing using the DHAT crate. It includes setting the global allocator and building a profiler instance within a test function.

```rust
#[global_allocator]
static ALLOC: dhat::Alloc = dhat::Alloc;

# // Tricky: comment out the `#[test]` because it's needed in an actual
# // test but messes up things here.
# /*
#[test]
# */
fn test() {
    let _profiler = dhat::Profiler::builder().testing().build();

    let _v1 = vec![1, 2, 3, 4];
    let v2 = vec![5, 6, 7, 8];
    drop(v2);

```

--------------------------------

### Get p50k_base Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the p50k_base tokenizer.

```rust
use tiktoken_rs::p50k_base_singleton;

let bpe = p50k_base_singleton();
```

--------------------------------

### Get o200k_harmony Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the o200k_harmony tokenizer.

```rust
use tiktoken_rs::o200k_harmony_singleton;

let bpe = o200k_harmony_singleton();
```

--------------------------------

### r50k_base_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the r50k_base tokenizer. This is preferred over `r50k_base()` for repeated tokenization operations.

```APIDOC
## r50k_base_singleton()

### Description
Get or initialize a cached singleton instance of the r50k_base tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls, making it more efficient for repeated tokenization operations.

### Signature
```rust
pub fn r50k_base_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.

### Example
```rust
use tiktoken_rs::r50k_base_singleton;

let bpe = r50k_base_singleton();
for text in &["hello", "world"] {
    let tokens = bpe.encode_with_special_tokens(text);
    println!("Tokens: {:?}", tokens);
}
```
```

--------------------------------

### Get o200k_base Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the o200k_base tokenizer. Recommended for GPT-4o and GPT-5 models.

```rust
use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
```

--------------------------------

### Ad Hoc Profiling Output Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Example output from DHAT when performing ad hoc profiling. This output summarizes the number of units and events recorded, and provides a filename for the saved data.

```text
dhat: Total:     141 units in 11 events
dhat: The data has been saved to dhat-ad-hoc.json, and is viewable with dhat/dh_view.html
```

--------------------------------

### o200k_harmony_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the o200k_harmony tokenizer. Returns a static reference to a `CoreBPE` instance.

```APIDOC
## o200k_harmony_singleton()

### Description
Get or initialize a cached singleton instance of the o200k_harmony tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls.

### Signature
```rust
pub fn o200k_harmony_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.
```

--------------------------------

### Get r50k_base Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the r50k_base tokenizer. Preferred over `r50k_base()` for repeated tokenization operations.

```rust
use tiktoken_rs::r50k_base_singleton;

let bpe = r50k_base_singleton();
for text in &["hello", "world"] {
    let tokens = bpe.encode_with_special_tokens(text);
    println!("Tokens: {:?}", tokens);
}
```

--------------------------------

### p50k_edit_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the p50k_edit tokenizer. Returns a static reference to a `CoreBPE` instance.

```APIDOC
## p50k_edit_singleton()

### Description
Get or initialize a cached singleton instance of the p50k_edit tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls.

### Signature
```rust
pub fn p50k_edit_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.
```

--------------------------------

### ChatCompletionRequestMessage Usage Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Illustrates how to create a `ChatCompletionRequestMessage` instance. This is useful for constructing input for chat-based language models.

```rust
use tiktoken_rs::ChatCompletionRequestMessage;

let message = ChatCompletionRequestMessage {
    role: "user".to_string(),
    content: Some("What is the capital of France?".to_string()),
    name: None,
    function_call: None,
    tool_calls: vec![],
    refusal: None,
};
```

--------------------------------

### Catching and Handling anyhow::Error

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/05-errors.md

Demonstrates how to catch and handle `anyhow::Error` returned from initialization functions like `cl100k_base()`. This is a common pattern for dealing with potential errors during tokenizer setup.

```rust
use tiktoken_rs::cl100k_base;

match cl100k_base() {
    Ok(bpe) => println!("Tokenizer loaded"),
    Err(e) => eprintln!("Failed to load tokenizer: {}", e),
}
```

--------------------------------

### o200k_base_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the o200k_base tokenizer. Recommended for GPT-4o and GPT-5 models. Returns a static reference to a `CoreBPE` instance.

```APIDOC
## o200k_base_singleton()

### Description
Get or initialize a cached singleton instance of the o200k_base tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls. It is recommended for GPT-4o and GPT-5 models.

### Signature
```rust
pub fn o200k_base_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.
```

--------------------------------

### cl100k_base_singleton()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the cl100k_base tokenizer. Recommended for chat completion token counting. Returns a static reference to a `CoreBPE` instance.

```APIDOC
## cl100k_base_singleton()

### Description
Get or initialize a cached singleton instance of the cl100k_base tokenizer. This function provides a static reference to a `CoreBPE` instance that is reused across all calls. It is recommended for chat completion token counting.

### Signature
```rust
pub fn cl100k_base_singleton() -> &'static CoreBPE
```

### Returns
A static reference to a `CoreBPE` instance.

### Example
```rust
use tiktoken_rs::cl100k_base_singleton;

let bpe = cl100k_base_singleton();
let token_count = bpe.count_with_special_tokens("Chat message");
```
```

--------------------------------

### Simple Token Counting for a Single Message

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/06-async-openai-integration.md

This example shows the basic usage of `num_tokens_from_messages` to count tokens for a simple text message using the 'gpt-4o' model.

```rust
use tiktoken_rs::async_openai::num_tokens_from_messages;
use async_openai::types::chat::{
    ChatCompletionRequestMessage,
    ChatCompletionRequestUserMessage,
    ChatCompletionRequestUserMessageContent,
};

#[tokio::main]
async fn main() -> Result<()> {
    let messages = &[ChatCompletionRequestMessage::User(
        ChatCompletionRequestUserMessage {
            content: ChatCompletionRequestUserMessageContent::Text(
                "What is Rust?".to_string(),
            ),
            name: None,
        },
    )];

    let count = num_tokens_from_messages("gpt-4o", messages)?;
    println!("This message uses {} tokens", count);
    Ok(())
}
```

--------------------------------

### Rust Example: Using tiktoken-rs for Model and Tokenizer Operations

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/04-model-tokenizer-mapping.md

This snippet demonstrates how to use the tiktoken-rs library in Rust to interact with different models and their tokenizers. It shows how to retrieve tokenizer details, context sizes, tokenize strings, and calculate maximum response tokens for chat completions.

```rust
use tiktoken_rs::{
    tokenizer::get_tokenizer,
    model::get_context_size,
    bpe_for_model,
    get_chat_completion_max_tokens,
    ChatCompletionRequestMessage,
};

fn main() -> Result<()> {
    let model = "gpt-4o";
    
    // Determine tokenizer
    let tokenizer = get_tokenizer(model).unwrap();
    println!("Tokenizer: {:?}", tokenizer);
    
    // Get context size
    let ctx_size = get_context_size(model).unwrap();
    println!("Context size: {}", ctx_size);
    
    // Get BPE and tokenize
    let bpe = bpe_for_model(model)?;
    let tokens = bpe.encode_with_special_tokens("Hello, world!");
    println!("Token count: {}", tokens.len());
    
    // Calculate remaining tokens for chat
    let messages = vec![
        ChatCompletionRequestMessage {
            role: "system".to_string(),
            content: Some("You are helpful.".to_string()),
            ..Default::default()
        },
    ];
    let max_response = get_chat_completion_max_tokens(model, &messages)?;
    println!("Max response tokens: {}", max_response);
    
    Ok(())
}
```

--------------------------------

### Ad Hoc Event Profiling Example

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Illustrates how to use `ad_hoc_event` for custom profiling. Use an integer argument to represent the weight of the event; use 1 if no meaningful weight is applicable.

```rust
//! sites. You might want to know how often it is called and which other
//! functions called it the most. In that case, you would add an
//! [`ad_hoc_event`] call to that function, and the data collected by this
//! crate and viewed with DHAT's viewer would show you exactly what you want to
//! know.
//!
//! The meaning of the integer argument to `ad_hoc_event` will depend on
//! exactly what you are measuring. If there is no meaningful weight to give to
//! an event, you can just use `1`.
```

--------------------------------

### Initialize Ad Hoc Profiler in Main Function

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt

Initialize ad hoc profiling by adding this code to the start of your main function. This enables manual annotation of code points.

```rust
let _profiler = dhat::Profiler::new_ad_hoc();
```

--------------------------------

### Encode Text to Tokens using Rank

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/03-types.md

Example of encoding a string into a vector of token ranks using a specific tokenizer (o200k_base). The result is a `Vec<Rank>`.

```rust
use tiktoken_rs::{o200k_base, Rank};

let bpe = o200k_base()?;
let tokens: Vec<Rank> = bpe.encode_with_special_tokens("hello");
// tokens is Vec<u32>
```

--------------------------------

### Run Main Binary

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/CONTRIBUTING.md

Use this command to run the main binary of the project. You can optionally pass arguments to the binary.

```bash
just run
just r [...]
```

--------------------------------

### Initialize a Tokenizer (cl100k_base)

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Demonstrates the basic initialization of a tokenizer using the cl100k_base encoding. This is useful for general-purpose tokenization.

```rust
use tiktoken_rs::cl100k_base;

fn main() -> Result<()> {
    let bpe = cl100k_base()?;
    let tokens = bpe.encode_with_special_tokens("Hello, world!");
    println!("Token count: {}", tokens.len());
    Ok(())
}
```

--------------------------------

### Get Chat Completion Max Tokens with async-openai

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md

Use this function to determine the maximum token count for a chat completion request. Ensure the `async-openai` feature is enabled in your `Cargo.toml`. This example uses specific message types from the `async-openai` crate.

```rust
use tiktoken_rs::async_openai::get_chat_completion_max_tokens;
use async_openai::types::chat::{
    ChatCompletionRequestMessage,
    ChatCompletionRequestSystemMessage,
    ChatCompletionRequestSystemMessageContent,
    ChatCompletionRequestUserMessage,
    ChatCompletionRequestUserMessageContent,
};

let messages = vec![
    ChatCompletionRequestMessage::System(ChatCompletionRequestSystemMessage {
        content: ChatCompletionRequestSystemMessageContent::Text(
            "You are a helpful assistant that only speaks French.".to_string(),
        ),
        name: None,
    }),
    ChatCompletionRequestMessage::User(ChatCompletionRequestUserMessage {
        content: ChatCompletionRequestUserMessageContent::Text(
            "Hello, how are you?".to_string(),
        ),
        name: None,
    }),
];
let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap();
println!("max_tokens: {}", max_tokens);
```

--------------------------------

### Get p50k_edit Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the p50k_edit tokenizer.

```rust
use tiktoken_rs::p50k_edit_singleton;

let bpe = p50k_edit_singleton();
```

--------------------------------

### Initialization & Singletons

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-index.md

Provides access to pre-defined tokenizers for different GPT models. Singleton versions offer efficient, shared access.

```APIDOC
## Initialization & Singletons

### `r50k_base()` / `r50k_base_singleton()`

- **Description**: Get the tokenizer for GPT-3 models.
- **Usage**: `r50k_base()` or `r50k_base_singleton()`

### `p50k_base()` / `p50k_base_singleton()`

- **Description**: Get the tokenizer for Code models.
- **Usage**: `p50k_base()` or `p50k_base_singleton()`

### `p50k_edit()` / `p50k_edit_singleton()`

- **Description**: Get the tokenizer for Edit models.
- **Usage**: `p50k_edit()` or `p50k_edit_singleton()`

### `cl100k_base()` / `cl100k_base_singleton()`

- **Description**: Get the tokenizer for Chat models.
- **Usage**: `cl100k_base()` or `cl100k_base_singleton()`

### `o200k_base()` / `o200k_base_singleton()`

- **Description**: Get the tokenizer for GPT-4o, GPT-5, o1, and o3 models.
- **Usage**: `o200k_base()` or `o200k_base_singleton()`

### `o200k_harmony()` / `o200k_harmony_singleton()`

- **Description**: Get the tokenizer for gpt-oss models.
- **Usage**: `o200k_harmony()` or `o200k_harmony_singleton()`
```

--------------------------------

### Handle Tokenizer Loading with Try-Catch

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Illustrates a basic try-catch pattern for loading a tokenizer and encoding text. This approach explicitly handles potential errors during tokenizer initialization.

```rust
use tiktoken_rs::cl100k_base;

fn main() {
    match cl100k_base() {
        Ok(bpe) => {
            let tokens = bpe.encode_with_special_tokens("test");
            println!("Success: {} tokens", tokens.len());
        }
        Err(e) => eprintln!("Error: {}", e),
    }
}
```

--------------------------------

### Get cl100k_base Singleton Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get or initialize a cached singleton instance of the cl100k_base tokenizer. Recommended for chat completion token counting.

```rust
use tiktoken_rs::cl100k_base_singleton;

let bpe = cl100k_base_singleton();
let token_count = bpe.count_with_special_tokens("Chat message");
```

--------------------------------

### Run All Tests

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/CONTRIBUTING.md

Execute all tests in the project using this command.

```bash
just test
just t
```

--------------------------------

### Initialize o200k_harmony Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the o200k_harmony tokenizer for open-source models. Use for gpt-oss models like gpt-oss-20b and gpt-oss-120b. Returns a new CoreBPE instance with extended special tokens.

```rust
use tiktoken_rs::o200k_harmony;

let bpe = o200k_harmony()?;
let special = bpe.special_tokens();
```

--------------------------------

### Initialize o200k_base Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the o200k_base tokenizer for modern OpenAI models. Use for GPT-5 series, GPT-4.1, GPT-4o, o1, o3, and o4 models. Returns a new CoreBPE instance.

```rust
use tiktoken_rs::o200k_base;

let bpe = o200k_base()?;
let tokens = bpe.encode_with_special_tokens("Hello, GPT-4o!");
```

--------------------------------

### Initialize p50k_base Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the p50k_base tokenizer for code models. Use for text-davinci-002 and text-davinci-003. Returns a new CoreBPE instance.

```rust
use tiktoken_rs::p50k_base;

let bpe = p50k_base()?;
let token_count = bpe.count_with_special_tokens("def hello():");
```

--------------------------------

### Logging and Recovery on Initialization Failure

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/05-errors.md

Shows how to log an error during tokenizer initialization and then abort the operation. This pattern is useful for critical failures where continuing is not possible.

```rust
use tiktoken_rs::cl100k_base;

let bpe = match cl100k_base() {
    Ok(b) => b,
    Err(e) => {
        eprintln!("Failed to initialize tokenizer: {}", e);
        eprintln!("Aborting operation");
        return Err(e);
    }
};
```

--------------------------------

### o200k_base()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the o200k_base tokenizer for modern OpenAI models, including the GPT-5 series and GPT-4o.

```APIDOC
## o200k_base()

### Description
Initializes the o200k_base tokenizer for modern OpenAI models. This tokenizer is suitable for the latest models.

### Signature
```rust
pub fn o200k_base() -> Result<CoreBPE>
```

### Returns
A new `CoreBPE` instance for the latest models.

### Use for
GPT-5 series, GPT-4.1, GPT-4o, o1, o3, o4 models.

### Example
```rust
use tiktoken_rs::o200k_base;

let bpe = o200k_base()?;
let tokens = bpe.encode_with_special_tokens("Hello, GPT-4o!");
```
```

--------------------------------

### Initialize r50k_base Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the r50k_base tokenizer, also known as gpt2. Use for GPT-3 models like davinci, curie, babbage, and ada. Returns a new CoreBPE instance.

```rust
use tiktoken_rs::r50k_base;

let bpe = r50k_base()?;
let tokens = bpe.encode_with_special_tokens("Hello world");
println!("Token count: {}", tokens.len());
```

--------------------------------

### Instantiate CoreBPE with Custom Mappings

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/02-corebpe-methods.md

Create a new CoreBPE instance using custom encoder mappings and a regex pattern. This method is rarely used directly; prefer initialization functions. Ensure necessary imports are present.

```rust
use tiktoken_rs::CoreBPE;
use rustc_hash::FxHashMap as HashMap;

let encoder = HashMap::new(); // normally loaded from .tiktoken files
let special_tokens = HashMap::new();
let bpe = CoreBPE::new(encoder, special_tokens, r#"'(?:[sdmt]|ll|ve|re)""#)?;
```

--------------------------------

### Use a Singleton Tokenizer for Performance

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Shows how to use a singleton instance of a tokenizer for improved performance when performing multiple tokenization operations. This avoids repeated initialization.

```rust
use tiktoken_rs::cl100k_base_singleton;

fn main() {
    let bpe = cl100k_base_singleton();
    
    // Call multiple times without re-initializing
    for text in &["hello", "world", "test"] {
        let count = bpe.count_with_special_tokens(text);
        println!("'{}': {} tokens", text, count);
    }
}
```

--------------------------------

### Catching DecodeKeyError

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/05-errors.md

Shows how to catch DecodeKeyError when a token rank is not found during decoding. An example of an invalid token is provided.

```rust
use tiktoken_rs::cl100k_base;

let bpe = cl100k_base()?;
let tokens = vec![15339, 999999]; // 999999 may not exist

match bpe.decode_bytes(&tokens) {
    Ok(bytes) => println!("Decoded: {:?}", bytes),
    Err(e) => eprintln!("Invalid token: {}", e.token),
}
```

--------------------------------

### Compare Non-Singleton Tokenization (Less Efficient)

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Demonstrates the less efficient approach of initializing the tokenizer within a loop, leading to repeated initialization overhead. This is generally not recommended for performance-critical applications.

```rust
use tiktoken_rs::cl100k_base;

fn main() -> anyhow::Result<()> {
    for _ in 0..10000 {
        let bpe = cl100k_base()?; // Initialized 10,000 times
        let _tokens = bpe.encode_with_special_tokens("test");
    }
    Ok(())
}
```

--------------------------------

### Count Token Length with Singleton

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md

Utilize the `o200k_base_singleton` to get a shared instance of the tokenizer. This is more efficient for repeated calls as it avoids re-initialization.

```rust
use tiktoken_rs::o200k_base_singleton;

let bpe = o200k_base_singleton();
let tokens = bpe.encode_with_special_tokens(
  "This is a sentence   with spaces"
);
println!("Token count: {}", tokens.len());
```

--------------------------------

### r50k_base()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the r50k_base tokenizer, also known as gpt2. This tokenizer is suitable for older GPT-3 models.

```APIDOC
## r50k_base()

### Description
Initializes the r50k_base tokenizer (also known as `gpt2`). This tokenizer is suitable for older GPT-3 models like `davinci`, `curie`, `babbage`, and `ada`.

### Signature
```rust
pub fn r50k_base() -> Result<CoreBPE>
```

### Returns
A new `CoreBPE` instance configured for GPT-3 models.

### Example
```rust
use tiktoken_rs::r50k_base;

let bpe = r50k_base()?;
let tokens = bpe.encode_with_special_tokens("Hello world");
println!("Token count: {}", tokens.len());
```
```

--------------------------------

### bpe_for_tokenizer()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get a cached BPE tokenizer instance for a given tokenizer type. This is useful when you already know the specific tokenizer you want to use.

```APIDOC
## bpe_for_tokenizer()

### Description
Get a cached BPE tokenizer instance for a given tokenizer type.

### Signature
```rust
pub fn bpe_for_tokenizer(tokenizer: Tokenizer) -> Result<&'static CoreBPE>
```

### Parameters
#### Path Parameters
- **tokenizer** (Tokenizer) - Required - Enum variant from the Tokenizer enum

### Returns
A static reference to the appropriate `CoreBPE` instance.

### Request Example
```rust
use tiktoken_rs::{bpe_for_tokenizer, tokenizer::Tokenizer};

let bpe = bpe_for_tokenizer(Tokenizer::O200kBase)?;
let tokens = bpe.encode_with_special_tokens("hello");
```
```

--------------------------------

### o200k_harmony()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the o200k_harmony tokenizer for open-source models, providing extended special tokens for gpt-oss models.

```APIDOC
## o200k_harmony()

### Description
Initializes the o200k_harmony tokenizer for open-source models. This tokenizer includes extended special tokens for gpt-oss models.

### Signature
```rust
pub fn o200k_harmony() -> Result<CoreBPE>
```

### Returns
A new `CoreBPE` instance with extended special tokens for gpt-oss models.

### Use for
gpt-oss models like `gpt-oss-20b`, `gpt-oss-120b`.

### Example
```rust
use tiktoken_rs::o200k_harmony;

let bpe = o200k_harmony()?;
let special = bpe.special_tokens();
```
```

--------------------------------

### Count Ordinary Tokens

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/02-corebpe-methods.md

Use `count_ordinary` to get the token count without special token recognition. This is faster than encoding if only the count is needed.

```rust
use tiktoken_rs::p50k_base;

let bpe = p50k_base()?;
let count = bpe.count_ordinary("This is a sentence");
println!("Token count: {}", count);
```

--------------------------------

### p50k_base()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the p50k_base tokenizer, which is suitable for code models and models like `text-davinci-002` and `text-davinci-003`.

```APIDOC
## p50k_base()

### Description
Initializes the p50k_base tokenizer. This tokenizer is suitable for code models and models like `text-davinci-002` and `text-davinci-003`.

### Signature
```rust
pub fn p50k_base() -> Result<CoreBPE>
```

### Returns
A new `CoreBPE` instance for code models.

### Example
```rust
use tiktoken_rs::p50k_base;

let bpe = p50k_base()?;
let token_count = bpe.count_with_special_tokens("def hello():");
```
```

--------------------------------

### Get BPE Tokenizer for Model

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Retrieves a cached BPE tokenizer instance for a specified model name. Use this when you know the model identifier.

```rust
use tiktoken_rs::bpe_for_model;

let bpe = bpe_for_model("gpt-4o")?;
let tokens = bpe.encode_with_special_tokens("Hello world");
```

--------------------------------

### bpe_for_model()

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Get a cached BPE tokenizer instance for a given model name. This function is model-aware and simplifies tokenization by automatically selecting the correct tokenizer.

```APIDOC
## bpe_for_model()

### Description
Get a cached BPE tokenizer instance for a given model name.

### Signature
```rust
pub fn bpe_for_model(model: &str) -> Result<&'static CoreBPE>
```

### Parameters
#### Path Parameters
- **model** (string) - Required - Model name (e.g., "gpt-4o", "gpt-3.5-turbo", "o1-mini")

### Returns
A static reference to the appropriate `CoreBPE` instance, or an error if the model is unknown.

### Request Example
```rust
use tiktoken_rs::bpe_for_model;

let bpe = bpe_for_model("gpt-4o")?;
let tokens = bpe.encode_with_special_tokens("Hello world");
```
```

--------------------------------

### Initialize cl100k_base Tokenizer

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Initializes the cl100k_base tokenizer for chat and embedding models. Use for GPT-4, GPT-3.5-turbo, and text-embedding models. Returns a new CoreBPE instance.

```rust
use tiktoken_rs::cl100k_base;

let bpe = cl100k_base()?;
let tokens = bpe.encode_with_special_tokens("chat message");
```

--------------------------------

### Get Special Tokens from BPE

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/02-corebpe-methods.md

Retrieves a set of all special tokens recognized by the tokenizer. Useful for understanding special markers like end-of-text or fill-in-the-middle tokens.

```rust
use tiktoken_rs::cl100k_base;

let bpe = cl100k_base()?;
let special = bpe.special_tokens();
println!("Special tokens: {:?}", special);
```

--------------------------------

### Initialize Tokenizer for Repeated Use

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-index.md

Initialize a tokenizer for repeated use by retrieving a singleton instance. This is more efficient for multiple operations.

```rust
let bpe = cl100k_base_singleton();  // Reuse same instance
```

--------------------------------

### Get BPE Tokenizer for Tokenizer Type

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/01-core-api.md

Retrieves a cached BPE tokenizer instance for a given tokenizer enum variant. Use this when you have a specific tokenizer type in mind.

```rust
use tiktoken_rs::{bpe_for_tokenizer, tokenizer::Tokenizer};

let bpe = bpe_for_tokenizer(Tokenizer::O200kBase)?;
let tokens = bpe.encode_with_special_tokens("hello");
```

--------------------------------

### Initialize Tokenizer by Model Name

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-index.md

Initialize a tokenizer by specifying the model name. The library automatically selects the appropriate tokenizer.

```rust
let bpe = bpe_for_model("gpt-4o")?;  // By model name (automatic selection)
```

--------------------------------

### Count Tokens with Different Integer Types (p50k_base)

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Illustrates how to encode text into tokens and retrieve them as different integer types (u32, usize, u64). This is useful for compatibility with various systems and frameworks.

```rust
use tiktoken_rs::p50k_base;

fn main() -> Result<()> {
    let bpe = p50k_base()?;
    let text = "Hello";
    
    // Get tokens as u32 (default)
    let tokens_u32: Vec<u32> = bpe.encode_with_special_tokens(text);
    
    // Get tokens as usize (for indexing)
    let tokens_usize: Vec<usize> = bpe.encode_with_special_tokens_as(text);
    
    // Get tokens as u64 (for ML frameworks)
    let tokens_u64: Vec<u64> = bpe.encode_with_special_tokens_as(text);
    
    println!("u32: {:?}", tokens_u32);
    println!("usize: {:?}", tokens_usize);
    println!("u64: {:?}", tokens_u64);
    Ok(())
}
```

--------------------------------

### Unwrap Tokenizer Loading with Expect Message

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Shows how to load a tokenizer and unwrap the Result using `.expect()`, providing a custom error message if the operation fails. This is suitable when failure is considered unrecoverable.

```rust
use tiktoken_rs::o200k_base;

fn main() {
    let bpe = o200k_base()
        .expect("Failed to load o200k_base tokenizer");
    let tokens = bpe.encode_with_special_tokens("test");
    println!("Tokens: {}", tokens.len());
}
```

--------------------------------

### Count Tokens in Plain Text (o200k_base)

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Demonstrates how to count tokens in a given piece of plain text using the o200k_base encoding. This is a fundamental operation for understanding text length in tokens.

```rust
use tiktoken_rs::o200k_base;

fn main() -> Result<()> {
    let bpe = o200k_base()?;
    let text = "The quick brown fox jumps over the lazy dog";
    let count = bpe.count_with_special_tokens(text);
    println!("Token count: {}", count);
    Ok(())
}
```

--------------------------------

### Use `count_*` Methods for Token Counts

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/07-usage-examples.md

Utilize `count_*` methods when only the token count is needed, as this avoids allocating a token vector. This is more efficient than encoding the text and then getting the length of the resulting vector.

```rust
use tiktoken_rs::o200k_base;

fn main() -> anyhow::Result<()> {
    let bpe = o200k_base()?;
    
    // Efficient: no token vector allocated
    let count = bpe.count_with_special_tokens("long text");
    
    // Less efficient: allocates token vector
    let tokens = bpe.encode_with_special_tokens("long text");
    let count2 = tokens.len();
    
    assert_eq!(count, count2);
    Ok(())
}
```

--------------------------------

### Get Tokenizer for Model Name

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/04-model-tokenizer-mapping.md

Retrieves the tokenizer type for a given model name. Handles fine-tuned model prefixes and checks against known model mappings. Returns `None` if the model is unknown.

```rust
use tiktoken_rs::tokenizer::get_tokenizer;

assert_eq!(get_tokenizer("gpt-4o"), Some(Tokenizer::O200kBase));
assert_eq!(get_tokenizer("gpt-3.5-turbo"), Some(Tokenizer::Cl100kBase));
assert_eq!(get_tokenizer("unknown-model"), None);
```

--------------------------------

### Get Context Size for Model

Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/04-model-tokenizer-mapping.md

Retrieves the context window size (maximum tokens) for a given model. It strips fine-tuned prefixes and checks against known model mappings. Returns `None` if the model is unknown.

```rust
use tiktoken_rs::model::get_context_size;

assert_eq!(get_context_size("gpt-4o"), Some(128_000));
assert_eq!(get_context_size("gpt-4"), Some(8192));
assert_eq!(get_context_size("o1"), Some(200_000));
```