Maintainers
Pricing
Enterprise
More
More
Documentation
Try Live
Rankings
Install
Blog
Docs7
Ask
New
Add Docs
Tiktoken-rs
https://github.com/zurawiki/tiktoken-rs
Admin
Ready-made tokenizer library for working with GPT and tiktoken
Tokens:
34,190
Snippets:
407
Trust Score:
9.4
License:
MIT
Update:
1 month ago
Context
Chat
Benchmark
85.93
Latest
Show doc for...
Code
Info
Show Results
Tokens:
Raw
Copy
Link
Tokens:
Raw
Copy
Link
### Generate Site Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Example of generating a site using the gen_site function. This snippet is part of a larger script and is typically run when the script is executed directly. ```python if __name__ == "__main__": gen_site("basic-blog", [""], 250, paginate=True) ``` -------------------------------- ### Initialize custom CoreBPE instance Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Example of manually constructing a CoreBPE tokenizer with a defined vocabulary and regex pattern. ```rust use tiktoken_rs::CoreBPE; use rustc_hash::FxHashMap as HashMap; let mut encoder = HashMap::default(); encoder.insert(b"hello".to_vec(), 0); encoder.insert(b"world".to_vec(), 1); let mut special = HashMap::default(); special.insert("<|endoftext|>".to_string(), 2); let bpe = CoreBPE::new( encoder, special, r"'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+", )?; ``` -------------------------------- ### Heap Profiling Output Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Example output from DHAT when performing heap profiling. This output details memory usage and provides a filename for the saved data. ```text dhat: Total: 1,256 bytes in 6 blocks dhat: At t-gmax: 1,256 bytes in 6 blocks dhat: At t-end: 1,256 bytes in 6 blocks dhat: The data has been saved to dhat-heap.json, and is viewable with dhat/dh_view.html ``` -------------------------------- ### Heap Usage Testing Configuration Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt This example demonstrates how to configure a Rust project for heap usage testing using the DHAT crate. It includes setting the global allocator and building a profiler instance within a test function. ```rust #[global_allocator] static ALLOC: dhat::Alloc = dhat::Alloc; # // Tricky: comment out the `#[test]` because it's needed in an actual # // test but messes up things here. # /* #[test] # */ fn test() { let _profiler = dhat::Profiler::builder().testing().build(); let _v1 = vec![1, 2, 3, 4]; let v2 = vec![5, 6, 7, 8]; drop(v2); ``` -------------------------------- ### DHAT Heap Profiling Output Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt This snippet shows an example of the backtrace output generated by DHAT when heap profiling is enabled. It illustrates the format and the type of information provided, including allocation details and call stacks. ```rust #! #11: 0x10ae8441b: std::sync::once::Once::call_once_force::{{closure}} (src/sync/once.rs:320:40) //! #12: 0x10aea564c: std::sync::once::Once::call_inner (src/sync/once.rs:419:21) //! #13: 0x10ae81b1b: std::sync::once::Once::call_once_force (src/sync/once.rs:320:9) //! #14: 0x10ae81b1b: std::lazy::SyncOnceCell<T>::get_or_init_pin (std/src/lazy.rs:374:9) //! #15: 0x10ae81b1b: std::io::stdio::stdout (src/io/stdio.rs:679:16) //! #16: 0x10ae81b1b: std::io::stdio::print_to (src/io/stdio.rs:1196:21) //! #17: 0x10ae81b1b: std::io::stdio::_print (src/io/stdio.rs:1209:5) //! #18: 0x10ae2fe20: dhatter::main (dhatter/src/main.rs:8:5) //! } //! } ``` -------------------------------- ### Initialize Heap Profiler in Main Function Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Add this code to the start of your main function to initialize heap profiling. This should only be enabled while profiling. ```rust let _profiler = dhat::Profiler::new_heap(); ``` -------------------------------- ### Install tiktoken-rs with Cargo Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md Install the tiktoken-rs crate locally using Cargo. This is the first step before using the library in your Rust code. ```sh cargo add tiktoken-rs ``` -------------------------------- ### Example Token File Line Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Shows the format of a single line within the embedded token vocabulary files, consisting of a base64-encoded token and its corresponding rank. ```text aGVsbG8= 15339 ``` -------------------------------- ### DHAT Viewer Node Structure Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt An example of a node structure as displayed in the DHAT viewer. It shows memory statistics, allocation counts, average sizes, and lifetime information. ```text PP 1.1/2 { Total: 1,024 bytes (98.46%, 14,422,535.21/s) in 1 blocks (50%, 14,084.51/s), avg size 1,024 bytes, avg lifetime 35 µs (49.3% of program duration) Max: 1,024 bytes in 1 blocks, avg size 1,024 bytes At t-gmax: 1,024 bytes (98.46%) in 1 blocks (50%), avg size 1,024 bytes At t-end: 1,024 bytes (100%) in 1 blocks (100%), avg size 1,024 bytes Allocated at { #1: 0x10ae8441b: <alloc::alloc::Global as core::alloc::Allocator>::allocate (alloc/src/alloc.rs:226:9) #2: 0x10ae8441b: alloc::raw_vec::RawVec<T,A>::allocate_in (alloc/src/raw_vec.rs:207:45) #3: 0x10ae8441b: alloc::raw_vec::RawVec<T,A>::with_capacity_in (alloc/src/raw_vec.rs:146:9) #4: 0x10ae8441b: alloc::vec::Vec<T,A>::with_capacity_in (src/vec/mod.rs:609:20) #5: 0x10ae8441b: alloc::vec::Vec<T>::with_capacity (src/vec/mod.rs:470:9) #6: 0x10ae8441b: std::io::buffered::bufwriter::BufWriter<W>::with_capacity (io/buffered/bufwriter.rs:115:33) #7: 0x10ae8441b: std::io::buffered::linewriter::LineWriter<W>::with_capacity (io/buffered/linewriter.rs:109:29) #8: 0x10ae8441b: std::io::buffered::linewriter::LineWriter<W>::new (io/buffered/linewriter.rs:89:9) #9: 0x10ae8441b: std::io::stdio::stdout::{{closure}} (src/io/stdio.rs:680:58) #10: 0x10ae8441b: std::lazy::SyncOnceCell<T>::get_or_init_pin::{{closure}} (std/src/lazy.rs:375:25) } } ``` -------------------------------- ### Ad Hoc Profiling Output Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Example output from DHAT when performing ad hoc profiling. This output summarizes the number of units and events recorded, and provides a filename for the saved data. ```text dhat: Total: 141 units in 11 events dhat: The data has been saved to dhat-ad-hoc.json, and is viewable with dhat/dh_view.html ``` -------------------------------- ### Thread-Safe Tokenization Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Demonstrates using a singleton tokenizer across multiple threads with rayon, where each thread automatically utilizes its own cached regex instance. ```rust // Using singleton returns &'static CoreBPE // Safe to use across multiple threads let bpe = o200k_base_singleton(); // Each thread automatically gets its own regex instance rayon::scope(|s| { for i in 0..10 { s.spawn(|_| { let tokens = bpe.encode_with_special_tokens(&format!("text {}", i)); println!("Thread {} tokens: {}", i, tokens.len()); }); } }); ``` -------------------------------- ### Initialize ChatCompletionRequestMessage vector Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-chat-completion.md Example of creating a vector of messages using the ChatCompletionRequestMessage struct. ```rust use tiktoken_rs::ChatCompletionRequestMessage; let messages = vec![ ChatCompletionRequestMessage { role: "system".to_string(), content: Some("You are helpful.".to_string()), ..Default::default() }, ChatCompletionRequestMessage { role: "user".to_string(), content: Some("Hello!".to_string()), ..Default::default() }, ]; ``` -------------------------------- ### Get tokens as different integer types Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/examples-and-patterns.md Shows how to encode text into specific integer types like u32, usize, u64, or i64 using cl100k_base. ```rust use tiktoken_rs::cl100k_base; fn main() -> Result<(), Box<dyn std::error::Error>> { let bpe = cl100k_base()?; let text = "Hello world"; // As u32 (default Rank type) let tokens_u32: Vec<u32> = bpe.encode_ordinary_as(text); // As usize (for array indexing) let tokens_usize: Vec<usize> = bpe.encode_ordinary_as(text); // As u64 (for ML frameworks) let tokens_u64: Vec<u64> = bpe.encode_ordinary_as(text); // As i64 (if needed) let tokens_i64: Vec<i64> = bpe.encode_ordinary_as(text); println!("u32: {:?}", tokens_u32); println!("usize: {:?}", tokens_usize); println!("u64: {:?}", tokens_u64); println!("i64: {:?}", tokens_i64); Ok(()) } ``` -------------------------------- ### Ad Hoc Event Profiling Example Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Illustrates how to use `ad_hoc_event` for custom profiling. Use an integer argument to represent the weight of the event; use 1 if no meaningful weight is applicable. ```rust //! sites. You might want to know how often it is called and which other //! functions called it the most. In that case, you would add an //! [`ad_hoc_event`] call to that function, and the data collected by this //! crate and viewed with DHAT's viewer would show you exactly what you want to //! know. //! //! The meaning of the integer argument to `ad_hoc_event` will depend on //! exactly what you are measuring. If there is no meaningful weight to give to //! an event, you can just use `1`. ``` -------------------------------- ### Initialize Ad Hoc Profiler in Main Function Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Initialize ad hoc profiling by adding this code to the start of your main function. This enables manual annotation of code points. ```rust let _profiler = dhat::Profiler::new_ad_hoc(); ``` -------------------------------- ### Get Chat Completion Max Tokens with async-openai Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md Use this function to determine the maximum token count for a chat completion request. Ensure the `async-openai` feature is enabled in your `Cargo.toml`. This example uses specific message types from the `async-openai` crate. ```rust use tiktoken_rs::async_openai::get_chat_completion_max_tokens; use async_openai::types::chat::{ ChatCompletionRequestMessage, ChatCompletionRequestSystemMessage, ChatCompletionRequestSystemMessageContent, ChatCompletionRequestUserMessage, ChatCompletionRequestUserMessageContent, }; let messages = vec![ ChatCompletionRequestMessage::System(ChatCompletionRequestSystemMessage { content: ChatCompletionRequestSystemMessageContent::Text( "You are a helpful assistant that only speaks French.".to_string(), ), name: None, }), ChatCompletionRequestMessage::User(ChatCompletionRequestUserMessage { content: ChatCompletionRequestUserMessageContent::Text( "Hello, how are you?".to_string(), ), name: None, }), ]; let max_tokens = get_chat_completion_max_tokens("o1-mini", &messages).unwrap(); println!("max_tokens: {}", max_tokens); ``` -------------------------------- ### Run Main Binary Source: https://github.com/zurawiki/tiktoken-rs/blob/main/CONTRIBUTING.md Use this command to run the main binary of the project. You can optionally pass arguments to the binary. ```bash just run just r [...] ``` -------------------------------- ### Initialize BPE for a Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/types.md Demonstrates how to select a tokenizer variant and retrieve the corresponding BPE instance. ```rust use tiktoken_rs::tokenizer::Tokenizer; use tiktoken_rs::bpe_for_tokenizer; let tokenizer = Tokenizer::O200kBase; let bpe = bpe_for_tokenizer(tokenizer).unwrap(); ``` -------------------------------- ### Initialize p50k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the p50k_base tokenizer. ```rust pub fn p50k_base_singleton() -> &'static CoreBPE ``` -------------------------------- ### Initialize r50k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the r50k_base tokenizer. ```rust pub fn r50k_base_singleton() -> &'static CoreBPE ``` ```rust use tiktoken_rs::r50k_base_singleton; let bpe = r50k_base_singleton(); let tokens = bpe.encode_with_special_tokens("Reused tokenizer"); ``` -------------------------------- ### Initialize o200k_harmony_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the o200k_harmony tokenizer. ```rust pub fn o200k_harmony_singleton() -> &'static CoreBPE ``` -------------------------------- ### Initialize o200k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the o200k_base tokenizer. ```rust pub fn o200k_base_singleton() -> &'static CoreBPE ``` -------------------------------- ### Get Available Tokens for Response Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Determines the maximum number of tokens available for a chat completion response. ```rust use tiktoken_rs::get_chat_completion_max_tokens; let max_tokens = get_chat_completion_max_tokens("gpt-4o", &messages)?; println!("Max response: {} tokens", max_tokens); ``` -------------------------------- ### Initialization Functions Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Covers initialization functions like r50k_base, p50k_base, etc. Errors relate to data corruption or configuration issues. ```APIDOC ## Initialization Functions (e.g., r50k_base, p50k_base) ### Description Initializes specific tokenizer base configurations. ### Errors - Base64 decoding errors: Occurs if the embedded .tiktoken file is corrupted. - Regex compilation errors: Occurs if the internal pattern is invalid. - Duplicate token indices: Occurs if there are conflicts in the encoder/decoder mapping. ``` -------------------------------- ### Retrieve model context size Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/model-lookup.md Example usage of get_context_size to check token limits for known and unknown models. ```rust use tiktoken_rs::model::get_context_size; assert_eq!(get_context_size("gpt-4o"), Some(128_000)); assert_eq!(get_context_size("gpt-4"), Some(8192)); assert_eq!(get_context_size("gpt-5.4"), Some(1_050_000)); assert_eq!(get_context_size("unknown-model"), None); ``` -------------------------------- ### Count Token Length with Singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md Utilize the `o200k_base_singleton` to get a shared instance of the tokenizer. This is more efficient for repeated calls as it avoids re-initialization. ```rust use tiktoken_rs::o200k_base_singleton; let bpe = o200k_base_singleton(); let tokens = bpe.encode_with_special_tokens( "This is a sentence with spaces" ); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Initialize cl100k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the cl100k_base tokenizer. ```rust pub fn cl100k_base_singleton() -> &'static CoreBPE ``` -------------------------------- ### Initialize p50k_edit_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a static reference to the p50k_edit tokenizer. ```rust pub fn p50k_edit_singleton() -> &'static CoreBPE ``` -------------------------------- ### Run All Tests Source: https://github.com/zurawiki/tiktoken-rs/blob/main/CONTRIBUTING.md Execute all tests in the project using this command. ```bash just test just t ``` -------------------------------- ### Initialize p50k_base Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the p50k_base tokenizer for code models. ```rust pub fn p50k_base() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::p50k_base; let bpe = p50k_base().unwrap(); let tokens = bpe.encode_with_special_tokens("def hello(): pass"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Initialize o200k_base Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the o200k_base tokenizer for modern GPT-5 and o-series models. ```rust pub fn o200k_base() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::o200k_base; let bpe = o200k_base().unwrap(); let tokens = bpe.encode_with_special_tokens("Hello GPT-5"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Initialize o200k_harmony Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the o200k_harmony tokenizer for OSS models with an extended set of special tokens. ```rust pub fn o200k_harmony() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::o200k_harmony; let bpe = o200k_harmony().unwrap(); let tokens = bpe.encode_with_special_tokens("<|startoftext|>OSS model"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Initialize r50k_base Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the r50k_base tokenizer for GPT-3 models. ```rust pub fn r50k_base() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::r50k_base; let bpe = r50k_base().unwrap(); let tokens = bpe.encode_with_special_tokens("Hello GPT-3"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Create a custom tokenizer in Rust Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Initializes a CoreBPE instance by defining a vocabulary map, special tokens, and a regex pattern for tokenization. ```rust use tiktoken_rs::CoreBPE; use rustc_hash::FxHashMap as HashMap; fn create_test_tokenizer() -> Result<CoreBPE> { let mut encoder = HashMap::default(); // Build minimal vocabulary for i in 0..256 { encoder.insert(vec![i as u8], i as u32); } // Add some multi-byte tokens encoder.insert(b"hello".to_vec(), 256); encoder.insert(b"world".to_vec(), 257); let mut special = HashMap::default(); special.insert("<|eos|>".to_string(), 258); // Simple pattern for ASCII text let pattern = r"[a-zA-Z]+|[^a-zA-Z]"; CoreBPE::new(encoder, special, pattern) } ``` -------------------------------- ### Tokenizer Initialization Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Methods to initialize specific tokenizer instances via factory functions or singletons. ```APIDOC ## Tokenizer Initialization ### Description Initialize tokenizers for specific models using factory or singleton functions. ### Available Tokenizers - **O200kBase**: `o200k_base()`, `o200k_base_singleton()` - **Cl100kBase**: `cl100k_base()`, `cl100k_base_singleton()` - **O200kHarmony**: `o200k_harmony()`, `o200k_harmony_singleton()` - **P50kBase**: `p50k_base()`, `p50k_base_singleton()` - **P50kEdit**: `p50k_edit()`, `p50k_edit_singleton()` - **R50kBase**: `r50k_base()`, `r50k_base_singleton()` - **Gpt2**: `gpt2()` ``` -------------------------------- ### CoreBPE::new Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Constructor for creating a custom tokenizer with explicit configuration. ```APIDOC ## CoreBPE::new ### Description Creates a custom tokenizer from scratch. Requires a complete token vocabulary, a mapping of special token strings to their IDs, and a valid regex pattern for splitting text. ### Signature `pub fn new(encoder: HashMap<Vec<u8>, Rank>, special_tokens_encoder: HashMap<String, Rank>, pattern: &str) -> Result<Self>` ### Parameters - **encoder** (HashMap<Vec<u8>, Rank>) - Required - Map of token bytes to token IDs. Must be one-to-one with decoder. - **special_tokens_encoder** (HashMap<String, Rank>) - Required - Map of special token strings to token IDs. - **pattern** (&str) - Required - Regex pattern for text segmentation before BPE encoding. ### Returns - **Result<Self>** - New CoreBPE instance or error. ### Errors - Invalid regex pattern string - Encoder and decoder length mismatch (duplicate token indices) ### Example ```rust use tiktoken_rs::CoreBPE; use rustc_hash::FxHashMap as HashMap; let mut encoder = HashMap::default(); encoder.insert(b"hello".to_vec(), 0); encoder.insert(b"world".to_vec(), 1); let mut special = HashMap::default(); special.insert("<|endoftext|>".to_string(), 2); let bpe = CoreBPE::new( encoder, special, r"'(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+", )?; ``` ``` -------------------------------- ### o200k_harmony Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the o200k_harmony tokenizer for OSS models with an extended set of special tokens. ```APIDOC ## o200k_harmony ### Description Initializes the o200k_harmony tokenizer for OSS models. Uses the same encoding vocab as o200k_base but with a different set of special tokens optimized for the OSS ecosystem. ### Signature `pub fn o200k_harmony() -> Result<CoreBPE>` ### Returns `Result<CoreBPE>` — A new tokenizer instance. ### Usage Open-source models like `gpt-oss-20b`, `gpt-oss-120b`. ### Special Tokens Extended set of 1076 special tokens including `<|startoftext|>`, `<|endoftext|>`, `<|return|>`, `<|constrain|>`, `<|channel|>`, `<|start|>`, `<|end|>`, `<|message|>`, `<|call|>` and various reserved tokens. ``` -------------------------------- ### Query model context size Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Shows how to safely query context sizes, handling cases where the model is unknown or experimental. ```rust use tiktoken_rs::model::get_context_size; if let Some(size) = get_context_size("experimental-model") { println!("Context: {}", size); } else { println!("Model context size unknown"); } ``` -------------------------------- ### Singleton Initialization Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Retrieves a cached tokenizer instance. This is the recommended approach for performance. ```rust use tiktoken_rs::o200k_base_singleton; let bpe = o200k_base_singleton(); // Cached instance (fast) ``` -------------------------------- ### p50k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the p50k_base tokenizer. ```APIDOC ## p50k_base_singleton ### Description Returns a cached singleton instance of the p50k_base tokenizer for efficient reuse. ### Signature `pub fn p50k_base_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### Initialize cl100k_base Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the cl100k_base tokenizer for ChatGPT and embedding models. ```rust pub fn cl100k_base() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::cl100k_base; let bpe = cl100k_base().unwrap(); let tokens = bpe.encode_with_special_tokens("Hello GPT-4"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Direct Factory Initialization Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Creates a new tokenizer instance directly. Note that this is slower than using a singleton. ```rust use tiktoken_rs::o200k_base; let bpe = o200k_base()?; // New instance (slower) ``` -------------------------------- ### o200k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the o200k_base tokenizer. ```APIDOC ## o200k_base_singleton ### Description Returns a cached singleton instance of the o200k_base tokenizer for efficient reuse. ### Signature `pub fn o200k_base_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### r50k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the r50k_base tokenizer. ```APIDOC ## r50k_base_singleton ### Description Returns a cached singleton instance of the r50k_base tokenizer for efficient reuse. ### Signature `pub fn r50k_base_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### o200k_base Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the o200k_base tokenizer for GPT-5 and o-series models. ```APIDOC ## o200k_base ### Description Initializes the o200k_base tokenizer for GPT-5 and o-series models. This is the standard tokenizer for all current OpenAI models. ### Signature `pub fn o200k_base() -> Result<CoreBPE>` ### Returns `Result<CoreBPE>` — A new tokenizer instance. ### Usage Modern models including GPT-5, GPT-4.1, GPT-4o, o1, o3, o4-mini series. ### Special Tokens - `<|endoftext|>` (ID: 199999) - `<|endofprompt|>` (ID: 200018) ``` -------------------------------- ### Perform Basic Tokenization and Chat Message Counting Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/README.md Demonstrates counting tokens for raw text using a singleton tokenizer and calculating max tokens for chat completion requests. ```rust // Token count use tiktoken_rs::o200k_base_singleton; let bpe = o200k_base_singleton(); let tokens = bpe.count_with_special_tokens("Hello world"); println!("{}", tokens); // Chat message tokens use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage}; let messages = vec![ ChatCompletionRequestMessage { role: "user".to_string(), content: Some("Hello".to_string()), ..Default::default() }, ]; let max = get_chat_completion_max_tokens("gpt-4o", &messages)?; println!("Max response tokens: {}", max); ``` -------------------------------- ### r50k_base Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the r50k_base tokenizer for GPT-3 models such as davinci, curie, babbage, and ada. ```APIDOC ## r50k_base ### Description Initializes the r50k_base tokenizer for GPT-3 models. ### Signature `pub fn r50k_base() -> Result<CoreBPE>` ### Returns `Result<CoreBPE>` — A new tokenizer instance. ### Usage GPT-3 models like `davinci`, `curie`, `babbage`, `ada`. ### Special Tokens - `<|endoftext|>` (ID: 50256) ``` -------------------------------- ### p50k_base Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the p50k_base tokenizer for code models. ```APIDOC ## p50k_base ### Description Initializes the p50k_base tokenizer for code models. ### Signature `pub fn p50k_base() -> Result<CoreBPE>` ### Returns `Result<CoreBPE>` — A new tokenizer instance. ### Usage Code models like `text-davinci-002`, `text-davinci-003`, `code-davinci-001`, `code-davinci-002`. ### Special Tokens - `<|endoftext|>` (ID: 50256) ``` -------------------------------- ### Handle Unknown Models Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/model-lookup.md Unrecognized model strings return None for both tokenizer and context size lookups. ```rust assert_eq!(get_tokenizer("unknown-model"), None); assert_eq!(get_context_size("unknown-model"), None); ``` -------------------------------- ### Enable async-openai feature Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/README.md Add the async-openai feature flag to your Cargo.toml dependencies. ```toml tiktoken-rs = { version = "0.12", features = ["async-openai"] } ``` -------------------------------- ### o200k_harmony_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the o200k_harmony tokenizer. ```APIDOC ## o200k_harmony_singleton ### Description Returns a cached singleton instance of the o200k_harmony tokenizer for efficient reuse. ### Signature `pub fn o200k_harmony_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### Resolve Exact Model Names Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/model-lookup.md Use get_tokenizer and get_context_size to retrieve configuration for standard model names. ```rust use tiktoken_rs::tokenizer::get_tokenizer; use tiktoken_rs::model::get_context_size; // GPT-4o (current standard) assert_eq!(get_tokenizer("gpt-4o"), Some(Tokenizer::O200kBase)); assert_eq!(get_context_size("gpt-4o"), Some(128_000)); // GPT-3.5-turbo assert_eq!(get_tokenizer("gpt-3.5-turbo"), Some(Tokenizer::Cl100kBase)); assert_eq!(get_context_size("gpt-3.5-turbo"), Some(16_385)); // GPT-4 assert_eq!(get_tokenizer("gpt-4"), Some(Tokenizer::Cl100kBase)); assert_eq!(get_context_size("gpt-4"), Some(8192)); ``` -------------------------------- ### Calculate text completion token budget Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Utility function to determine the maximum token budget for a text completion prompt. ```rust let max_tokens = get_text_completion_max_tokens("gpt-3.5-turbo", "Prompt")?; ``` -------------------------------- ### cl100k_base_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the cl100k_base tokenizer. ```APIDOC ## cl100k_base_singleton ### Description Returns a cached singleton instance of the cl100k_base tokenizer for efficient reuse. ### Signature `pub fn cl100k_base_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### Using FromRank for Generic Encoding Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/types.md Demonstrates encoding text into different integer types using the FromRank trait. ```rust use tiktoken_rs::cl100k_base; let bpe = cl100k_base().unwrap(); let tokens_u32: Vec<u32> = bpe.encode_ordinary_as("hello"); let tokens_usize: Vec<usize> = bpe.encode_ordinary_as("hello"); let tokens_u64: Vec<u64> = bpe.encode_ordinary_as("hello"); let tokens_i64: Vec<i64> = bpe.encode_ordinary_as("hello"); ``` -------------------------------- ### Initialize p50k_edit Tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the p50k_edit tokenizer for text edit models, including Fill-In-Middle (FIM) special tokens. ```rust pub fn p50k_edit() -> Result<CoreBPE> ``` ```rust use tiktoken_rs::p50k_edit; let bpe = p50k_edit().unwrap(); let tokens = bpe.encode_with_special_tokens("<|fim_prefix|>def hello"); println!("Token count: {}", tokens.len()); ``` -------------------------------- ### Heap Usage Testing with Assertions Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Demonstrates how to use the profiler's testing mode to assert heap statistics. This mode saves a heap profile upon assertion failure, aiding in debugging. ```rust use dhat; // Assuming dhat is the crate providing these macros fn test() { let v1 = vec![1, 2, 3]; let v2 = vec![4, 5]; let v3 = vec![9, 10, 11, 12]; drop(v3); let stats = dhat::HeapStats::get(); // Three allocations were done in total. dhat::assert_eq!(stats.total_blocks, 3); dhat::assert_eq!(stats.total_bytes, 48); // At the point of peak heap size, two allocations totalling 32 bytes existed. dhat::assert_eq!(stats.max_blocks, 2); dhat::assert_eq!(stats.max_bytes, 32); // Now a single allocation remains alive. dhat::assert_eq!(stats.curr_blocks, 1); dhat::assert_eq!(stats.curr_bytes, 16); } # test() ``` -------------------------------- ### Visualize tokens Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-START-HERE.txt Split text into individual token strings for inspection. ```rust let token_strings = bpe.split_by_token("Hello world", true)?; for (i, token) in token_strings.iter().enumerate() { println!("Token {}: {}", i, token); } ``` -------------------------------- ### Visualize Tokens in Rust Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Splits a string into tokens and prints each token with its index. ```rust let tokens_str = bpe.split_by_token("Hello world", true)?; for (i, token) in tokens_str.iter().enumerate() { println!("Token {}: {:?}", i, token); } ``` -------------------------------- ### Integrate with Token Counting Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/model-lookup.md Use bpe_for_model for cached tokenizer access or get_chat_completion_max_tokens for automated context management. ```rust use tiktoken_rs::bpe_for_model; let bpe = bpe_for_model("gpt-4o").unwrap(); let tokens = bpe.encode_with_special_tokens("Hello world"); ``` ```rust use tiktoken_rs::{get_chat_completion_max_tokens, ChatCompletionRequestMessage}; let messages = vec![/* ... */]; let max_tokens = get_chat_completion_max_tokens("gpt-4o", &messages).unwrap(); // Internally: // 1. Calls get_tokenizer("gpt-4o") -> O200kBase // 2. Calls get_context_size("gpt-4o") -> 128_000 // 3. Tokenizes messages and subtracts from context size ``` -------------------------------- ### Format Code and Lint Source: https://github.com/zurawiki/tiktoken-rs/blob/main/CONTRIBUTING.md This command formats the code and addresses lint issues before uploading changes. It's recommended to run this before submitting your contributions. ```bash just fix ``` -------------------------------- ### p50k_edit_singleton Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Returns a cached singleton instance of the p50k_edit tokenizer. ```APIDOC ## p50k_edit_singleton ### Description Returns a cached singleton instance of the p50k_edit tokenizer for efficient reuse. ### Signature `pub fn p50k_edit_singleton() -> &'static CoreBPE` ### Returns - **CoreBPE** - A reference to the cached tokenizer instance. ``` -------------------------------- ### Handle errors for tokenizer initialization Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Matches the result of base tokenizer initialization to handle potential file corruption or regex errors. ```rust use tiktoken_rs::o200k_base; match o200k_base() { Ok(bpe) => println!("Tokenizer ready"), Err(e) => eprintln!("Initialization failed: {}", e), } ``` -------------------------------- ### Tokenizer Factory Functions Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/module-index.md Factory functions to initialize standard OpenAI tokenizer instances. ```APIDOC ## Tokenizer Factories ### Description Functions to create and return a `CoreBPE` instance for specific OpenAI models. ### Functions - `r50k_base() -> Result<CoreBPE>` - GPT-3 tokenizer - `p50k_base() -> Result<CoreBPE>` - Code model tokenizer - `p50k_edit() -> Result<CoreBPE>` - Edit model tokenizer - `cl100k_base() -> Result<CoreBPE>` - ChatGPT tokenizer - `o200k_base() -> Result<CoreBPE>` - GPT-5 and gpt-4o tokenizer - `o200k_harmony() -> Result<CoreBPE>` - OSS model tokenizer ``` -------------------------------- ### Calculate response budget Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-START-HERE.txt Determine the maximum tokens allowed for a chat completion response. ```rust use tiktoken_rs::get_chat_completion_max_tokens; let max = get_chat_completion_max_tokens("gpt-4o", &messages)?; ``` -------------------------------- ### Estimate API Costs Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Calculate estimated costs based on token counts and pricing per 1k tokens. ```rust use tiktoken_rs::get_chat_completion_max_tokens; let max_response = get_chat_completion_max_tokens("gpt-4o", &messages)?; let max_request = max_response; // They use same tokens let input_price_per_1k = 0.005; // Check current pricing let tokens_used = (messages_tokens as f64) / 1000.0; let estimated_cost = tokens_used * input_price_per_1k; ``` -------------------------------- ### Handle Errors with unwrap_or Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Provide a fallback tokenizer if the primary model initialization fails. ```rust let bpe = bpe_for_model("gpt-4o") .unwrap_or_else(|_| bpe_for_model("gpt-3.5-turbo").unwrap()); ``` -------------------------------- ### Use unwrap with default fallback Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Provides a default BPE instance if the requested model's tokenizer cannot be retrieved. ```rust use tiktoken_rs::bpe_for_model; fn get_tokens(model: &str, text: &str) -> usize { let bpe = bpe_for_model(model) .unwrap_or_else(|_| bpe_for_model("gpt-4o").unwrap()); bpe.count_with_special_tokens(text) } ``` -------------------------------- ### Handle errors for bpe_for_model Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Matches the result of BPE initialization to handle unsupported model names. ```rust use tiktoken_rs::bpe_for_model; match bpe_for_model("gpt-4o") { Ok(bpe) => { let tokens = bpe.encode_with_special_tokens("text"); }, Err(e) => eprintln!("Model not supported: {}", e), } ``` -------------------------------- ### cl100k_base Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-initialization.md Initializes the cl100k_base tokenizer for ChatGPT and embedding models. ```APIDOC ## cl100k_base ### Description Initializes the cl100k_base tokenizer for ChatGPT and embedding models. ### Signature `pub fn cl100k_base() -> Result<CoreBPE>` ### Returns `Result<CoreBPE>` — A new tokenizer instance. ### Usage ChatGPT models like `gpt-4`, `gpt-3.5-turbo`, and embeddings like `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large`. ### Special Tokens - `<|endoftext|>` (ID: 100257) - `<|fim_prefix|>` (ID: 100258) - `<|fim_middle|>` (ID: 100259) - `<|fim_suffix|>` (ID: 100260) - `<|endofprompt|>` (ID: 100276) ``` -------------------------------- ### Handle Errors with Match Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Use pattern matching to handle successful tokenizer initialization or errors explicitly. ```rust match bpe_for_model("gpt-4o") { Ok(bpe) => println!("Tokens: {}", bpe.count_with_special_tokens("text")), Err(e) => eprintln!("Error: {}", e), } ``` -------------------------------- ### Check context window size Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/README.md Verify if a model has a defined context size before accessing it. ```rust if let Some(context) = get_context_size("my-model") { /* ... */ } ``` -------------------------------- ### Batch Processing with Fallback Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Safely count tokens by attempting to load a model and falling back to a default if unavailable. ```rust fn count_safe(model: &str, text: &str) -> usize { bpe_for_model(model) .or_else(|_| bpe_for_model("gpt-4o")) .map(|bpe| bpe.count_with_special_tokens(text)) .unwrap_or(0) } ``` -------------------------------- ### Implement model fallback logic Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/errors.md Explicitly matches on the result of BPE retrieval to log errors and switch to a default model. ```rust use tiktoken_rs::bpe_for_model; fn count_with_fallback(model: &str, text: &str) -> usize { match bpe_for_model(model) { Ok(bpe) => bpe.count_with_special_tokens(text), Err(_) => { eprintln!("Model '{}' not found, using gpt-4o", model); let bpe = bpe_for_model("gpt-4o").unwrap(); bpe.count_with_special_tokens(text) } } } ``` -------------------------------- ### Convert tokens to different integer types Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Demonstrates encoding text into various integer types using the encode_ordinary_as method. ```rust // Default (u32) let tokens: Vec<u32> = bpe.encode_ordinary_as("text"); // As usize (for indexing) let tokens: Vec<usize> = bpe.encode_ordinary_as("text"); // As u64 (for ML) let tokens: Vec<u64> = bpe.encode_ordinary_as("text"); // As i64 let tokens: Vec<i64> = bpe.encode_ordinary_as("text"); ``` -------------------------------- ### Enable Global Allocator for Heap Profiling Source: https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/examples/example_text.txt Enable the global allocator by adding this code to your program to facilitate heap profiling. ```rust #[global_allocator] static ALLOC: dhat::Alloc = dhat::Alloc; ``` -------------------------------- ### Calculate max tokens for async-openai chat completion Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-chat-completion.md Calculates available response tokens using async-openai native message types. ```rust #[cfg(feature = "async-openai")] pub fn get_chat_completion_max_tokens( model: &str, messages: &[async_openai::types::ChatCompletionRequestMessage], ) -> Result<usize> ``` ```rust #[cfg(feature = "async-openai")] use tiktoken_rs::async_openai::get_chat_completion_max_tokens; let max_tokens = get_chat_completion_max_tokens("gpt-4o", &messages).unwrap(); ``` -------------------------------- ### Upgrade encode Calls with Result Handling Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md Demonstrates how to handle the `Result` returned by `CoreBPE::encode`. Propagate or unwrap the result before using the tokens. Requires Rust 1.85 or newer. ```rust use tiktoken_rs::o200k_base; let bpe = o200k_base().unwrap(); let allowed = bpe.special_tokens(); let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed).unwrap(); ``` -------------------------------- ### Auto-Detect Model Capability Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/quick-reference.md Verify if a model is supported and retrieve its context size dynamically. ```rust use tiktoken_rs::{get_tokenizer, get_context_size}; if let Some(tokenizer) = get_tokenizer("my-model") { if let Some(context) = get_context_size("my-model") { println!("Supported: {} context", context); } } ``` -------------------------------- ### Regex pattern for r50k_base, p50k_base, and p50k_edit Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Pattern used for GPT-3 and code-specific models to segment text. ```regex '(?:[sdmt]|ll|ve|re)| ?\p{L}++| ?\p{N}++| ?[^\s\p{L}\p{N}]++|\s++$|\s+(?!\S)|\s ``` -------------------------------- ### Define o200k_base special tokens Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Reserved special token definitions for the o200k_base tokenizer. ```rust ENDOFTEXT = "<|endoftext|>" (ID: 199999) ENDOFPROMPT = "<|endofprompt|>" (ID: 200018) ``` -------------------------------- ### Resolve Versioned Models Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/model-lookup.md Prefix matching allows resolution for specific model snapshots. ```rust // Prefix matches handle versioned snapshots assert_eq!(get_tokenizer("gpt-4o-2024-05-13"), Some(Tokenizer::O200kBase)); assert_eq!(get_tokenizer("gpt-4-0314"), Some(Tokenizer::Cl100kBase)); assert_eq!(get_tokenizer("gpt-3.5-turbo-0125"), Some(Tokenizer::Cl100kBase)); assert_eq!(get_context_size("gpt-3.5-turbo-0125"), Some(16_385)); ``` -------------------------------- ### Define CoreBPE constructor signature Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md The signature for initializing a custom CoreBPE instance with explicit encoder, special tokens, and regex pattern. ```rust pub fn new( encoder: HashMap<Vec<u8>, Rank>, special_tokens_encoder: HashMap<String, Rank>, pattern: &str, ) -> Result<Self> ``` -------------------------------- ### Auto-select tokenizer Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/00-START-HERE.txt Retrieve the appropriate tokenizer for a specific model and encode text. ```rust use tiktoken_rs::bpe_for_model; let bpe = bpe_for_model("gpt-4o")?; let tokens = bpe.encode_with_special_tokens("text"); ``` -------------------------------- ### Define o200k_harmony special tokens Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/configuration.md Reserved special token definitions for the o200k_harmony tokenizer. ```rust <|startoftext|> (ID: 199998) <|endoftext|> (ID: 199999) <|reserved_200000|> (ID: 200000) <|reserved_200001|> (ID: 200001) <|return|> (ID: 200002) <|constrain|> (ID: 200003) <|reserved_200004|> (ID: 200004) <|channel|> (ID: 200005) <|start|> (ID: 200006) <|end|> (ID: 200007) <|message|> (ID: 200008) <|reserved_200009|> - <|reserved_200011|> (IDs: 200009-200011) <|call|> (ID: 200012) <|reserved_200013|> - <|reserved_201087|> (IDs: 200013-201087) ``` -------------------------------- ### Calculate max tokens for text completion Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/api-reference-chat-completion.md Calculates available tokens for legacy text completions by tokenizing the prompt and subtracting from the model's context size. ```rust pub fn get_text_completion_max_tokens(model: &str, prompt: &str) -> Result<usize> ``` ```rust use tiktoken_rs::get_text_completion_max_tokens; let max_tokens = get_text_completion_max_tokens( "gpt-3.5-turbo", "Translate to French: '" ).unwrap(); println!("Max tokens: {}", max_tokens); ``` -------------------------------- ### Encode text to token IDs Source: https://github.com/zurawiki/tiktoken-rs/blob/main/_autodocs/examples-and-patterns.md Demonstrates encoding a string into a vector of token IDs using the o200k_base tokenizer. ```rust use tiktoken_rs::o200k_base; fn main() -> Result<(), Box<dyn std::error::Error>> { let bpe = o200k_base()?; let text = "The quick brown fox"; let tokens = bpe.encode_with_special_tokens(text); println!("Tokens: {:?}", tokens); println!("Count: {}", tokens.len()); Ok(()) } ``` -------------------------------- ### Count Token Length with o200k_base Source: https://github.com/zurawiki/tiktoken-rs/blob/main/README.md Use the `o200k_base` tokenizer to encode a string and count the number of tokens. This method initializes a new tokenizer instance for each call. ```rust use tiktoken_rs::o200k_base; let bpe = o200k_base().unwrap(); let tokens = bpe.encode_with_special_tokens( "This is a sentence with spaces" ); println!("Token count: {}", tokens.len()); ```
