### Setup file for tiktoken Extension

Source: https://github.com/openai/tiktoken/blob/main/README.md

Example `setup.py` content for packaging a tiktoken extension. It uses `setuptools` to define the package name, find namespace packages, and specify dependencies like 'tiktoken'.

```python
from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
    ...
)
```

--------------------------------

### Install tiktoken

Source: https://github.com/openai/tiktoken/blob/main/README.md

Command to install the tiktoken library from PyPI.

```bash
pip install tiktoken
```

--------------------------------

### Get Encoding and Encode/Decode Text

Source: https://github.com/openai/tiktoken/blob/main/README.md

Installs the tiktoken library and demonstrates how to get a specific encoding (e.g., 'o200k_base') and use it to encode and decode text. It also shows how to obtain the tokenizer corresponding to a specific OpenAI model like 'gpt-4o'.

```python
import tiktoken
enc = tiktoken.get_encoding("o200k_base")
assert enc.decode(enc.encode("hello world")) == "hello world"

# To get the tokeniser corresponding to a specific model in the OpenAI API:
enc = tiktoken.encoding_for_model("gpt-4o")
```

--------------------------------

### Install Custom Tiktoken Plugin

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Command to install the custom tiktoken extension package. Ensure not to use editable mode for installation.

```bash
pip install ./my_tiktoken_extension  # NOT editable mode
```

--------------------------------

### Setup for Custom Tiktoken Plugin

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Defines the necessary Python setup script and directory structure for a custom tiktoken extension. The plugin module must contain an ENCODING_CONSTRUCTORS dictionary mapping encoding names to constructor functions.

```python
# setup.py
from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
)

# Directory structure (NO __init__.py in tiktoken_ext)
my_tiktoken_extension/
├── tiktoken_ext/
│   └── my_encodings.py      # Contains ENCODING_CONSTRUCTORS
└── setup.py

# tiktoken_ext/my_encodings.py
def my_encoding():
    return {
        "name": "my_encoding",
        "pat_str": r"...",
        "mergeable_ranks": {...},
        "special_tokens": {...},
    }

ENCODING_CONSTRUCTORS = {
    "my_encoding": my_encoding,
}
```

--------------------------------

### Complete Example: Understanding Tokenization

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Demonstrates loading a tokenizer, encoding text with visualization, decoding tokens individually, and training a custom tokenizer from scratch. Use this to understand the BPE process.

```python
from tiktoken._educational import SimpleBytePairEncoding

# Load GPT-4's tokenizer in educational mode
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")

# Encode with visualization
print("=== Encoding with visualization ===")
tokens = enc.encode("hello world", visualise="colour")

# See how tokens decode individually
print("\n=== Token-by-token decoding ===")
token_bytes = enc.decode_tokens_bytes(tokens)
for i, tb in enumerate(token_bytes):
    print(f"Token {tokens[i]}: {tb}")

# Train a custom tokenizer from scratch
print("\n=== Training custom tokenizer ===")
training_text = """
The quick brown fox jumps over the lazy dog.
This is a test of the tokenizer.
BPE (Byte Pair Encoding) is a useful technique.
""" * 10

custom_enc = SimpleBytePairEncoding.train(
    training_data=training_text,
    vocab_size=500,
    pat_str=r"\b\w+\b|[^\w\s]|\s+"
)

# Use custom tokenizer
custom_tokens = custom_enc.encode("hello", visualise=None)
print(f"Custom tokens: {custom_tokens}")
```

--------------------------------

### Basic Text Encoding and Decoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Demonstrates how to get an encoding (e.g., cl100k_base used by GPT-4) and then encode text into tokens and decode tokens back into text.

```python
import tiktoken

# Get an encoding
enc = tiktoken.get_encoding("cl100k_base")  # Used by GPT-4, GPT-3.5-turbo

# Encode text
text = "hello world"
tokens = enc.encode(text)
# [31373, 995]

# Decode tokens
decoded = enc.decode(tokens)
# 'hello world'
```

--------------------------------

### Get and Use Tiktoken Encodings

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Retrieve standard and custom encodings by name. Encodings are cached after the first retrieval. This example also demonstrates how to handle errors when an unknown encoding name is provided.

```python
import tiktoken

# Get a standard encoding
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode("hello world")

# Get another encoding
enc_o200k = tiktoken.get_encoding("o200k_base")

# Error: unknown encoding
try:
    enc = tiktoken.get_encoding("unknown_encoding")
except ValueError as e:
    print(e)
    # ValueError: Unknown encoding unknown_encoding.
    # Plugins found: ['tiktoken_ext.openai_public']
    # tiktoken version: 0.13.0 (are you on latest?)
```

--------------------------------

### Load GPT-2 encoding using data_gym_to_mergeable_bpe_ranks

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/load.md

This example demonstrates how to load the GPT-2 encoding by providing URLs to the vocab.bpe and encoder.json files, along with their respective SHA-256 hashes for verification. This function is used internally for legacy encodings.

```python
from tiktoken.load import data_gym_to_mergeable_bpe_ranks

# Load GPT-2 encoding
ranks = data_gym_to_mergeable_bpe_ranks(
    vocab_bpe_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpe",
    encoder_json_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.json",
    vocab_bpe_hash="1ce1664773c50f3e0cc8842619a93edc4624525b728b188a9e0be33b7726adc5",
    encoder_json_hash="196139668be63f3b5d6574427317ae82f612a97c5d1cdaf36ed2256dbf636783",
)
```

--------------------------------

### Catching KeyError for Nonexistent Token String

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/errors.md

Use this example to catch a KeyError when trying to encode a string that does not represent a valid token in the current encoding. This handles cases where the input string is not a recognized token.

```python
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

try:
    token_value = enc.encode_single_token("nonexistent_token_xyz")
except KeyError as e:
    print(f"Token not in vocabulary: {e}")
```

--------------------------------

### List Available Encoding Names

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Call this function to get a sorted list of all encoding names that have been registered. It triggers registry initialization if it hasn't happened yet and is thread-safe. Use the returned names to retrieve specific encodings.

```python
import tiktoken

names = tiktoken.list_encoding_names()
# ['cl100k_base', 'gpt2', 'o200k_base', 'o200k_harmony', 'p50k_base', 'p50k_edit', 'r50k_base']

if "cl100k_base" in names:
    enc = tiktoken.get_encoding("cl100k_base")
```

--------------------------------

### Get Byte Values of Tokens

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/encoding.md

Retrieves the byte values for all tokens in the vocabulary in their respective order.

```python
def token_byte_values(self) -> list[bytes]
```

--------------------------------

### Get Encoding for a Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Obtain the appropriate encoding for a given OpenAI model name. This is the recommended way to get an encoding for a specific model.

```python
enc = tiktoken.encoding_for_model("gpt-4o")
```

--------------------------------

### Catch Blobfile ImportError

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/errors.md

Handle `ImportError` when accessing cloud storage URLs with `read_file_cached()` if the `blobfile` library is not installed. Instructs the user to install `blobfile`.

```python
from tiktoken.load import read_file_cached

try:
    data = read_file_cached("s3://bucket/file.bpe")
except ImportError as e:
    print("Install blobfile for cloud storage: pip install blobfile")
```

--------------------------------

### Example Structure for Mergeable Ranks

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/types.md

This snippet illustrates the structure of the `mergeable_ranks` dictionary, which maps token byte sequences to their merge priority ranks. It shows single bytes and merged tokens with their corresponding integer ranks.

```python
{
    b'\x00': 0,      # byte 0 — rank 0
    b'\x01': 1,      # byte 1 — rank 1
    # ... more single bytes ...
    b'\xff': 255,    # byte 255 — rank 255
    b'th': 256,      # "th" merge — rank 256
    b'he': 257,      # "he" merge — rank 257
    # ... more merges ...
}
```

--------------------------------

### List Available Encodings

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Retrieve a list of all available encoding names supported by tiktoken and get a specific encoding by its name.

```python
import tiktoken

# List all available encodings
encodings = tiktoken.list_encoding_names()
# ['cl100k_base', 'gpt2', 'o200k_base', 'o200k_harmony', 'p50k_base', 'p50k_edit', 'r50k_base']

# Get encoding by name
enc = tiktoken.get_encoding("o200k_base")  # o1, o3, GPT-4o
```

--------------------------------

### Handling Special Tokens in Encoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Demonstrates how to encode text containing special tokens. By default, encoding special tokens raises a ValueError. This example shows how to allow specific, all, or no special tokens during encoding.

```python
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

# By default, raises error if text contains special tokens
try:
    enc.encode("<|endoftext|>")
except ValueError:
    print("Cannot encode special token by default")

# Allow specific special tokens
tokens = enc.encode("<|endoftext|>", allowed_special={"<|endoftext|>"})
# [100257]

# Allow all special tokens
tokens = enc.encode("<|endoftext|>", allowed_special="all")
# [100257]

# Encode as normal text (no error)
tokens = enc.encode("<|endoftext|>", disallowed_special=())
# [27, 91, 437, 1659, 5239, 91, 29]
```

--------------------------------

### Define a Custom Encoding Plugin

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Example of how to define a custom encoding within the tiktoken_ext namespace. This involves creating a constructor function that returns a dictionary specifying the encoding details.

```python
# my_project/tiktoken_ext/my_encodings.py
from tiktoken.load import load_tiktoken_bpe

def my_custom_encoding():
    return {
        "name": "my_encoding",
        "pat_str": r"...",
        "mergeable_ranks": load_tiktoken_bpe("..."),
        "special_tokens": {"<|end|>": 1000},
    }

ENCODING_CONSTRUCTORS = {
    "my_encoding": my_custom_encoding,
}
```

--------------------------------

### Get Encoding by Name

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Retrieve a registered encoding by its name. Caching is applied for efficiency. Available names include 'gpt2', 'r50k_base', 'p50k_base', 'p50k_edit', 'cl100k_base', 'o200k_base', and 'o200k_harmony'.

```python
def get_encoding(encoding_name: str) -> Encoding:
    pass
```

--------------------------------

### Read File Content from Local Path or URL

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/load.md

Use `read_file` to fetch raw bytes from local files or remote URLs. For blob URLs (e.g., `gs://`, `s3://`), ensure the `blobfile` library is installed.

```python
from tiktoken.load import read_file

# Local file
data = read_file("/path/to/vocab.tiktoken")

# HTTP URL
data = read_file("https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken")
```

--------------------------------

### Configure Plugin Registration in setup.py

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Demonstrates how to configure setuptools in setup.py to include the tiktoken_ext namespace package for your custom encoding extension.

```python
from setuptools import setup, find_namespace_packages

setup(
    name="my_tiktoken_extension",
    packages=find_namespace_packages(include=['tiktoken_ext*']),
    install_requires=["tiktoken"],
)
```

--------------------------------

### Instantiate SimpleBytePairEncoding with Base Ranks

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Demonstrates how to create an instance of `SimpleBytePairEncoding` by first loading base BPE ranks from a URL and then passing these ranks along with a regex pattern to the constructor. This is useful for understanding how to initialize the tokenizer with specific encoding rules.

```python
from tiktoken._educational import SimpleBytePairEncoding
from tiktoken.load import load_tiktoken_bpe

# Load base ranks
ranks = load_tiktoken_bpe("https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken")

# Create educational encoder
enc = SimpleBytePairEncoding(
    pat_str=r"'(?i:[sdmt]|ll|ve|re)|[^

\p{L}\p{N}]?+\p{L}++",
    mergeable_ranks=ranks
)
```

--------------------------------

### Decode Tokens with Character Offsets

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/encoding.md

Use `decode_with_offsets` to decode token integers into a string and simultaneously get the starting character index for each token. Offsets point to the first character of a token, even if UTF-8 boundaries are misaligned.

```python
enc = tiktoken.get_encoding("cl100k_base")
text, offsets = enc.decode_with_offsets([31373, 995])
# ('hello world', [0, 5])
```

--------------------------------

### Educational BPE Visualization and Training

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Utilize the educational SimpleBytePairEncoding class to load a tokenizer, encode text with visual BPE merge information, and train a new tokenizer from scratch using provided training data.

```python
from tiktoken._educational import SimpleBytePairEncoding

# Load a tokenizer in educational mode
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")

# Encode with visualization of BPE merges
tokens = enc.encode("hello world", visualise="colour")
# Prints colored output showing token boundaries

# Train a tokenizer from scratch
training_text = "the quick brown fox jumps over the lazy dog " * 100
enc = SimpleBytePairEncoding.train(
    training_data=training_text,
    vocab_size=300,
    pat_str=r"\w+|\s"
)
```

--------------------------------

### Project Structure for tiktoken Extension

Source: https://github.com/openai/tiktoken/blob/main/README.md

Illustrates the required directory and file structure for creating a custom tiktoken extension using the plugin mechanism. This includes a namespace package `tiktoken_ext` and a `setup.py` file.

```text
my_tiktoken_extension
├── tiktoken_ext
│   └── my_encodings.py
└── setup.py
```

--------------------------------

### Get Encoding for a Specific Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Retrieve the encoding object for a given model name using the `encoding_for_model` function. This is the recommended way to get the correct encoding for different models.

```python
encoding_for_model("model-name")
```

--------------------------------

### encoding_name_for_model(model_name)

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Gets the name of the encoding used by a specific model.

```APIDOC
## encoding_name_for_model(model_name)

### Description
Gets the name of the encoding for a specific model.

### Parameters
#### Path Parameters
- **model_name** (string) - Required - The name of the model (e.g., "gpt-3.5-turbo").

### Returns
- **string** - The name of the encoding used by the model.
```

--------------------------------

### Train and Visualize BPE Encoding

Source: https://github.com/openai/tiktoken/blob/main/README.md

Uses the educational submodule of tiktoken to train a simple BPE tokenizer on sample text and visualize how the GPT-4 encoder handles text, including a long sequence of 'a's.

```python
from tiktoken._educational import *

# Train a BPE tokeniser on a small amount of text
enc = train_simple_encoding()

# Visualise how the GPT-4 encoder encodes text
enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
enc.encode("hello world aaaaaaaaaaaa")
```

--------------------------------

### Encoding Methods Quick Reference

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

This section provides a quick reference for methods used to encode strings into tokens.

```APIDOC
## Encoding Methods Quick Reference

### Encoding to Tokens

| Method | Input | Output | Special Tokens | Batch |
|--------|-------|--------|---|---|
| `encode` | `str` | `list[int]` | Configurable | Single |
| `encode_ordinary` | `str` | `list[int]` | Ignored | Single |
| `encode_to_numpy` | `str` | `numpy.ndarray` | Configurable | Single |
| `encode_single_token` | `str\|bytes` | `int` | All | Single |
| `encode_batch` | `list[str]` | `list[list[int]]` | Configurable | Parallel |
| `encode_ordinary_batch` | `list[str]` | `list[list[int]]` | Ignored | Parallel |
| `encode_with_unstable` | `str` | `tuple[list[int], list[list[int]]]` | Configurable | Single |
```

--------------------------------

### SimpleBytePairEncoding Constructor

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Creates a BPE tokenizer from a regex pattern for text splitting and a dictionary mapping token bytes to merge ranks. The `mergeable_ranks` parameter dictates the order of merges, with lower ranks being merged earlier.

```python
SimpleBytePairEncoding(
    *, 
    pat_str: str,
    mergeable_ranks: dict[bytes, int]
)
```

--------------------------------

### get_encoding(name)

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Retrieves a registered encoding by its name. This is the primary way to get an Encoding object for tokenization.

```APIDOC
## get_encoding(name)

### Description
Retrieves a registered encoding by its name.

### Parameters
#### Path Parameters
- **name** (string) - Required - The name of the encoding to retrieve (e.g., "cl100k_base").

### Returns
- **Encoding** - An Encoding object that can be used for tokenization and detokenization.
```

--------------------------------

### encoding_for_model(model_name)

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Gets the appropriate encoding for a given model name. This function maps model names to their corresponding encodings.

```APIDOC
## encoding_for_model(model_name)

### Description
Gets the encoding for a specific model.

### Parameters
#### Path Parameters
- **model_name** (string) - Required - The name of the model (e.g., "gpt-4").

### Returns
- **Encoding** - The Encoding object associated with the specified model.
```

--------------------------------

### bpe_train

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Trains a BPE tokenizer from scratch on provided text data.

```APIDOC
## bpe_train

### Description
Train BPE tokenizer from scratch on text data.

### Parameters
#### Request Body
- **data** (str) - Required - The training text data.
- **vocab_size** (int) - Required - The desired vocabulary size.
- **pat_str** (str) - Required - The regex pattern string for tokenization.
- **visualise** (str | None) - Optional - The visualization mode. Defaults to "colour".
```

--------------------------------

### Catch NumPy ImportError

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/errors.md

Catch `ImportError` or `ModuleNotFoundError` when calling `encode_to_numpy()` if NumPy is not installed. Provides a fallback to `encode()` for list output.

```python
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

try:
    tokens = enc.encode_to_numpy("hello world")
except (ImportError, ModuleNotFoundError) as e:
    print("NumPy is required for encode_to_numpy()")
    # Fallback to list
    tokens = enc.encode("hello world")
```

--------------------------------

### Tokens to Text Methods Quick Reference

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

This section provides a quick reference for methods used to decode tokens back into text.

```APIDOC
### Tokens to Text

| Method | Input | Output | Features |
|--------|-------|--------|----------||
| `decode` | `Sequence[int]` | `str` | UTF-8 error control |
| `decode_bytes` | `Sequence[int]` | `bytes` | — |
| `decode_single_token_bytes` | `int` | `bytes` | Single token |
| `decode_tokens_bytes` | `Sequence[int]` | `list[bytes]` | Visualize boundaries |
| `decode_with_offsets` | `Sequence[int]` | `tuple[str, list[int]]` | Character offsets |
| `decode_batch` | `Sequence[Sequence[int]]` | `list[str]` | Parallel, UTF-8 control |
| `decode_bytes_batch` | `Sequence[Sequence[int]]` | `list[bytes]` | Parallel |
```

--------------------------------

### Get Encoding Name for Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Retrieve the encoding name associated with a specific OpenAI model. This is useful for looking up the correct encoding before obtaining the encoding object itself.

```python
import tiktoken

name = tiktoken.encoding_name_for_model("gpt-4o")
# 'o200k_base'

# Then get the actual encoding
enc = tiktoken.get_encoding(name)
```

--------------------------------

### Educational API

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

A pure Python implementation of the Byte Pair Encoding (BPE) algorithm for learning purposes.

```APIDOC
## Educational API

### Description
Offers a pure Python implementation of the Byte Pair Encoding (BPE) algorithm. This API is designed to be educational, allowing users to understand and experiment with BPE by visualizing the process and training BPE models from scratch.

### API Surface
This documentation refers to a module or set of classes/functions intended for learning and experimentation with BPE.

### Usage
Ideal for users who want to understand the inner workings of BPE or need a reference implementation for educational purposes.

### Examples
(Specific function signatures and examples for the Educational API are not detailed in the provided overview. Refer to the `api-reference/educational.md` for specifics.)
```

--------------------------------

### Load tiktoken BPE File

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/load.md

Loads a tiktoken BPE file from a given path or URL into a dictionary mapping token bytes to rank integers. Supports optional SHA-256 hash verification for the file. Use `read_file_cached()` for repeated access to avoid redundant loading.

```python
from tiktoken.load import load_tiktoken_bpe

# Load OpenAI's cl100k_base encoding file
ranks = load_tiktoken_bpe(
    "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
    expected_hash="223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7"
)

# Use in custom encoding
from tiktoken import Encoding
enc = Encoding(
    name="cl100k_base",
    pat_str=r"...",
    mergeable_ranks=ranks,
    special_tokens={"<|endoftext|>": 100257}
)
```

--------------------------------

### Encode text to NumPy array

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/encoding.md

Use `encode_to_numpy` to encode text directly into a NumPy array, which can be more efficient by avoiding Python list overhead. Ensure NumPy is installed.

```python
import numpy as np
enc = tiktoken.get_encoding("cl100k_base")
tokens = enc.encode_to_numpy("hello world")
# array([ 31373,   995], dtype=uint32)
assert tokens.dtype == np.uint32
```

--------------------------------

### Get Encoding for a Specific Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Retrieves the appropriate `Encoding` object for a given OpenAI model name. This function is useful for ensuring compatibility with specific model tokenization requirements.

```python
def encoding_for_model(model_name: str) -> Encoding:
    pass
```

--------------------------------

### Loading Mergeable Ranks from Data Gym Format

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Loads BPE merge token definitions and their priority ranks from Data Gym format (vocab.bpe and encoder.json files). Includes expected hashes for verification.

```python
from tiktoken.load import data_gym_to_mergeable_bpe_ranks

ranks = data_gym_to_mergeable_bpe_ranks(
    vocab_bpe_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/vocab.bpe",
    encoder_json_file="https://openaipublic.blob.core.windows.net/gpt-2/encodings/main/encoder.json",
    vocab_bpe_hash="1ce1664773c50f3e0cc8842619a93edc4624525b728b188a9e0be33b7726adc5",
    encoder_json_hash="196139668be63f3b5d6574427317ae82f612a97c5d1cdaf36ed2256dbf636783",
)
```

--------------------------------

### Create System-Specific Encoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Extend an existing encoding (e.g., cl100k_base) with custom special tokens to create a system-specific encoding. This allows for custom control tokens.

```python
import tiktoken

# Extend cl100k_base with custom special tokens
base_enc = tiktoken.get_encoding("cl100k_base")

custom_enc = tiktoken.Encoding(
    name="my_system_encoding",
    pat_str=base_enc._pat_str,
    mergeable_ranks=base_enc._mergeable_ranks,
    special_tokens={
        **base_enc._special_tokens,
        "<|system|>": 100261,
        "<|assistant|>": 100262,
        "<|user|>": 100263,
    }
)

# Use custom tokens
tokens = custom_enc.encode(
    "<|system|>You are helpful",
    allowed_special={"<|system|>"}
)
```

--------------------------------

### SimpleBytePairEncoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

A pure Python implementation of the Byte Pair Encoding (BPE) algorithm for learning purposes.

```APIDOC
## SimpleBytePairEncoding

### Description
Pure Python BPE implementation for learning.

### Key Methods
- **encode(text, visualise)** → `list[int]`
- **decode_bytes(tokens)** → `bytes`
- **decode(tokens)** → `str`
- **decode_tokens_bytes(tokens)** → `list[bytes]`
- **train(training_data, vocab_size, pat_str)** → `SimpleBytePairEncoding` (static)
- **from_tiktoken(encoding)** → `SimpleBytePairEncoding` (static)
```

--------------------------------

### Get Encoding Name for a Specific Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Returns the name of the encoding used by a specific OpenAI model. This is useful for programmatic access to encoding names without needing to instantiate the encoding object directly.

```python
def encoding_name_for_model(model_name: str) -> str:
    pass
```

--------------------------------

### Lazy Initialization of Tiktoken Registry

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Demonstrates the lazy loading behavior of the tiktoken registry. The first call to `get_encoding()` triggers the initialization, which can be slow. Subsequent calls are fast as they use the cached registry. `list_encoding_names()` also triggers initialization if it hasn't occurred yet.

```python
import tiktoken

# First call triggers initialization
enc1 = tiktoken.get_encoding("cl100k_base")  # Slow (initializes registry)

# Subsequent calls are fast (uses cached registry)
enc2 = tiktoken.get_encoding("gpt2")         # Fast

# list_encoding_names also triggers initialization if needed
names = tiktoken.list_encoding_names()       # Fast (registry already loaded)
```

--------------------------------

### Handle UnicodeDecodeError during Decoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/errors.md

Catch a UnicodeDecodeError when decoding tokens results in invalid UTF-8 bytes and the errors parameter is set to 'strict'. This example shows how to implement a fallback to lossy decoding using 'replace'.

```python
import tiktoken

enc = tiktoken.get_encoding("cl100k_base")

try:
    text = enc.decode([some_invalid_tokens], errors="strict")
except UnicodeDecodeError as e:
    print("Tokens decode to invalid UTF-8")
    # Fallback to lossy decoding
    text = enc.decode([some_invalid_tokens], errors="replace")
```

--------------------------------

### Create a Custom Encoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Define and instantiate a custom encoding by providing its name, pattern string, mergeable ranks, and special tokens. This allows for unique tokenization schemes.

```python
from tiktoken import Encoding
from tiktoken.load import load_tiktoken_bpe

ranks = load_tiktoken_bpe("https://...")
enc = Encoding(
    name="custom",
    pat_str=r"...",
    mergeable_ranks=ranks,
    special_tokens={"<|end|>": 1000}
)
```

--------------------------------

### Create Custom Encoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Load a base encoding (like cl100k_base) from a URL and create a custom encoding by adding new special tokens. This allows for domain-specific tokenization.

```python
from tiktoken import Encoding
from tiktoken.load import load_tiktoken_bpe

# Load base encoding
ranks = load_tiktoken_bpe(
    "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
    expected_hash="223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7"
)

# Create custom encoding with additional special tokens
enc = Encoding(
    name="my_custom_cl100k",
    pat_str=r"'(?i:[sdmt]|ll|ve|re)|[^

	

€Ÿ­—€-Ÿ- -ÿĀ-ſƀ-ɏͰ-Ͽ  -ʯ̀-ͯ᷀-᷿ -⁰-₟₠-⃏℀-⅏←-⇿①-⓿─-▟■-◿☀-⛿✀-➿⟰-⟿⤀-⥿⬀-⯰Ⱡ-Ɀ⺀-⻿　-〿぀-ゟ゠-ヿ㄀-ㄯ㄰-㆏㆐-㆟ㆠ-ㆿ㈀-㋿㌀-㏿㐀-䶿一-鿿ꀀ-꒏꒐-꓏가-힯豈-﫿︰-﹏﹐-﹯＀-￯][^
- -ÿĀ-ſƀ-ɏͰ-Ͽ  -ʯ̀-ͯ᷀-᷿ -⁰-₟₠-⃏℀-⅏←-⇿①-⓿─-▟■-◿☀-⛿✀-➿⟰-⟿⤀-⥿⬀-⯰Ⱡ-Ɀ⺀-⻿　-〿぀-ゟ゠-ヿ㄀-ㄯ㄰-㆏㆐-㆟ㆠ-ㆿ㈀-㋿㌀-㏿㐀-䶿一-鿿ꀀ-꒏꒐-꓏가-힯豈-﫿︰-﹏﹐-﹯＀-￯]++|@?[#%&'()*+-./:;<=>?@[\]^_`{|}~]",
    mergeable_ranks=ranks,
    special_tokens={
        "<|endoftext|>": 100257,
        "<|custom_token|>": 100261,
    }
)

tokens = enc.encode("hello <|custom_token|>")
```

--------------------------------

### List Available Encoding Names

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Returns a list of all available encoding names that can be used with the `get_encoding` function.

```python
def list_encoding_names() -> list[str]:
    pass
```

--------------------------------

### bpe_train Function

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Trains a BPE tokenizer from scratch on provided text data. It requires the training data, a target vocabulary size, a regex pattern for text splitting, and an optional visualization mode.

```APIDOC
## bpe_train Function

### Description
Trains a BPE tokenizer from scratch on provided text data. It requires the training data, a target vocabulary size, a regex pattern for text splitting, and an optional visualization mode.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Method Signature
```python
def bpe_train(
    data: str,
    vocab_size: int,
    pat_str: str,
    visualise: str | None = "colour"
) -> dict[bytes, int]:
```

### Parameters
#### Parameters
- **data** (str) - Required - Training text corpus
- **vocab_size** (int) - Required - Target vocabulary size (must be ≥ 256)
- **pat_str** (str) - Required - Regex pattern for text splitting
- **visualise** (str | None) - Optional - Visualization mode (Default: "colour")

### Returns
`dict[bytes, int]` — token bytes to rank mapping

### Example
```python
from tiktoken._educational import bpe_train, SimpleBytePairEncoding

# Train from scratch
ranks = bpe_train(
    data="the quick brown fox jumps over the lazy dog " * 50,
    vocab_size=300,
    pat_str=r"\w+|\s",
    visualise="colour"  # Shows merge progress
)

# Use trained ranks
enc = SimpleBytePairEncoding(pat_str=r"\w+|\s", mergeable_ranks=ranks)
tokens = enc.encode("hello", visualise=None)
```
```

--------------------------------

### Loading Mergeable Ranks from Tiktoken BPE File

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Loads BPE merge token definitions and their priority ranks from a tiktoken BPE file using its URL. Includes expected hash for verification.

```python
from tiktoken.load import load_tiktoken_bpe

ranks = load_tiktoken_bpe(
    "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
    expected_hash="223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7"
)
```

--------------------------------

### load_tiktoken_bpe

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Loads a tiktoken BPE file in native format.

```APIDOC
## load_tiktoken_bpe

### Description
Load tiktoken BPE file (native format).

### Parameters
#### Request Body
- **tiktoken_bpe_file** (str) - Required - The path to the tiktoken BPE file.
- **expected_hash** (str | None) - Optional - The expected SHA-256 hash of the BPE file.
```

--------------------------------

### Get Encoding for a Specific Model

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Retrieve the encoding used by a given OpenAI model name. This function supports exact model name matching, prefix matching for versioned models (e.g., 'gpt-4o-2024-05-13' maps to 'gpt-4o'), and fine-tuned models. An unknown model name will raise a KeyError.

```python
import tiktoken

# Exact model name
enc = tiktoken.encoding_for_model("gpt-4o")
# Returns o200k_base encoding

# Versioned model (prefix matching)
enc = tiktoken.encoding_for_model("gpt-4o-2024-05-13")
# Returns o200k_base encoding

# Fine-tuned model
enc = tiktoken.encoding_for_model("ft:gpt-4o")
# Returns o200k_base encoding

# Unknown model raises KeyError
try:
    enc = tiktoken.encoding_for_model("unknown-model")
except KeyError as e:
    print(e)
    # KeyError: Could not automatically map unknown-model to a tokeniser...
```

--------------------------------

### Train New BPE Tokenizer

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Trains a new Byte Pair Encoding tokenizer from scratch using provided training data, a target vocabulary size, and a regex pattern for text splitting. The vocabulary size must be at least 256.

```python
from tiktoken._educational import SimpleBytePairEncoding

# Train a tokenizer on sample text
training_text = "the quick brown fox jumps over the lazy dog " * 100
enc = SimpleBytePairEncoding.train(
    training_data=training_text,
    vocab_size=300,
    pat_str=r"\w+|\s"
)

# Use trained tokenizer
tokens = enc.encode("hello world", visualise=None)
```

--------------------------------

### Train BPE Tokenizer from Scratch

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Use bpe_train to train a new BPE tokenizer from a text corpus. This function requires a vocabulary size and a regex pattern for splitting text.

```python
from tiktoken._educational import bpe_train, SimpleBytePairEncoding

# Train from scratch
ranks = bpe_train(
    data="the quick brown fox jumps over the lazy dog " * 50,
    vocab_size=300,
    pat_str=r"\w+|\s",
    visualise="colour"  # Shows merge progress
)

# Use trained ranks
enc = SimpleBytePairEncoding(pat_str=r"\w+|\s", mergeable_ranks=ranks)
tokens = enc.encode("hello", visualise=None)
```

--------------------------------

### list_encoding_names

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/registry.md

Retrieves a sorted list of all available encoding names from registered plugins. This function triggers registry initialization if it hasn't occurred already and is thread-safe.

```APIDOC
## list_encoding_names

### Description
Returns a list of all available encoding names from registered plugins.

### Returns
`list[str]` — sorted list of encoding name strings

### Behavior
Triggers registry initialization (loading all plugins) if not already done. Thread-safe.

### Example
```python
import tiktoken

names = tiktoken.list_encoding_names()
# ['cl100k_base', 'gpt2', 'o200k_base', 'o200k_harmony', 'p50k_base', 'p50k_edit', 'r50k_base']

if "cl100k_base" in names:
    enc = tiktoken.get_encoding("cl100k_base")
```
```

--------------------------------

### Convert Data Gym to Mergeable BPE Ranks Function Signature

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Signature for converting Data Gym format files to BPE ranks.

```python
def data_gym_to_mergeable_bpe_ranks(
    vocab_bpe_file: str,
    encoder_json_file: str,
    vocab_bpe_hash: str | None = None,
    encoder_json_hash: str | None = None,
    clobber_one_byte_tokens: bool = False,
) -> dict[bytes, int]
```

--------------------------------

### Tiktoken Load Exports

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Import functions for loading and handling BPE (Byte Pair Encoding) files from the tiktoken.load submodule. Includes utilities for reading files, checking hashes, and dumping BPE data.

```python
from tiktoken.load import (
    read_file,
    read_file_cached,
    check_hash,
    load_tiktoken_bpe,
    data_gym_to_mergeable_bpe_ranks,
    dump_tiktoken_bpe,
)
```

--------------------------------

### Encode Text with Visualization

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Encodes text into a list of token integers. Optionally visualizes the Byte Pair Encoding process by coloring different tokens or showing merge steps.

```python
from tiktoken._educational import SimpleBytePairEncoding

enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")

# Encode with visualization
tokens = enc.encode("hello world", visualise="colour")
# Prints colored output showing token merges

# Encode silently
tokens = enc.encode("hello world", visualise=None)
# [31373, 995]
```

--------------------------------

### Create Custom Encoding with Additional Special Tokens

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/load.md

Load base encoding data from a URL and create a custom Encoding object with new special tokens. Ensure the expected hash matches to verify the integrity of the loaded data.

```python
from tiktoken import Encoding
from tiktoken.load import load_tiktoken_bpe

# Load base encoding data
ranks = load_tiktoken_bpe(
    "https://openaipublic.blob.core.windows.net/encodings/cl100k_base.tiktoken",
    expected_hash="223921b76ee99bde995b7ff738513eef100fb51d18c93597a113bcffe865b2a7"
)

# Create custom encoding with additional special tokens
custom_enc = Encoding(
    name="my_custom_cl100k",
    pat_str=r"'(?i:[sdmt]|ll|ve|re)|[^

	

 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿက-￿]*|'[sdmt]|ll|ve|re)"|[

	

 !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~€‚ƒ„…†‡ˆ‰Š‹ŒŽ‘’“”•–—˜™š›œžŸ ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿက-￿]*|'[sdmt]|ll|ve|re)|+|-|.|/|0-9|A-Z|a-z|ª|º|À-Ö|Ø-ö|ø-ÿ|Ā-῿| -⿿|-￿]*",
    mergeable_ranks=ranks,
    special_tokens={
        "<|endoftext|>": 100257,
        "<|custom_token|>": 100261,
    }
)

# Use the custom encoding
tokens = custom_enc.encode("Hello <|custom_token|>")
```

--------------------------------

### Tiktoken Educational Exports

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Import experimental or educational classes and functions from tiktoken._educational. This includes a simple BPE implementation and training utilities.

```python
from tiktoken._educational import (
    SimpleBytePairEncoding,
    bpe_encode,
    bpe_train,
)
```

--------------------------------

### Encoding Constructor

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/encoding.md

Creates an Encoding object for tokenization and detokenization operations. This constructor allows for the definition of custom encodings with specific patterns, mergeable ranks, and special tokens.

```APIDOC
## Encoding Constructor

### Description
Creates an Encoding object for tokenization and detokenization operations.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **name** (str) - Required - The encoding name. Should reflect the encoding's behavior, especially regarding special tokens. Different special token sets must have different names.
- **pat_str** (str) - Required - A regex pattern string used to split input text before BPE encoding. Must be a valid regex pattern (supports `\p` Unicode properties).
- **mergeable_ranks** (dict[bytes, int]) - Required - Maps mergeable token bytes to their ranks (merge priority). Ranks must be ordered by merge priority (lower rank = earlier merge).
- **special_tokens** (dict[str, int]) - Required - Maps special token strings to their integer token values. Special tokens are not subject to BPE merging.
- **explicit_n_vocab** (int | None) - Optional - If provided, validates that the total vocabulary size equals this number. Must equal `len(mergeable_ranks) + len(special_tokens)`.

### Request Example
```python
from tiktoken import Encoding

# Create a custom encoding
cl100k_base = Encoding(
    name="custom_encoding",
    pat_str=r"'(?i:[sdmt]|ll|ve|re)|[^
\p{L}\p{N}]?+\p{L}++|\p{N}{1,3}+",
    mergeable_ranks=load_ranks_from_file(),  # dict[bytes, int]
    special_tokens={
        "<|endoftext|>": 100257,
        "<|startoftext|>": 100258,
    }
)
```

### Response
#### Success Response (200)
- **Encoding** (Encoding) - An instance of the Encoding class.
```

--------------------------------

### Model Prefix to Encoding Name Mapping

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/types.md

This dictionary maps model name prefixes to encoding names, enabling version-aware lookups for models with varying release dates.

```python
# Prefix-based matching (for version-aware lookups)
MODEL_PREFIX_TO_ENCODING = {
    "gpt-4o-": "o200k_base",        # matches gpt-4o-2024-05-13
    "gpt-4-": "cl100k_base",        # matches gpt-4-0314
    "gpt-3.5-turbo-": "cl100k_base",  # matches gpt-3.5-turbo-0301
    "ft:gpt-4o": "o200k_base",      # fine-tuned models
    # ... more prefixes
}
```

--------------------------------

### train

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/educational.md

Trains a new Byte Pair Encoding (BPE) tokenizer from scratch using a provided text corpus, vocabulary size, and splitting pattern. This method is a static method.

```APIDOC
## train (static method)

### Description
Trains a new BPE tokenizer on data from scratch.

### Method Signature
```python
@staticmethod
def train(training_data: str, vocab_size: int, pat_str: str) -> SimpleBytePairEncoding
```

### Parameters
#### Path Parameters
- **training_data** (str) - Required - Training text corpus
- **vocab_size** (int) - Required - Target vocabulary size (must be ≥ 256)
- **pat_str** (str) - Required - Regex pattern for splitting text

### Returns
`SimpleBytePairEncoding` — trained tokenizer

### Behavior
- Starts with 256 single-byte tokens
- Iteratively finds and merges most common byte pairs
- Continues until vocabulary reaches target size
- Prints merge progress if not suppressed

### Raises
`ValueError` if vocab_size < 256

### Example
```python
from tiktoken._educational import SimpleBytePairEncoding

# Train a tokenizer on sample text
training_text = "the quick brown fox jumps over the lazy dog " * 100
enc = SimpleBytePairEncoding.train(
    training_data=training_text,
    vocab_size=300,
    pat_str=r"\w+|\s"
)

# Use trained tokenizer
tokens = enc.encode("hello world", visualise=None)
```
```

--------------------------------

### data_gym_to_mergeable_bpe_ranks

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/REFERENCE_INDEX.md

Converts Data Gym format (vocab.bpe + encoder.json) to BPE ranks.

```APIDOC
## data_gym_to_mergeable_bpe_ranks

### Description
Convert Data Gym format (vocab.bpe + encoder.json) to BPE ranks.

### Parameters
#### Request Body
- **vocab_bpe_file** (str) - Required - Path to the vocab.bpe file.
- **encoder_json_file** (str) - Required - Path to the encoder.json file.
- **vocab_bpe_hash** (str | None) - Optional - Expected hash for vocab.bpe file.
- **encoder_json_hash** (str | None) - Optional - Expected hash for encoder.json file.
- **clobber_one_byte_tokens** (bool) - Optional - Whether to overwrite one-byte tokens. Defaults to False.
```

--------------------------------

### Inspect Token Boundaries

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/README.md

Visualize how text is tokenized by a specific encoding. This helps in understanding the tokenization process and identifying potential issues.

```python
import tiktoken

def show_tokens(text: str, encoding_name: str):
    """Show how text is tokenized."""
    enc = tiktoken.get_encoding(encoding_name)
    tokens = enc.encode(text)
    token_bytes = enc.decode_tokens_bytes(tokens)
    
    print(f"Text: {text}")
    print(f"Tokens: {tokens}")
    print("Token breakdown:")
    for token_id, token_bytes in zip(tokens, token_bytes):
        print(f"  {token_id}: {token_bytes}")

show_tokens("hello world", "cl100k_base")
```

--------------------------------

### Set Tiktoken Cache Directory and Disable Caching

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/configuration.md

Demonstrates how to use the TIKTOKEN_CACHE_DIR environment variable to specify a custom cache directory or disable caching by setting it to an empty string. It also shows how to reset to the default behavior.

```python
import os
from tiktoken.load import read_file_cached

# Use custom cache directory
os.environ["TIKTOKEN_CACHE_DIR"] = "/var/cache/tiktoken"
data = read_file_cached("https://example.com/encoding.tiktoken")

# Disable caching
os.environ["TIKTOKEN_CACHE_DIR"] = ""
data = read_file_cached("https://example.com/encoding.tiktoken")  # Always fetches fresh

# Reset to default
del os.environ["TIKTOKEN_CACHE_DIR"]
data = read_file_cached("https://example.com/encoding.tiktoken")  # Uses temp dir
```

--------------------------------

### Create Custom Encoding

Source: https://github.com/openai/tiktoken/blob/main/_autodocs/api-reference/encoding.md

Instantiate an Encoding object with custom parameters for name, regex pattern, mergeable ranks, and special tokens. This is useful for defining unique tokenization behaviors.

```python
from tiktoken import Encoding

# Create a custom encoding
cl100k_base = Encoding(
    name="custom_encoding",
    pat_str=r"'(?i:[sdmt]|ll|ve|re)|[^

p{L}
p{N}]?+\p{L}++|\p{N}{1,3}+",
    mergeable_ranks=load_ranks_from_file(),  # dict[bytes, int]
    special_tokens={
        "<|endoftext|>": 100257,
        "<|startoftext|>": 100258,
    }
)
```