### Build and Install to GOPATH/bin

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Install the tokenizer to your Go binary directory ($GOPATH/bin) using `go install`.

```bash
go install github.com/tiktoken-go/tokenizer/cmd/tokenizer@latest
tokenizer -list-models
```

--------------------------------

### Direct Tokenizer Usage

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Example of direct tokenizer usage without any configuration setup. Imports the library and encodes a string.

```go
package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	// No config files, no environment variables, no initialization
	
	// Direct usage:
	codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
	ids, _, _ := codec.Encode("hello")
	fmt.Println(ids)
	
	// That's it. Ready to use.
}
```

--------------------------------

### Install Tiktoken Go

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/INDEX.md

Use 'go get' to install the tiktoken-go package.

```bash
go get github.com/tiktoken-go/tokenizer
```

--------------------------------

### Configuration Guide

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/README.md

Explains the zero-configuration design of the tokenizer package and how codecs are created.

```APIDOC
## Configuration Guide

### Description

This guide explains the configuration aspects of the `tokenizer` Go package, emphasizing its zero-configuration design. It details how codecs are created and how model-to-encoding mappings are handled without requiring explicit user configuration files or environment variables.

### Zero-Configuration Design

- **Principle**: The package is designed to be used out-of-the-box with minimal to no setup. It automatically loads necessary data (like vocabularies) when a codec is requested.
- **No External Dependencies**: Users do not need to provide configuration files, set environment variables, or manage external resources to use the core tokenization functionality.

### Codec Creation Options

- **Factory Functions**: The primary way to obtain a `Codec` is through the factory functions `tokenizer.Get(encoding string)` and `tokenizer.ForModel(model string)`.
  - `tokenizer.Get()` allows direct selection of an encoding by its name (e.g., "cl100k_base").
  - `tokenizer.ForModel()` abstracts the encoding selection by mapping a given model name (e.g., "gpt-4") to its corresponding, pre-defined encoding.
- **Automatic Loading**: When these functions are called, the package automatically locates and loads the necessary vocabulary and merge rules for the requested encoding.

### Model-to-Encoding Mapping

- **Internal Tables**: The package maintains internal, hardcoded tables that map supported OpenAI model names to their respective encoding formats. For example, "gpt-4" maps to "cl100k_base".
- **No User Modification**: These mappings are not intended to be modified by the user at runtime. If you need to use a model with a custom encoding, you would typically use `tokenizer.Get()` with the desired encoding name directly.

### Runtime Configuration Patterns

- **Implicit Configuration**: The "configuration" is implicitly handled by the choice of model or encoding name passed to the factory functions.
- **No Runtime Overrides**: There are no mechanisms provided for overriding the default model-to-encoding mappings or vocabulary loading paths at runtime.

### Summary

The `tokenizer` package prioritizes ease of use through a zero-configuration approach. All necessary data and mappings are bundled within the package, allowing developers to instantiate tokenizers simply by specifying the desired model or encoding.
```

--------------------------------

### Get() Codec Configuration

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Creates a codec for a specific encoding format. Configuration is implicit based on the encoding choice.

```go
func Get(encoding Encoding) (Codec, error)
```

```go
codec, err := tokenizer.Get(tokenizer.Cl100kBase)
// codec is now configured with:
// - 100,256 tokens in vocabulary
// - 5 special tokens
// - Cl100k regex pattern
// No additional configuration needed
```

--------------------------------

### Get Codec for Encoding

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

Obtain a codec for a specific encoding scheme, such as `O200kBase`. Handle potential errors during codec retrieval. This example demonstrates counting tokens in a string using the obtained codec.

```go
codec, err := tokenizer.Get(tokenizer.O200kBase)
if err != nil {
	panic(err)
}
count, _ := codec.Count("text to count")
```

--------------------------------

### Get O200kBase Codec and Encode Text

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/codec-interface.md

Demonstrates how to get the O200kBase codec and encode a string into token IDs. This codec is suitable for models like O1, O3, O4Mini, GPT5, or GPT41.

```go
codec, _ := tokenizer.Get(tokenizer.O200kBase)
ids, tokens, _ := codec.Encode("hello world")
fmt.Println(codec.GetName()) // "o200k_base"
```

--------------------------------

### Get Codec and Encode Text

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Demonstrates how to retrieve a codec for a specific model and encode a string into token IDs. No external configuration is needed.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
ids, _, _ := codec.Encode("hello")
```

--------------------------------

### Get P50kBase Codec and Count Tokens

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/codec-interface.md

Demonstrates how to get the P50kBase codec and count the number of tokens in a given string. This codec is used with models like TextDavinci003 or CodeDavinci002.

```go
codec, _ := tokenizer.Get(tokenizer.P50kBase)
count, _ := codec.Count("How many tokens in this string?")
fmt.Println(count) // Token count
```

--------------------------------

### Get Codec by Model - Go

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/README.md

Obtain a codec instance for a specific OpenAI model. This is the recommended way to get a codec for a known model.

```go
codec, _ := tokenizer.ForModel(tokenizer.GPT4o)
```

--------------------------------

### Get Codec by Encoding - Go

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/README.md

Obtain a codec instance using a specific encoding format. Ensure the encoding format is supported.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
```

--------------------------------

### Encoding Output Format Example

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

When encoding, tokens are output as space-separated unsigned integers.

```text
24912 2375 220 4429
```

--------------------------------

### Get R50kBase Codec and Encode Text

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/codec-interface.md

Illustrates retrieving the R50kBase codec and encoding a string into token IDs. This codec is intended for legacy text and code models.

```go
codec, _ := tokenizer.Get(tokenizer.R50kBase)
ids, _, _ := codec.Encode("hello world")
fmt.Println(ids) // [31373 995]
```

--------------------------------

### Handle ErrEncodingNotSupported Error

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Demonstrates how to check for and handle the ErrEncodingNotSupported error when calling Get().

```go
codec, err := tokenizer.Get(tokenizer.Encoding("unsupported"))
if err == tokenizer.ErrEncodingNotSupported {
	// Handle unsupported encoding
}
```

--------------------------------

### Get Codec by Encoding

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Retrieves a Codec instance for a specified encoding format. Use this when you know the exact encoding you need. Ensure the encoding is supported to avoid errors.

```go
package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	codec, err := tokenizer.Get(tokenizer.Cl100kBase)
	if err != nil {
		panic(err)
	}
	fmt.Println(codec.GetName()) // Output: cl100k_base
}
```

--------------------------------

### Get Codec Directly

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Use this function when you know the exact encoding format you need. It maps encoding constants to codec constructors.

```go
// Direct encoding
codec, err := tokenizer.Get(tokenizer.O200kBase)

// Possible encodings:
// - O200kBase, Cl100kBase, R50kBase, P50kBase, P50kEdit, GPT2Enc
```

--------------------------------

### Get and Use Tokenizer Codec Concurrently

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Demonstrates how to retrieve a codec instance and safely use its Encode and Decode methods concurrently across multiple goroutines. The codec is obtained using tokenizer.Get() with a predefined encoding name.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)

// Safe to use concurrently
go codec.Encode("text1")
go codec.Encode("text2")
go codec.Decode([]uint{1, 2, 3})
```

--------------------------------

### Handle ErrModelNotSupported Error

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Example of how to check for and handle the ErrModelNotSupported error when calling ForModel().

```go
codec, err := tokenizer.ForModel(tokenizer.Model("unknown-model"))
if err == tokenizer.ErrModelNotSupported {
	// Handle unsupported model
}
```

--------------------------------

### Handle ErrModelNotSupported error

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

Example of how to check for and handle the ErrModelNotSupported error when retrieving a tokenizer for a model.

```go
codec, err := tokenizer.ForModel(tokenizer.Model("unknown"))
if err == tokenizer.ErrModelNotSupported {
	// Handle error
}
```

--------------------------------

### Get

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Retrieves a new codec instance configured for a specified encoding format. It takes an `Encoding` type as input and returns a `Codec` instance or an error if the encoding is unsupported.

```APIDOC
## Get

### Description
Returns a new codec instance configured for the specified encoding format.

### Function Signature
```go
func Get(encoding Encoding) (Codec, error)
```

### Parameters
#### Path Parameters
- **encoding** (Encoding) - Required - The encoding format (e.g., `Cl100kBase`, `O200kBase`)

### Return
A `Codec` instance ready for encoding/decoding operations, or an error if the encoding is unsupported.

### Error
Returns `ErrEncodingNotSupported` if the encoding is not recognized.

### Example
```go
package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	codec, err := tokenizer.Get(tokenizer.Cl100kBase)
	if err != nil {
		panic(err)
	}
	fmt.Println(codec.GetName())
}
```
```

--------------------------------

### Generated Vocabulary Structure

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Example of how vocabularies are generated and embedded as Go code, containing token-to-ID mappings. This ensures the binary is self-contained.

```go
// In codec/cl100k_base_vocab.go (generated)
var cl100kBaseVocabOnce sync.Once
var cl100kBaseVocab vocab

func cl100kBaseVocabInit() {
	cl100kBaseVocab = vocab{
		"hello":  15339,
		"world":  1917,
		// ... 100,256 entries total
	}
}
```

--------------------------------

### Get Tokenizer by Encoding Name

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Retrieves a tokenizer codec for a specific encoding name. Ensure the encoding name is valid.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
id, tokens, _ := codec.Encode("Hello, world!")
```

--------------------------------

### Get Codec Name

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Retrieves the human-readable name of the codec. Use this to identify the specific encoding scheme being used.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
name := codec.GetName()
fmt.Println(name) // Output: cl100k_base
```

--------------------------------

### Get P50kEdit Codec for FIM Operations

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/codec-interface.md

Shows how to obtain the P50kEdit codec, which includes special tokens for fill-in-the-middle operations. This codec is used with models like TextDavinciEdit001 or CodeDavinciEdit001.

```go
codec, _ := tokenizer.Get(tokenizer.P50kEdit)
ids, _, _ := codec.Encode("prefix code here")
// Supports FIM special tokens for code completion
```

--------------------------------

### Use ForModel to Get Codec

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

Obtain a tokenizer codec for a specific model using the ForModel function. This is the primary way to initialize a codec for encoding and decoding text.

```go
codec, err := tokenizer.ForModel(tokenizer.GPT4o)
if err != nil {
	panic(err)
}
ids, _, _ := codec.Encode("text")
```

--------------------------------

### Get Cl100kBase Codec, Encode, and Decode

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/codec-interface.md

Shows how to obtain the Cl100kBase codec, encode text to token IDs, and decode token IDs back to text. This codec is used with models like GPT4, GPT35Turbo, or TextEmbeddingAda002.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
ids, tokens, _ := codec.Encode("supercalifragilistic")
fmt.Println(len(ids)) // 7 tokens
text, _ := codec.Decode(ids)
fmt.Println(text)     // "supercalifragilistic"
```

--------------------------------

### Vocabulary Initialization

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Demonstrates vocabulary loading using sync.Once for efficient, single-time initialization.

```go
var cl100kBaseVocabOnce sync.Once

func cl100kBaseVocabInit() {
	// Load vocabulary from generated code
	// This runs only once, even if multiple codecs are created
}

func NewCl100kBase() *Codec {
	cl100kBaseVocabOnce.Do(cl100kBaseVocabInit)
	// ... rest of setup
}
```

--------------------------------

### Build the Tokenizer Command-Line Tool

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Builds the tokenizer command-line executable using Go. Navigate to the project directory and run the build command.

```bash
cd /workspace/home/tokenizer
go build -o tokenizer ./cmd/tokenizer
```

--------------------------------

### Lazy Loading Vocabulary with sync.Once

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Demonstrates the use of `sync.Once` for thread-safe, single-time initialization of the vocabulary. This pattern ensures fast startup and efficient memory usage.

```go
var cl100kBaseVocabOnce sync.Once
var cl100kBaseVocab vocab

func NewCl100kBase() *Codec {
	// Guaranteed to run exactly once, even with concurrent calls
	cl100kBaseVocabOnce.Do(cl100kBaseVocabInit)
	
	return &Codec{
		vocabulary: cl100kBaseVocab,
		// ...
	}
}
```

--------------------------------

### Build Tokenizer

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Standard Go build command for the tokenizer.

```bash
go build -o tokenizer ./cmd/tokenizer
```

--------------------------------

### Encode and Decode Text with Tokenizer

Source: https://github.com/tiktoken-go/tokenizer/blob/main/README.md

Demonstrates how to initialize a tokenizer and use it to encode text into token IDs and decode token IDs back into text. Ensure the tokenizer is retrieved using a valid encoding name.

```go
package main

import (
    "fmt"
    "github.com/tiktoken-go/tokenizer"
)

func main() {
    enc, err := tokenizer.Get(tokenizer.Cl100kBase)
    if err != nil {
        panic("oh oh")
    }

    // this should print a list of token ids
    ids, _, _ := enc.Encode("supercalifragilistic")
    fmt.Println(ids)

    // this should print the original string back
    text, _ := enc.Decode(ids)
    fmt.Println(text)
}
```

--------------------------------

### Lazy Initialization of Vocabulary

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Shows how vocabularies and reverse mappings are initialized lazily using `sync.Once` to minimize startup time. The vocabulary is loaded only once, even if the codec is created multiple times.

```go
var cl100kBaseVocabOnce sync.Once

func NewCl100kBase() *Codec {
	cl100kBaseVocabOnce.Do(cl100kBaseVocabInit)
	// Vocabulary loaded only once, even if codec created multiple times
	...
}
```

--------------------------------

### Application-Level Configuration Structure

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Define an application configuration struct that includes a default model and a tokenizer Codec. The Initialize method sets up the Codec based on the default model.

```go
type AppConfig struct {
	DefaultModel tokenizer.Model
	Tokenizer    tokenizer.Codec
}

func (c *AppConfig) Initialize() error {
	codec, err := tokenizer.ForModel(c.DefaultModel)
	if err != nil {
		return err
	}
	c.Tokenizer = codec
	return nil
}
```

--------------------------------

### Get Codec by OpenAI Model

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/INDEX.md

Obtain a tokenizer codec directly for a specified OpenAI model, such as GPT-4o.

```go
codec, _ := tokenizer.ForModel(tokenizer.GPT4o)
ids, _, _ := codec.Encode("text")
```

--------------------------------

### Basic Tiktoken Go Usage

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/INDEX.md

Demonstrates how to create a codec for a specific encoding (e.g., cl100k_base), encode text into token IDs and strings, decode token IDs back to text, and count tokens in a given string.

```go
package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	// Create a codec for a specific encoding
	codec, err := tokenizer.Get(tokenizer.Cl100kBase)
	if err != nil {
		panic(err)
	}
	
	// Encode text
	ids, tokens, _ := codec.Encode("hello world")
	fmt.Println(ids)    // Token IDs
	fmt.Println(tokens) // Token strings
	
	// Decode back
	text, _ := codec.Decode(ids)
	fmt.Println(text)   // "hello world"
	
	// Count tokens
	count, _ := codec.Count("how many tokens?")
	fmt.Println(count)
}
```

--------------------------------

### Encode and Decode Text

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/INDEX.md

Demonstrates the basic workflow of encoding text into token IDs and then decoding those IDs back to the original text using a specified encoding. Ensure the correct encoding is retrieved before proceeding.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)

// Encode
original := "hello world"
ids, _, _ := codec.Encode(original)

// Decode (round-trip)
text, _ := codec.Decode(ids)
// text == original
```

--------------------------------

### Tokenizer Package API

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/README.md

Provides factory functions to get tokenizers for specific models and defines the core Codec interface.

```APIDOC
## Package tokenizer

### Description

This package provides the core functionality for tokenizing text using various encoding schemes supported by OpenAI models. It includes factory functions to easily obtain codecs for specific models and defines the `Codec` interface that all encoding implementations must adhere to.

### Functions

#### Get(encoding string) (Codec, error)

- **Description**: Retrieves a `Codec` implementation based on the provided encoding name.
- **Parameters**:
  - `encoding` (string) - The name of the encoding to retrieve (e.g., "cl100k_base", "p50k_base").
- **Returns**:
  - `Codec` - An interface representing the tokenizer.
  - `error` - An error if the encoding is not supported.

#### ForModel(model string) (Codec, error)

- **Description**: Retrieves a `Codec` implementation suitable for the specified OpenAI model.
- **Parameters**:
  - `model` (string) - The name of the OpenAI model (e.g., "gpt-4", "text-davinci-003").
- **Returns**:
  - `Codec` - An interface representing the tokenizer.
  - `error` - An error if the model is not supported or does not have a corresponding encoding.

### Interface: Codec

#### Description

The `Codec` interface defines the contract for all tokenizer implementations within the package. It includes methods for encoding text to tokens, decoding tokens back to text, and managing special tokens.

#### Methods

- **Encode(input string, allowedSpecial map[string]bool, disallowedSpecial map[string]bool) ([]int, error)**
  - Encodes a string into a slice of token integers.
- **EncodeSingle(input string, allowedSpecial map[string]bool, disallowedSpecial map[string]bool) (int, error)**
  - Encodes a single string into a single token integer.
- **Decode(input []int) (string, error)**
  - Decodes a slice of token integers back into a string.
- **DecodeSingle(input int) (string, error)**
  - Decodes a single token integer back into a string.
- **Name() string**
  - Returns the name of the encoding.
- **Encoding() Encoding**
  - Returns the `Encoding` type associated with this codec.
- **SpecialTokens() map[string]int**
  - Returns a map of special token names to their integer representations.
- **SpecialTokensMap() map[int]string**
  - Returns a map of special token integers to their string representations.
- **IsSpecialToken(token int) bool**
  - Checks if a given token integer is a special token.

### Types

#### Model

- **Description**: Represents an OpenAI model. Contains a list of supported models.

#### Encoding

- **Description**: Represents an encoding format. Contains a list of supported encoding formats.

### Errors

- **ErrModelNotSupported**: Returned when a requested model is not supported.
- **ErrEncodingNotSupported**: Returned when a requested encoding is not supported.
```

--------------------------------

### Build Locally

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Build the tokenizer executable locally in the current directory.

```bash
cd /workspace/home/tokenizer
go build -o tokenizer ./cmd/tokenizer
./tokenizer -list-models
```

--------------------------------

### Tokenizer Command-Line Tool Usage

Source: https://github.com/tiktoken-go/tokenizer/blob/main/README.md

Shows how to use the included command-line tool for encoding and decoding text. Use the -h flag for a full list of options.

```sh
> tokenizer -h

Usage of tokenizer:
  -decode string
        tokens to decode
  -encode string
        text to encode
  -token string
        text to calculate token

> tokenizer -encode supercalifragilistic
```

--------------------------------

### Catch ErrEncodingNotSupported

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

Shows how to handle the ErrEncodingNotSupported error when using the Get function. If the encoding is not supported, it logs the error and defaults to a stable encoding.

```go
codec, err := tokenizer.Get(tokenizer.Encoding("gpt2"))
if err == tokenizer.ErrEncodingNotSupported {
	log.Printf("Encoding not supported: %v\n", err)
	// Fall back to a stable encoding
	codec, _ = tokenizer.Get(tokenizer.Cl100kBase)
}
```

--------------------------------

### Select Encoding by Name

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Use this command to directly specify the desired encoding format by its name.

```bash
./tokenizer -encoding o200k_base -encode "text"
```

--------------------------------

### Define Model Type

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

Represents an OpenAI model identifier as a string. Use this type with the ForModel() function to get a codec configured for a specific model.

```go
type Model string
```

--------------------------------

### Get Codec For Model

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Use this function for automatic encoding selection based on the OpenAI model name. It returns a codec configured for the latest generation by default.

```go
// Model-based selection
codec, err := tokenizer.ForModel(tokenizer.GPT4o)
// Returns codec configured for O200kBase (latest generation)
```

--------------------------------

### Get Tokenizer for a Specific Model

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Retrieves a tokenizer codec associated with a given model name. This is useful for using the correct encoding for a particular OpenAI model.

```go
codec, _ := tokenizer.ForModel(tokenizer.GPT4o)
id, _, _ := codec.Encode("Your text here")
```

--------------------------------

### Verify Codec Compatibility in Go

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

Check that you are using the correct codec for your encoding needs. This snippet demonstrates how to retrieve a codec and print its name.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
fmt.Println(codec.GetName()) // "cl100k_base"
```

--------------------------------

### List Supported Encodings

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Lists all supported encoding formats available for tokenization.

```bash
./tokenizer -list-encodings
```

--------------------------------

### Vocabulary Loading with sync.Once

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Vocabularies are pre-compiled and embedded, loaded at codec creation time using the sync.Once pattern to ensure they are loaded only once. This pattern guarantees efficient vocabulary loading even with multiple codec instances.

```go
func NewCl100kBase() *Codec {
	cl100kBaseVocabOnce.Do(cl100kBaseVocabInit)
	// ... rest of codec setup
}
```

--------------------------------

### Fail Fast with Panic on Initialization Error

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

Use this pattern when tokenizer initialization failure is considered a fatal condition for the application. It immediately stops execution if the tokenizer cannot be set up.

```go
codec, err := tokenizer.Get(tokenizer.Cl100kBase)
if err != nil {
	panic(fmt.Sprintf("Failed to initialize tokenizer: %v", err))
}
```

--------------------------------

### Implement Core Tokenization with BPE

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

The `tokenize` method splits input using a regex, attempts direct vocabulary lookup, and falls back to the BPE merge algorithm for unknown pieces. It yields tokens and their string representations.

```go
func (c *Codec) tokenize(input string, yield func(uint, string)) error {
	// 1. Split input using encoding-specific regex
	match, err := c.splitRegexp.FindStringMatch(input)
	
	for match != nil {
		piece := match.String()
		
		// 2. Try direct vocabulary lookup
		if id, ok := c.vocabulary[piece]; ok {
			yield(id, piece)
		} else {
			// 3. If not found, apply BPE merge algorithm
			parts := c.mergePairs(piece)
			
			// 4. Output merged tokens
			for i := range len(parts) - 1 {
				token := piece[parts[i].offset:parts[i+1].offset]
				yield(c.vocabulary[token], token)
			}
		}
		
		// 5. Continue with next regex match
		match, err = c.splitRegexp.FindNextMatch(match)
	}
	
	return nil
}
```

--------------------------------

### Get Token Strings Instead of IDs

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Encodes text and outputs the actual token strings (UTF-8 text of each token) instead of their IDs. Use the `-tokens` flag with `-encode`.

```bash
./tokenizer -model gpt-3.5-turbo -encode "hello world" -tokens
```

--------------------------------

### List Supported Models

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Lists all supported OpenAI model identifiers that can be used for tokenization.

```bash
./tokenizer -list-models
```

--------------------------------

### Validate Token IDs Before Decoding in Go

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

Before decoding, validate that token IDs are within the expected range for the chosen encoding to prevent errors. This example checks against the maximum ID for cl100k_base.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
maxTokenID := uint(100255) // Cl100kBase max

for _, id := range tokenIds {
	if id > maxTokenID {
		log.Printf("Invalid token ID: %d (exceeds max %d)", id, maxTokenID)
	}
}
```

--------------------------------

### Run Tokenizer with No Arguments

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Executing the tokenizer command without any arguments displays the help text, listing all available command-line flags and their usage.

```bash
./tokenizer
```

--------------------------------

### Get Codec for OpenAI Model

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Obtains a Codec instance suitable for a given OpenAI model. This function automatically maps model names to their corresponding encodings. It's useful for ensuring compatibility with specific models.

```go
package main

import (
	"fmt"
	"github.com/tiktoken-go/tokenizer"
)

func main() {
	codec, err := tokenizer.ForModel(tokenizer.GPT4o)
	if err != nil {
		panic(err)
	}
	
	ids, tokens, _ := codec.Encode("Hello, world!")
	fmt.Println(ids)    // Token IDs
	fmt.Println(tokens) // Token strings
}
```

--------------------------------

### Tokenization with Explicit Error Handling and Fallback

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

This function demonstrates explicit error checking for tokenizer initialization. It handles unsupported models by falling back to a default encoding and wraps other unexpected errors.

```go
func tokenizeWithFallback(text string, model tokenizer.Model) ([]uint, error) {
	codec, err := tokenizer.ForModel(model)
	if err != nil {
		if err == tokenizer.ErrModelNotSupported {
			// Use default
			codec, _ = tokenizer.Get(tokenizer.Cl100kBase)
		} else {
			return nil, fmt.Errorf("unexpected error: %w", err)
		}
	}
	
	ids, _, err := codec.Encode(text)
	return ids, err
}
```

--------------------------------

### Codec Interface

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

The Codec interface defines the contract for all codec implementations within the tokenizer package. It includes methods for getting the codec name, counting tokens in a string, encoding text into tokens, and decoding tokens back into text.

```APIDOC
## Codec Interface

### Description
Interface that all codec implementations must satisfy. Defines the contract for tokenization operations.

### Methods

- **GetName() string**: Returns the name of the encoding.
- **Count(string) (int, error)**: Counts the number of tokens in a given string.
- **Encode(string) ([]uint, []string, error)**: Encodes a string into a slice of token IDs and a slice of special tokens.
- **Decode([]uint) (string, error)**: Decodes a slice of token IDs back into a string.
```

--------------------------------

### Compare Encodings

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

See how the same text is tokenized differently across various encoding formats. This is useful for understanding encoding variations.

```bash
TEXT="hello world"
echo "O200k:"
./tokenizer -encoding o200k_base -encode "$TEXT"
echo "Cl100k:"
./tokenizer -encoding cl100k_base -encode "$TEXT"
echo "R50k:"
./tokenizer -encoding r50k_base -encode "$TEXT"
```

--------------------------------

### Codec Interface Definition

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

The primary interface for tokenization operations. All codec implementations satisfy this interface, providing methods for getting the codec name, counting tokens, encoding strings to token IDs, and decoding token IDs back to strings.

```go
type Codec interface {
	GetName() string
	Count(string) (int, error)
	Encode(string) ([]uint, []string, error)
	Decode([]uint) (string, error)
}
```

--------------------------------

### Cross-Compile for Linux AMD64

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Build the tokenizer for a different operating system and architecture, such as Linux AMD64.

```bash
GOOS=linux GOARCH=amd64 go build -o tokenizer ./cmd/tokenizer
```

--------------------------------

### Compile-Time Vocabulary Embedding

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Illustrates that vocabulary is loaded at codec creation time from compiled-in data, eliminating the need for runtime file I/O.

```go
// Vocabulary loaded at codec creation time from compiled-in data
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
// No disk access needed
```

--------------------------------

### ForModel() Codec Configuration

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Creates a codec based on an OpenAI model name. The encoding is determined automatically from model information.

```go
func ForModel(model Model) (Codec, error)
```

```go
switch model {
case O1, O1Preview, O1Mini, GPT5, GPT5Mini, GPT5Nano, GPT41, GPT4o, O3, O3Mini, O4Mini:
	// Uses O200kBase

case GPT4, GPT35, GPT35Turbo, TextEmbeddingAda002:
	// Uses Cl100kBase

case TextDavinci003, TextDavinci002, CodeDavinci001, CodeDavinci002, etc.:
	// Uses P50kBase

case TextDavinci001, TextCurie001, TextBabbage001, etc.:
	// Uses R50kBase

case TextDavinciEdit001, CodeDavinciEdit001:
	// Uses P50kEdit

case GPT2:
	// Uses GPT2Enc

default:
	// Check prefix patterns for fine-tuned models
}
```

```go
// Explicit model
codec, err := tokenizer.ForModel(tokenizer.GPT4o)
// codec is configured for O200kBase (latest generation)

// Model string with matching
codec, err := tokenizer.ForModel(tokenizer.Model("gpt-4o-mini"))
// codec is configured for O200kBase (matches "gpt-4o-" prefix)
```

--------------------------------

### Encode and Decode Text (Round-Trip)

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/package-tokenizer.md

Demonstrates encoding text into token IDs and then decoding those IDs back into text, verifying that the decoded text matches the original.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)
original := "Hello, world!"
id, _, _ := codec.Encode(original)
decoded, _ := codec.Decode(id)
// decoded == original
```

--------------------------------

### Verify Tokenization (Encode and Decode)

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Test if text encodes and decodes correctly by first encoding the text to token IDs and then decoding those IDs back to the original text. The output should match the input.

```bash
# Encode
IDS=$(./tokenizer -encoding cl100k_base -encode "hello world")

# Decode back
./tokenizer -encoding cl100k_base -decode "$IDS"
```

--------------------------------

### Codec Factory with Caching

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

Implement a codec factory with a cache to efficiently retrieve or create tokenizer Codecs. This pattern uses a mutex to ensure thread-safe access to the cache.

```go
var (
	codecCache = make(map[string]tokenizer.Codec)
	codecs     sync.RWMutex
)

func getOrCreateCodec(encoding string) (tokenizer.Codec, error) {
	codecs.RLock()
	if c, ok := codecCache[encoding]; ok {
		codecs.RUnlock()
		return c, nil
	}
	codecs.RUnlock()
	
	codec, err := tokenizer.Get(tokenizer.Encoding(encoding))
	if err != nil {
		return nil, err
	}
	
	codecs.Lock()
	codecCache[encoding] = codec
	codecs.Unlock()
	
	return codec, nil
}
```

--------------------------------

### Encoding Wrapper for Multiple Codecs

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Combine multiple codecs into a single wrapper to enable fallback mechanisms. This allows attempting encoding with a preferred encoding first and falling back to another if an error occurs.

```go
type MultiCodec struct {
	codecs map[string]tokenizer.Codec
}

func (mc *MultiCodec) EncodeWithFallback(text, preferredEncoding string) ([]uint, error) {
	// Try preferred encoding first, fall back on error
	codec, ok := mc.codecs[preferredEncoding]
	// ...
}
```

--------------------------------

### Run Tokenizer with Unknown Model

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/command-line-tool.md

Demonstrates an error when an unsupported model name is provided to the tokenizer command. The command exits with a non-zero status code.

```bash
./tokenizer -model unknown-model -encode "text"
```

--------------------------------

### Implement Merge Pairs Algorithm

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

The `mergePairs` method splits a piece into characters, ranks adjacent pairs based on vocabulary, and iteratively merges the lowest-ranked pairs. This ensures consistent tokenization by merging in vocabulary rank order.

```go
func (c *Codec) mergePairs(piece string) []part {
	// 1. Initialize parts array (one per character + final position)
	parts := make([]part, len(piece)+1)
	
	// 2. Rank adjacent pairs based on vocabulary
	for i := 0; i < len(parts)-2; i++ {
		parts[i].rank = getRank(i, 0)
	}
	
	// 3. Iteratively merge lowest-ranked pairs
	for {
		// Find lowest rank
		minRank, minIndex := findMinRank(parts)
		
		if minRank == math.MaxUint {
			break // No more pairs to merge
		}
		
		// Merge and update ranks
		parts[minIndex].rank = getRank(minIndex, 1)
		if minIndex > 0 {
			parts[minIndex-1].rank = getRank(minIndex-1, 1)
		}
		
		// Remove merged part
		parts = append(parts[:minIndex+1], parts[minIndex+2:]...)
	}
	
	return parts
}
```

--------------------------------

### Graceful Fallback to Default Encoding

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/errors.md

Employ this strategy when you wish to provide a default behavior if a user-specified model is not supported. It logs a warning and proceeds with a known encoding.

```go
codec, err := tokenizer.ForModel(userProvidedModel)
if err == tokenizer.ErrModelNotSupported {
	log.Printf("Model %s not supported, using GPT-3.5-turbo", userProvidedModel)
	codec, _ = tokenizer.Get(tokenizer.Cl100kBase)
}
```

--------------------------------

### Define part structure for BPE

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/types.md

Tracks character offset and merge priority for the byte-pair encoding algorithm. Lower rank indicates higher priority for merging.

```go
type part struct {
	offset int
	rank   uint
}
```

--------------------------------

### Custom Codec Wrapper for Logging

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Wrap an existing codec to add logging functionality before encoding. This is useful for debugging or monitoring encoding operations.

```go
type LoggingCodec struct {
	underlying tokenizer.Codec
}

func (lc *LoggingCodec) Encode(text string) ([]uint, []string, error) {
	log.Printf("Encoding: %q", text)
	return lc.underlying.Encode(text)
}
```

--------------------------------

### Stateless Tokenization for Concurrency

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Highlights that codec instances are stateless (except for lazy-loaded reverse vocab), making them safe to reuse and share across multiple goroutines.

```go
codec, _ := tokenizer.Get(tokenizer.Cl100kBase)

// Safe to call from multiple goroutines
go func() { codec.Encode("text1") }()
go func() { codec.Encode("text2") }()

```

--------------------------------

### Encoding Selection from User Input

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/configuration.md

This pattern allows selecting a tokenizer encoding based on a string input, typically from user input. It handles supported encodings and returns an error for unsupported ones.

```go
func getEncodingFromString(encStr string) (tokenizer.Encoding, error) {
	switch encStr {
	case "o200k_base":
		return tokenizer.O200kBase, nil
	case "cl100k_base":
		return tokenizer.Cl100kBase, nil
	default:
		return "", tokenizer.ErrEncodingNotSupported
	}
}
```

--------------------------------

### Reverse Vocabulary Lazy Initialization

Source: https://github.com/tiktoken-go/tokenizer/blob/main/_autodocs/api-reference/architecture.md

Illustrates the lazy creation of a reverse vocabulary (ID to string mapping) upon the first call to `Decode`. This optimizes memory by building the map only when needed.

```go
func (c *Codec) Decode(tokens []uint) (string, error) {
	if c.reverseVocabulary == nil {
		// Build on first decode
		c.reverseVocabulary = make(map[uint]string)
		for k, v := range c.vocabulary {
			c.reverseVocabulary[v] = k
		}
	}
	
	// Now use reverse vocab
	var out string
	for _, t := range tokens {
		piece, ok := c.reverseVocabulary[t]
		// ...
	}
	return out, nil
}
```