### Install gpt-tokenizer via npm

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Install the gpt-tokenizer package using npm for use in Node.js projects. This is the standard method for adding the library to your project dependencies.

```bash
npm install gpt-tokenizer
```

--------------------------------

### Import Tokenizers by Model Name

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Import tokenizers directly for specific models to get accurate encoding and model-specific features like pricing data. This allows for precise tokenization tailored to individual model architectures.

```typescript
// GPT-4o (uses o200k_base encoding)
import {
  encode,
  decode,
  encodeChat,
  isWithinTokenLimit,
  estimateCost,
} from 'gpt-tokenizer/model/gpt-4o'

// GPT-4 (uses cl100k_base encoding)
import { encode, decode } from 'gpt-tokenizer/model/gpt-4'

// GPT-3.5 Turbo
import { encode, decode, encodeChat } from 'gpt-tokenizer/model/gpt-3.5-turbo'

// O1 reasoning models
import { encode, decode } from 'gpt-tokenizer/model/o1'

// GPT-4.1 models
import { encode, decode } from 'gpt-tokenizer/model/gpt-4.1'
import { encode as encodeMini } from 'gpt-tokenizer/model/gpt-4.1-mini'

// Lazy loading for bundle optimization
async function getTokenizer() {
  const { encode, decode } = await import('gpt-tokenizer/model/gpt-4o')
  return { encode, decode }
}
```

--------------------------------

### Tokenize Chat Conversations (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Encode chat message arrays into tokens following OpenAI's ChatML format. This function correctly handles system, user, and assistant messages, including named system messages and examples.

```typescript
import { encodeChat } from 'gpt-tokenizer'

const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'What is the weather today?' },
  { role: 'assistant', content: 'I can help you check the weather.' },
]

const tokens = encodeChat(chat)
console.log(tokens.length) // Total token count for the conversation

// With named system examples
const chatWithNames = [
  {
    role: 'system',
    content: 'You translate corporate jargon into plain English.',
  },
  {
    role: 'system',
    name: 'example_user',
    content: 'New synergies will help drive top-line growth.',
  },
  {
    role: 'system',
    name: 'example_assistant',
    content: 'Things working well together will increase revenue.',
  },
  {
    role: 'user',
    content: "Let's circle back when we have more bandwidth.",
  },
]

const chatTokens = encodeChat(chatWithNames)
console.log(chatTokens.length)
```

--------------------------------

### Lazy Loading Tokenizer with Dynamic Import in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Demonstrates how to load the tokenizer asynchronously using dynamic `import()`. This approach is useful for lazy loading, where the tokenizer is only imported when needed, potentially improving initial load times.

```typescript
const {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} = await import('gpt-tokenizer/model/gpt-3.5-turbo')
```

--------------------------------

### Loading Custom BPE Encodings in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Explains how to load a Byte Pair Encoding (BPE) directly if the model is not explicitly supported by the package. This allows users to utilize custom or known BPE encodings for models not listed in the library's presets.

```typescript
import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/encoding/cl100k_base'
```

--------------------------------

### Encode, Decode, and Token Limit Checks in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Demonstrates how to encode text into tokens, decode tokens back into text, and check if text is within a specified token limit using the gpt-tokenizer library. It also shows encoding chat messages and handling special tokens.

```typescript
import {
  encode,
  encodeChat,
  decode,
  isWithinTokenLimit,
  encodeGenerator,
  decodeGenerator,
  decodeAsyncGenerator,
  ALL_SPECIAL_TOKENS,
} from 'gpt-tokenizer'
// note: depending on the model, import from the respective file, e.g.:
// import {...} from 'gpt-tokenizer/model/gpt-4o'

const text = 'Hello, world!'
const tokenLimit = 10

// Encode text into tokens
const tokens = encode(text)

// Decode tokens back into text
const decodedText = decode(tokens)

// Check if text is within the token limit
// returns false if the limit is exceeded, otherwise returns the actual number of tokens (truthy value)
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

// Allow special tokens when needed
const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

// Example chat:
const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },
] as const

// Encode chat into tokens
const chatTokens = encodeChat(chat)

// Check if chat is within the token limit
const chatWithinTokenLimit = isWithinTokenLimit(chat, tokenLimit)

const chatWithinTokenLimitWithSpecial = isWithinTokenLimit(chat, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})

// Encode text using generator
for (const tokenChunk of encodeGenerator(text)) {
  console.log(tokenChunk)
}

// Decode tokens using generator
for (const textChunk of decodeGenerator(tokens)) {
  console.log(textChunk)
}

// Decode tokens using async generator
// (assuming `asyncTokens` is an AsyncIterableIterator<number>)
for await (const textChunk of decodeAsyncGenerator(asyncTokens)) {
  console.log(textChunk)
}
```

--------------------------------

### Importing Specific Model Encodings in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Shows how to import tokenizer functions for specific OpenAI models, such as 'gpt-3.5-turbo', by importing from dedicated model files within the 'gpt-tokenizer' package. This ensures compatibility with the correct encoding scheme for each model.

```typescript
import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/model/gpt-3.5-turbo'
```

--------------------------------

### Load gpt-tokenizer as a UMD module

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

This HTML snippet demonstrates how to load the gpt-tokenizer as a UMD module via a CDN. The tokenizer is then available globally as GPTTokenizer_cl100k_base, providing access to encode and decode functions.

```html
<script src="https://unpkg.com/gpt-tokenizer"></script>

<script>
  // the package is now available as a global:
  const { encode, decode } = GPTTokenizer_cl100k_base
</script>
```

--------------------------------

### Encode Text to Tokens (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Convert plain text into a sequence of integer tokens using Byte Pair Encoding (BPE). This is the fundamental step for preparing text input for GPT models. It supports basic encoding and encoding with specific allowed special tokens.

```typescript
import { encode, ALL_SPECIAL_TOKENS, EndOfText, EndOfPrompt } from 'gpt-tokenizer'

// Basic encoding
const tokens = encode('Hello, world!')
console.log(tokens) // [24912, 11, 2375, 0]

// Encoding with special tokens allowed
const textWithSpecial = `Hello world${EndOfText} more text`
const tokensWithSpecial = encode(textWithSpecial, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})
console.log(tokensWithSpecial) // Includes special token ID for <|endoftext|>

// Using specific allowed special tokens
const allowedSet = new Set([EndOfPrompt])
const encoded = encode(`Some Text ${EndOfPrompt}`, {
  allowedSpecial: allowedSet,
})
```

--------------------------------

### Load gpt-tokenizer in Browser (UMD)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Include the gpt-tokenizer library in a browser environment using a UMD script tag. This makes the tokenizer functions available globally, typically under a specific namespace.

```html
<script src="https://unpkg.com/gpt-tokenizer"></script>
<script>
  const { encode, decode } = GPTTokenizer_cl100k_base
</script>
```

--------------------------------

### Importing from cjs or esm Directories in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Provides an alternative import path for environments that may not support package.json 'exports' resolution. This method imports directly from the CommonJS (cjs) or ECMAScript Modules (esm) directories within the 'gpt-tokenizer' package, ensuring broader compatibility.

```typescript
import {
  encode,
  decode,
  isWithinTokenLimit,
  // etc...
} from 'gpt-tokenizer/cjs/model/gpt-3.5-turbo'
```

--------------------------------

### Load specific encodings as UMD modules

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

These script tags show how to load specific encoding modules for gpt-tokenizer from a CDN. Each script provides a global variable named GPTTokenizer_${encoding} for accessing the tokenizer with that specific encoding, such as o200k_base for modern models or cl100k_base for gpt-4 and gpt-3.5.

```html
<script src="https://unpkg.com/gpt-tokenizer/dist/o200k_base.js"></script>
<script src="https://unpkg.com/gpt-tokenizer/dist/o200k_harmony.js"></script>
<script src="https://unpkg.com/gpt-tokenizer/dist/cl100k_base.js"></script>
<script src="https://unpkg.com/gpt-tokenizer/dist/p50k_base.js"></script>
<script src="https://unpkg.com/gpt-tokenizer/dist/p50k_edit.js"></script>
<script src="https://unpkg.com/gpt-tokenizer/dist/r50k_base.js"></script>
```

--------------------------------

### Configure Merge Cache for Performance (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Optimizes encoding performance by configuring an LRU cache for similar strings. Cache size can be adjusted, disabled, or cleared.

```typescript
import {
  encode,
  setMergeCacheSize,
  clearMergeCache,
} from 'gpt-tokenizer'

// Increase cache for applications with repetitive text
setMergeCacheSize(200000) // 200k entries

// Process many similar strings efficiently
const templates = [
  'Hello, {name}! Welcome to our service.',
  'Hello, {name}! Thank you for your purchase.',
  'Hello, {name}! Your order is ready.',
]

for (const template of templates) {
  const tokens = encode(template) // Cached merges speed up similar text
}

// Reduce cache for memory-constrained environments
setMergeCacheSize(10000)

// Disable caching entirely
setMergeCacheSize(0)

// Clear cache to free memory
clearMergeCache()

// Check vocabulary size
import { vocabularySize } from 'gpt-tokenizer'
console.log(`Vocabulary size: ${vocabularySize}`)
```

--------------------------------

### Encode Text to Tokens in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Shows the basic usage of the `encode` function from the 'gpt-tokenizer' library. This function takes a string as input and returns an array of numbers representing the tokens, suitable for input into GPT models.

```typescript
import { encode } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokens = encode(text)
```

--------------------------------

### Import Tokenizers by Encoding Name

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Import tokenizers by BPE encoding name when working with models not explicitly listed or for encoding-specific operations. This provides flexibility for custom model integrations and direct encoding manipulation.

```typescript
// o200k_base - for GPT-4o, GPT-4.1, o1, o3, o4 models
import { encode, decode } from 'gpt-tokenizer/encoding/o200k_base'

// cl100k_base - for GPT-4, GPT-3.5 models
import { encode, decode } from 'gpt-tokenizer/encoding/cl100k_base'

// p50k_base - for text-davinci-003, text-davinci-002
import { encode, decode } from 'gpt-tokenizer/encoding/p50k_base'

// r50k_base - for text-davinci-001
import { encode, decode } from 'gpt-tokenizer/encoding/r50k_base'

// o200k_harmony - for open-weight Harmony models (gpt-oss-*)
import { encode, decode } from 'gpt-tokenizer/encoding/o200k_harmony'

// Get encoding API instance
import { GptEncoding } from 'gpt-tokenizer/GptEncoding'
import { resolveEncoding } from 'gpt-tokenizer/resolveEncoding'

const encoding = GptEncoding.getEncodingApi('cl100k_base', resolveEncoding)
const tokens = encoding.encode('Hello, world!')
```

--------------------------------

### Decode Tokens to Text in TypeScript

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Demonstrates the use of the `decode` function from the 'gpt-tokenizer' library. This function takes an array of token numbers and converts them back into a human-readable string.

```typescript
import { decode } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
const text = decode(tokens)
```

--------------------------------

### Decode Tokens to Text (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Convert a sequence of integer tokens back into human-readable text. This is crucial for interpreting the output from GPT models. The function supports basic decoding and round-trip consistency checks.

```typescript
import { encode, decode } from 'gpt-tokenizer'

// Basic decoding
const tokens = [24912, 11, 2375, 0]
const text = decode(tokens)
console.log(text) // "Hello, world!"

// Round-trip encoding and decoding
const original = 'The quick brown fox jumps over the lazy dog.'
const roundTrip = decode(encode(original))
console.log(roundTrip === original) // true

// Handling emoji and unicode
const emojiText = 'hello 👋 world 🌍'
const emojiTokens = encode(emojiText)
console.log(decode(emojiTokens)) // "hello 👋 world 🌍"
```

--------------------------------

### Estimate API Usage Costs with gpt-tokenizer/model/gpt-4o

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Estimates the cost of processing tokens based on model-specific pricing. It returns cost breakdowns for the main API, batch API, and cached tokens where available. The function can also be used to estimate costs for different models by passing a model configuration object.

```typescript
import { estimateCost } from 'gpt-tokenizer/model/gpt-4o'

const tokenCount = 1000
const costEstimate = estimateCost(tokenCount)

console.log('Main API costs:')
console.log(`  Input: $${costEstimate.main?.input?.toFixed(6)}`)
console.log(`  Output: $${costEstimate.main?.output?.toFixed(6)}`)

console.log('Batch API costs:')
console.log(`  Input: $${costEstimate.batch?.input?.toFixed(6)}`)
console.log(`  Output: $${costEstimate.batch?.output?.toFixed(6)}`)

// Override model for cost estimation
import gpt4o from 'gpt-tokenizer/model/gpt-4o'
import * as models from 'gpt-tokenizer/models'

const gpt35Cost = gpt4o.estimateCost(1000, models['gpt-3.5-turbo'])
console.log(`GPT-3.5 input cost: $${gpt35Cost.main?.input?.toFixed(6)}`)
```

--------------------------------

### encodeChat

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Encodes an array of chat messages into a sequence of tokens. Requires specifying the model for accurate tokenization if not already configured.

```APIDOC
## encodeChat

### Description
Encodes the given chat into a sequence of tokens. The optional `encodeOptions` parameter lets you configure special token handling. If you didn't import the model version directly, or if `model` wasn't provided during initialization, it must be provided here to correctly tokenize the chat for a given model. Use this method when you need to transform a chat into the token format that the GPT models can process.

### Method
`encodeChat(chat: ChatMessage[], model?: ModelName, encodeOptions?: EncodeOptions): number[]`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **chat** (`ChatMessage[]`) - Required - An array of chat messages to encode.
- **model** (`ModelName`) - Optional - The name of the GPT model to use for encoding.
- **encodeOptions** (`EncodeOptions`) - Optional - Configuration for special token handling.

### Request Example
```typescript
import { encodeChat } from 'gpt-tokenizer'

const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },
]
const tokens = encodeChat(chat)
```

### Response
#### Success Response (200)
- `number[]`: An array of token IDs representing the encoded chat.
```

--------------------------------

### isWithinTokenLimit

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Checks if the input text or chat messages are within a specified token limit without fully encoding them. Returns `false` if the limit is exceeded, otherwise returns the token count.

```APIDOC
## isWithinTokenLimit

### Description
Checks if the input is within the token limit. Returns `false` if the limit is exceeded, otherwise returns the number of tokens. Use this method to quickly check if a given text or chat is within the token limit imposed by GPT models, without encoding the entire input. The optional `encodeOptions` parameter lets you configure special token handling.

### Method
`isWithinTokenLimit(text: string | Iterable<ChatMessage>, tokenLimit: number, encodeOptions?: EncodeOptions): false | number`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```typescript
import { isWithinTokenLimit, ALL_SPECIAL_TOKENS } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenLimit = 10
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})
```

### Response
#### Success Response (200)
- `false` or `number`: Returns `false` if the token limit is exceeded, otherwise returns the number of tokens.
```

--------------------------------

### Estimate Token Encoding Cost

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Estimates the cost of processing a given number of tokens using model-specific pricing. It calculates costs for main and batch API usage, including cached tokens, returning a PriceData object in USD.

```typescript
import { estimateCost } from 'gpt-tokenizer/model/gpt-4o'

const tokenCount = 1000
const costEstimate = estimateCost(tokenCount)

console.log('Main API input cost:', costEstimate.main?.input)
console.log('Main API output cost:', costEstimate.main?.output)
console.log('Batch API input cost:', costEstimate.batch?.input)
```

--------------------------------

### Customizing Allowed Special Tokens during Encoding

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Allows specifying custom sets of allowed special tokens when encoding text by passing a Set of tokens to the `encode` function. Special tokens like EndOfPrompt can be included.

```typescript
import {
  EndOfPrompt,
  encode,
} from 'gpt-tokenizer'

const inputText = `Some Text ${EndOfPrompt}`
const allowedSpecialTokens = new Set([EndOfPrompt])
const encoded = encode(inputText, { allowedSpecialTokens })
const expectedEncoded = [8538, 2991, 220, 100276]

// expect(encoded).toBe(expectedEncoded)
```

--------------------------------

### Adjusting LRU Merge Cache Size

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Manages the LRU cache size for encoding performance. Increasing the size speeds up encoding similar strings at the cost of memory, while setting it to 0 disables caching.

```typescript
import { setMergeCacheSize } from 'gpt-tokenizer'

// Set to 5000 entries
setMergeCacheSize(5000)

// Disable caching completely
setMergeCacheSize(0)
```

--------------------------------

### encodeGenerator

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Encodes text into tokens using a generator, yielding chunks of tokens. Ideal for processing large texts or streaming data.

```APIDOC
## encodeGenerator

### Description
Encodes the given text using a generator, yielding chunks of tokens. Use this method when you want to encode text in chunks, which can be useful for processing large texts or streaming data.

### Method
`encodeGenerator(text: string): Generator<number[], void, undefined>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **text** (`string`) - Required - The text to encode.

### Request Example
```typescript
import { encodeGenerator } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokens = []
for (const tokenChunk of encodeGenerator(text)) {
  tokens.push(...tokenChunk)
}
```

### Response
#### Success Response (200)
- `Generator<number[], void, undefined>`: A generator that yields chunks of token IDs.
```

--------------------------------

### Async Token Decoding with Generator

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Decodes a sequence of tokens asynchronously using a generator. This is useful for processing large outputs or streaming data in an asynchronous context, yielding chunks of decoded text.

```javascript
import { decodeAsyncGenerator } from 'gpt-tokenizer'

async function processTokens(asyncTokensIterator) {
  let decodedText = ''
  for await (const textChunk of decodeAsyncGenerator(asyncTokensIterator)) {
    decodedText += textChunk
  }
}
```

--------------------------------

### Async Stream Token Decoding with decodeAsyncGenerator

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Decodes tokens asynchronously using an async generator. Perfect for processing streaming API responses or async token streams. It accepts an async iterable of tokens and yields text chunks asynchronously.

```typescript
import { decodeAsyncGenerator } from 'gpt-tokenizer'

// Simulate async token stream from API
async function* simulateStreamingResponse() {
  const tokens = [24912, 11, 2375, 0]
  for (const token of tokens) {
    await new Promise((resolve) => setTimeout(resolve, 100))
    yield token
  }
}

async function processStreamingResponse() {
  let fullText = ''

  for await (const textChunk of decodeAsyncGenerator(simulateStreamingResponse())) {
    fullText += textChunk
    console.log(`Received: "${textChunk}"`) 
  }

  console.log(`Complete response: "${fullText}"`) 
}

processStreamingResponse()

// Real-world usage with OpenAI streaming
async function handleOpenAIStream(stream: AsyncIterable<number>) {
  const decoder = decodeAsyncGenerator(stream)

  for await (const text of decoder) {
    // Update UI or process text incrementally
    appendToDisplay(text)
  }
}
```

--------------------------------

### Check Token Limits Efficiently (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Determine if input text or chat messages remain within a specified token limit without fully encoding the content. This function is optimized for quick checks and returns the token count if within limits, or `false` if exceeded.

```typescript
import { isWithinTokenLimit, ALL_SPECIAL_TOKENS } from 'gpt-tokenizer'

// Check plain text
const text = 'This is a sample text for token counting.'
const limit = 100

const result = isWithinTokenLimit(text, limit)
if (result === false) {
  console.log('Token limit exceeded!')
} else {
  console.log(`Within limit: ${result} tokens used`)
}

// Check chat messages
const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Hello, how are you?' },
]

const chatResult = isWithinTokenLimit(chat, 50)
console.log(chatResult) // false if over 50 tokens, otherwise token count

// With special tokens allowed
const textWithSpecial = `Hello <|endoftext|> world`
const resultWithSpecial = isWithinTokenLimit(textWithSpecial, 100, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})
```

--------------------------------

### Count Chat Completion Tokens for Function Calling (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Estimates the token count for a chat completion request that includes function definitions or pinned function calls. This helper is specifically for models supporting `function_calling`. It accounts for message overhead, function definitions, and pinned calls.

```typescript
import {
  countChatCompletionTokens,
  type ChatCompletionRequest,
} from 'gpt-tokenizer/model/gpt-4o'

const request: ChatCompletionRequest = {
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Find the weather for San Francisco.' },
  ],
  functions: [
    {
      name: 'get_weather',
      description: 'Look up the weather for a city.',
      parameters: {
        type: 'object',
        required: ['city'],
        properties: {
          city: { type: 'string' },
          unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
      },
    },
  ],
}

const promptTokenEstimate = countChatCompletionTokens(request)
```

```typescript
import gpt4o from 'gpt-tokenizer/model/gpt-4o'

// Reuse the `request` defined above
const tokenCount = gpt4o.countChatCompletionTokens?.(request)
```

--------------------------------

### countChatCompletionTokens

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Calculates the token count for a chat completion request, including message overhead, function definitions, and pinned function calls. This is specifically for models supporting function calling.

```APIDOC
## countChatCompletionTokens

### Description
Counts the tokens that a function-calling chat completion request will consume, including message overhead, optional function definitions, and pinned function calls. This helper is only available on models that support the `function_calling` feature.

### Method
`countChatCompletionTokens(request: ChatCompletionRequest): number`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **request** (`ChatCompletionRequest`) - Required - The chat completion request object, including messages and optional function definitions.

### Request Example
```typescript
import {
  countChatCompletionTokens,
  type ChatCompletionRequest,
} from 'gpt-tokenizer/model/gpt-4o'

const request: ChatCompletionRequest = {
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Find the weather for San Francisco.' },
  ],
  functions: [
    {
      name: 'get_weather',
      description: 'Look up the weather for a city.',
      parameters: {
        type: 'object',
        required: ['city'],
        properties: {
          city: { type: 'string' },
          unit: { type: 'string', enum: ['celsius', 'fahrenheit'] },
        },
      },
    },
  ],
}

const promptTokenEstimate = countChatCompletionTokens(request)
```

### Response
#### Success Response (200)
- `number`: The estimated number of tokens the chat completion request will consume.
```

--------------------------------

### decodeGenerator

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Decodes a sequence of token IDs into text using a generator, yielding chunks of decoded text. Useful for processing large outputs or streaming data.

```APIDOC
## decodeGenerator

### Description
Decodes a sequence of tokens using a generator, yielding chunks of decoded text. Use this method when you want to decode tokens in chunks, which can be useful for processing large outputs or streaming data.

### Method
`decodeGenerator(tokens: Iterable<number>): Generator<string, void, undefined>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **tokens** (`Iterable<number>`) - Required - An iterable sequence of token IDs to decode.

### Request Example
```typescript
import { decodeGenerator } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
let decodedText = ''
for (const textChunk of decodeGenerator(tokens)) {
  decodedText += textChunk
}
```

### Response
#### Success Response (200)
- `Generator<string, void, undefined>`: A generator that yields chunks of decoded text.
```

--------------------------------

### countTokens

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Counts the number of tokens in the input text or chat messages. Useful for determining token counts without checking against a specific limit.

```APIDOC
## countTokens

### Description
Counts the number of tokens in the input text or chat. Use this method when you need to determine the number of tokens without checking against a limit. The optional `encodeOptions` parameter allows you to specify custom sets of allowed or disallowed special tokens.

### Method
`countTokens(text: string | Iterable<ChatMessage>, encodeOptions?: EncodeOptions): number`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```typescript
import { countTokens } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenCount = countTokens(text)
```

### Response
#### Success Response (200)
- `number`: The total number of tokens in the input text or chat.
```

--------------------------------

### Encode Text to Tokens Using a Generator (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Encodes a given text into tokens, yielding chunks of token IDs. This generator-based approach is efficient for processing large texts or streaming scenarios, allowing tokenization in manageable parts.

```typescript
import { encodeGenerator } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokens = []
for (const tokenChunk of encodeGenerator(text)) {
  tokens.push(...tokenChunk)
}
```

--------------------------------

### encodeChatGenerator

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Encodes chat messages into tokens using a generator, yielding chunks of tokens. Accepts any iterator as input for chat messages.

```APIDOC
## encodeChatGenerator

### Description
Same as `encodeChat`, but uses a generator as output, and may use any iterator as the input `chat`.

### Method
`encodeChatGenerator(chat: Iterator<ChatMessage>, model?: ModelName): Generator<number[], void, undefined>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **chat** (`Iterator<ChatMessage>`) - Required - An iterator yielding chat messages to encode.
- **model** (`ModelName`) - Optional - The name of the GPT model to use for encoding.

### Request Example
(No specific example provided in source text, but usage would be similar to `encodeGenerator` with chat messages)

### Response
#### Success Response (200)
- `Generator<number[], void, undefined>`: A generator that yields chunks of token IDs representing the encoded chat.
```

--------------------------------

### Handle Special Tokens in Encoding (TypeScript)

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Configures how special tokens are processed during encoding. By default, special tokens are disallowed and will throw an error. You can allow all, specific, or disallow specific tokens using sets.

```typescript
import {
  encode,
  decode,
  ALL_SPECIAL_TOKENS,
  EndOfText,
  EndOfPrompt,
  FimPrefix,
  FimMiddle,
  FimSuffix,
  ImStart,
  ImEnd,
} from 'gpt-tokenizer'

// Allow all special tokens
const text = `Hello ${EndOfText} world ${EndOfPrompt}`
const tokens = encode(text, { allowedSpecial: ALL_SPECIAL_TOKENS })
console.log(decode(tokens)) // "Hello <|endoftext|> world <|endofprompt|>"

// Allow specific special tokens only
const allowedSet = new Set([EndOfText])
const tokensPartial = encode(`Text ${EndOfText}`, {
  allowedSpecial: allowedSet,
})

// Disallow specific special tokens (throws error if found)
const disallowedSet = new Set([EndOfText])
try {
  encode(`Text ${EndOfText}`, { disallowedSpecial: disallowedSet })
} catch (error) {
  console.log('Disallowed special token found!')
}

// Fill-in-the-middle tokens for code completion
const codeCompletion = `${FimPrefix}function hello() {${FimSuffix}}${FimMiddle}`
const codeTokens = encode(codeCompletion, {
  allowedSpecial: new Set([FimPrefix, FimSuffix, FimMiddle]),
})
```

--------------------------------

### Stream Token Decoding with decodeGenerator

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Decodes tokens using a generator, yielding chunks of decoded text. Ideal for processing large outputs or streaming model responses. It takes an iterable of tokens as input and returns an async iterable of text chunks.

```typescript
import { decodeGenerator } from 'gpt-tokenizer'

const tokens = [24912, 11, 2375, 0, 2500, 382, 261, 6150, 2201, 13]
let decodedText = ''

for (const textChunk of decodeGenerator(tokens)) {
  decodedText += textChunk
  console.log(`Decoded chunk: "${textChunk}"`) 
}

console.log(`Full text: "${decodedText}"`) 

// Streaming decode from token array
function* tokenSource() {
  const tokens = [24912, 61138, 233, 2375, 130321, 235] // "hello 👋 world 🌍"
  for (const token of tokens) {
    yield token
  }
}

for (const text of decodeGenerator(tokenSource())) {
  process.stdout.write(text)
}
```

--------------------------------

### Count Function-Calling Chat Completion Tokens with gpt-tokenizer/model/gpt-4o

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Counts tokens specifically for OpenAI function-calling chat completion requests. This includes overhead for messages, function definitions, and pinned function calls. This functionality is only available on models that support function calling. The function can be imported directly or accessed via the default export of the model module.

```typescript
import {
  countChatCompletionTokens,
  type ChatCompletionRequest,
} from 'gpt-tokenizer/model/gpt-4o'

const request: ChatCompletionRequest = {
  messages: [
    { role: 'system', content: 'You are a helpful assistant.' },
    { role: 'user', content: 'Find the weather for San Francisco.' },
  ],
  functions: [
    {
      name: 'get_weather',
      description: 'Look up the weather for a city.',
      parameters: {
        type: 'object',
        required: ['city'],
        properties: {
          city: { type: 'string', description: 'The city name' },
          unit: {
            type: 'string',
            enum: ['celsius', 'fahrenheit'],
            description: 'Temperature unit',
          },
        },
      },
    },
  ],
  function_call: 'auto',
}

const promptTokens = countChatCompletionTokens(request)
console.log(`Prompt tokens: ${promptTokens}`)

// Using default export
import gpt4o from 'gpt-tokenizer/model/gpt-4o'

const tokenCount = gpt4o.countChatCompletionTokens?.(request)
console.log(`Token count: ${tokenCount}`)
```

--------------------------------

### Decode Tokens to Text Using a Generator (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Decodes a sequence of token IDs back into human-readable text, yielding chunks of decoded strings. This generator function is ideal for handling large token sequences or streaming decoded output efficiently.

```typescript
import { decodeGenerator } from 'gpt-tokenizer'

const tokens = [18435, 198, 23132, 328]
let decodedText = ''
for (const textChunk of decodeGenerator(tokens)) {
  decodedText += textChunk
}
```

--------------------------------

### Customizing Disallowed Special Tokens during Encoding

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Enables specifying custom sets of disallowed special tokens during text encoding. Passing a Set of tokens to `encode` will cause an error if any disallowed token is present in the input.

```typescript
import { encode, EndOfText } from 'gpt-tokenizer'

const inputText = `Some Text ${EndOfText}`
const disallowedSpecial = new Set([EndOfText])
// throws an error: 
// const encoded = encode(inputText, { disallowedSpecial })
```

--------------------------------

### Count Tokens in Text or Chat with gpt-tokenizer

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Counts the exact number of tokens for plain text or chat messages. This function is useful when the precise token count is needed without checking against a specific limit. It supports counting tokens in strings and arrays of chat message objects. Special tokens can be handled by providing an 'allowedSpecial' option.

```typescript
import { countTokens } from 'gpt-tokenizer'

// Count tokens in text
const text = 'Hello, world! How are you today?'
const tokenCount = countTokens(text)
console.log(`Token count: ${tokenCount}`)

// Count tokens in chat
const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'What is 2 + 2?' },
]

const chatTokenCount = countTokens(chat)
console.log(`Chat token count: ${chatTokenCount}`)

// With special tokens
import { countTokens, ALL_SPECIAL_TOKENS, EndOfText } from 'gpt-tokenizer'

const textWithSpecial = `Hello ${EndOfText} world`
const count = countTokens(textWithSpecial, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})
console.log(`Token count with special: ${count}`)
```

--------------------------------

### Stream Chat Encoding with encodeChatGenerator

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Encodes chat messages using a generator, yielding token chunks. Accepts any iterable as input chat messages. This function is useful for processing chat histories efficiently.

```typescript
import { encodeChatGenerator } from 'gpt-tokenizer/model/gpt-4o'

const messages = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Hello!' },
  { role: 'assistant', content: 'Hi there! How can I help you today?' },
]

const allTokens: number[] = []

for (const tokenChunk of encodeChatGenerator(messages)) {
  allTokens.push(...tokenChunk)
}

console.log(`Total chat tokens: ${allTokens.length}`)

// Stream from message iterator
function* messageStream() {
  yield { role: 'system', content: 'You are a helpful assistant.' }
  yield { role: 'user', content: 'What is TypeScript?' }
}

for (const tokens of encodeChatGenerator(messageStream())) {
  console.log(`Message tokens: ${tokens.length}`)
}
```

--------------------------------

### Encode Chat Messages to Tokens (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Converts an array of chat messages into a sequence of token IDs. This is essential for preparing chat data for GPT models. The function allows specifying the target model and configuring special token handling.

```typescript
import { encodeChat } from 'gpt-tokenizer'

const chat = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'assistant', content: 'gpt-tokenizer is awesome.' },
]
const tokens = encodeChat(chat)
```

--------------------------------

### Count Tokens in Text or Chat (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Counts the number of tokens in the input text or chat messages. This function is useful when you need to know the token count without checking against a specific limit. It supports custom handling of special tokens via `encodeOptions`.

```typescript
import { countTokens } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenCount = countTokens(text)
```

--------------------------------

### Clearing the Merge Cache

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Explicitly clears the LRU merge cache to free up memory. This function is useful when memory management is critical or after significant changes in the data being processed.

```typescript
import { clearMergeCache } from 'gpt-tokenizer'

clearMergeCache()
```

--------------------------------

### Check if Text is Within Token Limit (TypeScript)

Source: https://github.com/niieani/gpt-tokenizer/blob/main/README.md

Checks if the input text or chat messages are within a specified token limit. Returns `false` if the limit is exceeded, otherwise returns the token count. It can optionally configure special token handling using `encodeOptions`.

```typescript
import { isWithinTokenLimit, ALL_SPECIAL_TOKENS } from 'gpt-tokenizer'

const text = 'Hello, world!'
const tokenLimit = 10
const withinTokenLimit = isWithinTokenLimit(text, tokenLimit)

const withinTokenLimitWithSpecial = isWithinTokenLimit(text, tokenLimit, {
  allowedSpecial: ALL_SPECIAL_TOKENS,
})
```

--------------------------------

### Stream Token Encoding with encodeGenerator in gpt-tokenizer

Source: https://context7.com/niieani/gpt-tokenizer/llms.txt

Encodes text into tokens using a generator, yielding chunks of tokens. This is ideal for processing large texts or streaming data efficiently without consuming excessive memory. The function returns an iterable that provides token chunks as they are generated.

```typescript
import { encodeGenerator } from 'gpt-tokenizer'

const text = 'Hello, world! This is a longer text for streaming encoding.'
const allTokens: number[] = []

for (const tokenChunk of encodeGenerator(text)) {
  allTokens.push(...tokenChunk)
  console.log(`Received chunk of ${tokenChunk.length} tokens`)
}

console.log(`Total tokens: ${allTokens.length}`)

// Process large text in chunks
function processLargeText(largeText: string, maxTokensPerChunk: number) {
  let currentChunk: number[] = []

  for (const tokens of encodeGenerator(largeText)) {
    currentChunk.push(...tokens)
    if (currentChunk.length >= maxTokensPerChunk) {
      processChunk(currentChunk)
      currentChunk = []
    }
  }

  if (currentChunk.length > 0) {
    processChunk(currentChunk)
  }
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.