TokenLens

https://github.com/xn1cklas/tokenlens
Admin
TokenLens provides typed model metadata and utilities for AI applications to manage context, cost,...
Tokens:7,235
Snippets:30
Trust Score:8
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Run Next.js Development Server

Source: https://github.com/xn1cklas/tokenlens/blob/main/apps/web-demo/README.md

Commands to start the Next.js development server using different package managers. Assumes Node.js and a package manager (npm, yarn, pnpm, or bun) are installed. Opens the application at http://localhost:3000.

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

--------------------------------

### Quick Start: TokenLens Usage and Cost Calculation

Source: https://github.com/xn1cklas/tokenlens/blob/main/README.md

Demonstrates how to import and use core TokenLens functions like `fetchModels`, `getContext`, `getTokenCosts`, and `getUsage` to retrieve model context limits, calculate token costs, and get overall usage data for a given model and usage parameters.

```typescript
import { fetchModels } from 'tokenlens';
import { getContext, getTokenCosts, getUsage } from '@tokenlens/helpers';

// 1) Get providers (or import getModels() for a static set)
const openai = await fetchModels('openai'); // or getModels() synchronously

// 2) Use flexible ids: 'provider/model', 'provider:id', or providerless 'model'
const modelId = 'openai/gpt-4o-mini';
const usage = { prompt_tokens: 3200, completion_tokens: 400 };

// Context caps only
const { maxInput, maxOutput, maxTotal } = getContext(modelId, openai);

// Cost breakdown only
const { inputUSD, outputUSD, reasoningUSD, cacheReadUSD, cacheWriteUSD, totalUSD } = getTokenCosts(modelId, usage, openai);

// Or both together
const usageData = getUsage(modelId, usage, openai);

console.log({
  maxInput,
  maxOutput,
  maxTotal,
  inputUSD,
  outputUSD,
  reasoningUSD,
  cacheReadUSD,
  cacheWriteUSD,
  totalUSD,
  usageData,
});

```

--------------------------------

### Quick Start: TokenLens Model Utilities

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Demonstrates basic usage of TokenLens for getting model metadata, calculating percentage of context used, remaining tokens, and estimated cost from usage data. It supports both direct provider usage and Vercel AI SDK fields.

```typescript
import {
  type ModelId,
  modelMeta,
  percentOfContextUsed,
  tokensRemaining,
  costFromUsage,
} from 'tokenlens';

const id: ModelId = 'openai:gpt-4.1';
// Works with provider usage or Vercel AI SDK usage
const usage = { prompt_tokens: 3200, completion_tokens: 400 };

const meta = modelMeta(id);
const used = percentOfContextUsed({ id, usage, reserveOutput: 256 });
const remaining = tokensRemaining({ id, usage, reserveOutput: 256 });
const costUSD = costFromUsage({ id, usage });

console.log({ meta, used, remaining, costUSD });
```

--------------------------------

### Fetch Models and Estimate Token Usage with TokenLens Helpers

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/helpers/README.md

Demonstrates how to fetch model information from OpenAI and then use @tokenlens/helpers functions to get context window details and estimate token costs based on usage. It also shows how to get a summary of usage.

```typescript
import { fetchModels } from 'tokenlens';
import { getContext, getTokenCosts, getUsage } from '@tokenlens/helpers';

const openai = await fetchModels('openai');
const modelId = 'openai/gpt-4o-mini';
const usage = { prompt_tokens: 1000, completion_tokens: 500 };

const { maxInput, maxOutput, maxTotal } = getContext(modelId, openai);
const {
  inputUSD,
  outputUSD,
  reasoningUSD,
  cacheReadUSD,
  cacheWriteUSD,
  totalUSD,
} = getTokenCosts(modelId, usage, openai);
const summary = getUsage(modelId, usage, openai);

```

--------------------------------

### Context Budgeting and Compaction Utilities

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Provides examples of using TokenLens functions for context management, including calculating remaining context with `remainingContext`. The example shows how to specify the model, usage details, and a reserved output size.

```typescript
import { remainingContext, shouldCompact, tokensToCompact, contextHealth } from 'tokenlens';

const rc = remainingContext({ modelId: 'anthropic:claude-3-5-sonnet-20240620', usage: u1, reserveOutput: 256 });
```

--------------------------------

### Get Context Window Limits with TokenLens Helpers

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Illustrates how to retrieve model context window limits (input, output, total tokens) using the `getContext` helper function from '@tokenlens/helpers'. Examples show usage with different formats for model IDs and provider objects, enabling effective budget planning.

```typescript
import { getContext } from '@tokenlens/helpers';
import { fetchModels } from 'tokenlens';

const providers = await fetchModels();

// Object syntax
const context1 = getContext({ modelId: 'openai/gpt-4o', providers });
console.log(context1);
// {
//   inputMax: undefined,
//   outputMax: 16384,
//   combinedMax: 128000,
//   totalMax: 128000,
//   maxInput: undefined,
//   maxOutput: 16384,
//   maxTotal: 128000
// }

// Positional syntax
const context2 = getContext('anthropic/claude-3-5-sonnet', providers);
console.log(context2.maxTotal); // 200000

// Works with provider:model format
const context3 = getContext('anthropic:claude-3-5-haiku', providers);
console.log(context3.maxTotal); // 200000

// Works with providerless model names (searches across providers)
const context4 = getContext('gpt-4o-mini', providers);
console.log(context4.maxTotal); // 128000
```

--------------------------------

### Import and Fetch Models with TokenLens (JavaScript)

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/fetch/README.md

Demonstrates how to import and use the `fetchModels` function from the @tokenlens/fetch package to retrieve model data. It shows examples of fetching the entire catalog, filtering by provider, and filtering by a specific model. Error handling using `FetchModelsError` is also included.

```javascript
import { fetchModels, getModelMeta, FetchModelsError } from '@tokenlens/fetch';

const catalog = await fetchModels();
const openai = await fetchModels({ provider: 'openai' });
const gpt4o = await fetchModels({ provider: 'openai', model: 'gpt-4o' });

// Pick raw metadata directly from the fetched catalog
const prov = getModelMeta({ providers: catalog, provider: 'openai' });
const model = getModelMeta({ providers: catalog, id: 'openai:gpt-4o' });

try {
  await fetchModels();
} catch (err) {
  if (err instanceof FetchModelsError) {
    // 'UNAVAILABLE' | 'NETWORK' | 'HTTP' | 'PARSE'
    console.error(err.code, err.status, err.message);
  }
}
```

--------------------------------

### Get Model Metadata from TokenLens API

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/models/README.md

Demonstrates how to retrieve metadata for specific models using the 'getModelMeta' function from the @tokenlens/models/api module. It shows how to get metadata for a single model, multiple models, or an entire provider.

```typescript
import { getModelMeta } from '@tokenlens/models/api';

const providers = getModels();

// Single model
const singleModelMeta = getModelMeta(providers, 'openai', 'gpt-4o');

// Multiple models
const multipleModelsMeta = getModelMeta(providers, 'openai', ['gpt-4o', 'o3-mini']);

// Whole provider
const allOpenaiModelsMeta = getModelMeta(providers, 'openai');
```

--------------------------------

### Using TokenLens Helpers via 'tokenlens' Package (Back-compat)

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/helpers/README.md

Shows how to import and use context, cost, and usage helper functions directly from the main 'tokenlens' package, which provides backward compatibility by injecting a default set of providers if none are supplied.

```typescript
import { getContext, getTokenCosts, getUsage } from 'tokenlens';
// These wrappers inject a default providers set if not supplied.

```

--------------------------------

### Create Small Catalog with Multiple Providers in TokenLens

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/models/README.md

Shows how to manually construct a small catalog object by importing models from different providers, such as OpenAI and Anthropic.

```typescript
import openai from '@tokenlens/models/providers/openai';
import anthropic from '@tokenlens/models/providers/anthropic';

const small = { providers: { openai, anthropic } };
```

--------------------------------

### Picking Model Metadata with Static Catalog

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Illustrates retrieving model metadata from a statically built catalog using `getModels` and `getModelMeta`. It shows how to access information for providers, specific models, multiple models, or by a provider-agnostic ID.

```typescript
import { getModels, getModelMeta } from 'tokenlens';

// Build a static catalog (or use fetchModels() dynamically)
const providers = getModels();

// Provider info
const openai = getModelMeta({ providers, provider: 'openai' });

// Single model
const gpto = getModelMeta({ providers, provider: 'openai', model: 'gpt-4o' });

// Multiple models
const picks = getModelMeta({ providers, provider: 'openai', models: ['gpt-4o', 'o3-mini'] });

// Providerless id
const viaId = getModelMeta({ providers, id: 'openai:gpt-4o' }); // or 'openai/gpt-4o'
```

--------------------------------

### Fetch AI Model Metadata with TokenLens

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Demonstrates how to fetch AI model metadata using the `fetchModels` function from the 'tokenlens' library. It covers fetching all providers, specific providers or models, searching across providers, and includes error handling for network or HTTP issues. The static `getModels` function is also shown for zero-latency access.

```typescript
import { fetchModels, getModels, FetchModelsError } from 'tokenlens';

// Fetch all providers from models.dev (requires Node 18+ or modern browsers)
const allProviders = await fetchModels();
console.log(Object.keys(allProviders)); // ['openai', 'anthropic', 'deepseek', ...]

// Fetch a specific provider
const openai = await fetchModels({ provider: 'openai' });
console.log(openai?.models['gpt-4o']?.limit?.context); // 128000

// Fetch a specific model within a provider
const gpt4o = await fetchModels({ provider: 'openai', model: 'gpt-4o' });
console.log(gpt4o?.name); // 'GPT-4o'

// Search for a model across all providers
const matches = await fetchModels({ model: 'claude-3-5-sonnet' });
console.log(matches.map(m => m.provider)); // ['anthropic', 'google-vertex-anthropic', ...]

// Error handling with typed error codes
try {
  await fetchModels({ provider: 'openai' });
} catch (err) {
  if (err instanceof FetchModelsError) {
    // err.code is: 'UNAVAILABLE' | 'NETWORK' | 'HTTP' | 'PARSE'
    console.error(`Fetch failed: ${err.code} - ${err.message}`);
    if (err.status) console.error(`HTTP status: ${err.status}`);
  }
}

// Use static registry (synchronous, no network call)
const staticProviders = getModels();
const staticOpenAI = staticProviders.openai;
```

--------------------------------

### Async Fetching of Model Catalogs

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Demonstrates fetching model metadata dynamically from models.dev using `fetchModels`. It covers fetching the entire catalog, by provider, specific models, searching across providers, and handling potential fetch errors. Includes an option for custom fetch implementations.

```typescript
import {
  fetchModels,
  FetchModelsError,
  type ModelCatalog,
  type ProviderInfo,
  type ProviderModel,
} from 'tokenlens';

// 1) Fetch the full catalog (Node 18+ or modern browsers with global fetch)
const catalog: ModelCatalog = await fetchModels();

// 2) Fetch by provider key (e.g. 'openai', 'anthropic', 'deepseek')
const openai: ProviderInfo | undefined = await fetchModels({ provider: 'openai' });

// 3) Fetch a specific model within a provider
const gpto: ProviderModel | undefined = await fetchModels({ provider: 'openai', model: 'gpt-4o' });

// 4) Search for a model across providers when provider is omitted
const matches: Array<{ provider: string; model: ProviderModel }> = await fetchModels({ model: 'gpt-4.1' });

// 5) Error handling with typed error codes
try {
  await fetchModels();
} catch (err) {
  if (err instanceof FetchModelsError) {
    // err.code is one of: 'UNAVAILABLE' | 'NETWORK' | 'HTTP' | 'PARSE'
    console.error('Fetch failed:', err.code, err.status, err.message);
  } else {
    throw err;
  }
}

// 6) Provide a custom fetch (for Node < 18 or custom runtimes)
// import fetch from 'cross-fetch' or 'undici'
// const catalog = await fetchModels({ fetch });
```

--------------------------------

### Async Fetching Models with TokenLens

Source: https://github.com/xn1cklas/tokenlens/blob/main/README.md

Shows how to asynchronously fetch model information using `fetchModels` from TokenLens. It covers fetching all providers, specific providers, specific models, searching across providers, and robust error handling for network or HTTP issues.

```typescript
import {
  fetchModels,
  FetchModelsError,
  type ModelCatalog,
  type ProviderInfo,
  type ProviderModel,
} from 'tokenlens';

// 1) Fetch all providers (Node 18+ or modern browsers with global fetch)
const providers: ModelCatalog = await fetchModels();

// 2) Fetch by provider key (e.g. 'openai', 'anthropic', 'deepseek')
const openai: ProviderInfo | undefined = await fetchModels({ provider: 'openai' });

// 3) Fetch a specific model within a provider
const gpto: ProviderModel | undefined = await fetchModels({ provider: 'openai', model: 'gpt-4o' });

// 4) Search for a model across providers when provider is omitted
const matches: Array<{ provider: string; model: ProviderModel }> = await fetchModels({ model: 'gpt-4.1' });

// 5) Error handling with typed error codes
try {
  await fetchModels();
} catch (err) {
  if (err instanceof FetchModelsError) {
    // err.code is one of: 'UNAVAILABLE' | 'NETWORK' | 'HTTP' | 'PARSE'
    console.error('Fetch failed:', err.code, err.status, err.message);
  } else {
    throw err;
  }
}

// 6) Provide a custom fetch (for Node < 18 or custom runtimes)
// import fetch from 'cross-fetch' or 'undici'

```

--------------------------------

### Analyze Token Usage: Context Caps and Cost Estimates (TypeScript)

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Utilize the `getUsage` function to obtain both context window capacities and cost estimations in a single API call. This function supports object and positional syntax for input and can optionally reserve output tokens. It relies on `@tokenlens/helpers` and `tokenlens` packages.

```typescript
import { getUsage } from '@tokenlens/helpers';
import { fetchModels } from 'tokenlens';

const providers = await fetchModels();
const usage = { prompt_tokens: 15000, completion_tokens: 2500 };

// Object syntax with output reservation
const usageData1 = getUsage({
  modelId: 'openai/gpt-4o',
  usage,
  providers,
  reserveOutput: 4096
});
console.log(usageData1);
// {
//   context: {
//     inputMax: undefined,
//     outputMax: 16384,
//     combinedMax: 128000,
//     totalMax: 128000,
//     maxInput: undefined,
//     maxOutput: 16384,
//     maxTotal: 128000
//   },
//   costUSD: {
//     inputUSD: 0.075,
//     outputUSD: 0.025,
//     totalUSD: 0.100,
//     reasoningUSD: undefined,
//     cacheReadUSD: undefined,
//     cacheWriteUSD: undefined
//   }
// }

// Positional syntax
const usageData2 = getUsage('anthropic/claude-3-opus', usage, providers);
console.log(`Context: ${usageData2.context?.maxTotal}, Cost: $${usageData2.costUSD?.totalUSD}`);
// Context: 200000, Cost: $0.425

// Check if response fits in remaining budget
const currentUsed = { prompt_tokens: 50000, completion_tokens: 8000 };
const analysis = getUsage('gpt-4o', currentUsed, providers, 4096);
const remaining = (analysis.context?.maxTotal ?? 0) - 58000 - 4096;
console.log(`Remaining tokens: ${remaining}`); // Remaining tokens: 65904

```

--------------------------------

### Import Specific Provider Models from TokenLens

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/models/README.md

Imports models for a specific provider (e.g., OpenAI) and optionally creates a small catalog using a helper function. This allows for focused imports.

```typescript
import openai from '@tokenlens/models/providers/openai';
```

```typescript
import { sourceFromModels } from '@tokenlens/helpers';

const source = sourceFromModels(openai);
```

--------------------------------

### Build Models.dev Compatible Catalog with TokenLens

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/models/README.md

Imports the 'getModels' function to build a models.dev-compatible object. This function is used to create a catalog of models.

```typescript
import { getModels } from '@tokenlens/models';
```

--------------------------------

### List Models with Tokenlens

Source: https://github.com/xn1cklas/tokenlens/blob/main/README.md

Demonstrates how to list models from a specific provider with a stable status using the `listModels` function from the 'tokenlens' library. This function is useful for filtering and retrieving model information based on provider and status.

```typescript
import { listModels } from 'tokenlens';

const stableOpenAI = listModels({ provider: 'openai', status: 'stable' });
```

--------------------------------

### Calculate AI Token Costs with TokenLens Helpers

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Shows how to estimate USD costs for AI API calls using the `getTokenCosts` function from '@tokenlens/helpers'. It covers calculating costs based on token usage and model pricing, supporting various input formats for usage data (e.g., OpenAI, Anthropic, Vercel AI SDK) and different model ID specifications.

```typescript
import { getTokenCosts } from '@tokenlens/helpers';
import { fetchModels } from 'tokenlens';

const providers = await fetchModels();
const usage = {
  prompt_tokens: 5000,
  completion_tokens: 1500,
  reasoning_tokens: 2000,
  cache_read_input_tokens: 10000,
  cache_creation_input_tokens: 3000
};

// Object syntax
const costs1 = getTokenCosts({
  modelId: 'openai/gpt-4o',
  usage,
  providers
});
console.log(costs1);
// {
//   inputUSD: 0.025,
//   outputUSD: 0.015,
//   reasoningUSD: 0.120,
//   cacheReadUSD: 0.012,
//   cacheWriteUSD: 0.037,
//   totalUSD: 0.209,
//   inputTokenUSD: 0.025,  // aliases
//   outputTokenUSD: 0.015,
//   reasoningTokenUSD: 0.120,
//   cacheReadsUSD: 0.012,
//   cacheWritesUSD: 0.037
// }

// Positional syntax
const costs2 = getTokenCosts('anthropic/claude-3-5-sonnet', usage, providers);
console.log(`Total cost: $${costs2.totalUSD?.toFixed(4)}`); // Total cost: $0.0895

// Works with various usage field formats (OpenAI, Anthropic, Vercel AI SDK)
const aiSdkUsage = { inputTokens: 3200, outputTokens: 400 };
const costs3 = getTokenCosts('deepseek/deepseek-chat', aiSdkUsage, providers);
console.log(costs3.totalUSD); // 0.00012
```

--------------------------------

### List and Filter Models by Provider and Status (TypeScript)

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Query the model registry using `listModels` and `getModels` from the `tokenlens` package. Filter models by provider (e.g., 'openai', 'anthropic') and status (e.g., 'stable', 'deprecated'). The `getModels` function retrieves the entire catalog, while `listModels` allows for specific filtering.

```typescript
import { listModels } from 'tokenlens';
import { getModels } from 'tokenlens';

const catalog = getModels();

// List all stable OpenAI models
const stableOpenAI = listModels({ provider: 'openai', status: 'stable' });
console.log(stableOpenAI.map(m => m.id));
// ['openai:gpt-4o', 'openai:gpt-4o-mini', 'openai:gpt-4-turbo', ...]

// List all Anthropic models
const allAnthropic = listModels({ provider: 'anthropic' });
console.log(allAnthropic.length); // 15+

// List deprecated models
const deprecated = listModels({ status: 'deprecated' });
console.log(deprecated.map(m => `${m.provider}:${m.vendorId}`));

// Access model details
const gpt4o = stableOpenAI.find(m => m.vendorId === 'gpt-4o');
console.log({
  id: gpt4o?.id,
  displayName: gpt4o?.displayName,
  contextWindow: gpt4o?.context.combinedMax,
  inputPrice: gpt4o?.pricing?.inputPerMTokens,
  outputPrice: gpt4o?.pricing?.outputPerMTokens
});
// {
//   id: 'openai:gpt-4o',
//   displayName: 'GPT-4o',
//   contextWindow: 128000,
//   inputPrice: 5,
//   outputPrice: 15
// }

```

--------------------------------

### Provider-Agnostic Usage Normalization

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Shows how to normalize diverse usage object shapes into a standard format using `normalizeUsage` and break down token counts with `breakdownTokens`. This is useful for handling input from various sources, including the Vercel AI SDK.

```typescript
import { normalizeUsage, breakdownTokens } from 'tokenlens';

// Works with many shapes, including Vercel AI SDK fields
const u1 = normalizeUsage({ prompt_tokens: 1000, completion_tokens: 150 });
const u2 = normalizeUsage({ inputTokens: 900, outputTokens: 200, totalTokens: 1100 });
const b = breakdownTokens({ inputTokens: 900, cachedInputTokens: 300, reasoningTokens: 120 });
```

--------------------------------

### Calculate Remaining Context with TokenLens Helpers

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Shows how to use the `remainingContext` and `percentRemaining` functions from `@tokenlens/helpers` to calculate available context tokens and usage percentages for a given model. It supports different strategies like 'input-only' and handles various usage formats. Note: These specific API functions are deprecated in favor of `getContext` and manual calculations.

```typescript
import { remainingContext, percentRemaining } from '@tokenlens/helpers';
import { getModels } from 'tokenlens';

const catalog = getModels();
const usage = { input_tokens: 80000, output_tokens: 5000 };

// Get detailed remaining context info
const remaining = remainingContext({
  modelId: 'openai/gpt-4o',
  usage,
  reserveOutput: 4096,
  catalog
});
console.log(remaining);
// {
//   remainingCombined: 38904,
//   remainingInput: undefined,
//   percentUsed: 0.696,
//   model: { id: 'openai:gpt-4o', ... }
// }

// Get percent remaining (inverse of percentUsed)
const pctRemaining = percentRemaining({
  modelId: 'anthropic/claude-3-5-sonnet',
  usage: { input_tokens: 50000, output_tokens: 10000 },
  catalog
});
console.log(`${(pctRemaining * 100).toFixed(1)}% remaining`); // 70.0% remaining

// Strategy options: 'provider-default', 'combined', 'input-only'
const inputOnlyRemaining = remainingContext({
  modelId: 'openai/gpt-4o',
  usage: { input_tokens: 30000 },
  strategy: 'input-only',
  catalog
});
console.log(inputOnlyRemaining.remainingInput); // Depends on inputMax
```

--------------------------------

### Check Model Context Health and Compactness

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Determines the health status of a model's context window and checks if it should be compacted. It takes a model ID and usage object as input.

```javascript
if (shouldCompact({ modelId: 'anthropic:claude-3-5-sonnet-20240620', usage: u1 })) {
  const n = tokensToCompact({ modelId: 'anthropic:claude-3-5-sonnet-20240620', usage: u1 });
  // summarize oldest messages by ~n tokens
}
const badge = contextHealth({ modelId: 'openai:gpt-4.1', usage: u2 }); // { status: 'ok'|'warn'|'compact' }
```

--------------------------------

### Normalize Model IDs with TokenLens

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Demonstrates how TokenLens automatically normalizes various model ID formats (gateway-style, Anthropic versioning, canonical, providerless) into a canonical format (provider:model). This normalization simplifies model handling across different providers and is automatically applied in helper functions like getContext.

```typescript
import { toModelId } from '@tokenlens/core';
import { getContext } from '@tokenlens/helpers';
import { getModels } from 'tokenlens';

const providers = getModels();

// Gateway-style format (provider/model) → canonical format (provider:model)
const canonical1 = toModelId('openai/gpt-4o');
console.log(canonical1); // 'openai:gpt-4o'

// Anthropic version normalization: dots → dashes
const canonical2 = toModelId('anthropic/claude-3.5-sonnet');
console.log(canonical2); // 'anthropic:claude-3-5-sonnet'

// Already canonical format passes through
const canonical3 = toModelId('deepseek:deepseek-chat');
console.log(canonical3); // 'deepseek:deepseek-chat'

// Providerless format searches across providers
const canonical4 = toModelId('gpt-4o-mini');
console.log(canonical4); // 'gpt-4o-mini'

// All formats work in helper functions automatically
const ctx1 = getContext('openai/gpt-4o', providers);
const ctx2 = getContext('openai:gpt-4o', providers);
const ctx3 = getContext('gpt-4o', providers);
console.log(ctx1.maxTotal === ctx2.maxTotal && ctx2.maxTotal === ctx3.maxTotal); // true
```

--------------------------------

### Normalize Token Usage Across SDKs (TypeScript)

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Standardize token usage data from various sources like OpenAI, Anthropic, and Vercel AI SDK into a consistent format using `normalizeUsage`. The `breakdownTokens` function provides a more granular analysis including cache and reasoning tokens. Both functions are part of the `@tokenlens/helpers` package.

```typescript
import { normalizeUsage, breakdownTokens } from '@tokenlens/helpers';

// OpenAI format
const openaiUsage = { prompt_tokens: 1000, completion_tokens: 200 };
const normalized1 = normalizeUsage(openaiUsage);
console.log(normalized1); // { input: 1000, output: 200, total: 1200 }

// Anthropic format
const anthropicUsage = { input_tokens: 1000, output_tokens: 200 };
const normalized2 = normalizeUsage(anthropicUsage);
console.log(normalized2); // { input: 1000, output: 200, total: 1200 }

// Vercel AI SDK v2 format
const aiSdkUsage = { inputTokens: 1000, outputTokens: 200, totalTokens: 1200 };
const normalized3 = normalizeUsage(aiSdkUsage);
console.log(normalized3); // { input: 1000, output: 200, total: 1200 }

// Extended breakdown with cache and reasoning tokens
const detailedUsage = {
  prompt_tokens: 5000,
  completion_tokens: 1000,
  reasoning_tokens: 2000,
  cache_read_input_tokens: 10000,
  cache_creation_input_tokens: 3000
};
const breakdown = breakdownTokens(detailedUsage);
console.log(breakdown);
// {
//   input: 5000,
//   output: 1000,
//   total: 6000,
//   cacheReads: 10000,
//   cacheWrites: 3000,
//   reasoningTokens: 2000
// }

```

--------------------------------

### Create and Resolve Model Registry - TypeScript

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/core/README.md

Demonstrates how to create a model registry using `createRegistry` and resolve model identifiers with aliases. This function takes an array of Model objects and returns a registry object with a `resolveModel` method. The `resolveModel` method helps in looking up canonical model entries using shorthands or aliases.

```typescript
import { createRegistry, type Model } from '@tokenlens/core';

const reg = createRegistry([
  {
    id: 'openai:gpt-4o',
    provider: 'openai',
    status: 'stable',
    context: { combinedMax: 128000 },
    source: 'docs',
  } satisfies Model,
]);

// Resolve shorthands/aliases to the canonical entry
const m = reg.resolveModel('gpt-4o');
console.log(m?.id); // 'openai:gpt-4o'
```

--------------------------------

### Estimate Conversation Cost with TokenLens Helpers

Source: https://context7.com/xn1cklas/tokenlens/llms.txt

Illustrates how to use `sumUsage` to aggregate token counts across multiple conversation turns and `estimateConversationCost` to calculate the monetary cost in USD. These functions help in understanding and optimizing the financial impact of AI conversations. Note: These specific API functions are deprecated.

```typescript
import { sumUsage, estimateConversationCost } from '@tokenlens/helpers';
import { getModels } from 'tokenlens';

const catalog = getModels();

// Conversation with 5 turns
const turn1 = { prompt_tokens: 500, completion_tokens: 150 };
const turn2 = { prompt_tokens: 800, completion_tokens: 200 };
const turn3 = { prompt_tokens: 1200, completion_tokens: 250 };
const turn4 = { prompt_tokens: 1500, completion_tokens: 300 };
const turn5 = { prompt_tokens: 2000, completion_tokens: 400 };

// Sum all usage
const totalUsage = sumUsage([turn1, turn2, turn3, turn4, turn5]);
console.log(totalUsage); // { input: 6000, output: 1300, total: 7300 }

// Estimate total conversation cost
const conversationCost = estimateConversationCost({
  modelId: 'openai/gpt-4o-mini',
  usages: [turn1, turn2, turn3, turn4, turn5],
  catalog
});
console.log(conversationCost);
// {
//   inputUSD: 0.0009,
//   outputUSD: 0.00078,
//   totalUSD: 0.00168
// }
console.log(`Total conversation cost: $${conversationCost.totalUSD?.toFixed(4)}`);
// Total conversation cost: $0.0017
```

--------------------------------

### Import Full Model Array from TokenLens

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/models/README.md

Imports the entire 'modelsDev' array, which contains all available models. This is a legacy import pattern.

```typescript
import { modelsDev } from '@tokenlens/models';
```

--------------------------------

### Estimate Conversation Cost and Context Rotation

Source: https://github.com/xn1cklas/tokenlens/blob/main/packages/tokenlens/README.md

Calculates the total token usage for a conversation and estimates the context rotation. It takes an array of usage objects and message token counts as input.

```typescript
import { sumUsage, estimateConversationCost, computeContextRot, nextTurnBudget } from 'tokenlens';

const totals = sumUsage([turn1.usage, turn2.usage, turn3.usage]);
const cost = estimateConversationCost({ modelId: 'openai:gpt-4.1', usages: [turn1.usage, turn2.usage] });
const rot = computeContextRot({ messageTokens: [800, 600, 400, 300, 200], keepRecentTurns: 2, modelId: 'openai:gpt-4.1' });
const nextBudget = nextTurnBudget({ modelId: 'openai:gpt-4.1', usage: totals, reserveOutput: 256 });
```
Tokens:
Raw