### Installation

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Commands to install the tokenx package using npm, pnpm, or yarn.

```bash
# npm
npm install tokenx

# pnpm
pnpm add tokenx

# yarn
yarn add tokenx
```

--------------------------------

### Usage

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Example demonstrating how to use the main functions of the tokenx library, including estimation, limit checking, slicing, and splitting, with custom options.

```typescript
import { estimateTokenCount, isWithinTokenLimit, sliceByTokens, splitByTokens } from 'tokenx'

const text = 'Your text goes here.'

// Estimate the number of tokens in the text
const estimatedTokens = estimateTokenCount(text)
console.log(`Estimated token count: ${estimatedTokens}`)

// Check if text is within a specific token limit
const tokenLimit = 1024
const withinLimit = isWithinTokenLimit(text, tokenLimit)
console.log(`Is within token limit: ${withinLimit}`)

// Slice text by token positions (like Array.slice)
const firstTokens = sliceByTokens(text, 0, 5)
console.log(`First ~5 tokens: ${firstTokens}`)

// Split text into token-based chunks
const chunks = splitByTokens(text, 100)
console.log(`Split into ${chunks.length} chunks`)

// Use custom options for different languages or models
const customOptions = {
  defaultCharsPerToken: 4, // More conservative estimation
  languageConfigs: [
    { pattern: /[你我他]/g, averageCharsPerToken: 1.5 }, // Custom Chinese rule
  ]
}

const customEstimate = estimateTokenCount(text, customOptions)
console.log(`Custom estimate: ${customEstimate}`)
```

--------------------------------

### isWithinTokenLimit API Usage

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Example of using the isWithinTokenLimit function with and without custom options.

```typescript
const withinLimit = isWithinTokenLimit('Check this text against a limit', 100)
// With custom options
const customCheck = isWithinTokenLimit('Text', 50, { defaultCharsPerToken: 3 })
```

--------------------------------

### estimateTokenCount API Usage

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Example of using the estimateTokenCount function with and without custom options.

```typescript
const estimatedTokens = estimateTokenCount('Hello, world!')

// With custom options
const customEstimate = estimateTokenCount('Bonjour le monde!', {
  defaultCharsPerToken: 4,
  languageConfigs: [
    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
  ]
})
```

--------------------------------

### splitByTokens Usage

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Illustrates how to use the splitByTokens function for basic splitting, with overlap, and with custom options.

```typescript
const text = 'Long text that needs to be split into smaller chunks...'

// Basic splitting
const chunks = splitByTokens(text, 100)
console.log(`Split into ${chunks.length} chunks`)

// With overlap for semantic continuity
const overlappedChunks = splitByTokens(text, 100, { overlap: 10 })

// With custom options
const customChunks = splitByTokens(text, 50, {
  defaultCharsPerToken: 4,
  overlap: 5
})
```

--------------------------------

### sliceByTokens Usage

Source: https://github.com/johannschopplich/tokenx/blob/main/README.md

Demonstrates how to use the sliceByTokens function with various index combinations and custom options.

```typescript
const text = 'Hello, world! This is a test sentence.'

const firstThree = sliceByTokens(text, 0, 3)
const fromSecond = sliceByTokens(text, 2)
const lastTwo = sliceByTokens(text, -2)
const middle = sliceByTokens(text, 1, -1)

// With custom options
const customSlice = sliceByTokens(text, 0, 5, {
  defaultCharsPerToken: 4,
  languageConfigs: [
    { pattern: /[éèêëàâîï]/i, averageCharsPerToken: 3 }
  ]
})
```