### Install js-tiktoken via npm

Source: https://github.com/dqbd/tiktoken/blob/main/js/README.md

Installs the js-tiktoken library using npm. This is the primary method for integrating the tokenizer into your JavaScript project.

```bash
npm install js-tiktoken
```

--------------------------------

### Install tiktoken WASM

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Installs the WebAssembly version of the tiktoken library using npm. This provides full feature parity with the original Python library.

```bash
npm install tiktoken
```

--------------------------------

### Basic tiktoken Usage (TypeScript)

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Demonstrates basic usage of the tiktoken library in TypeScript, including getting an encoding by name, encoding and decoding text, and retrieving an encoder for a specific OpenAI model. It also shows how to extend an encoding with custom special tokens and the importance of freeing the encoder when done.

```typescript
import assert from "node:assert";
import { get_encoding, encoding_for_model } from "tiktoken";

const enc = get_encoding("gpt2");
assert(
  new TextDecoder().decode(enc.decode(enc.encode("hello world"))) ===
    "hello world"
);

// To get the tokeniser corresponding to a specific model in the OpenAI API:
const enc = encoding_for_model("text-davinci-003");

// Extend existing encoding with custom special tokens
const enc = encoding_for_model("gpt2", {
  "<|im_start|>": 100264,
  "<|im_end|>": 100265,
});

// don't forget to free the encoder after it is not used
enc.free();
```

--------------------------------

### Use Tiktoken in Next.js API Route

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

This example shows how to use the `tiktoken` library within a Next.js API route to encode a string into tokens and return them as a JSON response. It also demonstrates proper resource management by calling `encoding.free()`.

```typescript
import { get_encoding } from "tiktoken";
import { NextApiRequest, NextApiResponse } from "next";

export default function handler(req: NextApiRequest, res: NextApiResponse) {
  const encoding = get_encoding("cl100k_base");
  const tokens = encoding.encode("hello world");
  encoding.free();
  return res.status(200).json({ tokens });
}
```

--------------------------------

### Full Usage: Get Encoding for GPT2

Source: https://github.com/dqbd/tiktoken/blob/main/js/README.md

Illustrates the full usage of js-tiktoken by importing the `getEncoding` function to retrieve the tokenizer for a specific model like 'gpt2'. This method includes all necessary tokenizers.

```typescript
import assert from "node:assert";
import { getEncoding, encodingForModel } from "js-tiktoken";

const enc = getEncoding("gpt2");
assert(enc.decode(enc.encode("hello world")) === "hello world");
```

--------------------------------

### Lightweight tiktoken Usage (JavaScript)

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Shows how to use the lightweight WASM binary of tiktoken for constrained environments like Edge Runtimes or Cloudflare Workers. It involves importing the Tiktoken class from 'tiktoken/lite' and initializing it with pre-defined encoder data.

```javascript
const { Tiktoken } = require("tiktoken/lite");
const cl100k_base = require("tiktoken/encoders/cl100k_base.json");

const encoding = new Tiktoken(
  cl100k_base.bpe_ranks,
  cl100k_base.special_tokens,
  cl100k_base.pat_str
);
const tokens = encoding.encode("hello world");
encoding.free();
```

--------------------------------

### Load Latest tiktoken Ranks (JavaScript)

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Demonstrates how to dynamically load the latest BPE ranks and model-to-encoding mappings for tiktoken. This approach uses the `load` function and a registry to fetch data for a specified model, suitable for applications needing up-to-date tokenization configurations.

```javascript
const { Tiktoken } = require("tiktoken/lite");
const { load } = require("tiktoken/load");
const registry = require("tiktoken/registry.json");
const models = require("tiktoken/model_to_encoding.json");

async function main() {
  const model = await load(registry[models["gpt-3.5-turbo"]]);
  const encoder = new Tiktoken(
    model.bpe_ranks,
    model.special_tokens,
    model.pat_str
  );
  const tokens = encoder.encode("hello world");
  encoder.free();
}

main();
```

--------------------------------

### Cloudflare Workers Wrangler TOML Configuration

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

A TOML configuration rule for Cloudflare Workers' `wrangler.toml` file. This rule ensures that all `.wasm` files are compiled and uploaded as WebAssembly modules during the build process.

```toml
[[rules]]
globs = ["**/*.wasm"]
type = "CompiledWasm"
```

--------------------------------

### Cloudflare Workers Tiktoken Initialization

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Initializes the tiktoken encoder for Cloudflare Workers, adhering to the 1 MB WASM limit. It requires manual import of the WASM binary and a specific `wrangler.toml` configuration to upload WASM files during the build process.

```javascript
import { init, Tiktoken } from "tiktoken/lite/init";
import wasm from "./node_modules/tiktoken/lite/tiktoken_bg.wasm";
import model from "tiktoken/encoders/cl100k_base.json";

export default {
  async fetch() {
    await init((imports) => WebAssembly.instantiate(wasm, imports));
    const encoder = new Tiktoken(
      model.bpe_ranks,
      model.special_tokens,
      model.pat_str
    );
    const tokens = encoder.encode("test");
    encoder.free();
    return new Response(`${tokens}`);
  },
};
```

--------------------------------

### Electron Forge Webpack Configuration for WASM

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Configures Webpack for Electron Forge to copy the tiktoken WASM binary into the application package. This ensures the WASM file is available at runtime for the Electron main process.

```javascript
const CopyPlugin = require("copy-webpack-plugin");

module.exports = {
  // ...
  plugins: [
    new CopyPlugin({
      patterns: [
        { from: "./node_modules/tiktoken/tiktoken_bg.wasm" },
      ],
    }),
  ],
};
```

--------------------------------

### Vercel Edge Runtime Tiktoken Initialization

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Initializes the tiktoken encoder for use within the Vercel Edge Runtime. It imports the WASM module with a '?module' suffix and a JSON model file, then instantiates the Tiktoken class to encode text.

```typescript
// @ts-expect-error
import wasm from "tiktoken/lite/tiktoken_bg.wasm?module";
import model from "tiktoken/encoders/cl100k_base.json";
import { init, Tiktoken } from "tiktoken/lite/init";

export const config = { runtime: "edge" };

export default async function (req: Request) {
  await init((imports) => WebAssembly.instantiate(wasm, imports));

  const encoding = new Tiktoken(
    model.bpe_ranks,
    model.special_tokens,
    model.pat_str
  );

  const tokens = encoding.encode("hello world");
  encoding.free();

  return new Response(`${tokens}`);
}
```

--------------------------------

### Lite Tokenizer with Local Ranks

Source: https://github.com/dqbd/tiktoken/blob/main/js/README.md

Demonstrates the 'lite' usage of js-tiktoken by importing ranks locally. This approach minimizes bundle size by only including required encoding data.

```typescript
import { Tiktoken } from "js-tiktoken/lite";
import o200k_base from "js-tiktoken/ranks/o200k_base";

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
```

--------------------------------

### Configure Vite for WASM

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

This snippet shows how to configure Vite to support WASM modules by adding `vite-plugin-wasm` and `vite-plugin-top-level-await` to your `vite.config.js` file.

```javascript
import wasm from "vite-plugin-wasm";
import topLevelAwait from "vite-plugin-top-level-await";
import { defineConfig } from "vite";

export default defineConfig({
  plugins: [wasm(), topLevelAwait()],
});
```

--------------------------------

### Create tiktoken with Custom Configuration (TypeScript)

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Illustrates creating a tiktoken encoder instance directly with custom BPE ranks, special tokens, and a regex pattern. This method reads rank data from a file and allows for fine-grained control over the tokenization process.

```typescript
import { Tiktoken } from "../pkg";
import { readFileSync } from "fs";

const encoder = new Tiktoken(
  readFileSync("./ranks/gpt2.tiktoken").toString("utf-8"),
  { "<|endoftext|>": 50256, "<|im_start|>": 100264, "<|im_end|>": 100265 },
  "'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"
);

```

--------------------------------

### Configure Create React App for WASM with Craco

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

This configuration for `craco.config.js` enables WASM support in Create React App by modifying the Webpack configuration. It sets `asyncWebAssembly` and `layers` experiments and excludes WASM files from static file serving to allow Webpack to handle them.

```javascript
module.exports = {
  webpack: {
    configure: (config) => {
      config.experiments = {
        asyncWebAssembly: true,
        layers: true,
      };

      // turn off static file serving of WASM files
      // we need to let Webpack handle WASM import
      config.module.rules
        .find((i) => "oneOf" in i)
        .oneOf.find((i) => i.type === "asset/resource")
        .exclude.push(/\.wasm$/);

      return config;
    },
  },
};
```

--------------------------------

### Configure Next.js for WASM

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

This configuration for `next.config.js` enables WASM support in Next.js by setting `asyncWebAssembly` and `layers` in webpack experiments. It's applicable for both API routes and pages.

```typescript
// next.config.json
const config = {
  webpack(config, { isServer, dev }) {
    config.experiments = {
      asyncWebAssembly: true,
      layers: true,
    };

    return config;
  },
};
```

--------------------------------

### Custom WASM Initialization for tiktoken (TypeScript)

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

Provides a method to override the default WASM initialization logic in tiktoken, which is useful for bundlers that do not support WASM ESM integration. It involves fetching the WASM binary and providing a custom instantiation function.

```typescript
import { get_encoding, init } from "tiktoken/init";

async function main() {
  const wasm = "..."; // fetch the WASM binary somehow
  await init((imports) => WebAssembly.instantiate(wasm, imports));

  const encoding = get_encoding("cl100k_base");
  const tokens = encoding.encode("hello world");
  encoding.free();
}

main();
```

--------------------------------

### Lite Tokenizer with CDN Ranks

Source: https://github.com/dqbd/tiktoken/blob/main/js/README.md

Shows how to use the 'lite' version of js-tiktoken by dynamically loading encoding ranks from a CDN. This offers flexibility in managing dependencies.

```typescript
import { Tiktoken } from "js-tiktoken/lite";

const res = await fetch(`https://tiktoken.pages.dev/js/o200k_base.json`);
const o200k_base = await res.json();

const enc = new Tiktoken(o200k_base);
assert(enc.decode(enc.encode("hello world")) === "hello world");
```

--------------------------------

### Use Tiktoken in Next.js Page

Source: https://github.com/dqbd/tiktoken/blob/main/README.md

This React component demonstrates how to use the `tiktoken` library to encode text into tokens within a Next.js application. It includes state management for user input and displays the resulting tokens.

```typescript
import { get_encoding } from "tiktoken";
import { useState } from "react";

const encoding = get_encoding("cl100k_base");

export default function Home() {
  const [input, setInput] = useState("hello world");
  const tokens = encoding.encode(input);

  return (
    <div>
      <input
        type="text"
        value={input}
        onChange={(e) => setInput(e.target.value)}
      />
      <div>{tokens.toString()}</div>
    </div>
  );
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.