### Install SharpToken via NuGet

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Commands to install the SharpToken package using the NuGet Package Manager or .NET CLI.

```powershell
# Using NuGet Package Manager
Install-Package SharpToken

# Using .NET CLI
dotnet add package SharpToken
```

--------------------------------

### Install SharpToken via .NET CLI

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Use the .NET CLI command to add the SharpToken package to your project.

```bash
dotnet add package SharpToken
```

--------------------------------

### Install SharpToken via NuGet Package Manager

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Use the NuGet Package Manager in Visual Studio to add the SharpToken library to your project.

```powershell
Install-Package SharpToken
```

--------------------------------

### Get Encoding by Name

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Instantiate GptEncoding by providing the specific encoding name, such as 'cl100k_base'.

```csharp
// Get encoding by encoding name
var encoding = GptEncoding.GetEncoding("cl100k_base");
```

--------------------------------

### Get Encoding Instances for Supported Models

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Demonstrates how to obtain GptEncoding instances for various supported model types.

```csharp
var r50kBaseEncoding = GptEncoding.GetEncoding("r50k_base");
var p50kBaseEncoding = GptEncoding.GetEncoding("p50k_base");
var p50kEditEncoding = GptEncoding.GetEncoding("p50k_edit");
var cl100kBaseEncoding = GptEncoding.GetEncoding("cl100k_base");
var o200kBaseEncoding = GptEncoding.GetEncoding("o200k_base");
var o200kHarmonyEncoding = GptEncoding.GetEncoding("o200k_harmony");
var claudeEncoding = GptEncoding.GetEncoding("claude");
```

--------------------------------

### Get Encoding for Model

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Instantiate GptEncoding by specifying the GPT model name, like 'gpt-4'.

```csharp
// Get encoding by model name
var encoding = GptEncoding.GetEncodingForModel("gpt-4");
```

--------------------------------

### Claude Model Tokenization Example

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Use 'claude' encoding for Anthropic's models. Note: This is accurate for pre-Claude 3 models and an approximation for Claude 3+.

```csharp
var encoding = GptEncoding.GetEncodingForModel("claude-3.5-sonnet");
var count = encoding.CountTokens("Hello, Claude!");
```

--------------------------------

### Get O200K Harmony Encoding

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Instantiate the O200K Harmony encoding, which extends O200K base with additional special tokens for advanced use cases like function calling.

```csharp
using SharpToken;

var harmonyEncoding = GptEncoding.GetEncoding("o200k_harmony");
```

--------------------------------

### Benchmark Code for Tokenizer Libraries

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

This C# code sets up a BenchmarkDotNet benchmark to compare the performance of SharpToken, TiktokenSharp, TokenizerLib, and MLTokenizers. It includes setup for different .NET runtimes and memory diagnostics.

```csharp
[SimpleJob(RuntimeMoniker.Net60)]
[SimpleJob(RuntimeMoniker.Net80)]
[SimpleJob(RuntimeMoniker.Net471)]
[RPlotExporter]
[MemoryDiagnoser]
public class CompareBenchmark
{
    private GptEncoding _sharpToken;
    private TikToken _tikToken;
    private ITokenizer _tokenizer;
    private Tokenizer _mlTokenizer;
    private string _kLongText;

    [GlobalSetup]
    public async Task Setup()
    {
        _sharpToken = GptEncoding.GetEncoding("cl100k_base");
        _tikToken = await TikToken.GetEncodingAsync("cl100k_base").ConfigureAwait(false);
        _tokenizer = await TokenizerBuilder.CreateByModelNameAsync("gpt-4").ConfigureAwait(false);
        _kLongText = "King Lear, one of Shakespeare's darkest and most savage plays, tells the story of the foolish and Job-like Lear, who divides his kingdom, as he does his affections, according to vanity and whim. Lear’s failure as a father engulfs himself and his world in turmoil and tragedy.";
    }

    [Benchmark]
    public int SharpToken()
    {
        var sum = 0;
        for (var i = 0; i < 10000; i++)
        {
            var encoded = _sharpToken.Encode(_kLongText);
            var decoded = _sharpToken.Decode(encoded);
            sum += decoded.Length;
        }

        return sum;
    }

    [Benchmark]
    public int TiktokenSharp()
    {
        var sum = 0;
        for (var i = 0; i < 10000; i++)
        {
            var encoded = _tikToken.Encode(_kLongText);
            var decoded = _tikToken.Decode(encoded);
            sum += decoded.Length;
        }

        return sum;
    }

    [Benchmark]
    public int TokenizerLib()
    {
        var sum = 0;
        for (var i = 0; i < 10000; i++)
        {
            var encoded = _tokenizer.Encode(_kLongText);
            var decoded = _tokenizer.Decode(encoded.ToArray());
            sum += decoded.Length;
        }

        return sum;
    }

    [Benchmark]
    public int MLTokenizers()
    {
        var sum = 0;
        for (var i = 0; i < 10000; i++)
        {
            var encoded = _mlTokenizer.EncodeToIds(_kLongText);
            var decoded = _mlTokenizer.Decode(encoded);
            sum += decoded.Length;
        }

        return sum;
    }
}
```

--------------------------------

### Parallel Text Encoding, Decoding, and Counting

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Use this snippet to process multiple texts concurrently. Ensure the encoding instance is created once and shared across threads. This example demonstrates parallel encoding, decoding, and token counting for batch operations.

```csharp
using SharpToken;

// Create encoding instance once (thread-safe)
var encoding = GptEncoding.GetEncoding("cl100k_base");

// Sample texts to process in parallel
var texts = new[]
{
    "First document to encode.",
    "Second document with different content.",
    "Third document for testing parallel processing.",
    "Fourth document to complete the batch."
};

// Process texts in parallel
var tasks = texts.Select(text => Task.Run(() =>
{
    var encoded = encoding.Encode(text);
    var decoded = encoding.Decode(encoded);
    return new { Original = text, Tokens = encoded, Decoded = decoded };
}));

var results = await Task.WhenAll(tasks);

foreach (var result in results)
{
    Console.WriteLine($"Original: {result.Original}");
    Console.WriteLine($"Token count: {result.Tokens.Count}");
    Console.WriteLine($"Verified: {result.Original == result.Decoded}");
    Console.WriteLine();
}

// Parallel token counting for batch operations
var tokenCounts = await Task.WhenAll(
    texts.Select(text => Task.Run(() => encoding.CountTokens(text)))
);

Console.WriteLine($"Total tokens across all documents: {tokenCounts.Sum()}");
```

--------------------------------

### Get Encoding Name for Model

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Use this static method to get the encoding name for a given model without instantiating an encoding object. It supports exact model names and prefix matching for versioned models.

```csharp
using SharpToken;

// Get encoding name without creating an encoding instance
string cl100kName = Model.GetEncodingNameForModel("gpt-4");           // Returns "cl100k_base"
string o200kName = Model.GetEncodingNameForModel("gpt-4o");           // Returns "o200k_base"
string claudeName = Model.GetEncodingNameForModel("claude-3.5-sonnet"); // Returns "claude"

// Prefix matching for versioned models
string gpt4Prefix = Model.GetEncodingNameForModel("gpt-4-0314");      // Returns "cl100k_base"
string gpt5Prefix = Model.GetEncodingNameForModel("gpt-5-thinking");  // Returns "o200k_base"
string claudePrefix = Model.GetEncodingNameForModel("claude-3-opus-20240229"); // Returns "claude"

Console.WriteLine($"GPT-4 uses: {Model.GetEncodingNameForModel("gpt-4")}");
Console.WriteLine($"GPT-4o uses: {Model.GetEncodingNameForModel("gpt-4o")}");
Console.WriteLine($"GPT-5 uses: {Model.GetEncodingNameForModel("gpt-5")}");
Console.WriteLine($"Claude 3.5 uses: {Model.GetEncodingNameForModel("claude-3.5-sonnet")}");

// Throws exception for unknown models
try
{
    Model.GetEncodingNameForModel("unknown-model");
}
catch (Exception ex)
{
    Console.WriteLine($"Error: {ex.Message}");
    // Output: Error: Could not automatically map unknown-model to a tokenizer...
}
```

--------------------------------

### Get Claude Encoding

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Obtain the Claude encoding directly or by specifying a model name. Supports various Claude versions including Claude 3 and Claude 4.

```csharp
using SharpToken;

// Get Claude encoding directly
var claudeEncoding = GptEncoding.GetEncoding("claude");

// Get Claude encoding by model name
var claude3Opus = GptEncoding.GetEncodingForModel("claude-3-opus");
var claude35Sonnet = GptEncoding.GetEncodingForModel("claude-3.5-sonnet");
var claude37Sonnet = GptEncoding.GetEncodingForModel("claude-3.7-sonnet");
var claude4Sonnet = GptEncoding.GetEncodingForModel("claude-4-sonnet");
```

--------------------------------

### Initialize GptEncoding by Name

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Obtain an encoding instance by specifying the encoding scheme name directly.

```csharp
using SharpToken;

// Get encoding by encoding name
var cl100kEncoding = GptEncoding.GetEncoding("cl100k_base");
var o200kEncoding = GptEncoding.GetEncoding("o200k_base");
var claudeEncoding = GptEncoding.GetEncoding("claude");

// Available encodings:
// - r50k_base: Used by older GPT-3 models (text-davinci-001, text-curie-001, etc.)
// - p50k_base: Used by code models and text-davinci-002/003
// - p50k_edit: Used by edit models (text-davinci-edit-001, code-davinci-edit-001)
// - cl100k_base: Used by GPT-4, GPT-3.5-turbo, and embedding models
// - o200k_base: Used by GPT-4o and GPT-5 models
// - o200k_harmony: Extended o200k_base with additional special tokens
// - claude: Used by Anthropic Claude models

var encoding = GptEncoding.GetEncoding("cl100k_base");
var text = "Hello, world!";
var tokens = encoding.Encode(text);
Console.WriteLine($"Tokens: [{string.Join(", ", tokens)}]");
// Output: Tokens: [9906, 11, 1917, 0]
```

--------------------------------

### Import SharpToken Library

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Include the SharpToken namespace in your C# code to access its functionalities.

```csharp
using SharpToken;
```

--------------------------------

### Basic Claude Encoding and Decoding

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Demonstrates fundamental encoding of text to tokens and decoding tokens back to text using the Claude encoding. Ensure NFKC normalization is considered for fullwidth characters.

```csharp
using SharpToken;

// Basic encoding and decoding
var text = "Hello, Claude!";
var tokens = claudeEncoding.Encode(text);
var decoded = claudeEncoding.Decode(tokens);
Console.WriteLine($"Text: {text}");
Console.WriteLine($"Tokens: [{string.Join(", ", tokens)}]");
Console.WriteLine($"Decoded: {decoded}");
```

```csharp
// NFKC normalization - fullwidth characters normalize to ASCII
var fullwidthHello = "Ｈｅｌｌｏ";  // Fullwidth characters
var asciiHello = "Hello";
var fullwidthTokens = claudeEncoding.Encode(fullwidthHello);
var asciiTokens = claudeEncoding.Encode(asciiHello);
Console.WriteLine($"Fullwidth and ASCII produce same tokens: {fullwidthTokens.SequenceEqual(asciiTokens)}");
// Output: Fullwidth and ASCII produce same tokens: True
```

--------------------------------

### Initialize GptEncoding by Model Name

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Automatically map a model name to its corresponding encoding, including support for versioned names and Azure deployment names.

```csharp
using SharpToken;

// Get encoding by model name - automatically maps to correct encoding
var gpt4Encoding = GptEncoding.GetEncodingForModel("gpt-4");           // Uses cl100k_base
var gpt4oEncoding = GptEncoding.GetEncodingForModel("gpt-4o");         // Uses o200k_base
var gpt5Encoding = GptEncoding.GetEncodingForModel("gpt-5");           // Uses o200k_base
var turboEncoding = GptEncoding.GetEncodingForModel("gpt-3.5-turbo");  // Uses cl100k_base
var claudeEncoding = GptEncoding.GetEncodingForModel("claude-3.5-sonnet"); // Uses claude

// Prefix matching works for versioned model names
var gpt4VersionedEncoding = GptEncoding.GetEncodingForModel("gpt-4-0314");      // Uses cl100k_base
var turboVersionedEncoding = GptEncoding.GetEncodingForModel("gpt-3.5-turbo-0301"); // Uses cl100k_base
var gpt5VersionedEncoding = GptEncoding.GetEncodingForModel("gpt-5-2024-08-07");   // Uses o200k_base
var claudeVersionedEncoding = GptEncoding.GetEncodingForModel("claude-3-opus-20240229"); // Uses claude

// Azure deployment names are also supported
var azureEncoding = GptEncoding.GetEncodingForModel("gpt-35-turbo");   // Uses cl100k_base

// Example usage
var encoding = GptEncoding.GetEncodingForModel("gpt-4");
var tokens = encoding.Encode("Hello, GPT-4!");
var decoded = encoding.Decode(tokens);
Console.WriteLine($"Original: Hello, GPT-4!");
Console.WriteLine($"Tokens: [{string.Join(", ", tokens)}]");
Console.WriteLine($"Decoded: {decoded}");
// Output:
// Original: Hello, GPT-4!
// Tokens: [9906, 11, 480, 2898, 12, 19, 0]
// Decoded: Hello, GPT-4!
```

--------------------------------

### Basic O200K Harmony Encoding and Decoding

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Perform standard text encoding and decoding using the O200K Harmony encoding. This process is identical to the base O200K encoding for regular text.

```csharp
// Basic encoding works the same as o200k_base
var text = "Hello, world!";
var tokens = harmonyEncoding.Encode(text);
var decoded = harmonyEncoding.Decode(tokens);
Console.WriteLine($"Decoded: {decoded}");
// Output: Decoded: Hello, world!
```

--------------------------------

### Encode and Decode Text

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Convert text to integer tokens and back to text using the same encoding instance.

```csharp
var encoded = cl100kBaseEncoding.Encode("Hello world!"); // Output: [9906, 1917, 0]
```

```csharp
var decoded = cl100kBaseEncoding.Decode(encoded); // Output: "Hello world!"
```

--------------------------------

### Encode String to Tokens

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Convert a string into a list of integer tokens using the specified encoding.

```csharp
var encoded = encoding.Encode("Hello, world!"); // Output: [9906, 11, 1917, 0]
```

--------------------------------

### Decode Tokens to String

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Convert a list of integer tokens back into a human-readable string.

```csharp
var decoded = encoding.Decode(encoded); // Output: "Hello, world!"
```

--------------------------------

### Token Counting for Claude

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Calculate the number of tokens a given prompt will consume using the Claude encoding. This is useful for estimating costs and managing context window limits.

```csharp
using SharpToken;

// Token counting for Claude
var prompt = "Please write a poem about the ocean.";
var tokenCount = claudeEncoding.CountTokens(prompt);
Console.WriteLine($"Claude token count: {tokenCount}");
```

--------------------------------

### Encode with Custom Allowed Special Tokens

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Pass a HashSet of allowed special tokens to the Encode method to include them in the tokenization process.

```csharp
const string encodingName = "cl100k_base";
const string inputText = "Some Text <|endofprompt|>";
var allowedSpecialTokens = new HashSet<string> { "<|endofprompt|>" };

var encoding = GptEncoding.GetEncoding(encodingName);
var encoded = encoding.Encode(inputText, allowedSpecialTokens);
var expectedEncoded = new List<int> { 8538, 2991, 220, 100276 };

Assert.Equal(expectedEncoded, encoded);
```

--------------------------------

### O200K Harmony Special Tokens

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Encode and decode text containing special tokens specific to the O200K Harmony encoding, such as <|startoftext|> and <|call|>. Ensure these tokens are included in the `allowedSpecial` set.

```csharp
// Additional special tokens available in o200k_harmony
var specialTokens = new HashSet<string> { "<|startoftext|", "<|call|", "<|reserved_200020|>" };
var textWithSpecial = "Hello <|startoftext|> world <|call|> test <|reserved_200020|>";
var encodedSpecial = harmonyEncoding.Encode(textWithSpecial, allowedSpecial: specialTokens);
var decodedSpecial = harmonyEncoding.Decode(encodedSpecial);
Console.WriteLine($"Decoded with special: {decodedSpecial}");
// Output: Decoded with special: Hello <|startoftext|> world <|call|> test <|reserved_200020|>
```

--------------------------------

### Retrieve Encoding Name by Model Prefix

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Use GetEncodingNameForModel to resolve an encoding name from a model name or its prefix. An exception is thrown if no match is found.

```csharp
string encodingName = Model.GetEncodingNameForModel("claude-3.5-sonnet"); // Returns "claude"
string encodingName = Model.GetEncodingNameForModel("gpt-4-0314");        // Returns "cl100k_base"
```

--------------------------------

### Encode String to Tokens

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Encodes a string into a list of token integers using a specified encoding. Supports optional allowed and disallowed special token sets for precise control over tokenization.

```csharp
using SharpToken;

var encoding = GptEncoding.GetEncoding("cl100k_base");

// Basic encoding
var text = "Hello, world!";
var tokens = encoding.Encode(text);
Console.WriteLine($"Text: {text}");
Console.WriteLine($"Tokens: [{string.Join(", ", tokens)}]");
// Output:
// Text: Hello, world!
// Tokens: [9906, 11, 1917, 0]

// Encoding longer text
var paragraph = "King Lear, one of Shakespeare's darkest plays.";
var paragraphTokens = encoding.Encode(paragraph);
Console.WriteLine($"Token count: {paragraphTokens.Count}");
// Output: Token count: 11

// Encoding with allowed special tokens
var textWithSpecial = "Some Text<|endofprompt|>";
var allowedSpecialTokens = new HashSet<string> { "<|endofprompt|>" };
var specialTokens = encoding.Encode(textWithSpecial, allowedSpecialTokens);
Console.WriteLine($"Tokens with special: [{string.Join(", ", specialTokens)}]");
// Output: Tokens with special: [8538, 2991, 100276]

// Allow all special tokens
var fimText = "<|fim_prefix|>lorem ipsum<|fim_suffix|>dolor sit amet<|fim_middle|>";
var allAllowed = new HashSet<string> { "all" };
var fimTokens = encoding.Encode(fimText, allAllowed);
Console.WriteLine($"FIM tokens: [{string.Join(", ", fimTokens)}]");
// Output: FIM tokens: [100258, 385, 1864, 27439, 100260, 67, 795, 2503, 28311, 100259]

// Disallowed special tokens throw exceptions
try
{
    encoding.Encode("Some Text<|endofprompt|>"); // No allowed set = special tokens disallowed
}
catch (ArgumentException ex)
{
    Console.WriteLine($"Error: Disallowed special token found");
}

// Custom disallowed set
try
{
    encoding.Encode("Some Text", disallowedSpecial: new HashSet<string> { "Some" });
}
catch (ArgumentException ex)
{
    Console.WriteLine($"Error: Custom disallowed token found");
}
```

--------------------------------

### Count tokens with GptEncoding.CountTokens

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Provides high-performance token counting without full list allocation. Useful for prompt validation and RAG chunking.

```csharp
using SharpToken;

var encoding = GptEncoding.GetEncoding("cl100k_base");

// Basic token counting
var text = "Hello, world!";
var count = encoding.CountTokens(text);
Console.WriteLine($"Text: {text}");
Console.WriteLine($"Token count: {count}");
// Output:
// Text: Hello, world!
// Token count: 4

// Token counting for prompt size validation
var prompt = "You are a helpful assistant. Please answer the following question: What is the capital of France?";
var promptTokens = encoding.CountTokens(prompt);
var maxTokens = 4096;
Console.WriteLine($"Prompt tokens: {promptTokens}");
Console.WriteLine($"Remaining tokens: {maxTokens - promptTokens}");
// Output:
// Prompt tokens: 20
// Remaining tokens: 4076

// Counting tokens with special tokens
var textWithSpecial = "Hello<|endofprompt|>World";
var allowedSpecial = new HashSet<string> { "<|endofprompt|>" };
var specialCount = encoding.CountTokens(textWithSpecial, allowedSpecial);
Console.WriteLine($"Token count with special: {specialCount}");
// Output: Token count with special: 3

// Performance comparison: CountTokens vs Encode().Count
// CountTokens is optimized for counting only - minimal heap allocations
var largeText = string.Join(" ", Enumerable.Repeat("This is a test sentence for benchmarking.", 1000));
var stopwatch = System.Diagnostics.Stopwatch.StartNew();
var countResult = encoding.CountTokens(largeText);
stopwatch.Stop();
Console.WriteLine($"CountTokens: {countResult} tokens in {stopwatch.ElapsedMilliseconds}ms");

// Use for RAG text chunking
int maxChunkTokens = 500;
string[] sentences = new[] {
    "First sentence of the document.",
    "Second sentence with more content.",
    "Third sentence to complete the example."
};
int currentChunkTokens = 0;
var currentChunk = new List<string>();

foreach (var sentence in sentences)
{
    var sentenceTokens = encoding.CountTokens(sentence);
    if (currentChunkTokens + sentenceTokens <= maxChunkTokens)
    {
        currentChunk.Add(sentence);
        currentChunkTokens += sentenceTokens;
    }
}
Console.WriteLine($"Chunk size: {currentChunkTokens} tokens");
```

--------------------------------

### Decode token integers with GptEncoding.Decode

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Decodes lists or enumerables of token integers back into strings using UTF-8. Supports standard token lists and special tokens.

```csharp
using SharpToken;

var encoding = GptEncoding.GetEncoding("cl100k_base");

// Basic decoding
var tokens = new List<int> { 9906, 11, 1917, 0 };
var decodedText = encoding.Decode(tokens);
Console.WriteLine($"Tokens: [{string.Join(", ", tokens)}]");
Console.WriteLine($"Decoded: {decodedText}");
// Output:
// Tokens: [9906, 11, 1917, 0]
// Decoded: Hello, world!

// Round-trip encoding and decoding
var originalText = "The quick brown fox jumps over the lazy dog.";
var encoded = encoding.Encode(originalText);
var decoded = encoding.Decode(encoded);
Console.WriteLine($"Original: {originalText}");
Console.WriteLine($"Encoded: [{string.Join(", ", encoded)}]");
Console.WriteLine($"Decoded: {decoded}");
Console.WriteLine($"Match: {originalText == decoded}");
// Output:
// Original: The quick brown fox jumps over the lazy dog.
// Encoded: [791, 4062, 14198, 39935, 35308, 927, 279, 16053, 5679, 13]
// Decoded: The quick brown fox jumps over the lazy dog.
// Match: True

// Decoding with special tokens
var specialTokens = new List<int> { 8538, 2991, 100276 };
var specialDecoded = encoding.Decode(specialTokens);
Console.WriteLine($"Special decoded: {specialDecoded}");
// Output: Special decoded: Some Text<|endofprompt|>

// Decoding IEnumerable<int>
IEnumerable<int> tokenEnumerable = new[] { 9906, 11, 1917, 0 };
var enumerableDecoded = encoding.Decode(tokenEnumerable);
Console.WriteLine($"Enumerable decoded: {enumerableDecoded}");
// Output: Enumerable decoded: Hello, world!
```

--------------------------------

### GptEncoding.GetEncoding

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Retrieves a GptEncoding instance by specifying the exact encoding scheme name.

```APIDOC
## GptEncoding.GetEncoding

### Description
Creates a GptEncoding instance by specifying the encoding name directly. This is the primary method for obtaining an encoding when you know the specific encoding scheme you need.

### Parameters
#### Method Parameters
- **encodingName** (string) - Required - The name of the encoding scheme (e.g., "cl100k_base", "o200k_base", "claude").

### Request Example
```csharp
var encoding = GptEncoding.GetEncoding("cl100k_base");
```

### Response
- **GptEncoding** (object) - An instance configured for the specified encoding scheme.
```

--------------------------------

### Encode with Custom Disallowed Special Tokens

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Pass a HashSet of disallowed special tokens to the Encode method to trigger an ArgumentException if they appear in the input.

```csharp
const string encodingName = "cl100k_base";
const string inputText = "Some Text";

var encoding = GptEncoding.GetEncoding(encodingName);

void TestAction()
{
    encoding.Encode(inputText, disallowedSpecial: new HashSet<string> { "Some" });
}

Assert.Throws<ArgumentException>(TestAction);
```

--------------------------------

### GptEncoding.GetEncodingForModel

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Retrieves a GptEncoding instance by specifying a model name, which the library automatically maps to the correct encoding.

```APIDOC
## GptEncoding.GetEncodingForModel

### Description
Creates a GptEncoding instance by specifying a model name. The library automatically maps model names to their appropriate encodings, including support for prefix matching for versioned model names.

### Parameters
#### Method Parameters
- **modelName** (string) - Required - The name of the model (e.g., "gpt-4", "gpt-4o", "claude-3.5-sonnet").

### Request Example
```csharp
var encoding = GptEncoding.GetEncodingForModel("gpt-4");
```

### Response
- **GptEncoding** (object) - An instance configured for the encoding scheme associated with the provided model name.
```

--------------------------------

### Count Tokens in a String

Source: https://github.com/dmitry-brazhenko/sharptoken/blob/main/README.md

Efficiently count the number of tokens a given string will produce without actual encoding.

```csharp
var count = encoding.CountTokens("Hello, world!"); // Output: 4
```

--------------------------------

### GptEncoding.Encode

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Encodes a string into a list of token integers, supporting optional allowed and disallowed special token sets.

```APIDOC
## GptEncoding.Encode

### Description
Encodes a string into a list of token integers. Supports optional allowed and disallowed special token sets for handling special tokens in the input text.

### Parameters
#### Method Parameters
- **text** (string) - Required - The input text to encode.
- **allowedSpecial** (HashSet<string>) - Optional - A set of special tokens allowed in the input.
- **disallowedSpecial** (HashSet<string>) - Optional - A set of special tokens that should trigger an exception if encountered.

### Response
- **Returns** (List<int>) - A list of token integers representing the encoded text.
- **Throws** (ArgumentException) - Thrown if a disallowed special token is encountered.
```

--------------------------------

### GptEncoding.Decode

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Decodes a list of token integers back into the original string using UTF-8 encoding.

```APIDOC
## GptEncoding.Decode

### Description
Decodes a list of token integers back into the original string. The decoding uses UTF-8 encoding to convert the byte representation back to text.

### Parameters
#### Request Body
- **tokens** (List<int> or IEnumerable<int>) - Required - The list of token integers to decode.
```

--------------------------------

### GptEncoding.CountTokens

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

High-performance method for counting tokens without allocating memory for the full token list.

```APIDOC
## GptEncoding.CountTokens

### Description
High-performance method for counting tokens without allocating memory for the full token list. Ideal for validating prompt sizes before sending to LLMs or implementing text splitters for RAG applications.

### Parameters
#### Request Body
- **text** (string) - Required - The input text to count tokens for.
- **allowedSpecial** (HashSet<string>) - Optional - A set of special tokens to be included in the count.
```

--------------------------------

### O200K Harmony Specific Special Token IDs

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Retrieve the integer token IDs for specific special tokens within the O200K Harmony encoding. This is useful for direct manipulation or verification of token values.

```csharp
// Specific special token IDs
var startoftext = harmonyEncoding.Encode("<|startoftext|>", new HashSet<string> { "<|startoftext|>" });
var call = harmonyEncoding.Encode("<|call|>", new HashSet<string> { "<|call|>" });
var reserved = harmonyEncoding.Encode("<|reserved_200020|>", new HashSet<string> { "<|reserved_200020|>" });
Console.WriteLine($"<|startoftext|> = {startoftext[0]}");  // 199998
Console.WriteLine($"<|call|> = {call[0]}");                // 200012
Console.WriteLine($"<|reserved_200020|> = {reserved[0]}"); // 200020
```

--------------------------------

### Claude Special Tokens

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

Encode specific special tokens like <EOT> for Claude models. Only tokens present in the `allowedSpecial` set will be encoded.

```csharp
using SharpToken;

// Claude special tokens
var allowedSpecial = new HashSet<string> { "<EOT>" };
var specialTokens = claudeEncoding.Encode("<EOT>", allowedSpecial);
Console.WriteLine($"<EOT> token: [{string.Join(", ", specialTokens)}]");
// Output: <EOT> token: [0]
```

--------------------------------

### Model.GetEncodingNameForModel

Source: https://context7.com/dmitry-brazhenko/sharptoken/llms.txt

A static utility method that returns the encoding name for a given model without creating an encoding instance.

```APIDOC
## Model.GetEncodingNameForModel

### Description
A static utility method that returns the encoding name for a given model without creating an encoding instance. Useful for determining which encoding scheme a model uses.

### Parameters
#### Method Parameters
- **modelName** (string) - Required - The name of the model to look up (e.g., "gpt-4", "gpt-4o").

### Response
- **Returns** (string) - The name of the encoding (e.g., "cl100k_base", "o200k_base").
- **Throws** (Exception) - Thrown if the model name cannot be mapped to a known tokenizer.
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.