### Install FastBertTokenizer via CLI

Source: https://github.com/georg-jung/fastberttokenizer/blob/master/README.md

Commands to initialize a new .NET console project and add the required NuGet package.

```bash
dotnet new console
dotnet add package FastBertTokenizer
```

--------------------------------

### Install FastBertTokenizer via NuGet

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Use the .NET CLI to add the package to your project.

```bash
dotnet add package FastBertTokenizer
```

--------------------------------

### Tokenize text with BertTokenizer

Source: https://github.com/georg-jung/fastberttokenizer/blob/master/README.md

Example showing how to load a model from HuggingFace, encode text into IDs, and decode them back to strings.

```csharp
using FastBertTokenizer;

var tok = new BertTokenizer();
await tok.LoadFromHuggingFaceAsync("bert-base-uncased");
var (inputIds, attentionMask, tokenTypeIds) = tok.Encode("Lorem ipsum dolor sit amet.");
Console.WriteLine(string.Join(", ", inputIds.ToArray()));
var decoded = tok.Decode(inputIds.Span);
Console.WriteLine(decoded);

// Output:
// 101, 19544, 2213, 12997, 17421, 2079, 10626, 4133, 2572, 3388, 1012, 102
// [CLS] lorem ipsum dolor sit amet. [SEP]
```

--------------------------------

### Initialize BertTokenizer

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Create a new instance of the tokenizer. A vocabulary must be loaded before encoding text.

```csharp
using FastBertTokenizer;

// Create a new tokenizer instance
var tokenizer = new BertTokenizer();

// The tokenizer is ready for vocabulary loading
// but cannot encode text until a vocabulary is loaded
```

--------------------------------

### Load Vocabulary from HuggingFace

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Download and load tokenizer configurations directly from the HuggingFace Hub. Requires .NET Core 3.0 or higher.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();

// Load the bert-base-uncased tokenizer from HuggingFace
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Load other popular models
await tokenizer.LoadFromHuggingFaceAsync("bert-base-cased");
await tokenizer.LoadFromHuggingFaceAsync("bert-base-multilingual-cased");
await tokenizer.LoadFromHuggingFaceAsync("WhereIsAI/UAE-Large-V1");

// Now the tokenizer is ready to encode text
var (inputIds, attentionMask, tokenTypeIds) = tokenizer.Encode("Hello, world!");
Console.WriteLine(string.Join(", ", inputIds.ToArray()));
// Output: 101, 7592, 1010, 2088, 999, 102
```

--------------------------------

### LoadFromHuggingFaceAsync

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Downloads and initializes a tokenizer configuration directly from the HuggingFace Hub.

```APIDOC
## LoadFromHuggingFaceAsync

### Description
Downloads and loads a tokenizer configuration directly from HuggingFace Hub. This is the easiest way to initialize a tokenizer with a pre-trained model's vocabulary.

### Parameters
#### Request Body
- **modelName** (string) - Required - The name of the model on HuggingFace Hub (e.g., "bert-base-uncased").

### Request Example
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");
```

--------------------------------

### Benchmark environment details

Source: https://github.com/georg-jung/fastberttokenizer/blob/master/README.md

System information used for performance benchmarking of the library.

```text
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3527/23H2/2023Update/SunValley3)
AMD Ryzen 7 PRO 4750U with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK 8.0.204
```

--------------------------------

### Perform Semantic Search with ONNX Runtime

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Demonstrates encoding text and running inference using an ONNX model to generate embeddings. Requires Microsoft.ML.OnnxRuntime for model execution.

```csharp
using FastBertTokenizer;
using Microsoft.ML.OnnxRuntime;

// Initialize tokenizer
var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("WhereIsAI/UAE-Large-V1");

// Load embedding model
using var session = new InferenceSession("model.onnx");

// Encode a query
string query = "What is machine learning?";
var (inputIds, attentionMask, tokenTypeIds) = tokenizer.Encode(query, maximumTokens: 512, padTo: 512);

// Prepare model inputs
using var inputIdsTensor = OrtValue.CreateTensorValueFromMemory(
    OrtMemoryInfo.DefaultInstance, inputIds, [1, 512]);
using var attentionTensor = OrtValue.CreateTensorValueFromMemory(
    OrtMemoryInfo.DefaultInstance, attentionMask, [1, 512]);
using var tokenTypeTensor = OrtValue.CreateTensorValueFromMemory(
    new long[512], [1, 512]);
using var outputTensor = OrtValue.CreateTensorValueFromMemory(
    new float[512 * 1024], [1, 512, 1024]);

// Run inference
await session.RunAsync(
    new RunOptions(),
    ["input_ids", "attention_mask", "token_type_ids"],
    [inputIdsTensor, attentionTensor, tokenTypeTensor],
    ["last_hidden_state"],
    [outputTensor]);

// Extract embedding (first 1024 floats = CLS token embedding)
var embedding = outputTensor.GetTensorDataAsSpan<float>().Slice(0, 1024).ToArray();
Console.WriteLine($"Generated embedding with {embedding.Length} dimensions");

// Decode tokens back to verify
string decoded = tokenizer.Decode(inputIds.Span);
Console.WriteLine($"Query tokens: {decoded}");
```

--------------------------------

### Load Tokenizer Configuration from JSON

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Load a tokenizer configuration from a local tokenizer.json file or stream.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();

// Load from a file path
await tokenizer.LoadTokenizerJsonAsync("/path/to/tokenizer.json");

// Or load from a stream
using var stream = File.OpenRead("/path/to/tokenizer.json");
await tokenizer.LoadTokenizerJsonAsync(stream);

// Synchronous loading is also available
using var syncStream = File.OpenRead("/path/to/tokenizer.json");
tokenizer.LoadTokenizerJson(syncStream);

// Tokenizer is now ready for encoding
var result = tokenizer.Encode("Example text to tokenize");
Console.WriteLine($"Token count: {result.InputIds.Length}");
```

--------------------------------

### Load Custom Vocabulary from Text File

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Load a vocabulary from a vocab.txt file, specifying case sensitivity and special tokens.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();

// Load vocabulary for an uncased model (converts input to lowercase)
await tokenizer.LoadVocabularyAsync(
    vocabTxtFilePath: "/path/to/vocab.txt",
    convertInputToLowercase: true,  // Use true for uncased models
    unknownToken: "[UNK]",
    clsToken: "[CLS]",
    sepToken: "[SEP]",
    padToken: "[PAD]"
);

// Load vocabulary for a cased model (preserves case)
var casedTokenizer = new BertTokenizer();
await casedTokenizer.LoadVocabularyAsync(
    vocabTxtFilePath: "/path/to/cased-vocab.txt",
    convertInputToLowercase: false  // Use false for cased models
);

// Load from a TextReader
using var reader = new StreamReader("/path/to/vocab.txt");
var anotherTokenizer = new BertTokenizer();
await anotherTokenizer.LoadVocabularyAsync(reader, convertInputToLowercase: true);
```

--------------------------------

### Process Async Batches with ChannelReader

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Utilizes a bounded channel for producer-consumer scenarios to achieve high-throughput parallel tokenization. Requires the experimental FBERTTOK001 warning suppression.

```csharp
using System.Threading.Channels;
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Create a bounded channel for backpressure
var channel = Channel.CreateBounded<(int Key, string Content)>(100);

// Producer task
var producerTask = Task.Run(async () =>
{
    for (int i = 0; i < 1000; i++)
    {
        await channel.Writer.WriteAsync((i, $"Article {i} content goes here..."));
    }
    channel.Writer.Complete();
});

#pragma warning disable FBERTTOK001
// Parallel batch consumer with configurable parallelism
await foreach (var batch in tokenizer.CreateAsyncBatchEnumerator(
    channel.Reader,
    tokensPerInput: 512,
    batchSize: 8,
    stride: 25,
    maxDegreeOfParallelism: Environment.ProcessorCount))
{
    // High-throughput batch processing
    // Tokenization happens in parallel across multiple threads

    var validItems = batch.OutputCorrelation.Span
        .ToArray()
        .Count(x => x.HasValue);

    Console.WriteLine($"Processed batch with {validItems} valid items");

    // Feed to model for inference...
}
#pragma warning restore FBERTTOK001

await producerTask;
```

--------------------------------

### LoadVocabularyAsync

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Loads a vocabulary from a vocab.txt file for custom tokenizer configurations.

```APIDOC
## LoadVocabularyAsync

### Description
Loads a vocabulary from a vocab.txt file where each line represents a token. Allows configuration of case sensitivity and special tokens.

### Parameters
#### Request Body
- **vocabTxtFilePath** (string) - Required - Path to the vocab.txt file.
- **convertInputToLowercase** (bool) - Required - Whether to convert input to lowercase.
- **unknownToken** (string) - Optional - The token used for unknown characters.
- **clsToken** (string) - Optional - The classification token.
- **sepToken** (string) - Optional - The separator token.
- **padToken** (string) - Optional - The padding token.

### Request Example
await tokenizer.LoadVocabularyAsync("/path/to/vocab.txt", true, "[UNK]", "[CLS]", "[SEP]", "[PAD]");
```

--------------------------------

### CreateAsyncBatchEnumerator for Batch Tokenization

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Tokenizes content from an async source in batches, handling long documents with configurable stride for overlap. Ideal for processing large datasets or streaming content. Requires experimental API usage.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Simulate async content source (e.g., from database or API)
async IAsyncEnumerable<(int Key, string Content)> GetArticlesAsync()
{
    yield return (1, "First article with some content that might be long.");
    yield return (2, "Second article discussing various topics in detail.");
    yield return (3, "Third article about natural language processing and transformers.");
    await Task.CompletedTask;
}

// Process in batches with stride for overlapping context
const int tokensPerInput = 512;  // Max tokens per model input
const int batchSize = 4;         // Inputs per batch
const int stride = 50;           // Overlap tokens when splitting long content

#pragma warning disable FBERTTOK001  // Experimental API
await foreach (var batch in tokenizer.CreateAsyncBatchEnumerator(
    GetArticlesAsync(),
    tokensPerInput,
    batchSize,
    stride))
{
    Console.WriteLine($"Batch with {batchSize} inputs:");

    // Access tokenized data
    var inputIds = batch.InputIds;       // Shape: [batchSize * tokensPerInput]
    var attentionMask = batch.AttentionMask;
    var correlation = batch.OutputCorrelation;  // Maps outputs to source keys

    // Process each item in batch
    for (int i = 0; i < batchSize; i++)
    {
        var range = correlation.Span[i];
        if (range.HasValue)
        {
            Console.WriteLine($"  Item {i}: Key={range.Value.Key}, " +
                $"Offset={range.Value.Offset}, " +
                $"More={range.Value.LastTokenizedWordStartIndex.HasValue}");
        }
    }

    // Batch data is ready for model inference
    // Pass inputIds and attentionMask to your ONNX/ML model
}
#pragma warning restore FBERTTOK001
```

--------------------------------

### CreateBatchEnumerator for Synchronous Batch Tokenization

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Creates a synchronous enumerable for batch tokenization from in-memory collections. Handles splitting documents with configurable stride for overlap. Requires experimental API usage.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Source data with keys for correlation
var documents = new List<(string Id, string Text)>
{
    ("doc1", "Short document."),
    ("doc2", "A much longer document that contains multiple sentences. " +
             "It might need to be split across multiple tokenized inputs."),
    ("doc3", "Another document for processing.")
};

#pragma warning disable FBERTTOK001
foreach (var batch in tokenizer.CreateBatchEnumerator(
    documents.Select(d => (d.Id, d.Text)),
    tokensPerInput: 128,
    batchSize: 2,
    stride: 20))
{
    // Process batch synchronously
    for (int i = 0; i < 2; i++)
    {
        var range = batch.OutputCorrelation.Span[i];
        if (range.HasValue)
        {
            var start = i * 128;
            var realTokens = batch.AttentionMask.Span
                .Slice(start, 128)
                .ToArray()
                .Count(x => x == 1);

            Console.WriteLine($"Document '{range.Value.Key}': {realTokens} tokens, " +
                $"continues: {range.Value.LastTokenizedWordStartIndex.HasValue}");
        }
    }
}
#pragma warning restore FBERTTOK001
```

--------------------------------

### LoadTokenizerJsonAsync

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Loads a tokenizer configuration from a local tokenizer.json file in the HuggingFace format.

```APIDOC
## LoadTokenizerJsonAsync

### Description
Loads a tokenizer configuration from a local tokenizer.json file. Supports version 1.0 of the tokenizer.json format with WordPiece model type.

### Parameters
#### Request Body
- **pathOrStream** (string/Stream) - Required - The file path or stream containing the tokenizer.json data.

### Request Example
await tokenizer.LoadTokenizerJsonAsync("/path/to/tokenizer.json");
```

--------------------------------

### Encode Batch of Strings in Parallel

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Encodes multiple input strings in parallel using multiple threads, writing results to contiguous memory arrays. This is ideal for batch processing with AI models that support batched inputs, providing results in a [batch_size, sequence_length] layout.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Prepare batch of inputs
string[] inputs = {
    "The weather is beautiful today.",
    "Machine learning is transforming industries.",
    "Natural language processing enables text understanding.",
    "BERT models require tokenized input.",
    "FastBertTokenizer provides high performance."
};

const int batchSize = 5;
const int maxTokens = 512;
int totalElements = batchSize * maxTokens;

// Pre-allocate contiguous memory for batch
var inputIds = new long[totalElements];
var attentionMask = new long[totalElements];
var tokenTypeIds = new long[totalElements];

// Parallel encoding of entire batch
tokenizer.Encode(
    inputs: inputs.AsMemory(),
    inputIds: inputIds.AsMemory(),
    attentionMask: attentionMask.AsMemory(),
    tokenTypeIds: tokenTypeIds.AsMemory(),
    maximumTokens: maxTokens
);

// Access results for each input
for (int i = 0; i < batchSize; i++)
{
    int start = i * maxTokens;
    var ids = inputIds.AsSpan(start, maxTokens);
    var mask = attentionMask.AsSpan(start, maxTokens);

    // Count actual tokens (attention_mask = 1)
    int actualTokens = 0;
    for (int j = 0; j < maxTokens && mask[j] == 1; j++)
        actualTokens++;

    Console.WriteLine($"Input {i}: {actualTokens} tokens");
}

// Results can be passed directly to ONNX Runtime or other inference engines
// The memory layout is [batch_size, sequence_length] = [5, 512]
```

--------------------------------

### Encode Single String to Memory Buffers

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Encodes a single string into input_ids, attention_mask, and token_type_ids using memory buffers. The internal buffers are reused, so copy data if needed. Supports specifying maximum tokens and padding.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Basic encoding with default 512 max tokens
var (inputIds, attentionMask, tokenTypeIds) = tokenizer.Encode("Lorem ipsum dolor sit amet.");

Console.WriteLine("Input IDs: " + string.Join(", ", inputIds.ToArray()));
// Output: 101, 19544, 2213, 12997, 17421, 2079, 10626, 4133, 2572, 3388, 1012, 102

Console.WriteLine("Attention Mask: " + string.Join(", ", attentionMask.ToArray()));
// Output: 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1

Console.WriteLine("Token Type IDs: " + string.Join(", ", tokenTypeIds.ToArray()));
// Output: 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0

// Encoding with specific maximum tokens and padding
var result = tokenizer.Encode(
    input: "Short text",
    maximumTokens: 128,
    padTo: 128  // Pad output to exactly 128 tokens
);

Console.WriteLine($"Padded length: {result.InputIds.Length}");
// Output: Padded length: 128

// The attention mask shows which tokens are real (1) vs padding (0)
var realTokenCount = result.AttentionMask.Span.ToArray().Count(x => x == 1);
Console.WriteLine($"Real tokens: {realTokenCount}, Padding tokens: {128 - realTokenCount}");
```

--------------------------------

### Decode Token IDs to Text

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Converts token IDs back to human-readable text, handling WordPiece continuation tokens and common artifacts. Load the tokenizer and then use the Decode method with either encoded input IDs or specific token IDs.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Encode then decode
var (inputIds, _, _) = tokenizer.Encode("Hello, how are you doing today?");
string decoded = tokenizer.Decode(inputIds.Span);

Console.WriteLine($"Decoded: {decoded}");
// Output: [CLS] hello, how are you doing today? [SEP]

// Decode specific token IDs
ReadOnlySpan<long> tokenIds = new long[] { 101, 7592, 1010, 2088, 999, 102 };
string text = tokenizer.Decode(tokenIds);
Console.WriteLine($"Decoded IDs: {text}");
// Output: [CLS] hello, world! [SEP]

// Decode handles contractions properly
var (contractIds, _, _) = tokenizer.Encode("I can't believe it's working!");
string contractDecoded = tokenizer.Decode(contractIds.Span);
Console.WriteLine($"Contractions: {contractDecoded}");
// Output: [CLS] i can't believe it's working! [SEP]
```

--------------------------------

### Encode Single String to Span Buffers

Source: https://context7.com/georg-jung/fastberttokenizer/llms.txt

Encodes a single string directly into pre-allocated Span buffers for maximum efficiency, especially when encoding multiple inputs. Supports encoding with or without token type IDs and padding.

```csharp
using FastBertTokenizer;

var tokenizer = new BertTokenizer();
await tokenizer.LoadFromHuggingFaceAsync("bert-base-uncased");

// Pre-allocate buffers for maximum efficiency
const int maxTokens = 512;
Span<long> inputIds = stackalloc long[maxTokens];
Span<long> attentionMask = stackalloc long[maxTokens];
Span<long> tokenTypeIds = stackalloc long[maxTokens];

// Encode with all three outputs
int tokenCount = tokenizer.Encode(
    input: "The quick brown fox jumps over the lazy dog.",
    inputIds: inputIds,
    attentionMask: attentionMask,
    tokenTypeIds: tokenTypeIds,
    padTo: maxTokens
);

Console.WriteLine($"Produced {tokenCount} tokens (including padding)");

// Encode without token type IDs for better performance
int tokenCount2 = tokenizer.Encode(
    input: "Another sentence to encode",
    inputIds: inputIds,
    attentionMask: attentionMask,
    padTo: 128
);

// Efficient batch processing with memory reuse
var sentences = new[] {
    "First sentence.",
    "Second sentence.",
    "Third sentence."
};

foreach (var sentence in sentences)
{
    int count = tokenizer.Encode(sentence, inputIds, attentionMask);
    Console.WriteLine($"'{sentence}' -> {count} tokens");
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.