FastBertTokenizer

https://georg-jung.github.io/FastBertTokenizer/

Admin

FastBertTokenizer is a fast and memory-efficient library for WordPiece tokenization as used by BERT,...

Tokens:827

Snippets:3

Trust Score:9.7

Update:6 months ago

Show doc for...

Tokens:

Raw

### Install FastBertTokenizer

Source: https://georg-jung.github.io/FastBertTokenizer

Add the FastBertTokenizer package to your .NET console application.

```bash
dotnet new console
dotnet add package FastBertTokenizer

```

--------------------------------

### BenchmarkDotNet Configuration

Source: https://georg-jung.github.io/FastBertTokenizer

Example of BenchmarkDotNet version and environment details used for performance testing. This is for informational purposes to understand benchmark conditions.

```text
BenchmarkDotNet v0.13.12, Windows 11 (10.0.22631.3527/23H2/2023Update/SunValley3)
AMD Ryzen 7 PRO 4750U with Radeon Graphics, 1 CPU, 16 logical and 8 physical cores
.NET SDK 8.0.204

```

--------------------------------

### Basic Tokenization with FastBertTokenizer

Source: https://georg-jung.github.io/FastBertTokenizer

Instantiate BertTokenizer, load a pre-trained model from HuggingFace, encode text into IDs, and decode IDs back to text. Ensure the model name is correct and available.

```csharp
using FastBertTokenizer;

var tok = new BertTokenizer();
await tok.LoadFromHuggingFaceAsync("bert-base-uncased");
var (inputIds, attentionMask, tokenTypeIds) = tok.Encode("Lorem ipsum dolor sit amet.");
Console.WriteLine(string.Join(", ", inputIds.ToArray()));
var decoded = tok.Decode(inputIds.Span);
Console.WriteLine(decoded);

// Output:
// 101, 19544, 2213, 12997, 17421, 2079, 10626, 4133, 2572, 3388, 1012, 102
// [CLS] lorem ipsum dolor sit amet. [SEP]

```

Tokens:

Raw