### Reload Tokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Demonstrates how to reload a tokenizer for use in the pipeline. This is a common setup step before processing text.
```python
from tokenizers import Tokenizer
# START reload_tokenizer
tokenizer = Tokenizer.from_file("path/to/your/tokenizer.json")
# END reload_tokenizer
```
```rust
use tokenizers::Tokenizer;
// START pipeline_reload_tokenizer
let tokenizer = Tokenizer::from_file("path/to/your/tokenizer.json").unwrap();
// END pipeline_reload_tokenizer
```
```js
const { Tokenizer } = require("tokenizers");
// START reload_tokenizer
const tokenizer = Tokenizer.fromFile("path/to/your/tokenizer.json");
// END reload_tokenizer
```
--------------------------------
### Setup Whitespace Pre-Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Demonstrates how to set up the Whitespace pre-tokenizer, which splits text into words based on spaces and punctuation. This is a common starting point for pre-tokenization.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders
def setup_pre_tokenizer():
tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
# Then, we can replace the pre_tokenizer
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
return tokenizer
```
```rust
use tokenizers::pre_tokenizers;
fn pipeline_setup_pre_tokenizer() {
let mut tokenizer = tokenizers::Tokenizer::new(tokenizers::models::WordPiece::new(true));
// Then, we can replace the pre_tokenizer
tokenizer.pre_tokenizer = Some(pre_tokenizers::Whitespace::new());
}
```
```js
const tokenizer = new Tokenizer(new models.WordPiece());
// Then, we can replace the pre_tokenizer
tokenizer.preTokenizer = PreTokenizer.whitespace();
```
--------------------------------
### Profile Rust Example
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Build a Rust example in release mode and profile its CPU usage using samply.
```bash
cd tokenizers
cargo build --release --example my_bench
samply record ./target/release/examples/my_bench
```
--------------------------------
### Initialize Tokenizer and Trainer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/training_from_memory.mdx
Sets up a tokenizer with a Unigram model, NFKC normalization, and a ByteLevel pre-tokenizer. This is a common setup for the examples that follow.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders, trainers
tokenizer = Tokenizer(models.Unigram())
tokenizer.normalizer = normalizers.Sequence([
normalizers.NFD(),
normalizers.Lowercase(),
normalizers.StripAccents(),
normalizers.NFKC(),
])
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()
tokenizer.decoder = decoders.ByteLevel()
trainer = trainers.UnigramTrainer()
```
--------------------------------
### Install Rust
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Install Rust using the official script. This is a prerequisite for building tokenizers from source.
```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
--------------------------------
### Install tokenizers from source
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Install the tokenizers library in editable mode from the Python bindings directory. Ensure your virtual environment is activated.
```bash
pip install -e .
```
--------------------------------
### Install Tokenizers from Source
Source: https://github.com/huggingface/tokenizers/blob/main/README.md
Install the tokenizers library directly from its GitHub repository. This is useful for development or when needing the latest unreleased features.
```bash
pip install git+https://github.com/huggingface/tokenizers.git#subdirectory=bindings/python
```
--------------------------------
### Install tokenizers with npm
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Install the tokenizers library using npm for Node.js projects.
```bash
npm install tokenizers
```
--------------------------------
### Install tokenizers from source with Rust
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Instructions for installing the tokenizers library from its source code, requiring Rust and Cargo.
```bash
# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"
```
```bash
git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python
# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate
# Install `tokenizers` in the current virtual env
pip install -e .
```
--------------------------------
### Install Documentation Dependencies
Source: https://github.com/huggingface/tokenizers/blob/main/docs/README.md
Install the required Python packages for building the documentation using pip.
```python
pip install sphinx sphinx_rtd_theme setuptools_rust
```
--------------------------------
### Install Tokenizers Package
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/node/README.md
Install the latest version of the tokenizers package using npm.
```bash
npm install tokenizers@latest
```
--------------------------------
### Sequence PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Composes multiple PreTokenizers to be run in a specified order.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Sequence([pre_tokenizers.Punctuation(), pre_tokenizers.WhitespaceSplit()])
```
--------------------------------
### Basic Tokenizer Usage
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/node/README.md
Load a tokenizer from a file and encode a string. This example demonstrates how to get various encoded outputs like length, tokens, IDs, attention mask, offsets, and more.
```typescript
import { Tokenizer } from "tokenizers";
const tokenizer = await Tokenizer.fromFile("tokenizer.json");
const wpEncoded = await tokenizer.encode("Who is John?");
console.log(wpEncoded.getLength());
console.log(wpEncoded.getTokens());
console.log(wpEncoded.getIds());
console.log(wpEncoded.getAttentionMask());
console.log(wpEncoded.getOffsets());
console.log(wpEncoded.getOverflowing());
console.log(wpEncoded.getSpecialTokensMask());
console.log(wpEncoded.getTypeIds());
console.log(wpEncoded.getWordIds());
```
--------------------------------
### Initialize Tokenizer and Trainer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/tutorials/python/training_from_memory.rst
Initializes a tokenizer with a Unigram model, NFKC normalization, and a ByteLevel pre-tokenizer, along with its corresponding trainer. This setup is used for subsequent training examples.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, trainers
tokenizer = Tokenizer(models.Unigram())
tokenizer.normalizer = normalizers.Sequence([
normalizers.NFD(),
normalizers.Lowercase(),
normalizers.StripAccents(),
normalizers.NFKC(),
])
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel()
tokenizer.decoder = pre_tokenizers.ByteLevel.default_decoder()
trainer = trainers.UnigramTrainer()
```
--------------------------------
### Install Released Tokenizers
Source: https://github.com/huggingface/tokenizers/blob/main/README.md
Install the latest stable version of the tokenizers library using pip. This is the recommended method for most users.
```bash
pip install tokenizers
```
--------------------------------
### Sequence Normalizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Composes multiple normalizers that will run in the provided order. Allows for complex normalization pipelines.
```rust
Sequence::new(vec![NFKC, Lowercase])
```
--------------------------------
### CharDelimiterSplit Pre-tokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Shows the CharDelimiterSplit pre-tokenizer, which splits text based on a specified character delimiter. This example uses 'x' as the delimiter.
```rust
CharDelimiterSplit::new('x')
```
--------------------------------
### Combine Pre-Tokenizers
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Shows how to combine multiple pre-tokenizers to create a more complex splitting strategy. This example combines whitespace, punctuation, and digit splitting.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders
def combine_pre_tokenizer():
tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
tokenizer.pre_tokenizer = pre_tokenizers.Sequence([
pre_tokenizers.Whitespace(),
pre_tokenizers.Punctuation(),
pre_tokenizers.Digits(individual_digits=True),
])
return tokenizer
```
```rust
use tokenizers::pre_tokenizers;
fn pipeline_combine_pre_tokenizer() {
let mut tokenizer = tokenizers::Tokenizer::new(tokenizers::models::WordPiece::new(true));
tokenizer.pre_tokenizer = Some(pre_tokenizers::Sequence::new(vec![
pre_tokenizers::Whitespace::new(),
pre_tokenizers::Punctuation::new(),
pre_tokenizers::Digits::new(true),
]));
}
```
```js
const tokenizer = new Tokenizer(new models.WordPiece());
tokenizer.preTokenizer = PreTokenizer.sequence([
PreTokenizer.whitespace(),
PreTokenizer.punctuation(),
PreTokenizer.digits({ individualDigits: true }),
]);
```
--------------------------------
### Digits PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits numbers from any other characters in the input string.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Digits()
```
--------------------------------
### Digits PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits numbers from any other characters in the input string.
```rust
use tokenizers::pre_tokenizers::digits::Digits;
let pre_tokenizer = Digits;
```
--------------------------------
### CharDelimiterSplit PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits the input string on a specified character delimiter.
```rust
use tokenizers::pre_tokenizers::char_delimiter_split::CharDelimiterSplit;
let pre_tokenizer = CharDelimiterSplit::new('x');
```
--------------------------------
### Whitespace PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on word boundaries using the regex '\w+|[^\w\s]+'.
```rust
use tokenizers::pre_tokenizers::whitespace::Whitespace;
let pre_tokenizer = Whitespace::new();
```
--------------------------------
### Punctuation PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Isolates all punctuation characters from the input string.
```rust
use tokenizers::pre_tokenizers::punctuation::Punctuation;
let pre_tokenizer = Punctuation;
```
--------------------------------
### ByteLevel PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on whitespaces and remaps bytes to visible characters. Useful for reducing alphabet size to 256 and ensuring no unknown tokens.
```rust
use tokenizers::pre_tokenizers::byte_level::ByteLevel;
let pre_tokenizer = ByteLevel::default();
```
--------------------------------
### Split Pre-tokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Demonstrates the Split pre-tokenizer with a space pattern and isolated behavior. This pre-tokenizer splits text based on a provided pattern and specified behavior.
```rust
Split::new( " ".to_string(), Behavior::Isolated, false)
```
--------------------------------
### Sequence Normalizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Chains multiple normalizers together to be applied sequentially in the specified order. Allows for complex normalization pipelines.
```python
Sequence([NFKC(), Lowercase()])
```
--------------------------------
### ByteLevel PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on whitespaces and remaps bytes to visible characters. Useful for reducing alphabet size to 256 and ensuring no unknown tokens.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.ByteLevel()
```
--------------------------------
### Punctuation PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Isolates all punctuation characters from the input string.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Punctuation()
```
--------------------------------
### CharDelimiterSplit PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits the input string on a specified character delimiter.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.CharDelimiterSplit('x')
```
--------------------------------
### Whitespace PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on word boundaries using the regex '\w+|[^\w\s]+'.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Whitespace()
```
--------------------------------
### WhitespaceSplit PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits the input string on any whitespace character.
```rust
use tokenizers::pre_tokenizers::whitespace::WhitespaceSplit;
let pre_tokenizer = WhitespaceSplit;
```
--------------------------------
### Update Rust
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Update your Rust installation using this command.
```bash
rustup update
```
--------------------------------
### WhitespaceSplit PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits the input string on any whitespace character.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.WhitespaceSplit()
```
--------------------------------
### Build and Test Python Bindings
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Install Python bindings in editable mode with development dependencies and run tests. This process uses maturin for building.
```bash
cd bindings/python
pip install -e ".[dev]" # install in editable mode with test deps (builds via maturin)
make test # run pytest, then cargo test
```
--------------------------------
### Setup Normalizer with Sequence
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Configures a normalizer that applies NFD Unicode normalization followed by accent removal. This is useful for cleaning text before tokenization.
```python
from tokenizers import normalizers
# START setup_normalizer
normalizer = normalizers.Sequence([
normalizers.NFD(),
normalizers.StripAccents(),
])
# END setup_normalizer
```
```rust
use tokenizers::normalizer::Sequence;
use tokenizers::normalizer::unicode::NFD;
use tokenizers::normalizer::unicode::StripAccents;
// START pipeline_setup_normalizer
let normalizer = Sequence::new(vec![Box::new(NFD{}), Box::new(StripAccents{})
]);
// END pipeline_setup_normalizer
```
```js
const { normalizers } = require("tokenizers");
// START setup_normalizer
const normalizer = normalizers.sequence([
normalizers.nfd(),
normalizers.strip_accents(),
]);
// END setup_normalizer
```
--------------------------------
### Metaspace PreTokenizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on whitespaces and replaces them with the special character '▁' (U+2581).
```rust
use tokenizers::pre_tokenizers::metaspace::Metaspace;
let pre_tokenizer = Metaspace;
```
--------------------------------
### Metaspace PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Splits on whitespaces and replaces them with the special character '▁' (U+2581).
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Metaspace()
```
--------------------------------
### Clone Template with Cargo Generate
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/examples/unstable_wasm/README.md
Use `cargo generate` to clone the wasm-pack template. Ensure you have `cargo-generate` installed.
```bash
cargo generate --git https://github.com/rustwasm/wasm-pack-template.git --name my-project
cd my-project
```
--------------------------------
### Apply Normalizer to a String
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/pipeline.md
Manually test a normalizer by applying it to any string. This example demonstrates applying NFD Unicode normalization and accent removal.
```python
from tokenizers import normalizers
normalizer = normalizers.Sequence([
normalizers.NFD(),
normalizers.StripAccents()
])
text = "Héllò Wörld!"
normalized_text = normalizer.normalize_str(text)
print(normalized_text)
```
--------------------------------
### Split PreTokenizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
A versatile pre-tokenizer that splits on a provided pattern with specified behavior (removed, isolated, merged_with_previous, merged_with_next, contiguous) and an optional invert flag.
```python
from tokenizers import pre_tokenizers
pre_tokenizer = pre_tokenizers.Split(pattern=' ', behavior='isolated', invert=False)
```
--------------------------------
### Lowercase Normalizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Converts all uppercase characters in the input string to lowercase. Useful for case-insensitive text processing.
```rust
Lowercase
```
--------------------------------
### Clone and Set Up Python Environment
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Clone the repository and set up a Python virtual environment for development. Ensure you activate the environment before proceeding.
```bash
git clone https://github.com/huggingface/tokenizers.git
cd tokenizers
# Create a virtualenv (using uv, venv, or your preferred tool)
python -m venv .venv
source .venv/bin/activate
```
--------------------------------
### Build All Documentation
Source: https://github.com/huggingface/tokenizers/blob/main/docs/README.md
Build the documentation for all supported languages from the /docs folder.
```bash
make html_all
```
--------------------------------
### BERT Post-processing
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/pipeline.md
Example of post-processing to make inputs suitable for the BERT model, which typically involves adding special tokens like [CLS] and [SEP].
```python
from tokenizers import processors
post_processor = processors.TemplateProcessing(
single='[CLS] $A [SEP]',
pair='[CLS] $A [SEP] $B [SEP]',
special_tokens=[
('[CLS]', 101),
('[SEP]', 102),
],
)
# Assuming 'tokenizer' is an initialized Tokenizer object
# tokenizer.post_processor = post_processor
```
--------------------------------
### Initialize Visualizer with RoBERTa Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Load a ByteLevelBPETokenizer from downloaded files and initialize the EncodingVisualizer with it. This allows visualization using RoBERTa's tokenization scheme.
```python
from tokenizers import ByteLevelBPETokenizer
roberta_tokenizer = ByteLevelBPETokenizer.from_file("/tmp/roberta-base-vocab.json", "/tmp/roberta-base-merges.txt")
roberta_visualizer = EncodingVisualizer(tokenizer=roberta_tokenizer, default_to_notebook=True)
roberta_visualizer(text, annotations=annotations)
```
--------------------------------
### Initialize a new Wasm App project
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/examples/unstable_wasm/www/README.md
Use this command to scaffold a new project with the `create-wasm-app` template. This sets up a project structure for using WebAssembly modules.
```bash
npm init wasm-app
```
--------------------------------
### Instantiate a BpeTrainer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/quicktour.md
Instantiate a BpeTrainer to configure the training process. Special tokens should be provided here.
```python
from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
```
--------------------------------
### Download and Unzip Dataset
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/quicktour.md
Download the wikitext-103 dataset and unzip it for training.
```bash
wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip
unzip wikitext-103-raw-v1.zip
```
--------------------------------
### Generate Tokenizer Stubs with Make Style
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Run the 'make style' command in the 'bindings/python' directory to build the extension, generate stub files, and format the code.
```bash
cd bindings/python
make style
```
--------------------------------
### Navigate to Python bindings
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Change directory to the Python bindings folder after cloning the repository.
```bash
cd tokenizers/bindings/python
```
--------------------------------
### Build Documentation for Specific Language
Source: https://github.com/huggingface/tokenizers/blob/main/docs/README.md
Build the documentation for a specific target language (e.g., python, rust, node).
```bash
make html O="-t python"
```
--------------------------------
### Initialize a BPE Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Instantiate a Tokenizer with a BPE model. This is the first step in building a custom tokenizer.
```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
# Initialize a Tokenizer with a BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
```
```rust
use tokenizers::tokenizer::Tokenizer;
use tokenizers::models::bpe::BPE;
// Initialize a Tokenizer with a BPE model
let tokenizer = Tokenizer::new(BPE::new(None));
```
```js
const { Tokenizer } = require("@xenova/transformers");
const { BPE } = require("@xenova/transformers");
// Initialize a Tokenizer with a BPE model
let tokenizer = new Tokenizer(new BPE({ unk_token: "[UNK]" }));
```
--------------------------------
### Download Legacy Vocabulary File
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Download the legacy vocabulary file for a pretrained tokenizer using wget.
```bash
wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt
```
--------------------------------
### Load Dataset with 🤗 Datasets
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/training_from_memory.mdx
Shows how to load a dataset using the 🤗 Datasets library. This is the first step towards training a tokenizer on a larger, pre-existing dataset.
```python
from datasets import load_dataset
dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
```
--------------------------------
### Initialize a BPE Trainer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Instantiate a BpeTrainer to configure training parameters like vocabulary size and special tokens. Special tokens should be listed in the order they are intended to be assigned IDs.
```python
from tokenizers.trainers import BpeTrainer
# Initialize a BpeTrainer
trainer = BpeTrainer(
vocab_size=30000,
special_tokens=["[UNK]", "[CLS]", "[SEP]", "[MASK]", "[PAD]"]
)
```
```rust
use tokenizers::trainers::BpeTrainer;
// Initialize a BpeTrainer
let trainer = BpeTrainer::new(30000, vec![
String::from("[UNK]"),
String::from("[CLS]"),
String::from("[SEP]"),
String::from("[MASK]"),
String::from("[PAD]"),
]);
```
```js
const { BpeTrainer } = require("@xenova/transformers");
// Initialize a BpeTrainer
let trainer = new BpeTrainer({
vocabSize: 30000,
specialTokens: ["[UNK]", "[CLS]", "[SEP]", "[MASK]", "[PAD]"]
});
```
--------------------------------
### Build Custom Byte-Level BPE Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Initialize a byte-level BPE tokenizer, customize its pre-tokenizer, decoder, and post-processor, then train and save it. Requires dataset files for training.
```python
from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors
# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())
# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)
# And then train
trainer = trainers.BpeTrainer(
vocab_size=20000,
min_frequency=2,
initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
"./path/to/dataset/1.txt",
"./path/to/dataset/2.txt",
"./path/to/dataset/3.txt"
], trainer=trainer)
# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)
```
--------------------------------
### Initialize Tokenizer and Visualizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Initializes a BertWordPieceTokenizer with a specified vocabulary file and then creates an EncodingVisualizer instance using this tokenizer.
```python
tokenizer = BertWordPieceTokenizer("/tmp/bert-base-uncased-vocab.txt", lowercase=True)
visualizer = EncodingVisualizer(tokenizer=tokenizer)
```
--------------------------------
### Lowercase Normalizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Converts all uppercase characters in the input string to lowercase. Useful for case-insensitive text processing.
```python
Lowercase()
```
--------------------------------
### Run Benchmarks
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Execute benchmarks for the Rust core. This command will download benchmark data if it's not already present.
```bash
cd tokenizers
make bench # downloads benchmark data if needed, then runs cargo bench
```
--------------------------------
### Download RoBERTa Tokenizer Files
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Download the vocabulary and merge files for the RoBERTa base tokenizer using wget. These files are necessary for initializing the tokenizer.
```bash
!wget "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-vocab.json" -O /tmp/roberta-base-vocab.json
!wget "https://s3.amazonaws.com/models.huggingface.co/bert/roberta-base-merges.txt" -O /tmp/roberta-base-merges.txt
```
--------------------------------
### Initialize and encode with CharBPETokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Initialize a CharBPETokenizer using pre-existing vocabulary and merges files, then encode text.
```python
from tokenizers import CharBPETokenizer
# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)
# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)
```
--------------------------------
### StripAccents Normalizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Removes accent symbols from Unicode characters. Recommended for use with NFD normalization for consistent results.
```python
StripAccents()
```
--------------------------------
### Rebuild Python Bindings
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Quickly rebuild the Python extension module after making changes to the Rust code. Ensure maturin is installed.
```bash
pip install maturin # if not already installed
maturin develop # fast rebuild of the extension module
```
--------------------------------
### StripAccents Normalizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Removes accent symbols from Unicode characters. Recommended for use with NFD normalization for consistent results.
```rust
StripAccents
```
--------------------------------
### Basic Training with a List
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/training_from_memory.mdx
Demonstrates the simplest way to train a tokenizer using a Python list of strings. This method is suitable for small datasets or quick testing.
```python
sentences = ["This is the first sentence.", "This is the second one."]
tokenizer.train_from_iterator(sentences, trainer=trainer)
```
--------------------------------
### Initialize a BPE Tokenizer in Python
Source: https://github.com/huggingface/tokenizers/blob/main/README.md
Instantiate a tokenizer using the Byte-Pair Encoding (BPE) model. This is a foundational step before customizing or training the tokenizer.
```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
tokenizer = Tokenizer(BPE())
```
--------------------------------
### Replace Normalizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Replaces occurrences of a specified string or regular expression with a given replacement string. Useful for custom text substitutions.
```python
Replace("a", "e")
```
--------------------------------
### Initialize Template Post-processing
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Set up post-processing using TemplateProcessing to automatically add special tokens like '[CLS]' and '[SEP]' to sentences and sentence pairs. This is common for BERT-style inputs.
```python
from tokenizers import AddedToken
from tokenizers.processors import TemplateProcessing
template = "[CLS] $A [SEP]"
post_processor = TemplateProcessing(
single=template,
pair=template + " $B [SEP]",
special_tokens=[
("[CLS]", 1),
("[SEP]", 2),
],
)
tokenizer.post_processor = post_processor
```
```rust
use tokenizers::processors::template::TemplateProcessing;
let mut template = TemplateProcessing::builder()
.single("[CLS] $A [SEP]")
.pair("[CLS] $A [SEP] $B [SEP]")
.special_tokens(vec![("[CLS]".to_string(), 1), ("[SEP]".to_string(), 2)])
.build();
tokenizer.with_post_processor(template);
```
```js
const { TemplateProcessing } = require("@xenova/transformers").tokenizers.processors;
const template = new TemplateProcessing({
single: "[CLS] $A [SEP]",
pair: "[CLS] $A [SEP] $B [SEP]",
specialTokens: [
["[CLS]", 1],
["[SEP]", 2],
],
});
tokenizer.postProcessor = template;
```
--------------------------------
### Import Pretrained Tokenizer from Legacy Vocabulary
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Import a pretrained tokenizer using its vocabulary file. Ensure the vocabulary file is downloaded.
```python
from tokenizers import BertWordPieceTokenizer
tokenizer = BertWordPieceTokenizer("bert-base-uncased-vocab.txt", lowercase=True)
```
--------------------------------
### Instantiate a BPE Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/quicktour.md
Instantiate a Tokenizer with a BPE model. This is the main API for creating tokenizers.
```python
from tokenizers import Tokenizer
from tokenizers.models import BPE
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
```
--------------------------------
### Visualize Text with Custom Annotations
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Use the initialized visualizer to render the text with the provided custom annotations. Ensure the 'annotations' parameter is used.
```python
visualizer(text, annotations=funnyAnnotations)
```
--------------------------------
### Configure BERT Post-Processor in Node.js
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Set up the post-processor in Node.js for BERT compatibility, adding '[CLS]' and '[SEP]' tokens. This prepares the tokenized output for BERT models.
```js
const tokenizer = require("../tokenizers").tokenizer;
// Customize post-processor to add special tokens for BERT
tokenizer.postProcessor = tokenizer.predefined.bertPostprocess(
"[SEP]",
"[CLS]",
);
```
--------------------------------
### Download Benchmark Data Manually
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Manually download specific benchmark data files required for running benchmarks directly with `cargo bench`.
```bash
cd tokenizers
make data/big.txt data/gpt2-vocab.json data/gpt2-merges.txt # etc.
```
--------------------------------
### Strip Normalizer Example
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Removes whitespace characters from the specified sides (left, right, or both) of the input string. Ensures clean text by removing unwanted spaces.
```python
Strip()
```
--------------------------------
### Visualize Tokens With Aligned Annotations
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Uses the EncodingVisualizer to display the tokenization of a given text, overlaying custom annotations defined by start and end positions and labels.
```python
visualizer(text, annotations=annotations)
```
--------------------------------
### Build BERT Tokenizer from Scratch
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/pipeline.md
Instantiate a BERT tokenizer from scratch using WordPiece model, NFD Unicode normalization, accent stripping, whitespace/punctuation pre-tokenization, and BERT-specific post-processing.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, processors
# Initialize a Tokenizer with the WordPiece model
tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
# Customize normalization
tokenizer.normalizer = normalizers.Sequence([
normalizers.NFD(),
normalizers.StripAccents()
])
# Customize pre-tokenization
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
# Customize post-processing
tokenizer.post_processor = processors.TemplateProcessing(
single='[CLS] $A [SEP]',
pair='[CLS] $A [SEP] $B [SEP]',
special_tokens=[
('[CLS]', 101),
('[SEP]', 102),
],
)
# Example usage (training would be a separate step)
# tokenizer.train_from_iterator(...)
print("BERT tokenizer initialized.")
```
--------------------------------
### Strip Normalizer Example (Rust)
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Removes whitespace characters from the specified sides (left, right, or both) of the input string. Ensures clean text by removing unwanted spaces.
```rust
Strip
```
--------------------------------
### Create a Batch Iterator for Training
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/tutorials/python/training_from_memory.rst
Defines a generator function to yield batches of text data from a 🤗 Datasets object. This allows for more efficient training by processing multiple examples at once.
```python
def batch_iterator(dataset, batch_size=100):
for i in range(0, len(dataset), batch_size):
yield dataset[i : i + batch_size]["text"]
```
--------------------------------
### Train Tokenizer from a Single Gzip File
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/training_from_memory.mdx
Demonstrates training a tokenizer directly from a single gzip file, which is treated as an iterator. Ensure the file contains text data.
```python
import gzip
with gzip.open("my_data.txt.gz", "rt", encoding="utf-8") as f:
tokenizer.train_from_iterator(f, trainer=trainer)
```
--------------------------------
### Define Custom Annotations
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Creates a list of Annotation objects, each specifying a start and end position in the text and a label. These annotations will be used to highlight specific parts of the tokenized text.
```python
anno1 = Annotation(start=0, end=2, label="foo")
anno2 = Annotation(start=2, end=4, label="bar")
anno3 = Annotation(start=6, end=8, label="poo")
anno4 = Annotation(start=9, end=12, label="shoe")
annotations = [
anno1,
anno2,
anno3,
anno4,
Annotation(start=23, end=30, label="random tandem bandem sandem landem fandom"),
Annotation(start=63, end=70, label="foo"),
Annotation(start=80, end=95, label="bar"),
Annotation(start=120, end=128, label="bar"),
Annotation(start=152, end=155, label="poo"),
]
```
--------------------------------
### Load Dataset with 🤗 Datasets
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/tutorials/python/training_from_memory.rst
Loads a dataset using the 🤗 Datasets library. This is the first step towards training a tokenizer on a dataset from the Hugging Face Hub.
```python
from datasets import load_dataset
dataset = load_dataset("wikitext", name="wikitext-2-raw-v1", split="train")
```
--------------------------------
### Clean and Rebuild Documentation
Source: https://github.com/huggingface/tokenizers/blob/main/docs/README.md
Clean the build directory before rebuilding the documentation, recommended for structural changes.
```bash
make clean && make html_all
```
--------------------------------
### Test with Wasm-Pack Headless Browsers
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/examples/unstable_wasm/README.md
Run your WebAssembly tests in headless browsers using `wasm-pack test`. Specify the browser (e.g., --firefox).
```bash
wasm-pack test --headless --firefox
```
--------------------------------
### Configure BERT Post-Processor in Python
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Set up the post-processor for a Tokenizer to make inputs suitable for the BERT model. This involves defining how special tokens like '[CLS]' and '[SEP]' are added.
```python
from tokenizers import Tokenizer
from tokenizers.processors import BertProcessing
tokenizer = Tokenizer.from_file("path/to/your/tokenizer.json")
# Customize post-processor to add special tokens for BERT
tokenizer.post_processor = BertProcessing(
"[SEP]",
"[CLS]",
)
```
--------------------------------
### Format and Check Python Bindings
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Auto-format the Python code and check for style compliance.
```bash
cd bindings/python
make style # auto-format
make check-style # check formatting
```
--------------------------------
### Troubleshoot Stub Generation with PYTHONHOME
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
If Python initialization errors occur during stub generation, manually set the PYTHONHOME environment variable before running the stub generator.
```bash
export PYTHONHOME=$(python3 -c 'import sys; print(sys.base_prefix)')
cargo run --manifest-path tools/stub-gen/Cargo.toml
```
--------------------------------
### Sequence Pre-tokenizer Composition
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Shows how to compose multiple pre-tokenizers using the Sequence pre-tokenizer. This allows for a series of pre-tokenization steps to be applied in order.
```rust
Sequence::new(vec![Punctuation, WhitespaceSplit])
```
--------------------------------
### Build and Test Rust Core
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Build and test the core Rust library. This command automatically downloads necessary test data using the huggingface_hub CLI.
```bash
cd tokenizers
make test # downloads test data automatically via the hf CLI, then runs cargo test
```
--------------------------------
### Train and Serialize Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/README.md
Trains a tokenizer using BPE and serializes it to a JSON file. Configures trainer options, normalizers, pre-tokenizers, post-processors, and decoders.
```rust
use tokenizers::decoders::DecoderWrapper;
use tokenizers::models::bpe::{BpeTrainerBuilder, BPE};
use tokenizers::normalizers::{strip::Strip, unicode::NFC, utils::Sequence, NormalizerWrapper};
use tokenizers::pre_tokenizers::byte_level::ByteLevel;
use tokenizers::pre_tokenizers::PreTokenizerWrapper;
use tokenizers::processors::PostProcessorWrapper;
use tokenizers::{AddedToken, Model, Result, TokenizerBuilder};
use std::path::Path;
fn main() -> Result<()> {
let vocab_size: usize = 100;
let mut trainer = BpeTrainerBuilder::new()
.show_progress(true)
.vocab_size(vocab_size)
.min_frequency(0)
.special_tokens(vec![
AddedToken::from(String::from(""), true),
AddedToken::from(String::from(""), true),
AddedToken::from(String::from(""), true),
AddedToken::from(String::from(""), true),
AddedToken::from(String::from(""), true),
])
.build();
let mut tokenizer = TokenizerBuilder::new()
.with_model(BPE::default())
.with_normalizer(Some(Sequence::new(vec![
Strip::new(true, true).into(),
NFC.into(),
])))
.with_pre_tokenizer(Some(ByteLevel::default()))
.with_post_processor(Some(ByteLevel::default()))
.with_decoder(Some(ByteLevel::default()))
.build()?;
let pretty = false;
tokenizer
.train_from_files(
&mut trainer,
vec!["path/to/vocab.txt".to_string()],
)?
.save("tokenizer.json", pretty)?;
Ok(())
}
```
--------------------------------
### Run Specific Rust Benchmarks
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Execute individual benchmarks for the Rust core by specifying the benchmark name.
```bash
cargo bench --bench bpe_benchmark
```
```bash
cargo bench --bench bert_benchmark
```
```bash
cargo bench --bench llama3_benchmark
```
```bash
cargo bench --bench layout_benchmark
```
--------------------------------
### Train a Tokenizer on Files
Source: https://github.com/huggingface/tokenizers/blob/main/README.md
Train a tokenizer using a list of files. This process builds the vocabulary based on the provided text data. Special tokens required by models can be specified.
```python
from tokenizers.trainers import BpeTrainer
trainer = BpeTrainer(special_tokens=["[UNK]", "[CLS]", "[SEP]", "[PAD]", "[MASK]"])
tokenizer.train(files=["wiki.train.raw", "wiki.valid.raw", "wiki.test.raw"], trainer=trainer)
```
--------------------------------
### Initialize a Whitespace Pre-tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Configure a pre-tokenizer to split input strings based on whitespace. This ensures that tokens do not span across multiple words.
```python
from tokenizers.pre_tokenizers import Whitespace
# Set the pre-tokenizer
tokenizer.pre_tokenizer = Whitespace()
```
```rust
use tokenizers::pre_tokenizers::whitespace::Whitespace;
// Set the pre-tokenizer
tokenizer.pre_tokenizer = Some(Box::new(Whitespace {}));
```
```js
const { Whitespace } = require("@xenova/transformers");
// Set the pre-tokenizer
tokenizer.set_pre_tokenizer(new Whitespace());
```
--------------------------------
### Train Tokenizer from Multiple Gzip Files
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/training_from_memory.mdx
Shows how to train a tokenizer from multiple gzip files by creating an iterator that yields from each file sequentially. This is useful for distributed or large datasets stored in gzip format.
```python
import glob
import gzip
def gzip_iterator(path):
for filename in glob.glob(path):
with gzip.open(filename, "rt", encoding="utf-8") as f:
for line in f:
yield line
tokenizer.train_from_iterator(gzip_iterator("my_data_*.txt.gz"), trainer=trainer)
```
--------------------------------
### ByteLevel Pre-tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/components.mdx
Illustrates the ByteLevel pre-tokenizer, which splits on whitespace and remaps bytes to visible characters. This method is efficient for tokenization as it uses a fixed alphabet size.
```rust
ByteLevel
```
--------------------------------
### Download BERT Vocabulary File
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
This command downloads the vocabulary file for the BERT base uncased model, which is required for initializing the tokenizer.
```bash
!wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt -O /tmp/bert-base-uncased-vocab.txt
```
--------------------------------
### Save Tokenizer in Python, Rust, and Node.js
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Demonstrates how to save a trained tokenizer to a file. This is useful for persisting your tokenizer configuration and vocabulary.
```python
tokenizer.save("my-tokenizer.json")
```
```rust
tokenizer.save("my-tokenizer.json").unwrap();
```
```js
tokenizer.save("my-tokenizer.json");
```
--------------------------------
### Train the Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Train the tokenizer on a list of files using the configured trainer. This process learns the merge rules for the BPE model.
```python
# Train the tokenizer on the files
tokenizer.train(["wikitext-103-raw-v1/train.txt"], trainer=trainer)
```
```rust
// Train the tokenizer on the files
tokenizer.train_from_files(vec!["wikitext-103-raw-v1/train.txt"], &mut trainer);
```
```js
// Train the tokenizer on the files
tokenizer.train(["wikitext-103-raw-v1/train.txt"], trainer);
```
--------------------------------
### Lint Rust Core
Source: https://github.com/huggingface/tokenizers/blob/main/CONTRIBUTING.md
Run Rust formatting (rustfmt) and linting (clippy) checks on the Rust core library.
```bash
cd tokenizers
make lint # rustfmt --check + clippy
```
--------------------------------
### Clone tokenizers repository
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/installation.mdx
Clone the tokenizers GitHub repository to build from source.
```bash
git clone https://github.com/huggingface/tokenizers
```
--------------------------------
### Load and Use Custom Byte-Level BPE Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Load a previously saved byte-level BPE tokenizer from a file and use it to encode text.
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")
encoded = tokenizer.encode("I can feel the magic, can you?")
```
--------------------------------
### Load Pretrained Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Load any tokenizer from the Hugging Face Hub if a `tokenizer.json` file is available in the repository.
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-uncased")
```
--------------------------------
### Train Tokenizer from Multiple Gzip Files
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/tutorials/python/training_from_memory.rst
Trains a tokenizer from multiple gzip-compressed files by iterating over a list of file paths. Each file is opened and read as an iterator.
```python
import glob
files = glob.glob("*.txt.gz")
def files_iterator(files):
for file in files:
with gzip.open(file, "rt", encoding="utf-8") as f:
for line in f:
yield line
tokenizer.train_from_iterator(files_iterator(files), trainer=trainer)
```
--------------------------------
### Load a pretrained tokenizer from the Hub
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Load a tokenizer that has been previously trained and uploaded to the Hugging Face Hub.
```python
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_pretrained("bert-base-cased")
```
--------------------------------
### Train and save a CharBPETokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Train a CharBPETokenizer from a list of text files and save the resulting tokenizer model.
```python
from tokenizers import CharBPETokenizer
# Initialize a tokenizer
tokenizer = CharBPETokenizer()
# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])
# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")
# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")
```
--------------------------------
### Basic Tokenizer Training with a List
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/tutorials/python/training_from_memory.rst
Trains a tokenizer using a simple Python list of strings as the training data. This is the most straightforward method for in-memory training.
```python
data = ["This is the first sentence.", "This is the second sentence."]
tokenizer.train_from_iterator(data, trainer=trainer)
```
--------------------------------
### Publish to NPM with Wasm-Pack
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/examples/unstable_wasm/README.md
Publish your compiled WebAssembly package to NPM using `wasm-pack publish`.
```bash
wasm-pack publish
```
--------------------------------
### Train Tokenizer on Files
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source/quicktour.md
Train the tokenizer using the `train` method, providing a list of files to process. This should be fast.
```python
files = ["wikitext-103-raw/wiki.train.raw", "wikitext-103-raw/wiki.valid.raw", "wikitext-103-raw/wiki.test.raw"]
tokenizer.train(files)
```
--------------------------------
### Replace Pre-Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Illustrates how to replace the existing pre-tokenizer of a Tokenizer instance. This is useful for customizing the initial text splitting behavior.
```python
from tokenizers import Tokenizer, models, normalizers, pre_tokenizers, decoders
def replace_pre_tokenizer():
tokenizer = Tokenizer(models.WordPiece(unk_token="[UNK]"))
# Replace the pre_tokenizer
tokenizer.pre_tokenizer = pre_tokenizers.Whitespace()
return tokenizer
```
```rust
use tokenizers::pre_tokenizers;
fn pipeline_replace_pre_tokenizer() {
let mut tokenizer = tokenizers::Tokenizer::new(tokenizers::models::WordPiece::new(true));
// Replace the pre_tokenizer
tokenizer.pre_tokenizer = Some(pre_tokenizers::Whitespace::new());
}
```
```js
const tokenizer = new Tokenizer(new models.WordPiece());
// Replace the pre_tokenizer
tokenizer.preTokenizer = PreTokenizer.whitespace();
```
--------------------------------
### Import Tokenizer and Visualizer Classes
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Imports the necessary BertWordPieceTokenizer and EncodingVisualizer classes from the tokenizers library.
```python
from tokenizers import BertWordPieceTokenizer
from tokenizers.tools import EncodingVisualizer
```
--------------------------------
### Generate Tokenizer Stubs Manually
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/README.md
Manually run the stub generator by executing 'cargo run' for the stub generator and then 'python stub.py'. This process builds the extension, copies it, and generates stubs.
```bash
cd bindings/python
cargo run --manifest-path tools/stub-gen/Cargo.toml
python stub.py
```
--------------------------------
### EncodingVisualizer Class
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/api/visualizer.mdx
A tool for visualizing tokenization encodings.
```APIDOC
## Class: tokenizers.tools.EncodingVisualizer
### Description
Provides methods to visualize the results of tokenization encodings.
### Method: __call__
This method is the primary way to invoke the visualizer, likely taking an encoding object as input and producing a visual representation.
```
--------------------------------
### Define Custom Annotations
Source: https://github.com/huggingface/tokenizers/blob/main/bindings/python/examples/using_the_visualizer.ipynb
Create a list of dictionaries representing custom annotations with 'startPlace', 'endPlace', and 'theTag' keys.
```python
funnyAnnotations = [dict(startPlace=i, endPlace=i + 3, theTag=str(i)) for i in range(0, 20, 4)]
funnyAnnotations
```
--------------------------------
### Reload Tokenizer from File in Python, Rust, and Node.js
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/quicktour.mdx
Shows how to reload a previously saved tokenizer from a file. Use this when you need to reuse a trained tokenizer without retraining.
```python
tokenizer = Tokenizer.from_file("my-tokenizer.json")
```
```rust
let tokenizer = Tokenizer::from_file("my-tokenizer.json").unwrap();
```
```js
const tokenizer = Tokenizer.from_file("my-tokenizer.json");
```
--------------------------------
### Build with Wasm-Pack
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/examples/unstable_wasm/README.md
Compile your Rust project into WebAssembly using `wasm-pack build`.
```bash
wasm-pack build
```
--------------------------------
### Load Pretrained Tokenizer
Source: https://github.com/huggingface/tokenizers/blob/main/tokenizers/README.md
Loads a pretrained tokenizer from the Hugging Face Hub. Requires the 'http' feature to be enabled.
```rust
use tokenizers::tokenizer::{Result, Tokenizer};
fn main() -> Result<()> {
# #[cfg(feature = "http")]
# {
// needs http feature enabled
let tokenizer = Tokenizer::from_pretrained("bert-base-cased", None)?;
let encoding = tokenizer.encode("Hey there!", false)?;
println!("{:?}", encoding.get_tokens());
# }
Ok(())
}
```
--------------------------------
### Initialize BERT Tokenizer with WordPiece Model
Source: https://github.com/huggingface/tokenizers/blob/main/docs/source-doc-builder/pipeline.mdx
Instantiate a new Tokenizer with the WordPiece model, which is fundamental for BERT tokenization. This sets up the core subword tokenization mechanism.
```python
from tokenizers import Tokenizer
from tokenizers.models import WordPiece
# Initialize a Tokenizer with a WordPiece model
tokenizer = Tokenizer(WordPiece.empty())
```
```rust
use tokenizers::tokenizer::Tokenizer;
use tokenizers::models::wordpiece::WordPiece;
// Initialize a Tokenizer with a WordPiece model
let tokenizer = Tokenizer::new(WordPiece::empty());
```
```js
const { Tokenizer } = require("@xenova/tokenizers");
const { WordPiece } = require("@xenova/tokenizers/models");
// Initialize a Tokenizer with a WordPiece model
const tokenizer = new Tokenizer(WordPiece.empty());
```