Kitoken

https://github.com/systemcluster/kitoken
Admin
Kitoken is a fast and versatile tokenizer for language models, compatible with SentencePiece,...
Tokens:72,984
Snippets:1,011
Trust Score:9.3
License:MIT
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install kitoken-cli with Cargo

Source: https://github.com/systemcluster/kitoken/blob/main/packages/cli/README.md

Install the kitoken-cli tool directly from its Git repository using Cargo, the Rust package manager.

```bash
cargo install --git https://github.com/Systemcluster/kitoken
```

--------------------------------

### Windows API Call Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/small_output_nai-t5.txt

Shows a Windows API call related to memory management, specifically GlobalUnlock.

```cpp
GlobalUnlock(medium.hGlobal);

```

--------------------------------

### XLNet Special Tokens Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Demonstrates the handling of special tokens like <|endoftext|> and <unk> by the tokenizer.

```text
<|endoftext|>
endoftext
Such joy <unk> let's run!<unk> <unk>
<unk>z
```

--------------------------------

### XLNet Mathematical Expression Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Illustrates the tokenizer's output for a mathematical expression.

```text
((V<unk>V)=<unk>V)/V<unk>,V <unk> <unk>
```

--------------------------------

### XLNet Punctuation and Numbers Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Shows how the tokenizer processes punctuation, numbers, and mixed character sequences.

```text
<unk>!? <unk>12<unk>1<unk>! <unk>......<unk>!<unk>
"<unk> <unk>- <unk>, <unk>, <unk> <unk> <unk>! <unk>!"
```

--------------------------------

### Double Newline Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_sqlpilot.txt

Shows how consecutive newline characters are handled.

```Plain Text
Double


Newline

```

--------------------------------

### Python Tokenizer Initialization and Usage

Source: https://github.com/systemcluster/kitoken/blob/main/packages/python/README.md

Initializes a Kitoken encoder from a web-based model and demonstrates encoding and decoding text. Ensure the 'kitoken' library is installed.

```python
from kitoken import Kitoken

encoder = Kitoken.from_web("hf:Qwen/Qwen3.5-9B")

tokens = encoder.encode("hello world!", True)
string = encoder.decode(tokens).decode("utf-8")

assert string == "hello world!"
```

--------------------------------

### Initialize and Tokenize Web Example

Source: https://github.com/systemcluster/kitoken/blob/main/packages/javascript/examples/web.html

This snippet initializes the Kitoken WebAssembly module, loads a tokenizer model, and provides functionality to tokenize and detokenize user input in a web page. It requires the 'kitoken/web' export and assumes 'web.js' and 'web_bg.wasm' are available.

```javascript
import init, { Kitoken } from "../dist/web.js";

const wasm_ = init()
const model_ = fetch('../../../tests/models/llama2.kit').then(res => res.arrayBuffer())

document.addEventListener('DOMContentLoaded', () => {
    const input = document.querySelector('#input');
    const submit = document.querySelector('#submit');
    const count = document.querySelector('#count span');
    const output = document.querySelector('#tokens');
    let encoder = null

    async function tokenize() {
        try {
            await wasm_
            const model = await model_

            if (!encoder) {
                encoder = new Kitoken(new Uint8Array(model))
            }

            const text = input.value;
            const tokens = encoder.encode(text);
            console.log("Encoded:", tokens)
            count.textContent = tokens.length;
            output.textContent = tokens.join(', ')

            const decoded = new TextDecoder().decode(encoder.decode(tokens));
            console.log("Decoded:", decoded);
        } catch (error) {
            console.error(error)
            output.textContent = 'Error: ' + error.message;
        }
    }

    submit.addEventListener('click', tokenize);
    tokenize();
});
```

--------------------------------

### Combining Characters Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_qwen35.txt

Demonstrates the use of combining characters in UTF-8, showing how diacritics and other marks can be applied to base characters to form new glyphs.

```plaintext
STARGΛ̊TE SG-1, a = v̇ = r̈, a⃑ ⊥ b⃑
```

--------------------------------

### APL Example in UTF-8

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_sqlpilot.txt

Demonstrates the use of APL characters within a UTF-8 encoded plain text file.

```APL
((V<unk>V)=<unk><unk>V)/V<unk>,V
```

--------------------------------

### XLNet Full Alphabet and Numbers Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Shows the tokenizer's output for the complete English alphabet (uppercase and lowercase) and numbers.

```text
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzACDGJKNOPQSTUVWXYZabcdfhijklmnpqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABDEFGJKLMNOPQSTUVWXYabcdefghijklmnopqrstuvwxyzABDEFGIJKLMOSTUVWXYabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzı<unk>α<unk>ι<unk>ο<unk>α<unk>ι<unk>ο<unk>α<unk>ι<unk>ο<unk>α<unk>ι<unk>ο<unk>α<unk>ι<unk>ο<unk>01234567890123456789012345678901234567890123456789
```

--------------------------------

### XLNet Mixed Characters Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Shows the tokenizer's output for a sequence containing mixed characters and tokens.

```text
<unk>oes<unk>tb<unk>
```

--------------------------------

### Russian Text Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_llama2.txt

Shows an example of Russian text encoded in UTF-8, suitable for multilingual applications and internationalization efforts.

```russian
Зарегистрируйтесь сейчас на Десятую Международную Конференцию по
Unicode, которая состоится 10-12 марта 1997 года в Майнце в Германии. 
Конференция соберет широкий круг экспертов по  вопросам глобального
Интернета и Unicode, локализации и интернационализации, воплощению и
применению Unicode в различных операционных системах и программных
приложениях, шрифтах, верстке и многоязычных компьютерных системах.
```

--------------------------------

### XLNet Unicode and ASCII Test Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Displays the tokenizer's handling of Unicode characters and an ASCII safety test.

```text
<unk> <unk> <unk> <unk> Unicode-<unk> <unk> <unk>
<unk> • ASCII safety test: 1lI|, 0OD, 8B <unk>
```

--------------------------------

### Linguistic and Dictionary Examples in UTF-8

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_sqlpilot.txt

Shows how UTF-8 can represent characters used in linguistics and dictionaries, including phonetic transcriptions.

```Plain Text
<unk>i <unk>nt<unk><unk>n<unk><unk><unk>n<unk>l f<unk><unk>n<unk>t<unk>k <unk>so<unk>si<unk>e<unk><unk>n
Yen [j<unk>n], Yoga [<unk>jo<unk>g<unk>]
```

--------------------------------

### Demosthenes Greek Text Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_mistral03.txt

This example presents a passage from Demosthenes in ancient Greek, showcasing the use of Polytonic Greek characters.

```greek
Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι,
ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς
λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ
τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿
εἰς τοῦτο προήκοντα,  ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ
πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν
οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι,
οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν
ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον
τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι
γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν
προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους
σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ
τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ
τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς
τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον.

  Δημοσθένους, Γ´ ᾿Ολυνθιακὸς
```

--------------------------------

### Combining Characters Example in UTF-8

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_sqlpilot.txt

Demonstrates the use of combining characters in UTF-8 for creating modified letters.

```Plain Text
STARG<unk><unk>TE SG-1, a = v<unk> = r̈, a<unk> <unk> b<unk>
```

--------------------------------

### Greek Text Example (Polytonic)

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/utf8_output_mistral03.txt

This snippet shows the Greek language in its Polytonic script, including accents and breathing marks. It's an example of how classical Greek can be represented using UTF-8.

```greek
Σὲ γνωρίζω ἀπὸ τὴν κόψη
τοῦ σπαθιοῦ τὴν τρομερή,
σὲ γνωρίζω ἀπὸ τὴν ὄψη
ποὺ μὲ βία μετράει τὴ γῆ.

᾿Απ᾿ τὰ κόκκαλα βγαλμένη
τῶν ῾Ελλήνων τὰ ἱερά
καὶ σὰν πρῶτα ἀνδρειωμένη
χαῖρε, ὦ χαῖρε, ᾿Ελευθεριά!
```

```greek
Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι,
ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς
λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ
τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿
εἰς τοῦτο προήκοντα,  ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ
πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν
οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι,
οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν
ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον
τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι
γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν
προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους
σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ
τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ
τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς
τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον.

Δημοσθένους, Γ´ ᾿Ολυνθιακὸς
```

--------------------------------

### Handling Unicode and ASCII Safety Test

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_hw_doc_summarizer.txt

Shows examples related to Unicode and an ASCII safety test with specific characters.

```text
unicode - [UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK][UNK]
[UNK][UNK]ascii safety test : 1li [UNK], 0od , 8b [UNK]
```

--------------------------------

### ASCII Safety Test Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/small_output_nai-t5.txt

Presents an ASCII safety test string used to verify character handling.

```text
 ⁇  • ASCII safety test: 1lI|, 0OD, 8B  ⁇ 

```

--------------------------------

### BERT Base Cased Tokenizer Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_bert_base_cased.txt

Demonstrates the use of the bert-base-cased tokenizer to process a sample text, including special tokens like <|endoftext|> and [UNK].

```text
Project: /systemcluster/kitoken
Content:
Hello my love!
I go a little more up and I find someone or something there. Yellow in color, with a head like a pointed top and a wire - like, slender body.
< | endoftext | >
endoftext
Such joy [UNK] let ' s run! [UNK]
[UNK] [UNK] [UNK]
I held the card up to the pad and hit " send. " Voilà, the door magically opened. On the other side was Blood, wearing a janitor ' s uniform for the Dallas airport and standing behind a large Rubbermaid trash receptacle.
GlobalUnlock ( medium. hGlobal ) ; * * *
[UNK] [ [UNK] ] 文 [UNK] [UNK]
「 なに ！ [UNK] [UNK] 長 [UNK] [UNK] [UNK] 、 [UNK] 本 [UNK] [UNK] ！ それは … … [UNK] [UNK] [UNK] 大 [UNK] な 事 になる ！ 」
" [UNK] [UNK] - [UNK], [UNK], [UNK] [UNK] [UNK]! [UNK]! "
( ( [UNK] ) = [UNK] ) / V←, V [UNK] [UNK] [UNK] [UNK]
[UNK] [UNK] [UNK] [UNK] Unicode - ის [UNK] [UNK]
[UNK] • ASCII safety test : 1lI |, 0OD, 8B [UNK]
[UNK]
Double Newline
[UNK]
< s > [PAD] < / s >
[UNK] eop [UNK]
むoesいtb [UNK]
```

--------------------------------

### XLNet Double Newline and Special Tokens Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt

Illustrates the tokenizer's output for a double newline and specific tokens like <s>, [PAD], and ?? eop ??.

```text
Double Newline
<unk>
<s> [PAD]</s>
?? eop ??
```

--------------------------------

### Character Set and Numerical Sequence Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/small_output_nai-t5.txt

Illustrates SentencePiece's ability to process a wide range of characters, including uppercase and lowercase alphabets, and numerical sequences.

```text
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzACDGJKNOPQSTUVWXYZabcdfhijklmnpqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz ⁇ 01234567890123456789012345678901234567890123456789
```

--------------------------------

### Nicer Typography in Plain Text

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_gpt_neox.txt

This example demonstrates how UTF-8 allows for enhanced typography in plain text files, including various quote styles, special characters, and symbols.

```text
╔══════════════════════════════════════════╗
║                                          ║
║   • ‘single’ and “double” quotes         ║
║                                          ║
║   • Curly apostrophes: “We’ve been here” ║
║                                          ║
║   • Latin-1 apostrophe and accents: '´`  ║
║                                          ║
║   • ‚deutsche‘ „Anführungszeichen“       ║
║                                          ║
║   • †, ‡, ‰, •, 3–4, —, −5/+5, ™, …      ║
║                                          ║
║   • ASCII safety test: 1lI|, 0OD, 8B     ║
║                      ╭─────────╮         ║
║   • the euro symbol: │ 14.95 € │         ║
║                      ╰─────────╯         ║
╚══════════════════════════════════════════╝
```

--------------------------------

### Georgian Text Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_gpt_neox.txt

This snippet displays text in Georgian, illustrating UTF-8's support for non-Latin scripts and its use in international communication.

```georgian
From a Unicode conference invitation:

გთხოვთ ახლავე გაიაროთ რეგისტრაცია Unicode-ის მეათე საერთაშორისო
კონფერენციაზე დასასწრებად, რომელიც გაიმართება 10-12 მარტს,
ქ. მაინცში, გერმანიაში. კონფერენცია შეჰკრებს ერთად მსოფლიოს
ექსპერტებს ისეთ დარგებში როგორიცაა ინტერნეტი და Unicode-ი,
ინტერნაციონალიზაცია და ლოკალიზაცია, Unicode-ის გამოყენება
ოპერაციულ სისტემებსა, და გამოყენებით პროგრამებში, შრიფტებში,
ტექსტების დამუშავებასა და მრავალენოვან კომპიუტერულ სისტემებში.
```

--------------------------------

### Nicely Formatted Plain Text with UTF-8 Characters

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/utf8_output_llama2.txt

This example shows how UTF-8 allows for enhanced plain text formatting using a variety of symbols, including quotes, apostrophes, and special characters for typography and currency.

```text
╔══════════════════════════════════════════╗
║                                          ║
║   • ‘single’ and “double” quotes         ║
║                                          ║
║   • Curly apostrophes: “We’ve been here” 
║                                          ║
║   • Latin-1 apostrophe and accents: '´`  ║
║                                          ║
║   • ‚deutsche‘ „Anführungszeichen“       ║
║                                          ║
║   • †, ‡, ‰, •, 3–4, —, −5/+5, ™, …      ║
║                                          ║
║   • ASCII safety test: 1lI|, 0OD, 8B     ║
║                      ╭─────────╮         ║
║   • the euro symbol: │ 14.95 € │         ║
║                      ╰─────────╯         ║
╚══════════════════════════════════════════╝
```

--------------------------------

### Extended Character Set Example

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_sqlpilot.txt

Demonstrates a long string with repeated sequences of the English alphabet, potentially for testing character set handling or repetition detection.

```Plain Text
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefgijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzACDGJKNOPQSTUVWXYZabcdfhijklmnpqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABDEFGJKLMNOPQSTUVWXYabcdefghijklmnopqrstuvwxyzABDEFGIJKLMOSTUVWXYabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>α<unk><unk><unk><unk><unk><unk>θ<unk><unk><unk>μ<unk><unk><unk>π<unk><unk><unk><unk><unk><unk><unk>ψ<unk><unk><unk>θ<unk><unk><unk>π01234567890123456789012345678901234567890123456789
```

--------------------------------

### Thai Text Example (UCS Level 2)

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_gpt_neox.txt

This snippet presents Thai text from a classic Chinese novel, demonstrating UTF-8's support for Southeast Asian scripts.

```thai
Excerpt from a poetry on The Romance of The Three Kingdoms (a Chinese
classic 'San Gua'):

[----------------------------|------------------------]
  ๏ แผ่นดินฮั่นเสื่อมโทรมแสนสังเวช  พระปกเกศกองบู๊กู้ขึ้นใหม่
สิบสองกษัตริย์ก่อนหน้าแลถัดไป       สององค์ไซร้โง่เขลาเบาปัญญา
  ทรงนับถือขันทีเป็นที่พึ่ง           บ้านเมืองจึงวิปริตเป็นนักหนา
โฮจิ๋นเรียกทัพทั่วหัวเมืองมา         หมายจะฆ่ามดชั่วตัวสำคัญ
  เหมือนขับไสไล่เสือจากเคหา      รับหมาป่าเข้ามาเลยอาสัญ
ฝ่ายอ้องุ้นยุแยกให้แตกกัน          ใช้สาวนั้นเป็นชนวนชื่นชวนใจ
  พลันลิฉุยกุยกีกลับก่อเหตุ          ช่างอาเพศจริงหนาฟ้าร้องไห้
ต้องรบราฆ่าฟันจนบรรลัย           ฤๅหาใครค้ำชูกู้บรรลังก์ ฯ
```

--------------------------------

### Compare, Convert, and Inspect Model Definitions with kitoken-cli

Source: https://github.com/systemcluster/kitoken/blob/main/packages/cli/README.md

These commands allow for comparing model files, converting models to the .kit format, and inspecting the details of .kit model files.

```bash
# Compare
kitoken compare llama4.json llama4.model
```

```bash
# Convert to .kit
kitoken convert llama4.model
```

```bash
# Inspect
kitoken inspect llama4.kit
```

--------------------------------

### Initialize and Use Kitoken in Node.js

Source: https://github.com/systemcluster/kitoken/blob/main/packages/javascript/README.md

This snippet demonstrates how to initialize the Kitoken tokenizer in a Node.js environment using a model file and then encode and decode text.

```javascript
import { Kitoken } from "kitoken/node"

const model = fs.readFileSync("models/llama4.model")
const encoder = new Kitoken(model)

const tokens = encoder.encode("hello world!", true)
const string = TextDecoder().decode(encoder.decode(tokens))
```

--------------------------------

### Encode and Decode Text with Kitoken in Rust

Source: https://github.com/systemcluster/kitoken/blob/main/README.md

Demonstrates how to load a tokenizer from the web and use it to encode and decode text. Ensure the 'kitoken' crate is added as a dependency.

```rust
use kitoken::Kitoken;
let encoder = Kitoken::from_web("hf:Qwen/Qwen3.5-9B")?;

let tokens = encoder.encode("Your future belongs to me.", true)?;
let string = String::from_utf8(encoder.decode(&tokens, true)?)?;

assert!(string == "Your future belongs to me.");
```

--------------------------------

### Load Tokenizer Model

Source: https://github.com/systemcluster/kitoken/blob/main/README.md

Loads a BPE tokenizer model from a JSON file. This is the initial step before performing tokenization.

```rust
let encoder = Kitoken::from_file("models/mistral.json")?;
```

--------------------------------

### Handling Unknown Tokens and End of Text

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_hw_doc_summarizer.txt

Demonstrates the use of '[UNK]' for unknown tokens and 'endoftext' markers in a sequence.

```text
[UNK][UNK]endoftext [UNK][UNK]
endoftext
```

--------------------------------

### Handling Multiple Unknown Tokens

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_hw_doc_summarizer.txt

Illustrates a sequence with a high density of unknown tokens.

```text
[UNK][UNK][UNK][UNK][UNK]
```

--------------------------------

### Load SentencePiece Model in Rust

Source: https://github.com/systemcluster/kitoken/blob/main/README.md

Use this to load SentencePiece models. Kitoken converts BPE and Unigram formats, adjusting normalization and encoding modes based on the model's definition.

```rust
let encoder = Kitoken::from_file("models/gemma.model")?;
```

--------------------------------

### Load HuggingFace Tokenizer in Rust

Source: https://github.com/systemcluster/kitoken/blob/main/README.md

Initialize Kitoken with HuggingFace Tokenizer definitions for BPE, Unigram, and WordPiece models. Kitoken converts these formats and applies normalization, pre-tokenization, and post-processing configurations.

```rust
let encoder = Kitoken::from_file("models/llama4.json")?;
```

--------------------------------

### Basic Word Tokenization

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_clip_base.txt

Illustrates basic word tokenization using a common delimiter. This is useful for simple text splitting.

```text
hello</w>my</w>love</w>!</w>
i</w>go</w>a</w>little</w>more</w>up</w>and</w>i</w>find</w>someone</w>or</w>something</w>there</w>.</w>yellow</w>in</w>color</w>,</w>with</w>a</w>head</w>like</w>a</w>pointed</w>top</w>and</w>a</w>wire</w>-
like</w>,</w>slender</w>body</w>.</w>.
<|endoftext|>
endoftext</w>
such</w>joy</w>😊</w>let</w>'s</w>run</w>!🏃‍♂️</w>
```

--------------------------------

### Encode and Decode Text with kitoken-cli

Source: https://github.com/systemcluster/kitoken/blob/main/packages/cli/README.md

Use these commands to encode text into tokens or decode tokens back into text using a specified model definition.

```bash
# Encode
kitoken encode deepseek.kit ./texts.txt
```

```bash
# Decode
kitoken decode deepseek.kit ./tokens.txt
```

--------------------------------

### GlobalUnlock Function Call

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_sqlpilot.txt

Demonstrates the usage of the GlobalUnlock function, often found in Windows API interactions.

```C++
GlobalUnlock(medium.hGlobal);
```

--------------------------------

### Mathematical and Symbolic Tokenization

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_clip_base.txt

Includes a snippet with mathematical symbols and operators, showcasing tokenization of complex symbolic expressions.

```text
((

v
⍳
v
)
=
⍳
⍴
v
)/
v
←,
v
⌷
←
⍳
→
⍴
∆∇⊃‾⍎⍕⌈

```

--------------------------------

### Special Token Handling in SentencePiece

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/small_output_nai-t5.txt

Demonstrates the presence and handling of special tokens like '<0xE2><0x98><0x86>' (endoftext) and '<0xE2><0x98><0x87>' (eop) within SentencePiece output.

```text
 ⁇ |endoftext|>
endoftext
Such joy  ⁇  let's run! ⁇   ⁇ 
 ⁇ 

```

```text
?? eop ??
```

--------------------------------

### Command-like Tokenization

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_clip_base.txt

Presents a sequence that resembles a command or function call, often seen in specific API interactions or scripting.

```text
i</w>held</w>the</w>card</w>up</w>to</w>the</w>pad</w>and</w>hit</w>"</w>send</w>."</w>voilà</w>,</w>the</w>door</w>magically</w>opened</w>.</w>on</w>the</w>other</w>side</w>was</w>blood</w>,</w>wearing</w>a</w>janitor</w>'s</w>uniform</w>for</w>the</w>dallas</w>airport</w>and</w>standing</w>behind</w>a</w>large</w>rubbermaid</w>trash</w>receptacle</w>.</w>
globalunlock</w>(</w>medium</w>.</w>hglobal</w>);</w>***</w>
```

--------------------------------

### Load Tiktoken Model in Rust

Source: https://github.com/systemcluster/kitoken/blob/main/README.md

Load Tiktoken definitions, which are BPE tokenizers used by OpenAI models. Kitoken uses BytePair tokenization in byte mode and infers special tokens and split regex for common models.

```rust
let encoder = Kitoken::from_file("models/o200k_base.tiktoken")?;
```

--------------------------------

### Georgian Language Sample

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/utf8_output_llama2.txt

This snippet shows a Unicode conference invitation in Georgian, demonstrating the encoding of the Georgian script for international communication.

```georgian
გთხოვთ ახლავე გაიაროთ რეგისტრაცია Unicode-ის მეათე საერთაშორისო
კონფერენციაზე დასასწრებად, რომელიც გაიმართება 10-12 მარტს,
ქ. მაინცში, გერმანიაში. კონფერენცია შეჰკრებს ერთად მსოფლიოს
ექსპერტებს ისეთ დარგებში როგორიცაა ინტერნეტი და Unicode-ი,
ინტერნაციონალიზაცია და ლოკალიზაცია, Unicode-ის გამოყენება
ოპერაციულ სისტემებსა, და გამოყენებით პროგრამებში, შრიფტებში,
ტექსტების დამუშავებასა და მრავალენოვან კომპიუტერულ სისტემებში.
```

--------------------------------

### Japanese Text Tokenization

Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_sqlpilot.txt

Illustrates tokenization of Japanese text, including handling of kanji and punctuation.

```Japanese
((V�V)=�V)/V�,V    ������� ������� �������� ������������� Unicode-�� ����������������
```
Tokens:
Raw