### Install kitoken-cli with Cargo Source: https://github.com/systemcluster/kitoken/blob/main/packages/cli/README.md Install the kitoken-cli tool directly from its Git repository using Cargo, the Rust package manager. ```bash cargo install --git https://github.com/Systemcluster/kitoken ``` -------------------------------- ### XLNet Special Tokens Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt Demonstrates the handling of special tokens like <|endoftext|> and by the tokenizer. ```text <|endoftext|> endoftext Such joy let's run! z ``` -------------------------------- ### Windows API Call Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/small_output_nai-t5.txt Shows a Windows API call related to memory management, specifically GlobalUnlock. ```cpp GlobalUnlock(medium.hGlobal); ``` -------------------------------- ### XLNet Mathematical Expression Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt Illustrates the tokenizer's output for a mathematical expression. ```text ((VV)=V)/V,V ``` -------------------------------- ### XLNet Punctuation and Numbers Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_xlnet_base_cased.txt Shows how the tokenizer processes punctuation, numbers, and mixed character sequences. ```text !? 121! ......! " - , , ! !" ``` -------------------------------- ### APL Example in UTF-8 Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_sqlpilot.txt Demonstrates the use of APL characters within a UTF-8 encoded plain text file. ```APL ((VV)=V)/V,V ``` -------------------------------- ### Russian Text Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_llama2.txt Shows an example of Russian text encoded in UTF-8, suitable for multilingual applications and internationalization efforts. ```russian Зарегистрируйтесь сейчас на Десятую Международную Конференцию по Unicode, которая состоится 10-12 марта 1997 года в Майнце в Германии. Конференция соберет широкий круг экспертов по вопросам глобального Интернета и Unicode, локализации и интернационализации, воплощению и применению Unicode в различных операционных системах и программных приложениях, шрифтах, верстке и многоязычных компьютерных системах. ``` -------------------------------- ### Initialize and Tokenize Web Example Source: https://github.com/systemcluster/kitoken/blob/main/packages/javascript/examples/web.html This snippet initializes the Kitoken WebAssembly module, loads a tokenizer model, and provides functionality to tokenize and detokenize user input in a web page. It requires the 'kitoken/web' export and assumes 'web.js' and 'web_bg.wasm' are available. ```javascript import init, { Kitoken } from "../dist/web.js"; const wasm_ = init() const model_ = fetch('../../../tests/models/llama2.kit').then(res => res.arrayBuffer()) document.addEventListener('DOMContentLoaded', () => { const input = document.querySelector('#input'); const submit = document.querySelector('#submit'); const count = document.querySelector('#count span'); const output = document.querySelector('#tokens'); let encoder = null async function tokenize() { try { await wasm_ const model = await model_ if (!encoder) { encoder = new Kitoken(new Uint8Array(model)) } const text = input.value; const tokens = encoder.encode(text); console.log("Encoded:", tokens) count.textContent = tokens.length; output.textContent = tokens.join(', ') const decoded = new TextDecoder().decode(encoder.decode(tokens)); console.log("Decoded:", decoded); } catch (error) { console.error(error) output.textContent = 'Error: ' + error.message; } } submit.addEventListener('click', tokenize); tokenize(); }); ``` -------------------------------- ### Python Tokenizer Initialization and Usage Source: https://github.com/systemcluster/kitoken/blob/main/packages/python/README.md Initializes a Kitoken encoder from a web-based model and demonstrates encoding and decoding text. Ensure the 'kitoken' library is installed. ```python from kitoken import Kitoken encoder = Kitoken.from_web("hf:Qwen/Qwen3.5-9B") tokens = encoder.encode("hello world!", True) string = encoder.decode(tokens).decode("utf-8") assert string == "hello world!" ``` -------------------------------- ### Demosthenes Greek Text Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_mistral03.txt This example presents a passage from Demosthenes in ancient Greek, showcasing the use of Polytonic Greek characters. ```greek Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι, ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿ εἰς τοῦτο προήκοντα, ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι, οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον. Δημοσθένους, Γ´ ᾿Ολυνθιακὸς ``` -------------------------------- ### Greek Text Example (Polytonic) Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/sentencepiece/utf8_output_mistral03.txt This snippet shows the Greek language in its Polytonic script, including accents and breathing marks. It's an example of how classical Greek can be represented using UTF-8. ```greek Σὲ γνωρίζω ἀπὸ τὴν κόψη τοῦ σπαθιοῦ τὴν τρομερή, σὲ γνωρίζω ἀπὸ τὴν ὄψη ποὺ μὲ βία μετράει τὴ γῆ. ᾿Απ᾿ τὰ κόκκαλα βγαλμένη τῶν ῾Ελλήνων τὰ ἱερά καὶ σὰν πρῶτα ἀνδρειωμένη χαῖρε, ὦ χαῖρε, ᾿Ελευθεριά! ``` ```greek Οὐχὶ ταὐτὰ παρίσταταί μοι γιγνώσκειν, ὦ ἄνδρες ᾿Αθηναῖοι, ὅταν τ᾿ εἰς τὰ πράγματα ἀποβλέψω καὶ ὅταν πρὸς τοὺς λόγους οὓς ἀκούω· τοὺς μὲν γὰρ λόγους περὶ τοῦ τιμωρήσασθαι Φίλιππον ὁρῶ γιγνομένους, τὰ δὲ πράγματ᾿ εἰς τοῦτο προήκοντα, ὥσθ᾿ ὅπως μὴ πεισόμεθ᾿ αὐτοὶ πρότερον κακῶς σκέψασθαι δέον. οὐδέν οὖν ἄλλο μοι δοκοῦσιν οἱ τὰ τοιαῦτα λέγοντες ἢ τὴν ὑπόθεσιν, περὶ ἧς βουλεύεσθαι, οὐχὶ τὴν οὖσαν παριστάντες ὑμῖν ἁμαρτάνειν. ἐγὼ δέ, ὅτι μέν ποτ᾿ ἐξῆν τῇ πόλει καὶ τὰ αὑτῆς ἔχειν ἀσφαλῶς καὶ Φίλιππον τιμωρήσασθαι, καὶ μάλ᾿ ἀκριβῶς οἶδα· ἐπ᾿ ἐμοῦ γάρ, οὐ πάλαι γέγονεν ταῦτ᾿ ἀμφότερα· νῦν μέντοι πέπεισμαι τοῦθ᾿ ἱκανὸν προλαβεῖν ἡμῖν εἶναι τὴν πρώτην, ὅπως τοὺς συμμάχους σώσομεν. ἐὰν γὰρ τοῦτο βεβαίως ὑπάρξῃ, τότε καὶ περὶ τοῦ τίνα τιμωρήσεταί τις καὶ ὃν τρόπον ἐξέσται σκοπεῖν· πρὶν δὲ τὴν ἀρχὴν ὀρθῶς ὑποθέσθαι, μάταιον ἡγοῦμαι περὶ τῆς τελευτῆς ὁντινοῦν ποιεῖσθαι λόγον. Δημοσθένους, Γ´ ᾿Ολυνθιακὸς ``` -------------------------------- ### Compare, Convert, and Inspect Model Definitions with kitoken-cli Source: https://github.com/systemcluster/kitoken/blob/main/packages/cli/README.md These commands allow for comparing model files, converting models to the .kit format, and inspecting the details of .kit model files. ```bash # Compare kitoken compare llama4.json llama4.model ``` ```bash # Convert to .kit kitoken convert llama4.model ``` ```bash # Inspect kitoken inspect llama4.kit ``` -------------------------------- ### Double Newline Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/small_output_sqlpilot.txt Shows how consecutive newline characters are handled. ```Plain Text Double Newline ``` -------------------------------- ### Combining Characters Example Source: https://github.com/systemcluster/kitoken/blob/main/tests/data/tokenizers/utf8_output_qwen35.txt Demonstrates the use of combining characters in UTF-8, showing how diacritics and other marks can be applied to base characters to form new glyphs. ```plaintext STARGΛ̊TE SG-1, a = v̇ = r̈, a⃑ ⊥ b⃑ ```