### Installing ttok CLI Tool via pip Source: https://github.com/simonw/ttok/blob/main/README.md This command installs the `ttok` command-line utility using pip, the Python package installer. It's the standard method for installing Python packages. ```bash pip install ttok ``` -------------------------------- ### Running ttok as a Python Module Source: https://github.com/simonw/ttok/blob/main/README.md This command demonstrates how to execute the `ttok` tool directly as a Python module, providing an alternative way to access its help documentation or any other functionality without relying on the installed `ttok` executable being in the system's PATH. ```Bash python -m ttok --help ``` -------------------------------- ### Installing ttok CLI Tool via Homebrew Source: https://github.com/simonw/ttok/blob/main/README.md This command installs the `ttok` command-line utility using Homebrew, a package manager for macOS and Linux. It provides an alternative installation method for users preferring Homebrew. ```bash brew install simonw/llm/ttok ``` -------------------------------- ### Installing ttok Development Dependencies Source: https://github.com/simonw/ttok/blob/main/README.md This command installs the project's core dependencies along with its test dependencies in editable mode. The editable installation (`-e`) allows for local code changes to be reflected without reinstallation, which is crucial for active development. ```Bash pip install -e '.[test]' ``` -------------------------------- ### Counting Tokens with the Default Model (GPT-3.5/4) Source: https://github.com/simonw/ttok/blob/main/README.md This command shows token counting using the default tokenizer model (GPT-3.5/4), which is typically `cl100k_base`. Comparing this output (e.g., 5 tokens) to the GPT-2 example highlights how tokenization varies across different models for the same input text. ```bash ttok boo Hello there this is ``` -------------------------------- ### Truncating Text by Token Count Source: https://github.com/simonw/ttok/blob/main/README.md This command demonstrates how to truncate input text to a specified maximum number of tokens using the `-t` or `--truncate` option. The example truncates the phrase 'This is too many tokens' to just the first 3 tokens, resulting in 'This is too'. ```bash ttok This is too many tokens -t 3 ``` -------------------------------- ### Setting Up Development Environment for ttok Source: https://github.com/simonw/ttok/blob/main/README.md These commands outline the initial steps for setting up a development environment for the `ttok` project. It involves navigating into the project directory, creating a new Python virtual environment, and activating it to isolate project dependencies. ```Bash cd ttok python -m venv venv source venv/bin/activate ``` -------------------------------- ### ttok Command Line Help Output Source: https://github.com/simonw/ttok/blob/main/README.md This snippet displays the complete help output for the `ttok` command-line tool, detailing its usage, available options for token counting, truncation, encoding, decoding, and viewing token details. It serves as a comprehensive reference for all `ttok` functionalities. ```Bash Usage: ttok [OPTIONS] [PROMPT]... Count and truncate text based on tokens To count tokens for text passed as arguments: ttok one two three To count tokens from stdin: cat input.txt | ttok To truncate to 100 tokens: cat input.txt | ttok -t 100 To truncate to 100 tokens using the gpt2 model: cat input.txt | ttok -t 100 -m gpt2 To view token integers: cat input.txt | ttok --encode To convert tokens back to text: ttok 9906 1917 --decode To see the details of the tokens: ttok "hello world" --tokens Outputs: [b'hello', b' world'] Options: --version Show the version and exit. -i, --input FILENAME -t, --truncate INTEGER Truncate to this many tokens -m, --model TEXT Which model to use --encode, --tokens Output token integers --decode Convert token integers to text --tokens Output full tokens --allow-special Do not error on special tokens --help Show this message and exit. ``` -------------------------------- ### Counting Tokens from Command-Line Arguments Source: https://github.com/simonw/ttok/blob/main/README.md This command demonstrates how to count tokens by providing the text directly as arguments to the `ttok` tool. The output '2' indicates that 'Hello world' consists of two tokens. ```bash ttok Hello world ``` -------------------------------- ### Running ttok Unit Tests Source: https://github.com/simonw/ttok/blob/main/README.md This command executes the test suite for the `ttok` project using `pytest`. Running tests is an essential step in the development workflow to ensure that code changes do not introduce regressions and that existing functionalities work as expected. ```Bash pytest ``` -------------------------------- ### Viewing Detailed Token Breakdown (Encoded) Source: https://github.com/simonw/ttok/blob/main/README.md This command combines the `--encode` and `--tokens` options to display a more detailed, byte-level breakdown of how the text is split into individual tokens. It shows the byte representation of each token, providing deeper insight into the tokenization process. ```bash ttok Hello world --encode --tokens ``` -------------------------------- ### Counting Tokens from Piped Input and Additional Arguments Source: https://github.com/simonw/ttok/blob/main/README.md This command shows how to combine piped input with additional text provided as command-line arguments. The `-i -` option instructs `ttok` to read from standard input before processing the subsequent arguments, allowing for flexible text concatenation. ```bash echo -n "Hello world" | ttok more text -i - ``` -------------------------------- ### Counting Tokens with a Specific Model (GPT-2) Source: https://github.com/simonw/ttok/blob/main/README.md This command demonstrates how to specify a different tokenizer model using the `-m` or `--model` option. Here, the `gpt2` model is used, which may result in a different token count (e.g., 6) compared to the default GPT-3.5/4 model for the same text. ```bash ttok boo Hello there this is -m gpt2 ``` -------------------------------- ### Counting Tokens from Piped Input Source: https://github.com/simonw/ttok/blob/main/README.md This command illustrates counting tokens by piping text into the `ttok` tool. The `echo -n` option is crucial to prevent an extra newline character from being counted as a token, ensuring accurate tokenization of the input string. ```bash echo -n "Hello world" | ttok ``` -------------------------------- ### Encoding Text to Token IDs Source: https://github.com/simonw/ttok/blob/main/README.md This command uses the `--encode` option to convert input text into its corresponding integer token IDs. This provides a numerical representation of the text as understood by the tokenizer, useful for understanding how models process text. ```bash ttok Hello world --encode ``` -------------------------------- ### Decoding Token IDs Back to Text Source: https://github.com/simonw/ttok/blob/main/README.md This command reverses the encoding process using the `--decode` option, converting a sequence of integer token IDs back into human-readable text. It's useful for verifying tokenization or reconstructing text from token sequences. ```bash ttok 9906 1917 --decode ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.