### Developer Installation for TokenDagger

Source: https://github.com/m4thyou/tokendagger/blob/main/README.md

Steps for cloning the repository, installing PCRE2 development libraries, updating git submodules, and installing Python development headers. Includes optional installation of 'tiktoken' for running tests.

```bash
git clone git@github.com:M4THYOU/TokenDagger.git
sudo apt install libpcre2-dev
git submodule update --init --recursive
sudo apt update && sudo apt install -y python3-dev

And optionally for running the tests:
pip3 install tiktoken
```

--------------------------------

### Install TokenDagger Package

Source: https://github.com/m4thyou/tokendagger/blob/main/README.md

Install the TokenDagger library using pip. This is the primary method for integrating TokenDagger into your Python projects.

```bash
pip install tokendagger
```

--------------------------------

### Run Tests for TokenDagger

Source: https://github.com/m4thyou/tokendagger/blob/main/README.md

Execute the test suite to verify TokenDagger's functionality against Tiktoken. Supports different tokenizers like llama and mistral. Also includes performance benchmarks for code tokenization.

```bash
make clean && make
pip3 install tiktoken
python3 tests/test_tokendagger_vs_tiktoken.py --tokenizer llama
python3 tests/test_tokendagger_vs_tiktoken.py --tokenizer mistral
python3 tests/performance_benchmark.py --tokenizer llama
python3 tests/performance_benchmark.py --tokenizer mistral
python3 tests/code_performance_benchmark.py --tokenizer llama
```

--------------------------------

### Replace Tiktoken Import with TokenDagger

Source: https://github.com/m4thyou/tokendagger/blob/main/README.md

Demonstrates how to replace the standard 'tiktoken' import with 'tokendagger'. This allows for a seamless drop-in replacement in existing codebases.

```python
# import tiktoken --- Remove this line!
import tokendagger as tiktoken

...

tokenizer = tiktoken.Encoding(
    name=name,
    pat_str=pattern,
    mergeable_ranks=vocab,
    special_tokens=special_tokens,
)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.