### Install count-tokens using pip

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Install the library using pip for quick setup.

```bash
pip install count-tokens
```

--------------------------------

### Install count-tokens from source

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Clone the repository and install from source for development or specific versions.

```bash
git clone https://github.com/izikeros/count_tokens
cd count_tokens
uv sync
```

--------------------------------

### Install count-tokens using uv

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Add the library to your project dependencies using uv.

```bash
uv add count-tokens
```

--------------------------------

### Install count-tokens with pipx

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Install the package globally using pipx for command-line usage.

```sh
pipx install count-tokens
```

--------------------------------

### Markdown Badge Examples

Source: https://github.com/izikeros/count_tokens/blob/main/VISIBILITY_CHECKLIST.md

Examples of Markdown syntax for creating badges commonly used in README files to display project status and information.

```markdown
[![CI](https://github.com/izikeros/count_tokens/actions/workflows/ci.yml/badge.svg)](https://github.com/izikeros/count_tokens/actions/workflows/ci.yml)
```

```markdown
[![codecov](https://codecov.io/gh/izikeros/count_tokens/branch/main/graph/badge.svg)](https://codecov.io/gh/izikeros/count_tokens)
```

```markdown
[![PyPI version](https://badge.fury.io/py/count-tokens.svg)](https://badge.fury.io/py/count-tokens)
```

```markdown
[![Downloads](https://pepy.tech/badge/count-tokens)](https://pepy.tech/project/count-tokens)
```

```markdown
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
```

```markdown
[![Documentation](https://img.shields.io/badge/docs-GitHub%20Pages-blue)](https://izikeros.github.io/count_tokens)
```

--------------------------------

### Run count-tokens with uvx

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Execute the count-tokens command without installation using uvx.

```sh
uvx count-tokens document.txt
```

--------------------------------

### Output token count in JSON format

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Get token counting results for a directory in JSON format.

```sh
count-tokens -d ./docs -p "*.md" --format json
```

--------------------------------

### Use the main count function in Python

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Utilize the primary 'count' function from the Python API to get token count with metadata.

```python
from count_tokens import count

# Returns token count with additional metadata
result = count("myfile.txt")
```

--------------------------------

### Output token count in CSV format

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Get token counting results for a directory in CSV format.

```sh
count-tokens -d ./docs -p "*.md" --format csv
```

--------------------------------

### Quiet token count

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Get only the token count for a file, suppressing other output.

```sh
count-tokens document.txt --quiet
```

--------------------------------

### Approximate token count by characters

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Get an approximate token count based on the number of characters, assuming 4 characters per token.

```sh
count-tokens document.txt --approx c
```

--------------------------------

### Approximate token count by words

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Get an approximate token count based on the number of words, assuming 1.33 tokens per word.

```sh
count-tokens document.txt --approx w
```

--------------------------------

### Development Commands

Source: https://github.com/izikeros/count_tokens/blob/main/AGENTS.md

Run these commands to set up the development environment, run tests, check code style, and build documentation.

```bash
make dev          # Set up development environment
```

```bash
make test         # Run tests
```

```bash
make test-cov     # Run tests with coverage
```

```bash
make lint         # Check code style
```

```bash
make format       # Auto-format code
```

```bash
make type-check   # Run type checker
```

```bash
make security     # Run security checks
```

```bash
make docs         # Build documentation
```

```bash
make serve-docs   # Serve docs locally
```

--------------------------------

### Display CLI help

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

View all available options and commands for the count-tokens CLI.

```bash
count-tokens --help
```

--------------------------------

### Project Structure

Source: https://github.com/izikeros/count_tokens/blob/main/AGENTS.md

Overview of the project's directory structure.

```text
count_tokens/
├── src/count_tokens/    # Source code
├── tests/               # Test files
├── docs/                # Documentation
├── pyproject.toml       # Project configuration
├── Makefile             # Development commands
└── README.md            # Project readme
```

--------------------------------

### Count Tokens in Directory (Default Patterns)

Source: https://context7.com/izikeros/count_tokens/llms.txt

Scans a directory for files matching default patterns (*.txt, *.py, *.md) and counts tokens in each. Handles potential errors by printing the result directly.

```python
from count_tokens.count import count_tokens_in_directory

# Process directory with default patterns (*.txt, *.py, *.md)
results = count_tokens_in_directory("./project")
for file_path, token_count in results.items():
    if isinstance(token_count, int):
        print(f"{file_path}: {token_count} tokens")
    else:
        print(f"{file_path}: {token_count}")  # Error message
```

--------------------------------

### Release Process Commands

Source: https://github.com/izikeros/count_tokens/blob/main/AGENTS.md

Commands to manage project releases by incrementing patch, minor, or major versions.

```bash
make release-patch  # 0.1.0 → 0.1.1
```

```bash
make release-minor  # 0.1.0 → 0.2.0
```

```bash
make release-major  # 0.1.0 → 1.0.0
```

--------------------------------

### Original API - Backward Compatibility

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Provides access to the original functions for backward compatibility, allowing token counting in files and strings with encoding and approximation options.

```APIDOC
## Programmatic Usage: Original API

### Description
The original functions are still available for backward compatibility.

### Methods
- `count_tokens_in_file(file, encoding_name=None, approximate=None, tokens_per_word=None, characters_per_token=None)`
- `count_tokens_in_string(text, encoding_name=None, approximate=None, tokens_per_word=None, characters_per_token=None)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from count_tokens.count import count_tokens_in_file, count_tokens_in_string

# Count tokens in a file
num_tokens = count_tokens_in_file("document.txt")

# Count tokens in a string
num_tokens = count_tokens_in_string("This is a string.")

# Use specific encoding
num_tokens = count_tokens_in_string("This is a string.", encoding_name="cl100k_base")

# Word-based approximation with custom tokens per word ratio
num_tokens = count_tokens_in_file("document.txt", approximate="w", tokens_per_word=1.5)

# Character-based approximation with custom characters per token ratio
num_tokens = count_tokens_in_file("document.txt", approximate="c", characters_per_token=3.5)
```

### Response
#### Success Response (200)
- **num_tokens** (int) - The number of tokens counted.
```

--------------------------------

### Count Tokens in Directory with Custom Patterns

Source: https://context7.com/izikeros/count_tokens/llms.txt

Scans a directory for files matching custom patterns (e.g., *.md, *.rst, *.txt) and calculates the total token count for documentation files.

```python
# Custom file patterns for documentation
results = count_tokens_in_directory(
    "./docs",
    file_patterns=["*.md", "*.rst", "*.txt"]
)
total = sum(c for c in results.values() if isinstance(c, int))
print(f"Total documentation tokens: {total}")
```

--------------------------------

### Simple API - Count Tokens

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Provides a simplified API for token counting operations on strings and files, with options for approximation.

```APIDOC
## Programmatic Usage: Simple API

### Description
The `count` function offers a simplified interface for token counting.

### Method
`count(text=None, file=None, encoding=None, approximate=None, tokens_per_word=None, characters_per_token=None)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from count_tokens import count

# Count tokens in a string
result = count(text="This is a string")

# Count tokens in a file
result = count(file="document.txt", encoding="cl100k_base")

# Count tokens with approximation
result = count(file="document.txt", approximate="w", tokens_per_word=1.5)
```

### Response
#### Success Response (200)
- **result** (int or dict) - The token count or a dictionary containing approximation details.
```

--------------------------------

### Process Directory with File Patterns (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function to process all files in a directory matching specified patterns. Supports recursive searching.

```python
from count_tokens import count

# Process a directory
results = count(
    directory="./docs",
    file_patterns=["*.md", "*.txt"],
    recursive=True
)

# Print results
for file_path, token_count in results.items():
    print(f"{file_path}: {token_count} tokens")
```

--------------------------------

### Count Tokens with Approximation

Source: https://context7.com/izikeros/count_tokens/llms.txt

Uses an approximation method ('w' for words) with a specified tokens-per-word ratio to quickly estimate token counts across markdown files in a directory.

```python
# Use approximation for quick estimates across many files
results = count_tokens_in_directory(
    "./content",
    file_patterns=["*.md"],
    recursive=True,
    approximate="w",
    tokens_per_word=1.4
)
```

--------------------------------

### Check token limit for a file

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Verify if a file's token count exceeds a specified maximum limit.

```sh
count-tokens document.txt --max-tokens 4096
```

--------------------------------

### Process directory with specific patterns

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Count tokens in files within a directory matching specified patterns.

```sh
count-tokens -d ./docs -p "*.md,*.txt"
```

--------------------------------

### Count tokens in a file using CLI

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Basic command to count tokens in a specified file using the CLI.

```bash
count-tokens myfile.txt
```

--------------------------------

### Check token limits

Source: https://context7.com/izikeros/count_tokens/llms.txt

Validate if files exceed a specified token limit, useful for LLM context window constraints.

```bash
# Check if file exceeds 4096 token limit
count-tokens document.txt --max-tokens 4096
# Output when under limit:
# File: document.txt
# Encoding: cl100k_base
# Number of tokens: 2048

# Output when over limit:
# File: document.txt
# Encoding: cl100k_base
# ⚠️ Token limit exceeded: 5120 > 4096
# Number of tokens: 5120
```

--------------------------------

### Count Tokens in a File (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function to count tokens in a file. Specify the file path and optionally the encoding.

```python
from count_tokens import count

# Count tokens in a file
result = count(file="document.txt", encoding="cl100k_base")
```

--------------------------------

### Stream Large Files for Token Counting (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function with 'use_streaming=True' to process large files without loading them entirely into memory. Define chunk size as needed.

```python
from count_tokens import count

# Process a large file with streaming
tokens = count(
    file="large_dataset.txt", 
    use_streaming=True,
    chunk_size=1024*1024  # 1MB chunks
)
```

--------------------------------

### Process directory recursively

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Count tokens in files within a directory and its subdirectories, matching a pattern.

```sh
count-tokens -d ./project -r -p "*.py"
```

--------------------------------

### Basic token count

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Count tokens in a single file using the default encoding.

```sh
count-tokens document.txt
```

--------------------------------

### Count tokens in a file using Python API

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Use the Python API to count tokens in a file and print the result.

```python
from count_tokens import count_tokens_in_file

num_tokens = count_tokens_in_file("myfile.txt")
print(f"File contains {num_tokens} tokens")
```

--------------------------------

### Count Tokens in File with Various Methods

Source: https://context7.com/izikeros/count_tokens/llms.txt

Use `count_tokens_in_file` for file token counting. Supports default encoding, specific encodings like 'o200k_base', and fast approximation methods using word ('w') or character ('c') counts. Custom approximation ratios can be provided.

```python
from count_tokens import count_tokens_in_file

# Basic file token counting with default encoding
num_tokens = count_tokens_in_file("document.txt")
print(f"File contains {num_tokens} tokens")
```

```python
# Specify encoding for different model families
num_tokens = count_tokens_in_file("document.txt", encoding_name="o200k_base")
print(f"GPT-4o tokens: {num_tokens}")
```

```python
# Fast word-based approximation (default: 1.33 tokens per word)
approx_tokens = count_tokens_in_file("document.txt", approximate="w")
print(f"Approximate tokens (words): {approx_tokens}")
```

```python
# Character-based approximation (default: 4 characters per token)
approx_tokens = count_tokens_in_file("document.txt", approximate="c")
print(f"Approximate tokens (chars): {approx_tokens}")
```

```python
# Custom approximation ratios for domain-specific content
approx_tokens = count_tokens_in_file(
    "technical_doc.txt",
    approximate="w",
    tokens_per_word=1.5  # Technical content may have more tokens per word
)
print(f"Technical doc tokens: {approx_tokens}")
```

```python
# Custom character ratio for different languages
approx_tokens = count_tokens_in_file(
    "chinese_text.txt",
    approximate="c",
    characters_per_token=2.0  # CJK characters often use fewer chars per token
)
```

--------------------------------

### Check Token Limits (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function with 'max_tokens' to check if content exceeds a specified token limit. The result indicates if the limit was exceeded.

```python
from count_tokens import count

# Check if a file exceeds token limit
result = count(file="document.txt", max_tokens=4096)

if isinstance(result, dict) and result.get("limit_exceeded"):
    print(f"⚠️ Token limit exceeded: {result['tokens']} > {result['max_tokens']}")
```

--------------------------------

### Word-Based Approximation (Original API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the original count_tokens_in_file function with approximate='w' and tokens_per_word to count tokens based on word approximation.

```python
# Word-based approximation with custom tokens per word ratio
num_tokens = count_tokens_in_file("document.txt", approximate="w", tokens_per_word=1.5)
```

--------------------------------

### Approximate token counts

Source: https://context7.com/izikeros/count_tokens/llms.txt

Use faster word or character-based approximation methods for token estimates instead of exact tokenization.

```bash
# Approximate using word count (default: 1.33 tokens per word)
count-tokens document.txt --approx w
# Output:
# File: document.txt
# Encoding: cl100k_base
# Approximation method: Words (tokens per word: 1.3333333333333333)
# Number of tokens: 89
```

```bash
# Approximate using character count (default: 4 characters per token)
count-tokens document.txt --approx c
```

```bash
# Custom tokens-per-word ratio for specific content types
count-tokens document.txt --approx w --tokens-per-word 1.5
```

```bash
# Custom characters-per-token ratio
count-tokens document.txt --approx c --characters-per-token 3.5
```

--------------------------------

### Process files in a directory

Source: https://context7.com/izikeros/count_tokens/llms.txt

Scan directories for files matching glob patterns, optionally recursively. Provides individual and total token counts.

```bash
# Count tokens in all markdown and text files in ./docs
count-tokens -d ./docs -p "*.md,*.txt"
```

```bash
# Process Python files recursively in a project
count-tokens -d ./project -r -p "*.py"
# Output:
# ./project/main.py: 245 tokens
# ./project/utils/helper.py: 128 tokens
# ./project/tests/test_main.py: 312 tokens
#
# Total: 685 tokens across 3 files
```

```bash
# Use default patterns (*.txt, *.py, *.md) without -p flag
count-tokens -d ./src -r
```

--------------------------------

### Adjust tokens per word ratio for approximation

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Customize the ratio of tokens per word used in approximation calculations.

```sh
count-tokens document.txt --approx w --tokens-per-word 1.5
```

--------------------------------

### Unified Token Counting API

Source: https://context7.com/izikeros/count_tokens/llms.txt

The `count` function provides a unified API for counting tokens in strings, files, or directories. It supports streaming for large files, approximation methods, and token limit checking. Directory processing can use glob patterns and recursive searching.

```python
from count_tokens import count

# Count tokens in a string
result = count(text="This is a simple text to count")
print(f"String tokens: {result}")  # Output: String tokens: 7
```

```python
# Count tokens in a file with custom encoding
result = count(file="document.txt", encoding="cl100k_base")
print(f"File tokens: {result}")
```

```python
# Count tokens with word-based approximation
result = count(file="large_doc.txt", approximate="w", tokens_per_word=1.4)
print(f"Approximate tokens: {result}")
```

```python
# Stream large files to avoid memory issues
result = count(
    file="very_large_file.txt",
    use_streaming=True,
    chunk_size=2 * 1024 * 1024  # 2MB chunks
)
print(f"Large file tokens: {result}")
```

```python
# Process entire directory with glob patterns
results = count(
    directory="./docs",
    file_patterns=["*.md", "*.txt", "*.rst"],
    recursive=True
)
for file_path, token_count in results.items():
    print(f"{file_path}: {token_count} tokens")
```

```python
# Check token limits - returns dict when limit exceeded
result = count(file="document.txt", max_tokens=4096)
if isinstance(result, dict) and result.get("limit_exceeded"):
    print(f"Warning: {result['tokens']} tokens exceeds limit of {result['max_tokens']}")
else:
    print(f"File is within limit: {result} tokens")
```

```python
# Directory processing with token limits
results = count(
    directory="./content",
    file_patterns=["*.md"],
    max_tokens=8000
)
for file_path, result in results.items():
    if isinstance(result, dict) and result.get("limit_exceeded"):
        print(f"⚠️ {file_path}: {result['tokens']} tokens (exceeds {result['max_tokens']})")
    elif isinstance(result, int):
        print(f"✓ {file_path}: {result} tokens")
    else:
        print(f"✗ {file_path}: {result}")  # Error message
```

--------------------------------

### Character-Based Approximation (Original API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the original count_tokens_in_file function with approximate='c' and characters_per_token to count tokens based on character approximation.

```python
# Character-based approximation with custom characters per token ratio
num_tokens = count_tokens_in_file("document.txt", approximate="c", characters_per_token=3.5)
```

--------------------------------

### Approximate Token Count by Words (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function with the 'approximate' option set to 'w' for word-based approximation. Specify the tokens_per_word ratio.

```python
from count_tokens import count

# Count tokens with approximation
result = count(file="document.txt", approximate="w", tokens_per_word=1.5)
```

--------------------------------

### Stream large file token count

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use streaming mode to count tokens in large files without consuming excessive memory.

```sh
count-tokens large_file.txt --stream
```

--------------------------------

### Count tokens in a file

Source: https://context7.com/izikeros/count_tokens/llms.txt

Use the `count-tokens` command to count tokens in a single file. Specify the encoding for different OpenAI models.

```bash
# Count tokens in a file with default cl100k_base encoding
count-tokens document.txt
# Output:
# File: document.txt
# Encoding: cl100k_base
# Number of tokens: 67
```

```bash
# Get only the token count (quiet mode)
count-tokens document.txt --quiet
# Output:
# 67
```

```bash
# Use a different encoding for GPT-4o models
count-tokens document.txt --encoding o200k_base
# Output:
# File: document.txt
# Encoding: o200k_base
# Number of tokens: 72
```

```bash
# Use p50k_base encoding for Codex models
count-tokens document.txt -e p50k_base
```

--------------------------------

### Count Tokens Recursively with Streaming

Source: https://context7.com/izikeros/count_tokens/llms.txt

Performs a recursive scan of a directory for Python files, using streaming mode with a specified chunk size for efficient handling of large codebases.

```python
# Recursive scan with streaming for large codebases
results = count_tokens_in_directory(
    "./src",
    file_patterns=["*.py"],
    recursive=True,
    use_streaming=True,
    chunk_size=1024 * 1024
)
print(f"Found {len(results)} Python files")
print(f"Total tokens: {sum(c for c in results.values() if isinstance(c, int))}")
```

--------------------------------

### Count tokens in a string using Python API

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Use the Python API to count tokens in a given string and print the result.

```python
from count_tokens import count_tokens_in_string

text = "Hello, world! This is a test."
num_tokens = count_tokens_in_string(text)
print(f"String contains {num_tokens} tokens")
```

--------------------------------

### Format token count output

Source: https://context7.com/izikeros/count_tokens/llms.txt

Obtain token counts in JSON or CSV formats for integration with other tools.

```bash
# JSON output for directory processing
count-tokens -d ./docs -p "*.md" --format json
# Output:
# {
#   "./docs/readme.md": 245,
#   "./docs/api.md": 512,
#   "./docs/guide.md": 328
# }
```

```bash
# CSV output for spreadsheet import
count-tokens -d ./docs -p "*.md" --format csv
# Output:
# file,tokens
# ./docs/readme.md,245
# ./docs/api.md,512
# ./docs/guide.md,328
```

--------------------------------

### Count tokens in a file using Python API

Source: https://github.com/izikeros/count_tokens/blob/main/docs/index.md

Import and use the `count_tokens_in_file` function from the count_tokens library to count tokens in a file programmatically.

```python
from count_tokens import count_tokens_in_file, count_tokens_in_string

# Count tokens in a file
num_tokens = count_tokens_in_file("myfile.txt")
```

--------------------------------

### Specify encoding for token counting CLI

Source: https://github.com/izikeros/count_tokens/blob/main/docs/getting-started.md

Count tokens in a file using a specific encoding, such as cl100k_base.

```bash
count-tokens myfile.txt --encoding cl100k_base
```

--------------------------------

### Count Tokens in a String (Simple API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the simplified count function to count tokens in a given string. No specific encoding is required by default.

```python
from count_tokens import count

# Count tokens in a string
result = count(text="This is a string")
```

--------------------------------

### Stream large file with custom chunk size

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Customize the chunk size for streaming large files, defaulting to 1MB.

```sh
count-tokens large_file.txt --stream --chunk-size 2097152
```

--------------------------------

### Specify encoding for token count

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Count tokens using a specific encoding like 'r50k_base'.

```sh
count-tokens document.txt -e r50k_base
```

--------------------------------

### Count Tokens in a File (Original API)

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the original count_tokens_in_file function for backward compatibility. This function counts tokens in a specified file.

```python
from count_tokens.count import count_tokens_in_file, count_tokens_in_string

# Count tokens in a file
num_tokens = count_tokens_in_file("document.txt")
```

--------------------------------

### Check Token Limits API

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Enables checking if content exceeds a specified token limit.

```APIDOC
## Programmatic Usage: Check Token Limits

### Description
Check if content exceeds token limits.

### Method
`count(file=None, text=None, max_tokens=None)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from count_tokens import count

# Check if a file exceeds token limit
result = count(file="document.txt", max_tokens=4096)

if isinstance(result, dict) and result.get("limit_exceeded"):
    print(f"⚠️ Token limit exceeded: {result['tokens']} > {result['max_tokens']}")
```

### Response
#### Success Response (200)
- **result** (dict) - A dictionary containing token count and limit status. Example: `{"tokens": 5000, "max_tokens": 4096, "limit_exceeded": true}`.
```

--------------------------------

### Directory Processing API

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Enables processing of multiple files within a directory based on specified patterns.

```APIDOC
## Programmatic Usage: Directory Processing

### Description
Process all files in a directory that match specific patterns, optionally recursively.

### Method
`count(directory, file_patterns, recursive=False)`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
from count_tokens import count

# Process a directory
results = count(
    directory="./docs",
    file_patterns=["*.md", "*.txt"],
    recursive=True
)

# Print results
for file_path, token_count in results.items():
    print(f"{file_path}: {token_count} tokens")
```

### Response
#### Success Response (200)
- **results** (dict) - A dictionary where keys are file paths and values are their token counts.
```

--------------------------------

### Adjust Characters Per Token Ratio

Source: https://github.com/izikeros/count_tokens/blob/main/README.md

Use the --approx c option to fine-tune token approximation based on character count. The default ratio is 4.0 characters per token.

```bash
count-tokens document.txt --approx c --characters-per-token 3.5
```

--------------------------------

### Count tokens in a string using Python API

Source: https://github.com/izikeros/count_tokens/blob/main/docs/index.md

Import and use the `count_tokens_in_string` function from the count_tokens library to count tokens in a string programmatically.

```python
from count_tokens import count_tokens_in_file, count_tokens_in_string

# Count tokens in a string
num_tokens = count_tokens_in_string("Hello, world!")
```

--------------------------------

### Count Tokens in Large Files with Streaming

Source: https://context7.com/izikeros/count_tokens/llms.txt

Use `count_tokens_in_large_file` to stream large files in chunks aligned to newline boundaries, ensuring accurate token counting without loading the entire file into memory. It supports custom chunk sizes, encodings, and approximation methods.

```python
from count_tokens.count import count_tokens_in_large_file

# Process a large file with default settings (1MB chunks)
total_tokens = count_tokens_in_large_file("large_dataset.txt")
print(f"Total tokens: {total_tokens}")
```

```python
# Custom chunk size for very large files
total_tokens = count_tokens_in_large_file(
    "massive_file.txt",
    encoding_name="cl100k_base",
    chunk_size=4 * 1024 * 1024  # 4MB chunks
)
print(f"Tokens in massive file: {total_tokens}")
```

```python
# Use approximation for even faster processing of huge files
approx_tokens = count_tokens_in_large_file(
    "huge_file.txt",
    approximate="w",
    tokens_per_word=1.33
)
print(f"Approximate tokens: {approx_tokens}")
```

--------------------------------

### Process File with Mixed Encoding

Source: https://context7.com/izikeros/count_tokens/llms.txt

Processes a file that may have mixed encoding issues, with an automatic fallback to latin-1.

```python
tokens = count_tokens_in_large_file("mixed_encoding_file.txt")
print(f"Tokens: {tokens}")
```

--------------------------------

### Count Tokens in String with Different Encodings

Source: https://context7.com/izikeros/count_tokens/llms.txt

Use `count_tokens_in_string` to count tokens in a string. Specify `encoding_name` for different model families like 'p50k_base' or 'o200k_base'. An empty string returns 0 tokens. Special characters are tokenized correctly.

```python
code_snippet = "def hello():\n    print('Hello, World!')"
num_tokens = count_tokens_in_string(code_snippet, encoding_name="p50k_base")
print(f"Code tokens: {num_tokens}")  # Output: Code tokens: 15
```

```python
num_tokens = count_tokens_in_string(text, encoding_name="o200k_base")
print(f"GPT-4o tokens: {num_tokens}")
```

```python
empty_count = count_tokens_in_string("")
print(f"Empty string tokens: {empty_count}")  # Output: Empty string tokens: 0
```

```python
special_text = "Special chars: !@#$%^&*()_+{}|:<>?"
num_tokens = count_tokens_in_string(special_text)
print(f"Special char tokens: {num_tokens}")
```

--------------------------------

### Stream large files for token counting

Source: https://context7.com/izikeros/count_tokens/llms.txt

Process large files by streaming chunks to avoid memory issues. Chunking is aligned to newlines for accuracy.

```bash
# Stream a large file with default 1MB chunks
count-tokens large_file.txt --stream
# Output:
# File: large_file.txt
# Encoding: cl100k_base
# Number of tokens: 1254893
```

```bash
# Customize chunk size (2MB) for better performance on very large files
count-tokens large_file.txt --stream --chunk-size 2097152
```