### Setup Virtual Environment and Install Dependencies

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Create a Python virtual environment and install necessary libraries for weight table generation.

```sh
python3 -m venv .venv
.venv/bin/pip install datasets numpy
```

--------------------------------

### Syntext Agent Harness Quick Installs

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Quick installation commands for Syntext agent harnesses, including global hooks and project-specific setups for various AI coding assistants.

```bash
# Claude Code project instructions only
st init
```

```bash
# Claude Code global Bash hook plus Grep blocker
st init -g
```

```bash
# RTK-style agent selectors
st init -g --agent cursor
```

```bash
st init -g --gemini
```

```bash
st --copilot        # project hook; `st init -g --copilot` is also accepted
```

```bash
st init --codex          # project rules
```

```bash
st init -g --codex       # global Codex rules
```

--------------------------------

### Manual Linux Installation (Generic Binary)

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Downloads and installs a Syntext binary for Linux (x86_64 or arm64) and places it in `/usr/local/bin`.

```bash
ARCH=amd64   # or arm64
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/st-${VERSION}-linux-${ARCH}" -o st
chmod +x st && sudo mv st /usr/local/bin/
```

--------------------------------

### Manual Linux Installation (Debian/Ubuntu)

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Downloads and installs the Syntext `.deb` package for Debian/Ubuntu systems.

```bash
VERSION=1.2.0

# Debian/Ubuntu (x86_64)
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/syntext_${VERSION}_amd64.deb" \
  -o "syntext_${VERSION}_amd64.deb"
sudo dpkg -i "syntext_${VERSION}_amd64.deb"
```

--------------------------------

### Customizing Installation with Environment Variables

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Allows overriding default installation paths and specifying a Syntext version during installation via environment variables.

```bash
INSTALL_DIR=~/.local/bin SYNTEXT_VERSION=1.2.0 \
  curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

--------------------------------

### Install Syntext via Homebrew (macOS)

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs Syntext using Homebrew by first tapping the repository and then installing the cask.

```bash
brew tap whit3rabbit/tap
brew install --cask whit3rabbit/tap/syntext
```

--------------------------------

### Syntext Installation

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Command to download and execute the Syntext installation script.

```sh
curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

--------------------------------

### Install Syntext on Windows via PowerShell

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs the Syntext executable (`st.exe`) to `%LOCALAPPDATA%\syntext` and adds it to the user's PATH.

```powershell
iwr -useb https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.ps1 | iex
```

--------------------------------

### Install Dependencies and Detect GPU

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Installs necessary Python packages and attempts to detect and configure a GPU with CuPy. Falls back to CPU if no GPU is available or CuPy fails.

```python
!pip install -q datasets huggingface_hub numpy pyarrow

import shutil, subprocess

GPU = False
if shutil.which("nvidia-smi"):
    try:
        !pip install -q cupy-cuda12x 2>/dev/null
        import cupy as cp
        name = cp.cuda.runtime.getDeviceProperties(0)("name").decode()
        mem_gb = cp.cuda.Device(0).mem_info[1] / 1e9
        GPU = True
        print(f"GPU: {name} ({mem_gb:.0f} GB)")
    except Exception as e:
        print(f"GPU detected but CuPy failed: {e}")
        print("Falling back to CPU (still fast with bulk download).")

if not GPU:
    print("No GPU. Using CPU with np.bincount (still 10-50x faster than streaming).")
```

--------------------------------

### Python script prerequisites and authentication

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Set up a Python virtual environment and install necessary libraries. Authenticate with Hugging Face to access datasets.

```shell
python3 -m venv .venv
.venv/bin/pip install datasets numpy
hf auth login
```

--------------------------------

### macOS Corpus Setup for Benchmarking

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Git clone commands to set up shallow clones of various repositories for benchmarking on macOS.

```sh
mkdir -p ./_syntext-bench
git clone --depth 1 -b v6.8 https://github.com/torvalds/linux.git ./_syntext-bench/linux
git clone --depth 1 -b 1.77.0 https://github.com/rust-lang/rust.git ./_syntext-bench/rust
git clone --depth 1 -b v18.2.0 https://github.com/facebook/react.git ./_syntext-bench/react
git clone --depth 1 -b v5.4.3 https://github.com/microsoft/TypeScript.git ./_syntext-bench/typescript
git clone --depth 1 -b v20.12.0 https://github.com/nodejs/node.git ./_syntext-bench/node
```

--------------------------------

### Install Syntext from Source using Cargo

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs the Syntext crate directly from source using the Rust package manager, Cargo.

```bash
cargo install syntext
```

--------------------------------

### Parse Query Example

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/edge_cases/nested/deep/path/file.txt

Illustrates a function call for parsing a query string. This is a basic example of interacting with the parsing functionality.

```python
parse_query("deeply nested content")
```

--------------------------------

### Syntext Agent Harness Explicit Install/Show/Uninstall

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Explicit commands for installing, showing, and uninstalling specific agent harnesses globally.

```bash
st agent install claude --global
```

```bash
st agent show claude --global
```

```bash
st agent uninstall claude --global
```

--------------------------------

### Restart from Scratch Command

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

A shell command to remove the existing checkpoint file, forcing the process to start from the beginning. Use this if you need to regenerate all weights without resuming.

```shell
!rm /content/bigram_checkpoint_v2.npz
```

--------------------------------

### Running Windows Install Script with Specific Execution Policy

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Allows running the Syntext PowerShell install script with a bypassed execution policy, useful for pinned versions or saved scripts.

```powershell
powershell -ExecutionPolicy Bypass -File install.ps1
```

--------------------------------

### Clone Git Repositories for macOS Benchmarks

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Shallow clones of large repositories like Linux, Rust, React, TypeScript, and Node.js are sufficient for search benchmarks on macOS. Ensure you have Git installed.

```shell
mkdir -p ./_syntext-bench
git clone --depth 1 -b v6.8 https://github.com/torvalds/linux.git ./_syntext-bench/linux
git clone --depth 1 -b 1.77.0 https://github.com/rust-lang/rust.git ./_syntext-bench/rust
git clone --depth 1 -b v18.2.0 https://github.com/facebook/react.git ./_syntext-bench/react
git clone --depth 1 -b v5.4.3 https://github.com/microsoft/TypeScript.git ./_syntext-bench/typescript
git clone --depth 1 -b v20.12.0 https://github.com/nodejs/node.git ./_syntext-bench/node
```

--------------------------------

### Syntext CLI Filtering and Output Modes

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Examples of using the Syntext CLI to filter search results by file type, glob pattern, specific files, and to control output format.

```bash
st -t rs "impl.*Iterator"           # restrict to Rust files
```

```bash
st -g "src/" "TODO"                 # restrict by glob
```

```bash
st -c "parse_query" src/lib.rs      # count matches in one file
```

```bash
st -l "parse_query"                 # print matching file paths
```

```bash
st --json "TODO"                    # NDJSON output for tooling
```

--------------------------------

### List Available Presets in Comparison Harness

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

List all available presets for the external repository comparison harness. This helps in selecting appropriate configurations for benchmarking.

```shell
python3 scripts/bench_compare.py --list-presets
```

--------------------------------

### Build Syntext Index

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Creates the initial search index for a repository. This should be run once per repository or after significant changes. The index is stored in the `.syntext/` directory.

```bash
st index
```

--------------------------------

### Rust Function to Parse Query

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/edge_cases/crlf_endings.txt

Parses a string slice into a Query struct. This function is part of the test setup.

```rust
fn parse_query(s: &str) -> Query {
    // TODO: validate input
    Query { raw: s.to_string() }
}
```

--------------------------------

### Run Preset Benchmarks

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Execute predefined benchmark configurations for various programming languages and applications.

```sh
python3 scripts/bench_compare.py --preset zed_mixed_app
python3 scripts/bench_compare.py --preset react_token_aligned
python3 scripts/bench_compare.py --preset rust_token_aligned
python3 scripts/bench_compare.py --preset linux_token_aligned
python3 scripts/bench_compare.py --preset typescript_compiler
python3 scripts/bench_compare.py --preset node_runtime
```

--------------------------------

### Optimize Benchmark Runs for Large Repositories

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Reduce build and search iterations to 1 and set warmups to 0 for benchmarking very large repositories or large-corpus presets. This focuses the benchmark on core performance without excessive repetition.

```shell
python3 scripts/bench_compare.py \
  --preset linux_token_aligned \
  --repo _syntext-bench/linux \
  --build-iterations 1 \
  --search-iterations 1 \
  --warmups 0
```

--------------------------------

### Run a Specific Preset in Comparison Harness

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Execute a benchmark run using a predefined preset, such as 'react_token_aligned', against a specified repository. This allows for consistent and repeatable benchmark tests.

```shell
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react
```

--------------------------------

### Compare Syntext Search Modes (Fork and Persistent)

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Report both 'fork' (default CLI-style) and 'persistent' (in-process) Syntext search modes side-by-side in a single run. This highlights the performance gap between cold and hot searches.

```shell
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --syntext-search-mode both
```

--------------------------------

### Generate weights using Python script with Hugging Face datasets

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Execute the Python script to generate weights, specifying the output file path. This method uses the 'bigcode/the-stack-smol' dataset via streaming.

```shell
HF_DATASETS_CACHE=/tmp/hf_cache .venv/bin/python scripts/weights_gen.py \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Run External Repository Comparison Harness

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Utilize the Python harness to compare Syntext against rg and grep on a real repository. This tests end-to-end performance for specified queries.

```shell
python3 scripts/bench_compare.py \
  --repo /path/to/repo \
  --query literal:token_aligned_symbol \
  --query literal:another_symbol
```

--------------------------------

### Generate Weight Table from HuggingFace Dataset

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Generate the weights.rs file using the 'the-stack-smol' dataset from HuggingFace. Ensure HF_DATASETS_CACHE is set for temporary storage.

```sh
HF_DATASETS_CACHE=/tmp/hf_cache .venv/bin/python scripts/weights_gen.py \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Build Syntext Index with Custom Index Directory

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Builds the Syntext index and stores it in a specified directory, rather than the default `.syntext/`.

```bash
st --index-dir /tmp/my-index index
```

--------------------------------

### Download Weights File from Colab

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Provides functionality to download the generated Rust weights file from Google Colab. Includes a fallback to print the file path if the `files` module is not available.

```python
try:
    from google.colab import files
    if os.path.exists(OUTPUT_PATH):
        files.download(OUTPUT_PATH)
except ImportError:
    print(f"File at: {OUTPUT_PATH}")
```

--------------------------------

### External Repository Comparison Harness

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Python scripts for comparing Syntext performance against ripgrep and grep on external repositories.

```sh
python3 scripts/bench_compare.py --preset react_token_aligned
python3 scripts/bench_compare.py --repo /path/to/repo --query literal:symbol
```

--------------------------------

### Generate weights using Python script with local directories

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Use the Python script to generate weights from local directories containing code. This is an alternative to using Hugging Face datasets.

```shell
.venv/bin/python scripts/weights_gen.py \
  --source local \
  --dirs ~/projects/linux ~/projects/rustc ~/projects/cpython \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Import Libraries and Define Configuration

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Imports essential libraries for data processing and defines configuration parameters including target data size, dataset name, language weights, file size limits, processing chunk size, download directory, checkpoint path, output path, and boundary threshold.

```python
import numpy as np
import pyarrow.parquet as pq
import os, time, logging, warnings
from pathlib import Path

logging.getLogger("datasets.load").setLevel(logging.CRITICAL)
warnings.filterwarnings("ignore", message=".*trust_remote_code.*")

# ── Target ─────────────────────────────────────────────────────────────────
# 100 GB recommended. At bulk download speeds, takes ~20-30 min.
# For a quick test: 2_000_000_000 (2 GB, ~2 min).
TARGET_BYTES = 100 * 1_000_000_000

DATASET_NAME = "bigcode/the-stack-dedup"

# the-stack-dedup shard names. C++ is "cpp" not "c++".
LANGUAGES = {
    # Tier 1 — agent-heavy
    "python": 2.0, "javascript": 2.0, "typescript": 2.0, "rust": 2.0,
    "go": 1.5, "java": 1.5, "c": 1.5, "cpp": 1.5,
    # Tier 2 — broader coverage
    "c-sharp": 1.0, "ruby": 1.0, "php": 1.0,
    "swift": 0.8, "kotlin": 0.8,
    "scala": 0.7, "shell": 0.7,
    "lua": 0.5, "haskell": 0.5, "r": 0.5,
    "perl": 0.4, "dart": 0.4,
    "elixir": 0.3, "julia": 0.3, "ocaml": 0.3, "zig": 0.3,
}

MAX_FILE_BYTES = 500_000       # Skip files > 500 KB (minified JS, generated)
PROCESS_CHUNK_MB = 256         # Max MB to concat before counting (bounds RAM)
DOWNLOAD_DIR = "/content/hf_shard"  # Temp dir for each shard, deleted after
CHECKPOINT_PATH = "/content/bigram_checkpoint_v2.npz"
OUTPUT_PATH = "/content/weights.rs"
BOUNDARY_THRESHOLD = 28000     # syntext's gram boundary threshold

print(f"Target:  {TARGET_BYTES / 1e9:.0f} GB | {len(LANGUAGES)} languages")
print(f"Dataset: {DATASET_NAME}")
print(f"GPU:     {'Yes (' + str(GPU) + ')' if GPU else 'No (CPU)'}")
print(f"Chunk:   {PROCESS_CHUNK_MB} MB per batch")
```

--------------------------------

### Build Syntext Index with Stats

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Builds the Syntext index and displays statistics about the created index, such as file count and size.

```bash
st index --stats
```

--------------------------------

### Build Syntext WASM from Source

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Builds a WebAssembly version of Syntext using `wasm-pack`. The output is suitable for bundlers.

```bash
cargo install wasm-pack
wasm-pack build --target bundler -- --features wasm --no-default-features
```

--------------------------------

### Generate Weight Table from Local Code Directories

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Generate the weights.rs file by analyzing code from specified local directories. This is an alternative to using HuggingFace datasets.

```sh
.venv/bin/python scripts/weights_gen.py \
  --source local \
  --dirs ~/projects/linux ~/projects/rustc ~/projects/cpython \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Syntext Development Commands

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Common commands for running tests, linters, and benchmarks for Syntext development.

```sh
cargo test                    # unit + integration tests
cargo clippy                  # lint, must pass with no warnings
cargo bench                   # criterion benchmarks
SYNTEXT_LOG_SELECTIVITY=1 cargo test --test correctness -- --nocapture
                              # show per-query selectivity stats
cargo test --test tokenizer   # unit: tokenizer only
cargo test --test posting     # unit: posting lists only
cargo test --test query       # unit: query router only
cargo test --test overlay     # unit: overlay/snapshot only
cargo test --test boundary_fuzz  # unit: boundary fuzzing
cargo test --test index_build # integration: index construction
cargo test --test incremental # integration: incremental updates
cargo test --test symbols     # integration: symbol index
cargo bench --bench query_latency -- --sample-size 10
cargo bench --bench index_build -- --sample-size 10
cargo bench --bench selectivity -- --sample-size 10
```

--------------------------------

### Copy generated weights and run tests/benchmarks

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

After generating the weights.rs file using the Colab notebook, copy it to the project's source directory and then run cargo test and benchmarks to ensure integration.

```shell
cp ~/Downloads/weights.rs src/tokenizer/weights.rs
cargo test
cargo bench --bench query_latency -- --sample-size 10
```

--------------------------------

### Main Processing Loop

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Iterates through languages and their respective shards, downloading, processing, and cleaning up each shard. It resumes from the last checkpoint if available and saves a new checkpoint after each shard is processed.

```python
# ── Resume or start fresh ──────────────────────────────────────────────────
counts, total_bytes, lang_bytes_done, lang_shard_done = load_checkpoint(CHECKPOINT_PATH)
if total_bytes > 0:
    pct = np.count_nonzero(counts) / 65536 * 100
    print(f"Resumed: {total_bytes / 1e9:.2f} GB, {pct:.1f}% pair coverage")
else:
    print("Starting fresh.")

# Per-language byte targets.
total_weight = sum(LANGUAGES.values())
lang_targets = {
    lang: int(TARGET_BYTES * w / total_weight)
    for lang, w in LANGUAGES.items()
}

wall_start = time.time()
total_files = 0
total_skipped = 0

for lang, byte_target in lang_targets.items():
    already = lang_bytes_done.get(lang, 0)
    if already >= byte_target:
        print(f"  {lang}: done ({already / 1e9:.2f} GB)")
        continue

    shards = lang_shards.get(lang, [])
    start_shard = lang_shard_done.get(lang, 0)
    if start_shard >= len(shards):
        print(f"  {lang}: all {len(shards)} shards processed ({already / 1e9:.2f} GB)")
        continue

    lang_bytes_local = already
    remaining = byte_target - already
    print(f"  {lang}: {remaining / 1e9:.2f} GB remaining, "
          f"shards {start_shard}–{len(shards)-1} of {len(shards)}")

    for shard_idx in range(start_shard, len(shards)):
        if lang_bytes_local >= byte_target:
            break

        shard_name = shards[shard_idx]
        shard_remaining = byte_target - lang_bytes_local

        try:
            # Download.
            t0 = time.time()
            local_path = hf_hub_download(
                DATASET_NAME,
                f"data/{lang}/{shard_name}",
                repo_type="dataset",
                local_dir=DOWNLOAD_DIR,
            )
            dl_sec = time.time() - t0
            dl_mb = os.path.getsize(local_path) / 1e6

            # Process.
            t1 = time.time()
            shard_counts, shard_bytes, shard_files, shard_skipped = process_parquet_shard(
                local_path, MAX_FILE_BYTES, PROCESS_CHUNK_MB, GPU,
                byte_limit=shard_remaining,
            )
            proc_sec = time.time() - t1

            counts += shard_counts
            total_bytes += shard_bytes
            lang_bytes_local += shard_bytes
            total_files += shard_files
            total_skipped += shard_skipped

        except Exception as e:
            print(f"    shard {shard_idx} error: {e}")
            continue
        finally:
            # Always clean up disk.
            shutil.rmtree(DOWNLOAD_DIR, ignore_errors=True)

        # Checkpoint after every shard.
        lang_bytes_done[lang] = lang_bytes_local
        lang_shard_done[lang] = shard_idx + 1
        save_checkpoint(CHECKPOINT_PATH, counts, total_bytes, lang_bytes_done, lang_shard_done)

        # Progress.
        elapsed = time.time() - wall_start
        pct = total_bytes / TARGET_BYTES * 100
        coverage = np.count_nonzero(counts) / 65536 * 100
        rate = total_bytes / elapsed / 1e6 if elapsed > 0 else 0
        eta = (TARGET_BYTES - total_bytes) / (total_bytes / elapsed) / 60 if total_bytes > 0 else 0
        print(
            f"    [{shard_idx+1}/{len(shards)}] "
            f"{dl_mb:.0f}MB dl:{dl_sec:.1f}s proc:{proc_sec:.1f}s | "
            f"{total_bytes / 1e9:.1f}GB ({pct:.0f}%) | "
            f"{rate:.0f} MB/s | ETA {eta:.0f}m | "
            f"{coverage:.1f}% coverage"
        )

    print(f"    {lang} done: {lang_bytes_local / 1e9:.2f} GB, {total_files:,} files")
```

--------------------------------

### Post-Generation Commands

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Shell commands to copy generated weights, run tests, and execute benchmarks. These commands are typically run after the weight generation process.

```bash
cp ~/Downloads/weights.rs src/tokenizer/weights.rs
cargo test
cargo bench --bench query_latency -- --sample-size 10
cargo bench --bench index_build -- --sample-size 10
python3 scripts/bench_compare.py --preset react_token_aligned
```

--------------------------------

### Directory Structure of Fixture Corpus

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/README.md

Illustrates the organization of source files and test cases within the syntext fixture repository.

```text
corpus/
  src/           Rust source files
  python/        Python source files
  typescript/    TypeScript/TSX files
  go/            Go source files
  java/          Java source files
  edge_cases/    Pathological files for edge case testing
  build/         .gitignored; should NOT be indexed
  .gitignore     Ignores build/ and common artifacts
```

--------------------------------

### Run Cargo Clippy for Linting

Source: https://github.com/whit3rabbit/syntext/blob/main/CLAUDE.md

Use Cargo Clippy to check for warnings and enforce coding style guidelines.

```shell
cargo clippy
```

--------------------------------

### Measure Build Time and Index Size Only

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Measure repeated build times and on-disk index sizes without running any search queries using the '--build-only' flag. This is useful for analyzing changes in tokenizer, segment layout, or index construction. The output can be in JSON format.

```shell
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --build-only \
  --build-iterations 5 \
  --json
```

--------------------------------

### Generate Markdown Table Output for Benchmarks

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Generate a copy-paste-friendly Markdown table of benchmark results. Optionally, specify an output file path for the Markdown report.

```shell
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --markdown-table-only
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --markdown-table-only \
  --output /tmp/react-bench.md
```

--------------------------------

### Build Syntext Index with Custom Repository Root

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Builds the Syntext index for a specified repository root directory, overriding the default behavior of finding the nearest `.git` ancestor.

```bash
st --repo-root /path/to/repo index
```

--------------------------------

### Probe Dataset Access and Download

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Checks if a dataset can be accessed and a small file downloaded. If access is denied, it provides instructions to grant access on Hugging Face.

```python
first_lang = next(iter(LANGUAGES))
print(f"Probing {DATASET_NAME} (lang={first_lang})...")
try:
    test_shards = list_shards(DATASET_NAME, first_lang)
    if not test_shards:
        raise RuntimeError("No shards found")
    # Try actually downloading a tiny piece
    test_path = hf_hub_download(
        DATASET_NAME, f"data/{first_lang}/{test_shards[0]}",
        repo_type="dataset", local_dir=DOWNLOAD_DIR,
    )
    test_tbl = pq.read_table(test_path, columns=["content"])
    shutil.rmtree(DOWNLOAD_DIR, ignore_errors=True)
    print(f"  Access confirmed. First shard: {test_tbl.num_rows:,} files.")
except Exception as e:
    err = str(e).lower()
    if "gated" in err or "access" in err or "401" in err or "403" in err:
        print("\n" + "=" * 64)
        print(f"  ACCESS DENIED: {DATASET_NAME}")
        print("=" * 64)
        print(f"\n  1. Visit https://huggingface.co/datasets/{DATASET_NAME}")
        print("  2. Click 'Agree and access repository'")
        print("  3. Re-run this cell")
        print("=" * 64)
    raise
```

--------------------------------

### Parse and Compare Existing Weights

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Parses an existing weights file and compares it against new weights, highlighting changes and boundary flips. Requires `numpy` and `re`. The `path` argument should point to the weights file.

```python
import re as _re
import os
import numpy as np

def parse_existing_weights(path):
    if not os.path.exists(path): return None
    with open(path) as f: text = f.read()
    match = _re.search(r'\\[[^\\]+\\]', text, _re.DOTALL)
    if not match: return None
    nums = [int(x.strip()) for x in match.group(1).split(',') if x.strip()]
    if len(nums) != 65536: return None
    return np.array(nums, dtype=np.uint16)

def compare_weights(old, new, bt=28000):
    diff = new.astype(np.int32) - old.astype(np.int32)
    changed = np.count_nonzero(diff)
    print(f"Changed: {changed:,} / 65,536 ({changed/655.36:.1f}%)")
    print(f"Max incr: {diff.max():+d}  Max decr: {diff.min():+d}  Mean |diff|: {np.abs(diff).mean():.1f}")

    flipped = (old >= bt) != (new >= bt)
    fc = flipped.sum()
    print(f"Boundary flips: {fc}")
    if fc > 0:
        top = np.argsort(np.abs(diff) * flipped.astype(np.int32))[::-1][:min(fc, 25)]
        for idx in top:
            if not flipped[idx]: continue
            b1, b2 = idx >> 8, idx & 0xFF
            c1 = chr(b1) if 32<=b1<127 else f"\\x{b1:02x}"
            c2 = chr(b2) if 32<=b2<127 else f"\\x{b2:02x}"
            d = "int->bnd" if new[idx]>=bt else "bnd->int"
            print(f"  '{c1}{c2}'  {old[idx]:5}->{new[idx]:5}  ({d})")

# Upload weights_old.rs to /content/, then uncomment:
# old = parse_existing_weights("/content/weights_old.rs")
# if old is not None: compare_weights(old, weights, BOUNDARY_THRESHOLD)
print("Upload weights_old.rs and uncomment above.")
```

--------------------------------

### Load Checkpoint Function

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Loads the weight generation state from a checkpoint file. If the file does not exist, it returns initial zeroed values. This function is crucial for resuming interrupted processes.

```python
def load_checkpoint(path):
    if not os.path.exists(path):
        return np.zeros(65536, dtype=np.int64), 0, {}, {}
    ckpt = np.load(path, allow_pickle=False)
    counts = ckpt["counts"]
    total_bytes = int(ckpt["total_bytes"])
    langs = [str(n) for n in ckpt["lang_names"]]
    lang_bytes = dict(zip(langs, [int(b) for b in ckpt["lang_bytes_vals"]]))
    lang_shard_idx = dict(zip(langs, [int(s) for s in ckpt["lang_shard_vals"]]))
    return counts, total_bytes, lang_bytes, lang_shard_idx
```

--------------------------------

### Build Syntext WASM Module

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Command to build the Syntext WebAssembly module using wasm-pack, targeting bundlers and enabling the 'wasm' feature.

```bash
wasm-pack build --target bundler -- --features wasm --no-default-features
```

--------------------------------

### Run Fuzz Target

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/ARCHITECTURE.md

Command to execute the fuzzing target for coverage invariant validation. Ensure you are using the nightly toolchain.

```sh
cargo +nightly fuzz run fuzz_coverage_invariant -- -max_len=4096
```

--------------------------------

### List Dataset Shards

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Lists the available Parquet shard filenames for a given language within the specified HuggingFace dataset. Handles potential exceptions during file system listing.

```python
from huggingface_hub import HfFileSystem, hf_hub_download

fs = HfFileSystem()

def list_shards(dataset_name, lang):
    """List Parquet shard filenames for a language."""
    prefix = f"datasets/{dataset_name}/data/{lang}"
    try:
        files = fs.ls(prefix, detail=False)
        shards = sorted([f.split("/")[-1] for f in files if f.endswith(".parquet")])
        return shards
    except Exception as e:
        return []
```

--------------------------------

### Search Repository with Regex

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Searches the entire indexed repository for lines matching the provided regular expression.

```bash
st "fn parse_query"
```