Syntext

https://github.com/whit3rabbit/syntext
Admin
A hybrid code search index for agent workflows, built in Rust, that offers significantly faster...
Tokens:27,005
Snippets:316
Trust Score:9.1
License:MIT
Update:3 months ago
Show doc for...
Tokens:
Raw
### Setup Python Environment

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Initialize a virtual environment and install necessary dependencies for weight generation.

```sh
python3 -m venv .venv
.venv/bin/pip install datasets numpy
```

--------------------------------

### Quick Install via Shell Script

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Downloads and executes the installation script to install the 'st' binary to /usr/local/bin.

```bash
curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

```bash
INSTALL_DIR=~/.local/bin SYNTEXT_VERSION=2.0.0 \
  curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

--------------------------------

### Install Syntext

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Executes the installation script to detect the platform and install the appropriate binary.

```sh
curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

--------------------------------

### Install from Source

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs the crate directly using Cargo.

```bash
cargo install syntext
```

--------------------------------

### Verify Installer

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/RELEASE.md

Validate the installation script with the new version.

```sh
SYNTEXT_VERSION=X.Y.Z curl -fsSL https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.sh | sh
```

--------------------------------

### Install and run fff comparison

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Builds the fff-mcp binary and runs the benchmark harness with fff integration.

```sh
git clone --depth 1 -b v0.9.4 https://github.com/dmtrKovalenko/fff /tmp/fff-bench
cargo build --release -p fff-mcp --no-default-features \
  --manifest-path /tmp/fff-bench/Cargo.toml
```

```sh
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --syntext-search-mode both \
  --fff-bin /tmp/fff-bench/target/release/fff-mcp
```

--------------------------------

### Manual Linux Installation

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Provides commands for installing via Debian package or direct binary download for Linux systems.

```bash
VERSION=2.0.0

# Debian/Ubuntu (x86_64)
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/syntext_${VERSION}_amd64.deb" \
  -o "syntext_${VERSION}_amd64.deb"
sudo dpkg -i "syntext_${VERSION}_amd64.deb"

# Any Linux (x86_64 or arm64)
ARCH=amd64   # or arm64
curl -L "https://github.com/whit3rabbit/syntext/releases/download/v${VERSION}/st-${VERSION}-linux-${ARCH}" -o st
chmod +x st && sudo mv st /usr/local/bin/
```

--------------------------------

### Install via Homebrew

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs Syntext on macOS using the Homebrew tap.

```bash
brew tap whit3rabbit/tap
brew install --cask whit3rabbit/tap/syntext
```

--------------------------------

### Windows Installation via PowerShell

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Installs the Windows executable and adds it to the user PATH.

```powershell
iwr -useb https://raw.githubusercontent.com/whit3rabbit/syntext/main/install.ps1 | iex
```

```powershell
powershell -ExecutionPolicy Bypass -File install.ps1
```

--------------------------------

### Manage Agent Installations

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Explicit commands for installing, displaying, and uninstalling specific agent harnesses.

```bash
st agent install claude --global
st agent show claude --global
st agent uninstall claude --global
```

--------------------------------

### Python script prerequisites and authentication

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Set up a Python virtual environment and install necessary libraries. Authenticate with Hugging Face to access datasets.

```shell
python3 -m venv .venv
.venv/bin/pip install datasets numpy
hf auth login
```

--------------------------------

### Install Dependencies and Detect GPU

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Installs necessary Python packages and attempts to detect and configure a GPU with CuPy. Falls back to CPU if no GPU is available or CuPy fails.

```python
!pip install -q datasets huggingface_hub numpy pyarrow

import shutil, subprocess

GPU = False
if shutil.which("nvidia-smi"):
    try:
        !pip install -q cupy-cuda12x 2>/dev/null
        import cupy as cp
        name = cp.cuda.runtime.getDeviceProperties(0)("name").decode()
        mem_gb = cp.cuda.Device(0).mem_info[1] / 1e9
        GPU = True
        print(f"GPU: {name} ({mem_gb:.0f} GB)")
    except Exception as e:
        print(f"GPU detected but CuPy failed: {e}")
        print("Falling back to CPU (still fast with bulk download).")

if not GPU:
    print("No GPU. Using CPU with np.bincount (still 10-50x faster than streaming).")
```

--------------------------------

### Bump Installer Defaults

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/RELEASE.md

Commit updated installer version constants after a successful release.

```sh
git commit -am "chore: bump installer defaults to X.Y.Z post-release"
git push origin main
```

--------------------------------

### Parse Query Example

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/edge_cases/nested/deep/path/file.txt

Illustrates a function call for parsing a query string. This is a basic example of interacting with the parsing functionality.

```python
parse_query("deeply nested content")
```

--------------------------------

### Restart from Scratch Command

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

A shell command to remove the existing checkpoint file, forcing the process to start from the beginning. Use this if you need to regenerate all weights without resuming.

```shell
!rm /content/bigram_checkpoint_v2.npz
```

--------------------------------

### Initialize Agent Harnesses

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Use these commands to set up agent integrations. Global flags (-g) apply settings system-wide, while project-level commands target the current directory.

```bash
# Claude Code project instructions only
st init

# Claude Code global Bash hook plus Grep blocker
st init -g

# RTK-style agent selectors
st init -g --agent cursor
st init -g --gemini
st init --copilot        # project hook; `st init -g --copilot` is also accepted
st init --codex          # project rules
st init -g --codex       # global Codex rules
```

--------------------------------

### List and run benchmark presets

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Commands to list available presets and execute a specific benchmark preset.

```sh
python3 scripts/bench_compare.py --list-presets
```

```sh
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react
```

--------------------------------

### Execute recommended benchmark presets

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Run standard benchmark presets to ensure reproducible performance results.

```sh
python3 scripts/bench_compare.py --preset zed_mixed_app
python3 scripts/bench_compare.py --preset react_token_aligned
python3 scripts/bench_compare.py --preset rust_token_aligned
python3 scripts/bench_compare.py --preset linux_token_aligned
python3 scripts/bench_compare.py --preset typescript_compiler
python3 scripts/bench_compare.py --preset node_runtime
```

--------------------------------

### Rust Function to Parse Query

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/edge_cases/crlf_endings.txt

Parses a string slice into a Query struct. This function is part of the test setup.

```rust
fn parse_query(s: &str) -> Query {
    // TODO: validate input
    Query { raw: s.to_string() }
}
```

--------------------------------

### Run external repository comparison harness

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Use the Python harness to compare syntext performance against other tools on real repositories.

```sh
python3 scripts/bench_compare.py --preset react_token_aligned
python3 scripts/bench_compare.py --repo /path/to/repo --query literal:symbol
```

--------------------------------

### Run development and testing commands

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Execute these commands in the project root to verify code quality, run unit/integration tests, and perform benchmarks.

```bash
cargo test                    # unit + integration tests
cargo clippy                  # lint, must pass with no warnings
cargo bench                   # criterion benchmarks
SYNTEXT_LOG_SELECTIVITY=1 cargo test --test correctness -- --nocapture
                              # show per-query selectivity stats
cargo test --test tokenizer   # unit: tokenizer only
cargo test --test posting     # unit: posting lists only
cargo test --test query       # unit: query router only
cargo test --test overlay     # unit: overlay/snapshot only
cargo test --test boundary_fuzz  # unit: boundary fuzzing
cargo test --test index_build # integration: index construction
cargo test --test incremental # integration: incremental updates
cargo test --test symbols     # integration: symbol index
cargo test --test oracle_self --features oracle  # integration: self-differential oracle
cargo test --test oracle_incremental --features oracle # integration: incremental overlay differential oracle
cargo test --test oracle_cli --features oracle   # integration: CLI subprocess differential oracle
cargo test --features oracle                     # run all tests including oracle targets
cargo bench --bench query_latency -- --sample-size 10
cargo bench --bench index_build -- --sample-size 10
cargo bench --bench selectivity -- --sample-size 10
cargo bench --bench freshness -- --sample-size 10
```

--------------------------------

### Run index_open benchmark

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Execute the index_open benchmark with a specific sample size.

```bash
cargo bench --bench index_open -- --sample-size 10
```

--------------------------------

### Clone Repositories for Benchmarking

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Creates a local directory and performs shallow clones of various large repositories to serve as a corpus for search benchmarks.

```sh
mkdir -p ./_syntext-bench
git clone --depth 1 -b v6.8 https://github.com/torvalds/linux.git ./_syntext-bench/linux
git clone --depth 1 -b 1.77.0 https://github.com/rust-lang/rust.git ./_syntext-bench/rust
git clone --depth 1 -b v18.2.0 https://github.com/facebook/react.git ./_syntext-bench/react
git clone --depth 1 -b v5.4.3 https://github.com/microsoft/TypeScript.git ./_syntext-bench/typescript
git clone --depth 1 -b v20.12.0 https://github.com/nodejs/node.git ./_syntext-bench/node
```

--------------------------------

### Run fff comparison benchmark

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Execute the benchmark comparison script using the react_token_aligned preset.

```bash
python3 scripts/bench_compare.py --preset react_token_aligned \
  --repo _syntext-bench/react --syntext-search-mode both \
  --fff-bin /tmp/fff-bench/target/release/fff-mcp \
  --build-iterations 3 --search-iterations 5 --warmups 2
```

--------------------------------

### Run freshness benchmark

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Runs the freshness criterion benchmark for 10 samples.

```sh
cargo bench --bench freshness -- --sample-size 10
```

--------------------------------

### Generate weights using Python script with Hugging Face datasets

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Execute the Python script to generate weights, specifying the output file path. This method uses the 'bigcode/the-stack-smol' dataset via streaming.

```shell
HF_DATASETS_CACHE=/tmp/hf_cache .venv/bin/python scripts/weights_gen.py \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Run external repository comparison

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Executes a performance comparison between syntext, rg, and grep on a specified repository.

```sh
python3 scripts/bench_compare.py \
  --repo /path/to/repo \
  --query literal:token_aligned_symbol \
  --query literal:another_symbol
```

--------------------------------

### Run open search end-to-end benchmark

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Runs the open_search_e2e criterion benchmark for 10 samples.

```sh
cargo bench --bench open_search_e2e -- --sample-size 10
```

--------------------------------

### Run external repository benchmark comparison

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Executes the Python benchmark comparison script against specified repositories using a preset configuration.

```bash
python3 scripts/bench_compare.py --repo ... --preset ... --json
```

--------------------------------

### Run fff comparison benchmark

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Executes a comparative benchmark between syntext and fff using a mixed-language corpus.

```sh
python3 scripts/bench_compare.py --preset zed_mixed_app \
  --repo /tmp/zed-bench --syntext-search-mode both \
  --fff-bin /tmp/fff-bench/target/release/fff-mcp
```

--------------------------------

### Download Weights File from Colab

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Provides functionality to download the generated Rust weights file from Google Colab. Includes a fallback to print the file path if the `files` module is not available.

```python
try:
    from google.colab import files
    if os.path.exists(OUTPUT_PATH):
        files.download(OUTPUT_PATH)
except ImportError:
    print(f"File at: {OUTPUT_PATH}")
```

--------------------------------

### Generate weights using Python script with local directories

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

Use the Python script to generate weights from local directories containing code. This is an alternative to using Hugging Face datasets.

```shell
.venv/bin/python scripts/weights_gen.py \
  --source local \
  --dirs ~/projects/linux ~/projects/rustc ~/projects/cpython \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Import Libraries and Define Configuration

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Imports essential libraries for data processing and defines configuration parameters including target data size, dataset name, language weights, file size limits, processing chunk size, download directory, checkpoint path, output path, and boundary threshold.

```python
import numpy as np
import pyarrow.parquet as pq
import os, time, logging, warnings
from pathlib import Path

logging.getLogger("datasets.load").setLevel(logging.CRITICAL)
warnings.filterwarnings("ignore", message=".*trust_remote_code.*")

# ── Target ─────────────────────────────────────────────────────────────────
# 100 GB recommended. At bulk download speeds, takes ~20-30 min.
# For a quick test: 2_000_000_000 (2 GB, ~2 min).
TARGET_BYTES = 100 * 1_000_000_000

DATASET_NAME = "bigcode/the-stack-dedup"

# the-stack-dedup shard names. C++ is "cpp" not "c++".
LANGUAGES = {
    # Tier 1 — agent-heavy
    "python": 2.0, "javascript": 2.0, "typescript": 2.0, "rust": 2.0,
    "go": 1.5, "java": 1.5, "c": 1.5, "cpp": 1.5,
    # Tier 2 — broader coverage
    "c-sharp": 1.0, "ruby": 1.0, "php": 1.0,
    "swift": 0.8, "kotlin": 0.8,
    "scala": 0.7, "shell": 0.7,
    "lua": 0.5, "haskell": 0.5, "r": 0.5,
    "perl": 0.4, "dart": 0.4,
    "elixir": 0.3, "julia": 0.3, "ocaml": 0.3, "zig": 0.3,
}

MAX_FILE_BYTES = 500_000       # Skip files > 500 KB (minified JS, generated)
PROCESS_CHUNK_MB = 256         # Max MB to concat before counting (bounds RAM)
DOWNLOAD_DIR = "/content/hf_shard"  # Temp dir for each shard, deleted after
CHECKPOINT_PATH = "/content/bigram_checkpoint_v2.npz"
OUTPUT_PATH = "/content/weights.rs"
BOUNDARY_THRESHOLD = 28000     # syntext's gram boundary threshold

print(f"Target:  {TARGET_BYTES / 1e9:.0f} GB | {len(LANGUAGES)} languages")
print(f"Dataset: {DATASET_NAME}")
print(f"GPU:     {'Yes (' + str(GPU) + ')' if GPU else 'No (CPU)'}")
print(f"Chunk:   {PROCESS_CHUNK_MB} MB per batch")
```

--------------------------------

### Create Git Tag

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/RELEASE.md

Create an annotated tag for the new release version.

```sh
git tag -a vX.Y.Z -m "Release vX.Y.Z"
```

--------------------------------

### Build WASM from Source

Source: https://github.com/whit3rabbit/syntext/blob/main/README.md

Compiles the project into a WASM package using wasm-pack.

```bash
cargo install wasm-pack
wasm-pack build --target bundler -- --features wasm --no-default-features
# output: pkg/  (JS glue + .wasm + TypeScript types)
```

--------------------------------

### Generate Weights from Local Directories

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Execute the generation script using local code directories as the source.

```sh
.venv/bin/python scripts/weights_gen.py \
  --source local \
  --dirs ~/projects/linux ~/projects/rustc ~/projects/cpython \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Generate Weights from HuggingFace

Source: https://github.com/whit3rabbit/syntext/blob/main/AGENTS.md

Execute the generation script using the recommended HuggingFace dataset source.

```sh
HF_DATASETS_CACHE=/tmp/hf_cache .venv/bin/python scripts/weights_gen.py \
  --output src/tokenizer/weights.rs
```

--------------------------------

### Copy generated weights and run tests/benchmarks

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/README.md

After generating the weights.rs file using the Colab notebook, copy it to the project's source directory and then run cargo test and benchmarks to ensure integration.

```shell
cp ~/Downloads/weights.rs src/tokenizer/weights.rs
cargo test
cargo bench --bench query_latency -- --sample-size 10
```

--------------------------------

### Measure index build performance

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Benchmarks index construction time and disk usage without executing search queries.

```sh
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --build-only \
  --build-iterations 5 \
  --json
```

--------------------------------

### Generate Markdown benchmark tables

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/BENCHMARKS.md

Outputs benchmark results as a Markdown table, optionally saving to a file.

```sh
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --markdown-table-only
```

```sh
python3 scripts/bench_compare.py \
  --preset react_token_aligned \
  --repo _syntext-bench/react \
  --markdown-table-only \
  --output /tmp/react-bench.md
```

--------------------------------

### Verify Benchmark Latency

Source: https://github.com/whit3rabbit/syntext/blob/main/docs/RELEASE.md

Run performance benchmarks to ensure no regressions before release.

```sh
cargo bench --bench query_latency -- --sample-size 10
cargo bench --bench freshness -- --sample-size 10
```

--------------------------------

### Main Processing Loop

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Iterates through languages and their respective shards, downloading, processing, and cleaning up each shard. It resumes from the last checkpoint if available and saves a new checkpoint after each shard is processed.

```python
# ── Resume or start fresh ──────────────────────────────────────────────────
counts, total_bytes, lang_bytes_done, lang_shard_done = load_checkpoint(CHECKPOINT_PATH)
if total_bytes > 0:
    pct = np.count_nonzero(counts) / 65536 * 100
    print(f"Resumed: {total_bytes / 1e9:.2f} GB, {pct:.1f}% pair coverage")
else:
    print("Starting fresh.")

# Per-language byte targets.
total_weight = sum(LANGUAGES.values())
lang_targets = {
    lang: int(TARGET_BYTES * w / total_weight)
    for lang, w in LANGUAGES.items()
}

wall_start = time.time()
total_files = 0
total_skipped = 0

for lang, byte_target in lang_targets.items():
    already = lang_bytes_done.get(lang, 0)
    if already >= byte_target:
        print(f"  {lang}: done ({already / 1e9:.2f} GB)")
        continue

    shards = lang_shards.get(lang, [])
    start_shard = lang_shard_done.get(lang, 0)
    if start_shard >= len(shards):
        print(f"  {lang}: all {len(shards)} shards processed ({already / 1e9:.2f} GB)")
        continue

    lang_bytes_local = already
    remaining = byte_target - already
    print(f"  {lang}: {remaining / 1e9:.2f} GB remaining, "
          f"shards {start_shard}–{len(shards)-1} of {len(shards)}")

    for shard_idx in range(start_shard, len(shards)):
        if lang_bytes_local >= byte_target:
            break

        shard_name = shards[shard_idx]
        shard_remaining = byte_target - lang_bytes_local

        try:
            # Download.
            t0 = time.time()
            local_path = hf_hub_download(
                DATASET_NAME,
                f"data/{lang}/{shard_name}",
                repo_type="dataset",
                local_dir=DOWNLOAD_DIR,
            )
            dl_sec = time.time() - t0
            dl_mb = os.path.getsize(local_path) / 1e6

            # Process.
            t1 = time.time()
            shard_counts, shard_bytes, shard_files, shard_skipped = process_parquet_shard(
                local_path, MAX_FILE_BYTES, PROCESS_CHUNK_MB, GPU,
                byte_limit=shard_remaining,
            )
            proc_sec = time.time() - t1

            counts += shard_counts
            total_bytes += shard_bytes
            lang_bytes_local += shard_bytes
            total_files += shard_files
            total_skipped += shard_skipped

        except Exception as e:
            print(f"    shard {shard_idx} error: {e}")
            continue
        finally:
            # Always clean up disk.
            shutil.rmtree(DOWNLOAD_DIR, ignore_errors=True)

        # Checkpoint after every shard.
        lang_bytes_done[lang] = lang_bytes_local
        lang_shard_done[lang] = shard_idx + 1
        save_checkpoint(CHECKPOINT_PATH, counts, total_bytes, lang_bytes_done, lang_shard_done)

        # Progress.
        elapsed = time.time() - wall_start
        pct = total_bytes / TARGET_BYTES * 100
        coverage = np.count_nonzero(counts) / 65536 * 100
        rate = total_bytes / elapsed / 1e6 if elapsed > 0 else 0
        eta = (TARGET_BYTES - total_bytes) / (total_bytes / elapsed) / 60 if total_bytes > 0 else 0
        print(
            f"    [{shard_idx+1}/{len(shards)}] "
            f"{dl_mb:.0f}MB dl:{dl_sec:.1f}s proc:{proc_sec:.1f}s | "
            f"{total_bytes / 1e9:.1f}GB ({pct:.0f}%) | "
            f"{rate:.0f} MB/s | ETA {eta:.0f}m | "
            f"{coverage:.1f}% coverage"
        )

    print(f"    {lang} done: {lang_bytes_local / 1e9:.2f} GB, {total_files:,} files")
```

--------------------------------

### Post-Generation Commands

Source: https://github.com/whit3rabbit/syntext/blob/main/scripts/notebooks/weights_gen_colab.ipynb

Shell commands to copy generated weights, run tests, and execute benchmarks. These commands are typically run after the weight generation process.

```bash
cp ~/Downloads/weights.rs src/tokenizer/weights.rs
cargo test
cargo bench --bench query_latency -- --sample-size 10
cargo bench --bench index_build -- --sample-size 10
python3 scripts/bench_compare.py --preset react_token_aligned
```

--------------------------------

### Directory Structure of Fixture Corpus

Source: https://github.com/whit3rabbit/syntext/blob/main/tests/fixtures/corpus/README.md

Illustrates the organization of source files and test cases within the syntext fixture repository.

```text
corpus/
  src/           Rust source files
  python/        Python source files
  typescript/    TypeScript/TSX files
  go/            Go source files
  java/          Java source files
  edge_cases/    Pathological files for edge case testing
  build/         .gitignored; should NOT be indexed
  .gitignore     Ignores build/ and common artifacts
```
Tokens:
Raw