### Prepare Data and Train Tokenizer

Source: https://github.com/karpathy/autoresearch/blob/master/README.md

Executes the `prepare.py` script to download training data and train a BPE tokenizer. This is a one-time setup step.

```bash
# 3. Download data and train tokenizer (one-time, ~2 min)
uv run prepare.py
```

--------------------------------

### Install uv and Project Dependencies

Source: https://github.com/karpathy/autoresearch/blob/master/README.md

Installs the uv project manager and then installs the project's dependencies. Ensure you have Python 3.10+ and a single NVIDIA GPU.

```bash
# 1. Install uv project manager (if you don't already have it)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 2. Install dependencies
uv sync
```

--------------------------------

### Initiate Autonomous Research Agent

Source: https://github.com/karpathy/autoresearch/blob/master/README.md

Provides an example prompt to start an AI agent for autonomous research. The agent should be pointed to `program.md` for instructions.

```bash
Hi have a look at program.md and let's kick off a new experiment! let's do the setup first.
```

--------------------------------

### Run a Single Training Experiment

Source: https://github.com/karpathy/autoresearch/blob/master/README.md

Manually initiates a single training experiment using the `train.py` script. This serves as a test to ensure the setup is working correctly.

```bash
# 4. Manually run a single training experiment (~5 min)
uv run train.py
```

--------------------------------

### MuonAdamW Optimizer Setup and Training Step

Source: https://context7.com/karpathy/autoresearch/llms.txt

Sets up the MuonAdamW optimizer with customized learning rates for different parameter groups and performs a training step. Includes learning rate scheduling.

```python
from train import GPT, GPTConfig

config = GPTConfig(n_layer=8, n_embd=512, n_head=4, n_kv_head=4)
model = GPT(config)
model.to(torch.device("cuda"))
model.init_weights()

# Setup optimizer with customized learning rates
optimizer = model.setup_optimizer(
    unembedding_lr=0.004,    # LR for lm_head (AdamW)
    embedding_lr=0.6,        # LR for token embeddings (AdamW)
    matrix_lr=0.04,          # LR for transformer matrices (Muon)
    scalar_lr=0.5,           # LR for per-layer scalars (AdamW)
    weight_decay=0.2,        # Cautious weight decay for Muon
    adam_betas=(0.8, 0.95)   # Adam momentum parameters
)

# Training step
loss = model(x, y)
loss.backward()
optimizer.step()
model.zero_grad(set_to_none=True)

# Learning rate scheduling
for group in optimizer.param_groups:
    group["lr"] = group["initial_lr"] * lr_multiplier
```

--------------------------------

### Running Training Experiments with uv

Source: https://context7.com/karpathy/autoresearch/llms.txt

Commands to run a training experiment using `uv run` and redirecting output for logging. Includes examples of extracting key metrics.

```bash
# Run single training experiment
uv run train.py

# Redirect output for logging (recommended for agent use)
uv run train.py > run.log 2>&1

# Extract key metrics from log
grep "^val_bpb:" run.log
grep "^peak_vram_mb:" run.log

# Expected output format:
# ---
# val_bpb:          0.997900
# training_seconds: 300.1
# total_seconds:    325.9
# peak_vram_mb:     45060.2
# mfu_percent:      39.80
# total_tokens_M:   499.6
# num_steps:        953
# num_params_M:     50.3
# depth:            8
```

--------------------------------

### GPT Model Forward Pass: Inference and Training

Source: https://context7.com/karpathy/autoresearch/llms.txt

Demonstrates how to perform model inference to get logits and compute loss during training. Supports configurable loss reduction.

```python
import torch
import torch.nn.functional as F

# Inference mode - get logits
model.eval()
input_ids = torch.randint(0, 8192, (1, 256), device="cuda")
with torch.no_grad():
    logits = model(input_ids)  # Shape: [1, 256, 8192]

# Training mode - compute loss
model.train()
x = torch.randint(0, 8192, (32, 2048), device="cuda")
y = torch.randint(0, 8192, (32, 2048), device="cuda")

# Mean reduction (default)
loss = model(x, y)  # Scalar tensor

# No reduction for per-token losses (used in BPB eval)
loss_flat = model(x, y, reduction='none')  # Shape: [32, 2048]

# Backpropagation
loss.backward()
```

--------------------------------

### Prepare Data and Train Tokenizer

Source: https://context7.com/karpathy/autoresearch/llms.txt

Run the preparation script to download data shards and train a BPE tokenizer. Customize the number of shards and download workers for testing or full downloads.

```bash
uv run prepare.py
```

```bash
uv run prepare.py --num-shards 8
```

```bash
uv run prepare.py --num-shards -1
```

```bash
uv run prepare.py --num-shards 20 --download-workers 16
```

--------------------------------

### Agent Experiment Loop Workflow

Source: https://context7.com/karpathy/autoresearch/llms.txt

Illustrates the agent workflow for running autonomous research experiments, including creating branches, initializing results tracking, running experiments, and evaluating/committing results.

```bash
# Agent workflow commands

# 1. Create experiment branch
git checkout -b autoresearch/mar5

# 2. Initialize results tracking (tab-separated)
echo -e "commit\tval_bpb\tmemory_gb\tstatus\tdescription" > results.tsv

# 3. Run baseline experiment
uv run train.py > run.log 2>&1

# 4. Extract and log results
VAL_BPB=$(grep "^val_bpb:" run.log | awk '{print $2}')
VRAM_MB=$(grep "^peak_vram_mb:" run.log | awk '{print $2}')
VRAM_GB=$(echo "scale=1; $VRAM_MB / 1024" | bc)
COMMIT=$(git rev-parse --short HEAD)
echo -e "$COMMIT\t$VAL_BPB\t$VRAM_GB\tkeep\tbaseline" >> results.tsv

# 5. Make changes, commit, run, evaluate
# If val_bpb improved: keep commit
# If val_bpb worse: git reset --hard HEAD~1

# Example results.tsv content:
# commit  val_bpb   memory_gb  status   description
# a1b2c3d 0.997900  44.0       keep     baseline
# b2c3d4e 0.993200  44.2       keep     increase LR to 0.04
# c3d4e5f 1.005000  44.0       discard  switch to GeLU activation
# d4e5f6g 0.000000  0.0        crash    double model width (OOM)
```

--------------------------------

### Data Preparation and Tokenizer Training

Source: https://context7.com/karpathy/autoresearch/llms.txt

Scripts and commands for preparing data and training a BPE tokenizer.

```APIDOC
## Data Preparation and Tokenizer Training

The `prepare.py` script handles one-time setup by downloading training data shards from HuggingFace and training a BPE tokenizer using rustbpe.

### Commands

*   **Full preparation (download 10 shards + train tokenizer)**
    ```bash
    uv run prepare.py
    ```
*   **Download only 8 shards for testing**
    ```bash
    uv run prepare.py --num-shards 8
    ```
*   **Download all available shards (6542 total)**
    ```bash
    uv run prepare.py --num-shards -1
    ```
*   **Customize parallel download workers**
    ```bash
    uv run prepare.py --num-shards 20 --download-workers 16
    ```
```

--------------------------------

### GPT Model Configuration and Initialization

Source: https://context7.com/karpathy/autoresearch/llms.txt

Define GPT model architecture parameters using GPTConfig and initialize the GPT model. The model can be built on a meta device before moving to GPU and weights can be initialized.

```python
from dataclasses import asdict
from train import GPTConfig, GPT

# Default configuration
config = GPTConfig(
    sequence_len=2048,
    vocab_size=8192,
    n_layer=12,
    n_head=6,
    n_kv_head=6,
    n_embd=768,
    window_pattern="SSSL"  # S=half context sliding window, L=full context
)

# Build model on meta device then move to GPU
with torch.device("meta"):
    model = GPT(config)
model.to_empty(device=torch.device("cuda"))
model.init_weights()

# Check parameter counts
param_counts = model.num_scaling_params()
print(f"Total parameters: {param_counts['total']:,}")
# Output: Total parameters: 50,331,648

# Estimate FLOPs per token
flops = model.estimate_flops()
print(f"FLOPs per token: {flops:e}")
```

--------------------------------

### Tokenizer Class Usage

Source: https://context7.com/karpathy/autoresearch/llms.txt

Demonstrates how to use the Tokenizer class for encoding and decoding text.

```APIDOC
## Tokenizer Class

The `Tokenizer` class wraps a tiktoken encoding with BPE tokenization, supporting single text and batch encoding with optional BOS token prepending.

### Initialization

```python
from prepare import Tokenizer

# Load tokenizer from default cache directory
tokenizer = Tokenizer.from_directory()
```

### Vocabulary Size

```python
# Get vocabulary size
vocab_size = tokenizer.get_vocab_size()  # Returns 8192
```

### Encoding

*   **Encode single text**
    ```python
    text = "Hello, world!"
    token_ids = tokenizer.encode(text)  # Returns list of integers
    ```
*   **Encode with BOS token prepended**
    ```python
    token_ids_with_bos = tokenizer.encode(text, prepend=tokenizer.get_bos_token_id())
    ```
*   **Batch encode multiple texts (parallelized)**
    ```python
    texts = ["First document", "Second document", "Third document"]
    batch_ids = tokenizer.encode(texts, prepend=tokenizer.get_bos_token_id(), num_threads=8)
    ```

### Decoding

```python
# Decode back to text
decoded = tokenizer.decode(token_ids)  # Returns "Hello, world!"
```
```

--------------------------------

### GPT Model Configuration

Source: https://context7.com/karpathy/autoresearch/llms.txt

Configuration options and initialization for the GPT model.

```APIDOC
## GPT Model Configuration

The `GPTConfig` dataclass defines model architecture parameters including layer count, attention heads, embedding dimensions, and sliding window patterns.

### Configuration and Initialization

```python
from dataclasses import asdict
from train import GPTConfig, GPT

# Default configuration
config = GPTConfig(
    sequence_len=2048,
    vocab_size=8192,
    n_layer=12,
    n_head=6,
    n_kv_head=6,
    n_embd=768,
    window_pattern="SSSL"  # S=half context sliding window, L=full context
)

# Build model on meta device then move to GPU
with torch.device("meta"):
    model = GPT(config)
model.to_empty(device=torch.device("cuda"))
model.init_weights()
```

### Parameter and FLOPs Estimation

```python
# Check parameter counts
param_counts = model.num_scaling_params()
print(f"Total parameters: {param_counts['total']:,}")
# Output: Total parameters: 50,331,648

# Estimate FLOPs per token
flops = model.estimate_flops()
print(f"FLOPs per token: {flops:e}")
```
```

--------------------------------

### Dataloader Factory

Source: https://context7.com/karpathy/autoresearch/llms.txt

How to create and use the efficient infinite dataloader for training.

```APIDOC
## Dataloader Factory

The `make_dataloader` function creates an efficient infinite dataloader with BOS-aligned best-fit packing for 100% token utilization.

### Creating Dataloaders

```python
from prepare import Tokenizer, make_dataloader, MAX_SEQ_LEN

tokenizer = Tokenizer.from_directory()
batch_size = 128
sequence_length = MAX_SEQ_LEN  # 2048

# Create training dataloader (infinite iterator)
train_loader = make_dataloader(tokenizer, batch_size, sequence_length, "train")

# Create validation dataloader
val_loader = make_dataloader(tokenizer, batch_size, sequence_length, "val")
```

### Getting Next Batch

```python
# Get next batch (inputs, targets, epoch_number)
x, y, epoch = next(train_loader)
# x.shape: torch.Size([128, 2048]) - input tokens on GPU
# y.shape: torch.Size([128, 2048]) - target tokens on GPU
# epoch: current epoch number (increments when data wraps)
```

### Training Loop Example

```python
for step in range(1000):
    x, y, epoch = next(train_loader)
    loss = model(x, y)
    loss.backward()
    optimizer.step()
```
```

--------------------------------

### Load and Prepare Experiment Data

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Reads the results.tsv file and cleans the columns for analysis.

```python
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# Load the TSV (tab-separated, 5 columns: commit, val_bpb, memory_gb, status, description)
df = pd.read_csv("results.tsv", sep="\t")
df["val_bpb"] = pd.to_numeric(df["val_bpb"], errors="coerce")
df["memory_gb"] = pd.to_numeric(df["memory_gb"], errors="coerce")
df["status"] = df["status"].str.strip().str.upper()

print(f"Total experiments: {len(df)}")
print(f"Columns: {list(df.columns)}")
df.head(10)
```

--------------------------------

### Dataloader Factory for Training

Source: https://context7.com/karpathy/autoresearch/llms.txt

Create an infinite dataloader with BOS-aligned best-fit packing for efficient training. The dataloader provides batches of input and target tokens, along with the current epoch number.

```python
from prepare import Tokenizer, make_dataloader, MAX_SEQ_LEN

tokenizer = Tokenizer.from_directory()
batch_size = 128
sequence_length = MAX_SEQ_LEN  # 2048

# Create training dataloader (infinite iterator)
train_loader = make_dataloader(tokenizer, batch_size, sequence_length, "train")

# Create validation dataloader
val_loader = make_dataloader(tokenizer, batch_size, sequence_length, "val")

# Get next batch (inputs, targets, epoch_number)
x, y, epoch = next(train_loader)
# x.shape: torch.Size([128, 2048]) - input tokens on GPU
# y.shape: torch.Size([128, 2048]) - target tokens on GPU
# epoch: current epoch number (increments when data wraps)

# Training loop example
for step in range(1000):
    x, y, epoch = next(train_loader)
    loss = model(x, y)
    loss.backward()
    optimizer.step()
```

--------------------------------

### Summarize Improvement Statistics

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Calculates and prints performance metrics comparing the baseline to the best achieved result.

```python
# Summary stats
kept = df[df["status"] == "KEEP"].copy()
baseline_bpb = df.iloc[0]["val_bpb"]
best_bpb = kept["val_bpb"].min()
best_row = kept.loc[kept["val_bpb"].idxmin()]

print(f"Baseline val_bpb:  {baseline_bpb:.6f}")
print(f"Best val_bpb:      {best_bpb:.6f}")
print(f"Total improvement: {baseline_bpb - best_bpb:.6f} ({(baseline_bpb - best_bpb) / baseline_bpb * 100:.2f}%)")
print(f"Best experiment:   {best_row['description']}")
print()

# How many experiments to find each improvement
print("Cumulative effort per improvement:")
kept_sorted = kept.reset_index()
for i, (_, row) in enumerate(kept_sorted.iterrows()):
    desc = str(row["description"]).strip()
    print(f"  Experiment #{row['index']:3d}: bpb={row['val_bpb']:.6f}  {desc}")
```

--------------------------------

### Hyperparameter Configuration for Training

Source: https://context7.com/karpathy/autoresearch/llms.txt

Defines key hyperparameters for model architecture, optimization, and scheduling within the `train.py` script. Adjust these for different model sizes and strategies.

```python
# Model architecture (in train.py)
ASPECT_RATIO = 64       # model_dim = depth * ASPECT_RATIO
HEAD_DIM = 128          # Target head dimension for attention
WINDOW_PATTERN = "SSSL" # Sliding window pattern: L=full, S=half context
DEPTH = 8               # Number of transformer layers

# Optimization
TOTAL_BATCH_SIZE = 2**19  # ~524K tokens per optimizer step
EMBEDDING_LR = 0.6        # Learning rate for token embeddings
UNEMBEDDING_LR = 0.004    # Learning rate for lm_head
MATRIX_LR = 0.04          # Learning rate for matrix parameters
SCALAR_LR = 0.5           # Learning rate for per-layer scalars
WEIGHT_DECAY = 0.2        # Cautious weight decay for Muon
ADAM_BETAS = (0.8, 0.95)  # Adam beta1, beta2

# Schedule
WARMUP_RATIO = 0.0        # Fraction of time budget for LR warmup
WARMDOWN_RATIO = 0.5      # Fraction of time budget for LR warmdown
FINAL_LR_FRAC = 0.0       # Final LR as fraction of initial

# Device
DEVICE_BATCH_SIZE = 128   # Per-device batch size (reduce if OOM)
```

--------------------------------

### Visualize Validation BPB Progress

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Generates a plot showing the evolution of validation BPB over time, highlighting kept improvements.

```python
fig, ax = plt.subplots(figsize=(16, 8))

# Filter out crashes for plotting
valid = df[df["status"] != "CRASH"].copy()
valid = valid.reset_index(drop=True)

baseline_bpb = valid.loc[0, "val_bpb"]

# Only plot points at or below baseline (the interesting region)
below = valid[valid["val_bpb"] <= baseline_bpb + 0.0005]

# Plot discarded as faint background dots
disc = below[below["status"] == "DISCARD"]
ax.scatter(disc.index, disc["val_bpb"],
           c="#cccccc", s=12, alpha=0.5, zorder=2, label="Discarded")

# Plot kept experiments as prominent green dots
kept_v = below[below["status"] == "KEEP"]
ax.scatter(kept_v.index, kept_v["val_bpb"],
           c="#2ecc71", s=50, zorder=4, label="Kept", edgecolors="black", linewidths=0.5)

# Running minimum step line
kept_mask = valid["status"] == "KEEP"
kept_idx = valid.index[kept_mask]
kept_bpb = valid.loc[kept_mask, "val_bpb"]
running_min = kept_bpb.cummin()
ax.step(kept_idx, running_min, where="post", color="#27ae60",
        linewidth=2, alpha=0.7, zorder=3, label="Running best")

# Label each kept experiment with its description
for idx, bpb in zip(kept_idx, kept_bpb):
    desc = str(valid.loc[idx, "description"]).strip()
    if len(desc) > 45:
        desc = desc[:42] + "..."

    ax.annotate(desc, (idx, bpb),
                textcoords="offset points",
                xytext=(6, 6), fontsize=8.0,
                color="#1a7a3a", alpha=0.9,
                rotation=30, ha="left", va="bottom")

n_total = len(df)
n_kept = len(df[df["status"] == "KEEP"])
ax.set_xlabel("Experiment #", fontsize=12)
ax.set_ylabel("Validation BPB (lower is better)", fontsize=12)
ax.set_title(f"Autoresearch Progress: {n_total} Experiments, {n_kept} Kept Improvements", fontsize=14)
ax.legend(loc="upper right", fontsize=9)
ax.grid(True, alpha=0.2)

# Y-axis: from just below best to just above baseline
best_bpb = kept_bpb.min()
margin = (baseline_bpb - best_bpb) * 0.15
ax.set_ylim(best_bpb - margin, baseline_bpb + margin)

plt.tight_layout()
plt.savefig("progress.png", dpi=150, bbox_inches="tight")
plt.show()
print("Saved to progress.png")
```

--------------------------------

### Evaluate Bits-Per-Byte (BPB)

Source: https://context7.com/karpathy/autoresearch/llms.txt

Compute the bits-per-byte metric using the evaluate_bpb function. This metric is independent of vocabulary size, allowing for fair comparison of different tokenizers. Ensure the model is in evaluation mode.

```python
import torch
from prepare import Tokenizer, evaluate_bpb

tokenizer = Tokenizer.from_directory()
batch_size = 128

# Model must be in eval mode
model.eval()

# Evaluate on validation set
with torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16):
    val_bpb = evaluate_bpb(model, tokenizer, batch_size)

print(f"Validation BPB: {val_bpb:.6f}")  # Lower is better, e.g., 0.997900
```

--------------------------------

### Tokenizer Class Usage

Source: https://context7.com/karpathy/autoresearch/llms.txt

Instantiate and use the Tokenizer class for encoding and decoding text. Supports single text, batch encoding with optional BOS token, and decoding back to text.

```python
from prepare import Tokenizer

# Load tokenizer from default cache directory
tokenizer = Tokenizer.from_directory()

# Get vocabulary size
vocab_size = tokenizer.get_vocab_size()  # Returns 8192

# Encode single text
text = "Hello, world!"
token_ids = tokenizer.encode(text)  # Returns list of integers

# Encode with BOS token prepended
token_ids_with_bos = tokenizer.encode(text, prepend=tokenizer.get_bos_token_id())

# Batch encode multiple texts (parallelized)
texts = ["First document", "Second document", "Third document"]
batch_ids = tokenizer.encode(texts, prepend=tokenizer.get_bos_token_id(), num_threads=8)

# Decode back to text
decoded = tokenizer.decode(token_ids)  # Returns "Hello, world!"
```

--------------------------------

### Print Sorted Results with Formatting

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Prints the sorted results from a pandas DataFrame in a formatted table. It includes rank, delta, BPB, and description. Ensure the DataFrame has 'delta', 'val_bpb', and 'description' columns.

```python
print(f"{ 'Rank':>4}  {'Delta':>8}  {'BPB':>10}  Description")
print("-" * 80)
for rank, (_, row) in enumerate(hits.iterrows(), 1):
    print(f"{rank:4d}  {row['delta']:+.6f}  {row['val_bpb']:.6f}  {row['description']}")

print(f"\n{ '':>4}  {hits['delta'].sum():+.6f}  { '':>10}  TOTAL improvement over baseline")
```

--------------------------------

### Calculate Experiment Outcomes

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Summarizes the distribution of experiment statuses and calculates the keep rate.

```python
counts = df["status"].value_counts()
print("Experiment outcomes:")
print(counts.to_string())

n_keep = counts.get("KEEP", 0)
n_discard = counts.get("DISCARD", 0)
n_crash = counts.get("CRASH", 0)
n_decided = n_keep + n_discard
if n_decided > 0:
    print(f"\nKeep rate: {n_keep}/{n_decided} = {n_keep / n_decided:.1%}")
```

--------------------------------

### List Kept Experiments

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Filters and displays details for all experiments marked as KEEP.

```python
# Show all KEPT experiments (the improvements that stuck)
kept = df[df["status"] == "KEEP"].copy()
print(f"KEPT experiments ({len(kept)} total):\n")
for i, row in kept.iterrows():
    bpb = row["val_bpb"]
    desc = row["description"]
    print(f"  #{i:3d}  bpb={bpb:.6f}  mem={row['memory_gb']:.1f}GB  {desc}")
```

--------------------------------

### Calculate Improvement Deltas

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Computes the performance delta between consecutive kept experiments.

```python
# Each kept experiment's delta is measured vs the previous kept experiment's bpb
# (since experiments are cumulative -- each one builds on the last kept state)
kept = df[df["status"] == "KEEP"].copy()
kept["prev_bpb"] = kept["val_bpb"].shift(1)
kept["delta"] = kept["prev_bpb"] - kept["val_bpb"]

# Drop baseline (no delta)
hits = kept.iloc[1:].copy()
```

--------------------------------

### BPB Evaluation Function

Source: https://context7.com/karpathy/autoresearch/llms.txt

Details on using the `evaluate_bpb` function for model evaluation.

```APIDOC
## BPB Evaluation Function

The `evaluate_bpb` function computes the bits-per-byte metric, a vocabulary-size-independent measure that enables fair comparison across different tokenizer configurations.

### Usage

```python
import torch
from prepare import Tokenizer, evaluate_bpb

tokenizer = Tokenizer.from_directory()
batch_size = 128

# Model must be in eval mode
model.eval()

# Evaluate on validation set
with torch.amp.autocast(device_type="cuda", dtype=torch.bfloat16):
    val_bpb = evaluate_bpb(model, tokenizer, batch_size)

print(f"Validation BPB: {val_bpb:.6f}")  # Lower is better, e.g., 0.997900
```
```

--------------------------------

### Sort DataFrame by Delta Improvement

Source: https://github.com/karpathy/autoresearch/blob/master/analysis.ipynb

Sorts a pandas DataFrame by the 'delta' column in descending order. This is useful for ranking items based on their improvement.

```python
hits = hits.sort_values("delta", ascending=False)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.