### Install Package

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

Installs the package in editable mode. This is a common setup step for development.

```bash
pip install -e .
```

--------------------------------

### Verify Installation with Random Baseline

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Test your installation by running a task with the random baseline adapter. This does not require API keys.

```bash
uv run main.py \
  --data_dir data/sample/tasks \
  --config random-baseline \
  --task_id 66e6c45b \
  --save_submission_dir submissions/test-random \
  --log-level INFO
```

--------------------------------

### Install Project Dependencies with uv

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

Install all project dependencies, including all adapters and SDKs, using uv.

```bash
uv sync
```

--------------------------------

### Install Package

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/AGENTS.md

Installs the package in editable mode. Use this for development.

```bash
make install
```

--------------------------------

### Clone Repository and Install Dependencies

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Clone the project repository and install dependencies using uv or pip. Ensure you have Python 3.10+ installed.

```bash
git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking

# Install dependencies with uv (recommended)
uv sync

# Or with pip
pip install -e .
```

--------------------------------

### Example Model Configuration in YAML

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

An example of a model configuration entry in `models.yml`. This defines parameters for a specific model, including its provider, pricing, and optional provider-specific arguments.

```yaml
- name: "gpt-4o-2024-11-20"   # config name you reference on the CLI; typically includes the reasoning level for clarity (e.g., "-basic", "-advanced")
    model_name: "gpt-4o-2024-11-20"  # provider’s actual model id
    provider: "openai"         # must match an adapter
    max_output_tokens: 4096    # optional; provider-specific
    temperature: 0.0           # optional; provider-specific
    pricing:
      date: "2024-11-20"
      input: 5.00              # USD per 1M input tokens
      output: 15.00            # USD per 1M output tokens
```

--------------------------------

### Set Up API Keys

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Copy the example environment file and edit it to include your API keys for services like OpenAI and Anthropic.

```bash
# Copy example env file
cp .env.example .env

# Edit .env and add your API keys
# OPENAI_API_KEY=sk-proj-...
# ANTHROPIC_API_KEY=sk-ant-...
# etc.
```

--------------------------------

### Token Cost Calculation Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Illustrates a practical example of calculating the total cost for a model attempt using specific token counts and pricing.

```plaintext
Model: gpt-4o-2024-11-20
Input: $5.00 per 1M tokens
Output: $15.00 per 1M tokens

Attempt usage:
  prompt_tokens: 500
  completion_tokens: 50
  reasoning_tokens: 0

Cost calculation:
  input_cost = 500 × (5.00 / 1_000_000) = $0.0025
  output_cost = 50 × (15.00 / 1_000_000) = $0.00075
  total_cost = $0.00325
```

--------------------------------

### Example Usage of ModelConfig

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/schemas.md

Demonstrates how to read model configuration and access its properties like model name, provider, and temperature.

```python
from arc_agi_benchmarking.utils.task_utils import read_models_config

config = read_models_config("gpt-4o-2024-11-20")
print(f"Model: {config.model_name}")
print(f"Provider: {config.provider}")
print(f"Temperature: {config.kwargs.get('temperature', 'default')}")
```

--------------------------------

### Make Commands

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

A set of convenience commands for common tasks like installation, testing, running benchmarks, and scoring.

```bash
make install
```

```bash
make test
```

```bash
make test-verbose
```

```bash
make run-sample
```

```bash
make run-batch
```

```bash
make score
```

```bash
make clean
```

--------------------------------

### Force Restart Command

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Demonstrates how to force a restart of a benchmarking run, ignoring any existing checkpoints and starting from scratch.

```bash
uv run cli/run_all.py --config gpt-4o --no-resume ...
# Ignores checkpoints, starts over
```

--------------------------------

### OpenAIAdapter Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Shows how to initialize the OpenAIAdapter with a specific model configuration and use it to make predictions. Ensure the OPENAI_API_KEY environment variable is set.

```python
from arc_agi_benchmarking.adapters import OpenAIAdapter

# Configure models.yml entry
# - name: "gpt-4o-2024-11-20"
#   model_name: "gpt-4o-2024-11-20"
#   provider: "openai"
#   temperature: 0.0
#   max_tokens: 4096
#   pricing: { input: 5.0, output: 15.0, date: "2024-11-20" }

adapter = OpenAIAdapter("gpt-4o-2024-11-20")
attempt = adapter.make_prediction(prompt)
```

--------------------------------

### ProviderAdapter Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Demonstrates how to instantiate and use a concrete provider adapter like AnthropicAdapter to generate predictions and access attempt details.

```python
from arc_agi_benchmarking.adapters import ProviderAdapter, AnthropicAdapter

# Create adapter for a specific model config
adapter = AnthropicAdapter("claude-3-5-sonnet")

# Generate a prediction
attempt = adapter.make_prediction(
    prompt="...",
    task_id="66e6c45b",
    pair_index=0
)

# Access results
print(f"Model: {attempt.metadata.model}")
print(f"Cost: ${attempt.metadata.cost.total_cost:.6f}")
print(f"Tokens: {attempt.metadata.usage.total_tokens}")
print(f"Answer: {attempt.answer}")
```

--------------------------------

### Local Storage Backend Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

Demonstrates how to use the LocalStorageBackend for writing, reading, listing, checking existence, and deleting data. Ensure the base directory is correctly specified.

```python
from pathlib import Path
from arc_agi_benchmarking.storage import LocalStorageBackend

# Create backend for submissions/checkpoints
storage = LocalStorageBackend(Path("submissions/gpt-4o"))

# Write checkpoint
checkpoint_data = json.dumps({...}).encode()
storage.write("checkpoints/task_123.json", checkpoint_data)

# Read back
data = storage.read("checkpoints/task_123.json")

# List all checkpoints
checkpoints = storage.list_keys("checkpoints/")

# Check existence
if storage.exists("checkpoints/task_123.json"):
    print("Checkpoint exists")

# Delete
storage.delete("checkpoints/task_123.json")
```

--------------------------------

### CLI Command for Single Task Execution

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/README.md

Example of how to run a single task using the CLI. Replace placeholders with actual paths and configuration details.

```bash
uv run main.py \
  --data_dir {path_to_tasks} \
  --config {model_config} \
  --task_id {task_id} \
  --save_submission_dir {output_dir} \
  [--num_attempts 2] \
  [--retry_attempts 2] \
  [--log-level INFO]
```

--------------------------------

### Setup Global Logging

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Configures global logging for benchmarks, allowing control over log level, metrics collection, and metrics filename prefix. Import from arc_agi_benchmarking.utils.logging_utils.

```python
from arc_agi_benchmarking.utils.logging_utils import setup_logging

setup_logging(
    log_level="INFO",
    metrics_enabled=True,
    metrics_prefix="gpt-4o"
)
```

--------------------------------

### Cost Estimation Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Demonstrates how to estimate the total cost for a large number of tasks using a specific model's pricing.

```plaintext
100 × 4 × 2 × 500 × rate
= 100 × 8 × 500 × rate
= 400,000 tokens × rate

With GPT-4o ($20 per 1M):
400,000 × (20.00 / 1_000_000) = $8.00 estimated
```

--------------------------------

### Model Pricing Configuration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/scoring.md

Example YAML configuration for model pricing, specifying rates per million tokens.

```yaml
pricing:
  input: 5.00       # USD per 1M input tokens
  output: 15.00     # USD per 1M output tokens
  date: "2024-11-20"
```

--------------------------------

### Score Submissions CLI Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Evaluate submitted solutions against ground truth tasks using the scoring script. Provide directories for tasks, submissions, and results.

```bash
uv run src/arc_agi_benchmarking/scoring/scoring.py \
  --task_dir data/sample/tasks \
  --submission_dir submissions/gpt-4o \
  --results_dir results/gpt-4o \
  --print-logs
```

--------------------------------

### Local Storage Path Security Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

Illustrates path security in LocalStorageBackend, showing how invalid keys attempting to escape the base directory are rejected with a ValueError.

```python
# These would raise ValueError
storage.write("../../etc/passwd", b"hack")
storage.read("../../../secret")
```

--------------------------------

### AnthropicAdapter Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Demonstrates how to instantiate and use the AnthropicAdapter to make predictions. Ensure the ANTHROPIC_API_KEY environment variable is set.

```python
from arc_agi_benchmarking.adapters import AnthropicAdapter

# models.yml entry:
# - name: "claude-3-5-sonnet"
#   model_name: "claude-3-5-sonnet-20241022"
#   provider: "anthropic"
#   max_tokens: 4096
#   temperature: 0.0
#   pricing: { input: 3.0, output: 15.0, date: "2024-10-22" }

adapter = AnthropicAdapter("claude-3-5-sonnet")
attempt = adapter.make_prediction(prompt, task_id="66e6c45b")

# Access reasoning
if attempt.metadata.reasoning_summary:
    print(f"Thinking: {attempt.metadata.reasoning_summary}")
```

--------------------------------

### TaskCheckpointManager Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

Demonstrates how to use the TaskCheckpointManager to check progress, resume tasks, record attempt results, and clean up checkpoints. Ensure to initialize the storage backend and TaskCheckpointManager with appropriate parameters.

```python
from arc_agi_benchmarking.checkpoint import TaskCheckpointManager
from arc_agi_benchmarking.storage import LocalStorageBackend
from decimal import Decimal

storage = LocalStorageBackend("submissions/gpt-4o")
manager = TaskCheckpointManager(storage, task_id="66e6c45b")

# Check progress
for pair_index in range(num_test_pairs):
    next_attempt = manager.get_next_attempt_index(pair_index, max_attempts=3)
    if next_attempt is None:
        print(f"Pair {pair_index}: Complete (all attempts done)")
    else:
        print(f"Pair {pair_index}: Resume at attempt {next_attempt}")

# Run attempt
response = adapter.make_prediction(...)

# Record result
manager.record_attempt(
    test_pair_index=0,
    attempt_index=0,
    response=response.answer,
    cost_usd=Decimal(str(response.metadata.cost.total_cost)),
    tokens_input=response.metadata.usage.prompt_tokens,
    tokens_output=response.metadata.usage.completion_tokens,
    duration_seconds=2.5
)

# After task complete, clean up
manager.delete_checkpoint()
```

--------------------------------

### Google Gemini Model Configuration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Example configuration for a Google Gemini model, specifying its name, provider, pricing, and parameters like temperature.

```yaml
- name: "gemini-2-flash"
  model_name: "gemini-2.0-flash"
  provider: "gemini"
  temperature: 0.7
  max_tokens: 4096
  pricing:
    date: "2024-12-19"
    input: 0.075
    output: 0.30
```

--------------------------------

### OpenAI GPT-4o Model Configuration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Example configuration for OpenAI's GPT-4o model, specifying its name, provider, pricing, and common parameters.

```yaml
- name: "gpt-4o-2024-11-20"
  model_name: "gpt-4o-2024-11-20"
  provider: "openai"
  temperature: 0.0
  max_tokens: 4096
  pricing:
    date: "2024-11-20"
    input: 5.00
    output: 15.00
```

--------------------------------

### Provider Initialization Function

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Illustrates how to initialize a specific provider adapter using its name and configuration. This function is typically used in main entry points to get an instance of the desired adapter.

```python
def init_provider(provider_name: str) -> ProviderAdapter:
    adapter_cls = PROVIDER_ADAPTERS[provider_name]
    return adapter_cls(config)
```

--------------------------------

### Run Single Task CLI Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Execute a single task using the main.py runner. Ensure all required parameters like data directory, config, task ID, and save directory are provided.

```bash
uv run main.py \
  --data_dir data/sample/tasks \
  --config gpt-4o-2024-11-20 \
  --task_id 66e6c45b \
  --save_submission_dir submissions/test \
  --num_attempts 2 \
  --log-level INFO
```

--------------------------------

### RandomAdapter Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Demonstrates how to instantiate and use the RandomAdapter for making predictions. This adapter is useful for smoke tests and debugging as it requires no API key and incurs no cost or token usage.

```python
from arc_agi_benchmarking.adapters import RandomAdapter

# No API key needed
adapter = RandomAdapter("random-baseline")
attempt = adapter.make_prediction(prompt)

print(f"Cost: ${attempt.metadata.cost.total_cost}")  # $0.00
print(f"Tokens: {attempt.metadata.usage.total_tokens}")  # 0
print(f"Answer: {attempt.answer}")  # Random grid
```

--------------------------------

### AsyncRequestRateLimiter Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/resilience.md

Demonstrates how to instantiate and use the AsyncRequestRateLimiter for throttling API calls, both with explicit acquire calls and as an asynchronous context manager.

```python
from arc_agi_benchmarking.utils.rate_limiter import AsyncRequestRateLimiter
import asyncio

# Create limiter: 10 requests per second, capacity of 20
limiter = AsyncRequestRateLimiter(rate=10.0, capacity=20.0)

async def call_api():
    # Wait until 1 request quota is available, then consume it
    await limiter.acquire(1)
    # API call here...
    print("Called API")

async def call_many():
    # Use as context manager
    async with limiter:
        # API call here...
        print("Called API")

# Run
asyncio.run(call_api())
```

--------------------------------

### Handle Google ResourceExhausted

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching Google's ResourceExhausted exception, indicating quota limits. This is automatically retried.

```python
from google.api_core.exceptions import ResourceExhausted

try:
    response = model.generate_content(...)
except ResourceExhausted as e:
    logger.warning(f"Google quota exhausted: {e}")
    # Caught by tenacity retry logic
```

--------------------------------

### Batch Runner Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

Demonstrates the typical usage pattern for BatchProgressManager in a batch runner. It shows how to initialize tasks, claim and process them in a loop, and mark their completion or failure, including cost and token tracking.

```python
from arc_agi_benchmarking.checkpoint import BatchProgressManager
from arc_agi_benchmarking.storage import LocalStorageBackend
from decimal import Decimal
import uuid

run_id = str(uuid.uuid4())
storage = LocalStorageBackend("submissions/gpt-4o")
progress_mgr = BatchProgressManager(storage, run_id=run_id)

# Initialize task list
task_ids = ["66e6c45b", "2dda855f", ...]
progress_mgr.initialize_tasks(task_ids, attempts_per_task=2)

# Process tasks in parallel workers
while True:
    # Worker tries to claim next task
    task_id = progress_mgr.claim_next_task()
    if not task_id:
        break  # All done
    
    try:
        # Process task
        results = run_task(task_id)
        
        # Mark complete with costs
        progress_mgr.mark_completed(
            task_id=task_id,
            cost_usd=Decimal(str(total_cost)),
            tokens_input=total_tokens_in,
            tokens_output=total_tokens_out
        )
    except Exception as e:
        # Mark failed with error
        progress_mgr.mark_failed(
            task_id=task_id,
            error=str(e),
            cost_usd=Decimal(str(partial_cost))
        )

# Get final stats
stats = progress_mgr.get_stats()
print(f"Completed: {stats['completed']}")
print(f"Total cost: ${stats['total_cost']}")
```

--------------------------------

### Run Batch Tasks CLI Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Process multiple tasks in parallel using the run_all.py script. Specify the data directory, model configuration, and output directory for submissions.

```bash
uv run cli/run_all.py \
  --data_dir data/arc-agi/data/evaluation \
  --config gpt-4o-2024-11-20 \
  --save_submission_dir submissions/gpt-4o \
  --workers 8 \
  --num_attempts 2 \
  --log-level INFO
```

--------------------------------

### CircuitBreaker Usage Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/resilience.md

Demonstrates how to instantiate and use the CircuitBreaker to protect API calls. It shows how to handle CircuitBreakerOpenError and record successes or failures.

```python
from arc_agi_benchmarking.resilience import CircuitBreaker, CircuitBreakerOpenError

breaker = CircuitBreaker(
    name="openai",
    failure_threshold=5,
    recovery_timeout=60.0,
    success_threshold=2,
    failure_exceptions={RateLimitError, Timeout},
    excluded_exceptions={ValidationError}
)

async def make_api_call():
    try:
        # Check state; may raise CircuitBreakerOpenError
        if breaker.state == CircuitBreakerState.OPEN:
            raise CircuitBreakerOpenError(
                "Circuit breaker is open, failing fast",
                provider="openai",
                recovery_time=breaker.stats.last_state_change_time
            )
        
        # Make API call
        response = await api.call(...)
        breaker.record_success()
        return response
    except Exception as e:
        breaker.record_failure(e)
        raise
```

--------------------------------

### Anthropic Claude 3.5 Sonnet Configuration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Example configuration for Anthropic's Claude 3.5 Sonnet model, including pricing and basic parameters.

```yaml
- name: "claude-3-5-sonnet"
  model_name: "claude-3-5-sonnet-20241022"
  provider: "anthropic"
  temperature: 0.0
  max_tokens: 4096
  pricing:
    date: "2024-10-22"
    input: 3.00
    output: 15.00
```

--------------------------------

### Load Training Pairs from Task

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Loads training example pairs (input/output) from a specified task JSON file. This utility function requires the directory containing the tasks and the task ID.

```python
from arc_agi_benchmarking.utils.task_utils import get_train_pairs_from_task

pairs = get_train_pairs_from_task("data/sample/tasks", "66e6c45b")
print(f"Loaded {len(pairs)} training pairs")
```

--------------------------------

### Single Task Execution Example

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Demonstrates how to use the ARCTester class to run a single ARC task. Ensure necessary utility functions for data retrieval are imported.

```python
from main import ARCTester
from arc_agi_benchmarking.utils.task_utils import (
    get_train_pairs_from_task,
    get_test_pairs_from_task
)

tester = ARCTester(
    config="gpt-4o-2024-11-20",
    save_submission_dir="submissions/test",
    num_attempts=2,
    retry_attempts=2
)

task_id = "66e6c45b"
training_pairs = get_train_pairs_from_task("data/sample/tasks", task_id)
test_pairs = get_test_pairs_from_task("data/sample/tasks", task_id)

results = tester.run_task(
    task_id=task_id,
    training_pairs=training_pairs,
    test_pairs=test_pairs,
    num_attempts=2
)

# Save submission
from arc_agi_benchmarking.utils.task_utils import save_submission
save_submission("submissions/test", task_id, results.model_dump())
```

--------------------------------

### Handle StorageReadError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Demonstrates catching StorageReadError during checkpoint loading. If an error occurs, it logs a warning and proceeds with starting fresh.

```python
from arc_agi_benchmarking.storage import StorageReadError

try:
    data = storage.read("checkpoints/task.json")
except StorageReadError as e:
    logger.warning(f"Can't load checkpoint: {e}, starting fresh")
    # Proceed without checkpoint
```

--------------------------------

### Convert Task Pairs to System Prompt

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Converts training example pairs and a test input into a formatted system prompt string. This function loads a template and structures the provided task data for model input.

```python
from arc_agi_benchmarking.prompts.prompt_manager import convert_task_pairs_to_prompt
from arc_agi_benchmarking.utils.task_utils import get_train_pairs_from_task, get_test_input_from_task

task_id = "66e6c45b"
training_pairs = get_train_pairs_from_task("data/sample/tasks", task_id)
test_pairs = get_test_input_from_task("data/sample/tasks", task_id)

prompt = convert_task_pairs_to_prompt(training_pairs, test_pairs[0])
print(f"Prompt length: {len(prompt)} characters")
```

--------------------------------

### ARC Task Visualization (Programmatic)

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Programmatically visualizes ARC tasks in the terminal using ASCII art. Renders training and testing examples. Import ARCViewer and ARCTask from arc_agi_benchmarking.utils and arc_agi_benchmarking.schemas respectively.

```python
from arc_agi_benchmarking.utils.viewer import ARCViewer
from arc_agi_benchmarking.schemas import ARCTask

task = ARCTask.load_from_file("data/sample/tasks/66e6c45b.json")
viewer = ARCViewer(task)
print(viewer.render_terminal())
```

--------------------------------

### setup_logging()

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Configure global logging for benchmarks, including log level and optional metrics collection settings.

```APIDOC
## setup_logging()

### Description
Configure global logging for benchmarks.

### Method
```python
def setup_logging(
    log_level: str = "INFO",
    metrics_enabled: bool = False,
    metrics_prefix: Optional[str] = None
) -> None
```

### Parameters
* **log_level** (str) - Optional - Default: "INFO" - Specifies the verbosity of logs (DEBUG, INFO, WARNING, ERROR, CRITICAL, NONE).
* **metrics_enabled** (bool) - Optional - Default: False - Enable metrics collection.
* **metrics_prefix** (str) - Optional - Default: None - Metrics output filename prefix.
```

--------------------------------

### get_train_pairs_from_task()

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Loads training pairs (input-output examples) from a task's JSON file.

```APIDOC
## get_train_pairs_from_task()

### Description
Loads training pairs (input-output examples) from a task's JSON file.

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Method Signature
`def get_train_pairs_from_task(data_dir: str, task_id: str) -> List[ARCPair]`

### Parameters
- **data_dir** (str) - Required - Directory containing the task files.
- **task_id** (str) - Required - The ID of the task (e.g., "66e6c45b").

### Returns
A list of `ARCPair` objects, each containing an input and output grid.

### File Format Expected
```json
{
  "train": [
    {"input": [[...]], "output": [[...]]},
    ...
  ]
}
```

### Usage Example
```python
from arc_agi_benchmarking.utils.task_utils import get_train_pairs_from_task

pairs = get_train_pairs_from_task("data/sample/tasks", "66e6c45b")
for i, pair in enumerate(pairs):
    print(f"Example {i}: {len(pair.input)}x{len(pair.input[0])} → {len(pair.output)}x{len(pair.output[0])}")
```
```

--------------------------------

### Handle Anthropic RateLimitError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching Anthropic's RateLimitError. This exception is automatically retried.

```python
from anthropic import RateLimitError

try:
    response = client.messages.create(...)
except RateLimitError as e:
    logger.warning(f"Anthropic rate limited: {e}")
    # Caught by tenacity retry logic
```

--------------------------------

### Handle TaskTimeoutError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching TaskTimeoutError and marking the task as failed. This indicates a timeout and does not trigger a retry.

```python
from arc_agi_benchmarking.resilience import TaskTimeoutError

try:
    result = await task_timeout(prediction, timeout_seconds=30)
except TaskTimeoutError:
    logger.error("API call timed out, marking task failed")
    # Fail task, don't retry (timeout indicates hangup)
```

--------------------------------

### Handle TokenMismatchError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching and logging a TokenMismatchError. This scenario is treated as an API error and triggers a retry.

```python
try:
    attempt = adapter.make_prediction(prompt)
except TokenMismatchError as e:
    logger.error(f"Token mismatch: {e}")
    # Treat as API error and retry
```

--------------------------------

### Run Tests

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

Executes all tests within the project. Use -q for quick, -v for verbose, or -x to stop on the first failure.

```bash
pytest -q
```

```bash
pytest -v
```

```bash
pytest -x
```

--------------------------------

### Calculate Per-Attempt Duration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/scoring.md

Formula for calculating the duration of a single attempt using start and end timestamps.

```python
duration = (end_timestamp - start_timestamp).total_seconds()
```

--------------------------------

### Handle StorageWriteError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching StorageWriteError when saving checkpoints. It logs an error, indicating potential loss of progress.

```python
from arc_agi_benchmarking.storage import StorageWriteError

try:
    storage.write("checkpoints/task.json", data)
except StorageWriteError as e:
    logger.error(f"Can't save checkpoint: {e}")
    # May cause loss of progress; alert user
```

--------------------------------

### Handle OpenAI RateLimitError

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Example of catching OpenAI's RateLimitError. This is automatically handled by tenacity's retry logic.

```python
from openai import RateLimitError

try:
    response = client.chat.completions.create(...)
except RateLimitError as e:
    logger.warning(f"OpenAI rate limited: {e}")
    # Caught by tenacity retry logic
```

--------------------------------

### Warning for Empty List Answers

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/scoring.md

Example warning message generated when a prediction results in an empty list, which is not counted as correct.

```text
Warning: Empty list prediction for {task_id}, pair {pair_index}, attempt {attempt_index}
```

--------------------------------

### View Available Models and Providers

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Check the `models.yml` file for available model configurations and `provider_config.yml` for supported providers.

```bash
# Check models.yml
cat src/arc_agi_benchmarking/models.yml

# Supported providers
cat provider_config.yml
```

--------------------------------

### Select Model Configuration for Token Efficiency

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Choose different model configurations to manage costs. More expensive models like 'gpt-4o' have higher token costs compared to 'gpt-4-turbo' or a 'random-baseline'.

```bash
# Expensive (gpt-4o)
uv run cli/run_all.py --config gpt-4o-2024-11-20 ...

# Cheaper (gpt-4-turbo)
uv run cli/run_all.py --config gpt-4-turbo-2024-04-09 ...

# Free (random baseline)
uv run cli/run_all.py --config random-baseline ...
```

--------------------------------

### Run Sample Benchmark

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

Executes a random baseline on a single sample task. No API key is required for this specific run.

```bash
python main.py --data_dir data/sample/tasks --config random-baseline --task_id 66e6c45b --save_submission_dir submissions/test
```

--------------------------------

### Run Batch Job for Full Sample Dataset

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Execute a batch run on the full sample dataset (3 tasks) with specified configuration, save directory, number of attempts, and workers.

```bash
# Full sample dataset (3 tasks)
uv run cli/run_all.py \
  --data_dir data/sample/tasks \
  --config gpt-4o-2024-11-20 \
  --save_submission_dir submissions/gpt-4o-sample \
  --num_attempts 2 \
  --workers 4 \
  --log-level INFO
```

--------------------------------

### Run Batch Benchmark

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

Executes a random baseline on all sample tasks. This is the main entry point for batch benchmarks.

```bash
python cli/run_all.py --data_dir data/sample/tasks --config random-baseline --save_submission_dir submissions/test
```

--------------------------------

### Run Batch Benchmark

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/AGENTS.md

Executes a random baseline on all sample tasks. This is the main entry point for batch benchmarks.

```bash
make run-batch
```

```bash
python cli/run_all.py --data_dir data/sample/tasks --config random-baseline --save_submission_dir submissions/test
```

--------------------------------

### Initialize and Run ARCScorer

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/scoring.md

Instantiate the ARCScorer with directories for tasks and submissions, then score all submissions. Prints overall accuracy, total cost, token usage, and duration.

```python
from arc_agi_benchmarking.scoring.scoring import ARCScorer

scorer = ARCScorer(
    task_dir="data/sample/tasks",
    submission_dir="submissions/gpt-4o",
    print_logs=True,
    results_dir="results/gpt-4o"
)

total_score, num_tasks = scorer.score_submission()

print(f"Overall accuracy: {total_score / num_tasks:.2%}")
print(f"Total cost: ${scorer.total_cost:.2f}")
print(f"Total tokens: {scorer.total_all_tokens}")
print(f"Total duration: {scorer.total_duration:.1f}s")
```

--------------------------------

### Run Sample Benchmark

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/AGENTS.md

Runs a random baseline on a single sample task. This command does not require an API key.

```bash
make run-sample
```

```bash
python main.py --data_dir data/sample/tasks --config random-baseline --task_id 66e6c45b --save_submission_dir submissions/test
```

--------------------------------

### Project Directory Structure Overview

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Understand the layout of the arc-agi-benchmarking project, including configuration files, source code, data directories, and output locations.

```text
arc-agi-benchmarking/
├── .env                          # API keys (created from .env.example)
├── models.yml                    # Model configurations
├── provider_config.yml           # Provider rate limits
├── main.py                       # Single-task runner
├── cli/
│   ├── run_all.py               # Batch runner
│   └── submission_cli.py         # Submission tools
├── src/arc_agi_benchmarking/
│   ├── models.yml               # Same as root (symlink or copy)
│   ├── adapters/                # Provider adapters
│   ├── checkpoint/              # Progress tracking
│   ├── scoring/                 # Scoring logic
│   ├── storage/                 # Storage backends
│   ├── prompts/                 # Prompt templates
│   ├── resilience/              # Rate limiting, circuit breaker
│   ├── utils/                   # Helper functions
│   └── tests/                   # Test suite
├── data/
│   ├── sample/tasks/            # Sample ARC tasks (provided)
│   └── arc-agi/                 # Clone full datasets here
├── submissions/                 # Output directory (auto-created)
├── checkpoints/                 # Progress checkpoints (auto-created)
├── results/                     # Scoring results (auto-created)
└── metrics_output/              # Timing metrics (auto-created)
```

--------------------------------

### Score a Run with Taskset and Submission

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

Use this command to score a benchmark run. Ensure you have the task directory containing solutions and the submission directory for your configuration.

```bash
uv run src/arc_agi_benchmarking/scoring/scoring.py \
  --task_dir <data_dir>/data/evaluation \
  --submission_dir submissions/<config> \
  --results_dir results/<config>
```

--------------------------------

### Run Batch Benchmark

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Execute the main benchmarking script with specified data, configuration, and output directories. Configure the number of attempts and log level for the run.

```bash
uv run cli/run_all.py \
  --data_dir data/arc-agi/data/evaluation \
  --config gpt-4o-2024-11-20 \
  --save_submission_dir submissions/gpt-4o-2024-11-20 \
  --num_attempts 2 \
  --log-level INFO
```

--------------------------------

### Calculate Requests Per Second (RPS)

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Demonstrates how to calculate the Requests Per Second (RPS) from the rate and period configuration.

```text
RPS = rate / period
Example: rate: 400, period: 60 → 6.67 requests/second
```

--------------------------------

### Local Filesystem Storage Backend

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

A concrete implementation of StorageBackend using the local filesystem. It supports atomic writes, path escaping, and automatic directory creation.

```python
class LocalStorageBackend(StorageBackend):
    def __init__(self, base_dir: Path | str)
    def read(self, key: str) -> Optional[bytes]
    def write(self, key: str, data: bytes) -> None
    def exists(self, key: str) -> bool
    def delete(self, key: str) -> None
    def list_keys(self, prefix: str) -> list[str]
```

--------------------------------

### Get Circuit Breaker

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/resilience.md

Retrieves or creates a singleton circuit breaker instance for a given provider. Use this to manage and track failures for specific API providers, preventing repeated calls to failing services.

```python
def get_circuit_breaker(provider: str) -> CircuitBreaker
```

```python
from arc_agi_benchmarking.resilience import get_circuit_breaker

breaker = get_circuit_breaker("openai")
```

--------------------------------

### Batch Processing with Checkpointing

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/README.md

Illustrates batch processing of tasks with checkpointing enabled using LocalStorageBackend and TaskCheckpointManager. This pattern helps manage progress and resume interrupted runs.

```python
from arc_agi_benchmarking.checkpoint import TaskCheckpointManager
from arc_agi_benchmarking.storage import LocalStorageBackend
from pathlib import Path

storage = LocalStorageBackend(Path("submissions/gpt-4o"))
checkpoint_mgr = TaskCheckpointManager(storage, task_id="66e6c45b")

# Check what's already done
for pair_idx in range(num_test_pairs):
    next_attempt = checkpoint_mgr.get_next_attempt_index(pair_idx, max_attempts=2)
    if next_attempt is None:
        print(f"Pair {pair_idx}: already complete")
    else:
        print(f"Pair {pair_idx}: run attempt {next_attempt}")

# Run an attempt
response = adapter.make_prediction(prompt)
checkpoint_mgr.record_attempt(
    test_pair_index=0,
    attempt_index=0,
    response=response.answer,
    cost_usd=Decimal(str(response.metadata.cost.total_cost)),
    tokens_input=response.metadata.usage.prompt_tokens,
    tokens_output=response.metadata.usage.completion_tokens,
    duration_seconds=2.5
)
```

--------------------------------

### Run All Bundled Sample Tasks with Random Solver

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

Execute all bundled sample ARC-AGI tasks using the 'random-baseline' solver. This is useful for a quick test of the batch processing capabilities.

```bash
uv run cli/run_all.py \
  --config random-baseline \
  --data_dir data/sample/tasks \
  --save_submission_dir submissions/random-baseline-sample \
  --log-level INFO
```

--------------------------------

### Single Prediction with OpenAIAdapter

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/README.md

Demonstrates how to make a single prediction using the OpenAIAdapter. Ensure necessary imports and task data are available.

```python
from arc_agi_benchmarking.adapters import OpenAIAdapter
from arc_agi_benchmarking.prompts.prompt_manager import convert_task_pairs_to_prompt
from arc_agi_benchmarking.utils.task_utils import (
    get_train_pairs_from_task,
    get_test_input_from_task
)

adapter = OpenAIAdapter("gpt-4o-2024-11-20")
training = get_train_pairs_from_task("data/tasks", "66e6c45b")
test_input = get_test_input_from_task("data/tasks", "66e6c45b")[0]

prompt = convert_task_pairs_to_prompt(training, test_input)
attempt = adapter.make_prediction(prompt, task_id="66e6c45b", pair_index=0)

print(f"Answer: {attempt.answer}")
print(f"Cost: ${attempt.metadata.cost.total_cost:.6f}")
print(f"Tokens: {attempt.metadata.usage.total_tokens}")
```

--------------------------------

### Troubleshoot 'No matching configuration found'

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

This error indicates the configuration name is not in `models.yml`. Check spelling or add the configuration to the file.

```bash
# Check spelling, add to models.yml, use `cat src/arc_agi_benchmarking/models.yml` to list available.
```

--------------------------------

### run_preflight()

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Perform pre-run validation and cost estimation for benchmark tasks. It checks various prerequisites and provides an estimated cost.

```APIDOC
## run_preflight()

### Description
Pre-run validation and cost estimation.

### Method
```python
async def run_preflight(
    config: str,
    data_dir: str,
    num_attempts: int = 1
) -> dict
```

### Parameters
* **config** (str) - Required - Model config name.
* **data_dir** (str) - Required - Task directory.
* **num_attempts** (int) - Optional - Default: 1 - Number of attempts per task for cost estimation.

### Returns
Dict with estimated cost, number of tasks, number of test pairs, total attempts, and warnings.
```

--------------------------------

### Basic Model Configuration Structure

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Defines the structure for configuring models, including name, provider, pricing, and optional parameters like temperature and max tokens.

```yaml
models:
  - name: "{config_name}"
    model_name: "{provider_model_id}"
    provider: "{provider_name}"
    pricing:
      date: "YYYY-MM-DD"
      input: {price_per_1m}
      output: {price_per_1m}
    # Provider-specific parameters (passed as kwargs):
    temperature: 0.0
    max_tokens: 4096
    top_p: 1.0
    # Batch/stream modes:
    stream: false
    batch: false
    # Rate limit override:
    rate_limit:
      rate: 50
      period: 60
```

--------------------------------

### Enable Metrics Collection via CLI

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Enables the collection of per-provider timings and per-attempt costs when running benchmark tasks. Results are saved to a CSV file.

```bash
uv run cli/run_all.py \
  --config gpt-4o \
  --enable-metrics
```

--------------------------------

### Provider Adapters Dictionary

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/adapters.md

Shows the structure of the PROVIDER_ADAPTERS dictionary used for registering and discovering available provider adapters within the framework.

```python
PROVIDER_ADAPTERS = {
    "anthropic": AnthropicAdapter,
    "openai": OpenAIAdapter,
    "deepseek": DeepseekAdapter,
    ...
}
```

--------------------------------

### Run Tests

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/AGENTS.md

Executes all tests within the project. Use `pytest -q` for a quick run or `pytest -v` for verbose output. `pytest -x` stops on the first failure.

```bash
make test
```

```bash
pytest -q
```

```bash
pytest -v
```

```bash
pytest -x
```

--------------------------------

### Add New Provider

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/AGENTS.md

Steps to integrate a new AI provider into the framework. This involves creating a new adapter file, exporting it, and updating the main runner logic and model configurations.

```python
# 1. Create adapters/<provider>.py extending ProviderAdapter or OpenAIBaseAdapter
# 2. Export in adapters/__init__.py
# 3. Add branch in main.py to recognize the provider
# 4. Add model config in models.yml
```

--------------------------------

### Clone the ARC-AGI Benchmarking Repository

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

Clone the repository to your local machine and navigate into the project directory.

```bash
git clone https://github.com/arcprize/arc-agi-benchmarking.git
cd arc-agi-benchmarking
```

--------------------------------

### Load Training Pairs from Task

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/utilities.md

Loads training input-output pairs from a specified task JSON file. Expects a file format with a 'train' key containing a list of pairs.

```python
from arc_agi_benchmarking.utils.task_utils import get_train_pairs_from_task

pairs = get_train_pairs_from_task("data/sample/tasks", "66e6c45b")
for i, pair in enumerate(pairs):
    print(f"Example {i}: {len(pair.input)}x{len(pair.input[0])} → {len(pair.output)}x{len(pair.output[0])}")
```

--------------------------------

### View Task Visually

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/CLAUDE.md

Launches a terminal-based visualizer for a specific ARC task. Useful for debugging and understanding task structure.

```bash
python -m arc_agi_benchmarking.utils --task data/sample/tasks/66e6c45b.json
```

--------------------------------

### Run All Tasks in Batch

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/README.md

Execute all tasks in a batch using the `uv run cli/run_all.py` command. Specify directories for data, model configuration, and submission saving. Optional parameters include the number of workers, attempts, and log level.

```bash
uv run cli/run_all.py \
  --data_dir {path_to_tasks} \
  --config {model_config} \
  --save_submission_dir {output_dir} \
  [--workers 8] \
  [--num_attempts 2] \
  [--log-level INFO]
```

--------------------------------

### Test Single Task with Random Baseline

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/errors.md

Execute the main script for a single task using specific data and configuration directories, and save output to a test directory. The random baseline requires no API keys and is suitable for basic testing.

```bash
uv run main.py \
  --data_dir data/sample/tasks \
  --config random-baseline \
  --task_id 66e6c45b \
  --save_submission_dir test-output \
  --log-level DEBUG
```

--------------------------------

### Full Benchmark Run with Workers

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/task-execution.md

Run a full benchmark with a specified number of parallel workers. This command configures the data directory, model configuration, submission output, worker count, attempts per task, and logging level.

```bash
uv run cli/run_all.py \
  --data_dir data/arc-agi/data/evaluation \
  --config gpt-4o-2024-11-20 \
  --save_submission_dir submissions/gpt-4o \
  --workers 8 \
  --num_attempts 3 \
  --log-level INFO
```

--------------------------------

### Resume Run Command

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/configuration.md

Shows the command to resume a previous benchmarking run, which automatically loads progress and skips completed tasks.

```bash
uv run cli/run_all.py --config gpt-4o ...
# Loads progress.json, skips completed tasks
```

--------------------------------

### Enable Metrics for Cost Tracking

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Enable metrics during batch runs to track actual costs and token usage. The metrics will be saved to a CSV file.

```bash
uv run cli/run_all.py \
  --config gpt-4o-2024-11-20 \
  --enable-metrics \
  ...
# Check: metrics_output/gpt-4o_metrics.csv
```

--------------------------------

### Abstract Storage Backend Interface

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/checkpointing-storage.md

Defines the interface for all storage backends. Implementations must provide methods for reading, writing, checking existence, deleting, and listing keys.

```python
class StorageBackend(abc.ABC):
    @abstractmethod
    def read(self, key: str) -> Optional[bytes]
    
    @abstractmethod
    def write(self, key: str, data: bytes) -> None
    
    @abstractmethod
    def exists(self, key: str) -> bool
    
    @abstractmethod
    def delete(self, key: str) -> None
    
    @abstractmethod
    def list_keys(self, prefix: str) -> list[str]
    
    def read_text(self, key: str, encoding: str = "utf-8") -> Optional[str]
    
    def write_text(self, key: str, text: str, encoding: str = "utf-8") -> None
```

--------------------------------

### Clone Full Dataset Repositories

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Clone the ARC-AGI datasets to your local machine. Use the appropriate path for your batch runs.

```bash
# ARC-AGI-1 (2019, ~400 tasks)
git clone https://github.com/fchollet/ARC-AGI.git data/arc-agi

# ARC-AGI-2 (2025, ~900 tasks)
git clone https://github.com/arcprize/ARC-AGI-2.git data/arc-agi-2
```

--------------------------------

### Define Model Configuration

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/schemas.md

Represents the configuration for a specific language model, including its name, provider, pricing, and optional API type and keyword arguments.

```python
class ModelConfig(BaseModel):
    name: str
    model_name: str
    provider: str
    pricing: ModelPricing
    api_type: Optional[str] = APIType.CHAT_COMPLETIONS
    kwargs: Dict[str, Any] = {}
```

--------------------------------

### Estimate Benchmark Costs

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Estimate the cost of running benchmarks using a Python script. Adjust the number of tasks, pairs per task, and average tokens as needed.

```python
from arc_agi_benchmarking.utils.task_utils import read_models_config
config = read_models_config('gpt-4o-2024-11-20')
tasks = 100  # Adjust
pairs_per_task = 4
attempts = 2
avg_tokens = 500

total_tokens = tasks * pairs_per_task * attempts * avg_tokens
input_cost = (total_tokens * 0.4) * (config.pricing.input / 1_000_000)
output_cost = (total_tokens * 0.1) * (config.pricing.output / 1_000_000)
print(f'Estimated: $"{input_cost + output_cost:.2f}"')

```

--------------------------------

### Run Tests

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/README.md

Execute the project's tests using pytest. This is useful for verifying changes and ensuring the project's integrity.

```bash
uv run pytest
```

--------------------------------

### Run a Single Task for Debugging

Source: https://github.com/arcprize/arc-agi-benchmarking/blob/main/_autodocs/quick-start.md

Execute a single task with a specific model configuration for debugging purposes. Use --num_attempts 1 and --log-level DEBUG for detailed output.

```bash
uv run main.py \
  --data_dir data/sample/tasks \
  --config gpt-4o-2024-11-20 \
  --task_id 66e6c45b \
  --save_submission_dir submissions/debug \
  --num_attempts 1 \
  --log-level DEBUG
```