### Install SIA MLE-Bench Extras

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Installs the necessary extras for SIA to work with MLE-Bench datasets, including the mle-bench package itself.

```bash
pip install 'sia-agent[mlebench]'
pip install git+https://github.com/openai/mle-bench
```

--------------------------------

### Clone Repository and Set Up Virtual Environment

Source: https://github.com/hexo-ai/sia/blob/main/CONTRIBUTING.md

Clone your fork of the SIA repository and set up a Python virtual environment. Install the project in development mode with all extras.

```bash
git clone https://github.com/<your-username>/sia.git
cd sia

python3 -m venv .venv
source .venv/bin/activate

pip install -e ".[dev]"
```

--------------------------------

### Install OpenHands Agent Implementation

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Installs the SIA agent with the OpenHands agent implementation, supporting multiple providers like Gemini, OpenAI, and Anthropic. Ensure you export the API key for each provider you intend to use.

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install 'sia-agent[openhands]'

# Export the key(s) for the provider(s) you'll use:
export ANTHROPIC_API_KEY="..."   # for anthropic/* models
export GEMINI_API_KEY="..."      # for gemini/* models (or GOOGLE_API_KEY)
export OPENAI_API_KEY="..."      # for openai/* models
```

--------------------------------

### Target Agent Command-Line Invocation Example

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/meta_prompt.txt

Illustrates how to invoke the target agent script from the command line, specifying dataset and working directories.

```bash
python target_agent.py --dataset_dir /path/to/dataset --working_dir /path/to/working
```

--------------------------------

### Install Claude Agent Implementation

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Installs the SIA agent with the Claude agent implementation, requiring an Anthropic API key. Use this if you are only running Claude models.

```bash
python3 -m venv .venv && source .venv/bin/activate
pip install 'sia-agent[claude]'
export ANTHROPIC_API_KEY="..."
```

--------------------------------

### Serve Sia Dashboard

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Start the Sia web dashboard to visualize run results. The dashboard serves the './runs' directory by default.

```bash
sia web                                  # serve ./runs at http://127.0.0.1:8000
sia web --runs-dir ./runs --port 8080    # custom directory / port
```

--------------------------------

### Start SIA Web Dashboard

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Launches the SIA web dashboard for visualizing results and monitoring runs. The dashboard auto-starts during `sia run` by default.

```bash
sia web                  # → http://127.0.0.1:8000
```

--------------------------------

### Start SandboxFusion Docker Container

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

This command starts the SandboxFusion service using Docker. It maps port 8080 for communication and runs the specified image. Ensure Docker is installed and sufficient disk space is available.

```bash
docker run \
  --rm \
  -it \
  -p 8080:8080 \
  --name sia-sandbox-fusion \
  volcengine/sandbox-fusion:server-20250609
```

--------------------------------

### Install Missing Python Packages

Source: https://github.com/hexo-ai/sia/blob/main/docs/troubleshooting.md

If packages are missing in the per-run virtual environment, use this command to install them manually. Ensure you are using the correct venv path.

```bash
runs/run_1/venv/bin/pip install anthropic
```

--------------------------------

### Complete Evaluation Script Example

Source: https://github.com/hexo-ai/sia/blob/main/EVALUATION_GUIDE.md

A full example of an `evaluate.py` script, including loading ground truth, processing submissions, calculating metrics, and saving results.json. This script is designed to be placed in `tasks/<task_name>/data/public/evaluate.py`.

```python
"""Evaluate predictions against ground truth."""

import pandas as pd
from pathlib import Path

# Path to ground truth (private data)
TASK_DIR = Path(__file__).parent.parent.parent  # Go up from data/public/
TRUTH_PATH = TASK_DIR / "data/private/test.csv"


def evaluate(submission_path: Path) -> dict:
    """Evaluate submission against ground truth."""

    # Load ground truth
    truth = pd.read_csv(TRUTH_PATH)

    # Load submission
    pred = pd.read_csv(submission_path)

    # Merge and calculate accuracy
    merged = truth.merge(pred, on="id", how="left")
    correct = (merged["label"] == merged["prediction"]).sum()
    total = len(merged)
    accuracy = correct / total

    return {
        "accuracy": float(accuracy),
        "n_correct": int(correct),
        "n_total": int(total)
    }


def main():
    """For manual testing."""
    import argparse
    import json
    import sys

    parser = argparse.ArgumentParser()
    parser.add_argument("--gen-dir", type=Path, required=True)
    args = parser.parse_args()

    # Find submission file (YOU handle this - check for whatever filename you expect)
    submission = args.gen_dir / "submission.csv"
    if not submission.exists():
        print(f"Error: {submission} not found")
        sys.exit(1)

    # Evaluate
    print("Evaluating...")
    results = evaluate(submission)

    # IMPORTANT: Save results.json (required by orchestrator)
    results_path = args.gen_dir / "results.json"
    with open(results_path, 'w') as f:
        json.dump(results, f, indent=2)
    print(f"Saved: {results_path}")

    # Print summary
    print(f"Accuracy: {results['accuracy']:.4f}")


if __name__ == "__main__":
    main()
```

--------------------------------

### Sample Agent Execution Trajectory

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/meta_prompt.txt

An example of the JSON structure for logging an agent's execution trajectory, including user messages.

```json
{
  "messages": [
    {
      "role": "user",
      "content": "hi"
    }
  ]
}
```

--------------------------------

### Submission Format Example

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/spaceship-titanic/reference/SAMPLE_TASK_DESCRIPTIONS.md

This is the required format for submitting your cluster assignments. Ensure CustomerId and ClusterId are correctly mapped.

```text
CustomerId,ClusterId
CUST_001,0
CUST_002,4
CUST_003,1
etc.
```

--------------------------------

### Visualize SIA Runs

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Starts the SIA web dashboard to visualize previous runs. By default, it serves the './runs' directory at http://127.0.0.1:8000. You can specify a different directory and port.

```bash
sia web                          # serve ./runs at http://127.0.0.1:8000
sia web --runs-dir ./runs --port 8080
```

--------------------------------

### Copy Reference Agent Template

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Copies the template for the reference agent into the task's reference directory. This serves as a starting point for the agent's implementation.

```bash
cp sia/tasks/_shared/reference_target_agent.py my-tasks/gpqa/reference/
```

--------------------------------

### Chess Hard Task Dataset Question Format

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/longcot-chess/data/public/task.md

Example JSON structure for questions in the Chess Hard Task dataset, illustrating prompts for 'best_3_moves' and 'knight_path' problems.

```json
[
  {
    "question_id": "best_3_moves_hard_1",
    "prompt": "You are given a chess position using this FEN: 8/8/4k2K/1pp5/3pP1P1/pP6/P1P5/8 b - - 0 38\nGive the 3 best next moves in this position using the Standard Algebraic Notation (SAN) format. Return your answer in the format: solution = [move1, move2, move3]",
    "problem": {
      "template": "best_3_moves"
    }
  },
  {
    "question_id": "knight_path_hard_1",
    "prompt": "There is a chess board of size 100x100 and certain target squares. Calculate the minimum number of moves it takes for the knight at the given starting position to touch all the target squares...\n\nReturn your answer in the format: solution = <integer>",
    "problem": {
      "template": "knight_path"
    }
  }
]
```

--------------------------------

### SIA Execution Evaluation Results

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_context_success_single.txt

Example of evaluation results returned upon successful execution. This JSON object contains metrics like accuracy, correct predictions, and total items.

```json
{
  "accuracy": 0.9,
  "correct": 9,
  "total": 10
}
```

--------------------------------

### Create Provider and Profile Directories

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Set up the necessary directories to store your custom provider and profile configurations.

```bash
mkdir -p providers profiles
```

--------------------------------

### Running SIA with Bundled Name and Path

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Demonstrates how to run SIA tasks using a target agent profile specified by a bundled name or an explicit file path.

```bash
sia run --task gpqa --target-agent-profile kimi-nebius-target   # bundled name
sia run --task gpqa --target-agent-profile ./profiles/mine.json # explicit path
```

--------------------------------

### Sample Target Agent Initialization

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/meta_prompt.txt

A basic Python script for a target agent. It prints a reference message and is intended to be expanded upon.

```python
print('reference target agent')
```

--------------------------------

### Prepare MLE-Bench Dataset with SIA

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Automates the preparation of an MLE-Bench dataset for SIA, including downloading data, renaming files, and generating task descriptions.

```bash
python -m sia.prepare_mlebench_dataset -c "spaceship-titanic"
```

--------------------------------

### Submission Format Example

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/spaceship-titanic/reference/SAMPLE_TASK_DESCRIPTIONS.md

This is the required format for submitting your predictions, including TranscriptId, Sentiment, and Summary.

```csv
TranscriptId,Sentiment,Summary
TR_990,Frustrated,Wants a refund for a late delivery.
TR_991,Neutral,Inquiring about store holiday hours.
etc.
```

--------------------------------

### Initializing UI Elements

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

This snippet appends the toolbar and host elements to the body and initiates the initial loading of trajectories.

```javascript
body.append(h("div", { class: "toolbar" }, h("span", { class:"muted" }, `${gen.trajectories.length} trajectories`), sel, filter, gen.has_openhands ? h("span", { class:"badge" }, "openhands available") : null));
body.append(host);
load();
```

--------------------------------

### Create Task Directory Structure

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Sets up the expected directory layout for a custom SIA task, including public and private data folders, and a reference directory.

```bash
mkdir -p my-tasks/gpqa/{data/public,data/private,reference}
```

--------------------------------

### Initialize OpenAI Client for Nebius Token Factory

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/meta_prompt_openai.txt

Configure the `openai` client to connect to the Nebius Token Factory API. Ensure the `NEBIUS_API_KEY` environment variable is set.

```python
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokenfactory.us-central1.nebius.com/v1/",
    api_key=os.environ["NEBIUS_API_KEY"],
)
```

--------------------------------

### Set Kaggle API Credentials

Source: https://github.com/hexo-ai/sia/blob/main/docs/troubleshooting.md

Provide Kaggle API credentials using environment variables for `mle-bench` to download competitions. Ensure you have also accepted the competition's rules on Kaggle's website.

```bash
export KAGGLE_USERNAME="..." KAGGLE_KEY="..."
```

--------------------------------

### Run Sia Agent with Kimi-K2.6 on Nebius

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Configure environment variables for Nebius and Anthropic API keys. Then, run the Sia agent specifying Kimi-K2.6 on Nebius as the target model for the GPQA task.

```bash
export NEBIUS_API_KEY="..."        # target provider
export ANTHROPIC_API_KEY="..."     # default-meta agent
sia run --task gpqa --target-agent-profile kimi-nebius-target --max_gen 5 --run_id 2
```

--------------------------------

### Get Current Route Information

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Parses the URL hash to extract the current run, generation, and tab identifiers. Returns an object representing the current route.

```javascript
function currentRoute() {
    const parts = location.hash.replace(/^#\/?/, "").split("/").filter(Boolean);
    return { run: parts[0], gen: parts[1], tab: parts[2] };
}
```

--------------------------------

### RLE Submission Format Example

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/spaceship-titanic/reference/SAMPLE_TASK_DESCRIPTIONS.md

Submissions for lesion segmentation must use Run-Length Encoding (RLE) for predicted masks. This format is a CSV with ImageId and EncodedPixels columns.

```text
ImageId,EncodedPixels
Scan001_Slice1,1 1 5 10
Scan001_Slice2,1 5 22 3
etc.
```

--------------------------------

### Malformed Execution Log Example

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_context_failure_single.txt

This JSON snippet represents a malformed or missing execution log, indicated by an 'error' field. It suggests focusing on agent robustness when such issues arise.

```json
[
  {
    "role": "user",
    "content": "attempt"
  }
]
```

--------------------------------

### SIA Target Agent Execution Trajectory

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_context_success_single.txt

Example of the target agent's execution trajectory, represented as a JSON array of message objects. This shows the conversation history or steps taken by the agent.

```json
[
  {
    "role": "user",
    "content": "solve it"
  }
]
```

--------------------------------

### Run SIA with MLE-Bench Task

Source: https://github.com/hexo-ai/sia/blob/main/README.md

After preparing an MLE-Bench task, run SIA using the generated task directory. This command initiates the self-improvement loop for the competition.

```bash
sia run --task_dir ./tasks/spaceship-titanic --max_gen 5 --run_id 1
```

--------------------------------

### Run SIA with a Custom Task Directory

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Execute the SIA run command, specifying the path to your prepared task directory. The `--max_gen` parameter controls the number of generation steps.

```bash
sia run --task_dir ./my-task --max_gen 5 --run_id 1
```

--------------------------------

### Render Run View

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Renders the main view for a specific run, including breadcrumbs, run name, start time, profile chips, and a generation chart. This function is called once when the run view is initially loaded.

```javascript
// ---- Run view (static chrome: built once per run) ---------------------------
function renderRun(run) {
  const main = $("#main");
  main.innerHTML = "";
  main.append(h("div", { class: "crumbs" }, "runs / " + run.name));
  main.append(h("h2", { style: "margin-bottom:2px" }, run.name));
  if (run.started) main.append(h("div", { class: "muted", style: "font-size:12px;margin-bottom:8px" }, "Started " + run.started));
  main.append(profileChips(run));
  main.append(genChart(run.generations));
  VIEW.pillsRow = h("div", { class: "pill-row" });
  VIEW.genHost = h("div", {});
  main.append(VIEW.pillsRow, VIEW.genHost);
}
```

--------------------------------

### Run SIA with Meta Agent on OpenHands + Gemini

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Executes the SIA orchestrator with a specified meta-agent profile, utilizing OpenHands and a Gemini model. Requires a pre-configured meta-agent profile.

```bash
sia run \
  --task_dir ./my-tasks/gpqa \
  --max_gen 5 \
  --run_id 1 \
  --meta-agent-profile gemini-meta
```

--------------------------------

### Run SIA in Weights Mode with SandboxFusion

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

This command runs SIA in Weights Mode, utilizing SandboxFusion as the training sandbox. Ensure the SandboxFusion service is running and accessible.

```bash
export TINKER_API_KEY="your-tinker-api-key"

sia run --task gpqa --max_gen 5 --run_id 1 \
    --focus weights \
    --training_sandbox sandboxfusion
```

--------------------------------

### Sample Trajectory 0

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_context_success_multi.txt

Represents the first user query or sample processed by the agent.

```json
[
  {
    "role": "user",
    "content": "q0"
  }
]
```

--------------------------------

### SIA Directory Layout

Source: https://github.com/hexo-ai/sia/blob/main/docs/architecture.md

Illustrates the file and directory structure of the SIA project, including core modules, agent implementations, task-specific directories, and the location for generated run artifacts.

```tree
sia/
├── sia/
│   ├── orchestrator.py             # Main orchestration logic
│   ├── context_manager.py          # Run/context tracking
│   ├── prompts.py                  # Meta and feedback prompt builders
│   ├── agent_impls/                # Agent runner backends (claude / openhands / pydantic-ai)
│   ├── prepare_mlebench_dataset.py # MLE-Bench dataset preparation
│   └── tasks/                      # Bundled with the wheel
│       ├── _shared/
│       │   ├── reference_target_agent.py
│       │   └── sample_agent_execution.json
│       └── {task-id}/              # gpqa, lawbench, longcot-chess, spaceship-titanic
│           ├── data/
│           │   ├── public/         # Public dataset
│           │   │   ├── task.md         # Task description
│           │   │   └── *.csv           # Data files
│           │   └── private/        # Held-out evaluation data
│           └── reference/
│               ├── SAMPLE_TASK_DESCRIPTIONS.md
│               └── reference_target_agent.py
└── runs/                           # Generated during execution
    └── run_{id}/
        ├── venv/                   # Isolated Python environment per run
        └── gen_{n}/                # Each generation's artifacts
            ├── target_agent.py
            ├── agent_execution.json
            └── improvement.md      # gen 2 onwards

```

--------------------------------

### Set Environment Variables for Weights Mode

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Before running SIA in Weights Mode, set the necessary environment variables. TINKER_API_KEY is always required. MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are needed if using Modal as the training sandbox.

```bash
export TINKER_API_KEY="your-tinker-api-key"
export MODAL_TOKEN_ID="your-modal-token-id"
export MODAL_TOKEN_SECRET="your-modal-token-secret"
```

--------------------------------

### Initialize Runs Data

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Fetches run data from the API on application boot. Handles potential errors during data loading and updates the UI accordingly.

```javascript
let RUNS = [];
async function boot() {
    try {
        RUNS = await api("/api/runs");
    } catch (e) {
        $("#runlist").innerHTML = `<div class="empty">Failed to load runs.<br>${esc(e.message)}</div>`;
        return;
    }
    renderSidebar();
    window.addEventListener("hashchange", route);
    route();
}
```

--------------------------------

### Run SIA with Custom Target Agent Profile

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Evaluates a specific model (e.g., Kimi-K2.6 on Nebius) as the target agent. This command requires setting the appropriate API key for the target provider and potentially the meta agent.

```bash
export NEBIUS_API_KEY="..."        # + ANTHROPIC_API_KEY for the default meta agent
sia run --task gpqa --target-agent-profile kimi-nebius-target --max_gen 5 --run_id 2
```

--------------------------------

### Task Structure for New Tasks

Source: https://github.com/hexo-ai/sia/blob/main/CONTRIBUTING.md

Defines the required directory structure and files for adding a new task to SIA. Includes specification, evaluation, agent template, and sample descriptions.

```bash
tasks/<task-name>/
  data/
    public/
      task.md          # Task specification (required)
      evaluate.py      # Evaluation script (recommended)
    private/
      <ground-truth>   # Private evaluation data
  reference/
    reference_target_agent.py      # Agent template (required)
    SAMPLE_TASK_DESCRIPTIONS.md    # Similar task examples (required)
```

--------------------------------

### Run Project Checks

Source: https://github.com/hexo-ai/sia/blob/main/CONTRIBUTING.md

Execute the test suite, linting, formatting checks, and type checking. These must all pass before submitting a pull request.

```bash
python -m pytest tests/ -v

ruff check sia/ tests/

ruff format --check sia/ tests/

ty check sia/
```

--------------------------------

### Define an OpenAI-Compatible Provider

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Configure a custom provider by specifying its ID, name, client kind, base URL, and API key environment variable.

```jsonc
// providers/my-endpoint.json   — an OpenAI-compatible provider
{
  "provider_id": "my-endpoint",
  "name": "My Endpoint",
  "client_kind": "openai",                 // anthropic | openai | google
  "base_url": "https://api.example.com/v1",
  "api_key_env": "MY_ENDPOINT_API_KEY"
}
```

--------------------------------

### Run SIA in Weights Mode with Modal Sandbox

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

This command runs SIA in Weights Mode, focusing on tuning model weights. It specifies Modal as the training sandbox environment for code execution during training rollouts.

```bash
sia run --task gpqa --max_gen 5 --run_id 1 --focus weights --training_sandbox modal
```

--------------------------------

### Run SIA with Custom SandboxFusion URL

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

This command runs SIA in Weights Mode with SandboxFusion, specifying a custom URL for the SandboxFusion service. This is useful if SandboxFusion is hosted at a non-default address.

```bash
export SANDBOX_URL="http://your-sandboxfusion-host:8080"
sia run --task gpqa --max_gen 5 --run_id 1 --focus weights --training_sandbox sandboxfusion
```

--------------------------------

### Knight Path Optimization Return Format

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/longcot-chess/reference/SAMPLE_TASK_DESCRIPTIONS.md

This format specifies how to return the minimum number of moves for a knight to visit all target squares on a 100x100 board.

```text
solution = <integer>
```

--------------------------------

### Set API Keys for LLM Providers

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Set environment variables for each LLM provider's API key. The orchestrator will warn at startup if any are unset.

```bash
export ANTHROPIC_API_KEY="..."   # anthropic provider (claude agent impl / claude target models)
export GEMINI_API_KEY="..."      # gemini provider  (or GOOGLE_API_KEY via openhands)
export OPENAI_API_KEY="..."      # openai provider
export TOGETHER_API_KEY="..."    # together provider
export NEBIUS_API_KEY="..."      # nebius provider
```

--------------------------------

### Sample Trajectory 1

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_context_success_multi.txt

Represents the second user query or sample processed by the agent.

```json
[
  {
    "role": "user",
    "content": "q1"
  }
]
```

--------------------------------

### Run SIA Orchestrator with Bundled Task

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Executes the SIA orchestrator for a pre-bundled task. This is used for comparison with custom tasks.

```bash
sia run --task gpqa --max_gen 5 --run_id 1
```

--------------------------------

### Compare Multiple LLMs on a Task

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Run the same task with different target agents or providers to compare their outputs. Each run is saved in a separate directory for side-by-side comparison.

```bash
sia run --task gpqa --max_gen 3 --run_id 1 --target-agent-profile default-target      # Claude
sia run --task gpqa --max_gen 3 --run_id 2 --target-agent-profile kimi-nebius-target  # Kimi on Nebius
```

--------------------------------

### Run Sia Agent with Gemini Meta Agent

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Execute the Sia agent, directing the meta agent to use the previously defined 'gemini-meta' profile for the GPQA task.

```bash
sia run --task gpqa --meta-agent-profile gemini-meta
```

--------------------------------

### Set Environment Variables for MLE-Bench

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Sets environment variables required for downloading datasets via the Kaggle API and for using the Gemini API (optional).

```bash
export KAGGLE_USERNAME="..." KAGGLE_KEY="..."   # mle-bench downloads via the Kaggle API
export GEMINI_API_KEY="..."                     # optional; required only without --skip-gemini
```

--------------------------------

### Set Gemini API Key Environment Variable

Source: https://github.com/hexo-ai/sia/blob/main/docs/troubleshooting.md

Set the GEMINI_API_KEY environment variable to enable the generation of similar task descriptions by `prepare_mlebench_dataset.py`. Alternatively, use the `--skip-gemini` flag.

```bash
export GEMINI_API_KEY="..."
```

--------------------------------

### Run SIA with a Custom Target Agent Profile

Source: https://github.com/hexo-ai/sia/blob/main/README.md

Execute an SIA task using a custom target agent profile, either by its name or explicit file path. Ensure the API key environment variable is set.

```bash
export MY_ENDPOINT_API_KEY="..."
sia run --task gpqa --target-agent-profile my-target      # by name (resolves ./profiles/my-target.json)
sia run --task gpqa --target-agent-profile ./profiles/my-target.json   # or by explicit path
```

--------------------------------

### Copy Dataset Files

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Copies the dataset files into the appropriate public and private directories within the task structure. Public inputs are visible to the agent, while private answers are held out for scoring.

```bash
# Public inputs — the agent is allowed to see these
cp questions.json my-tasks/gpqa/data/public/

# Private answers / ground truths — held out from the agent
cp answers.json my-tasks/gpqa/data/private/
```

--------------------------------

### Auto-fix Lint and Formatting Issues

Source: https://github.com/hexo-ai/sia/blob/main/CONTRIBUTING.md

Automatically fix linting and formatting issues in the project files using ruff.

```bash
ruff check --fix sia/ tests/
ruff format sia/ tests/
```

--------------------------------

### View Improvement Suggestions

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Shows the proposed improvements generated by the feedback agent for a specific generation. Helps in understanding how the agent is refined.

```bash
cat runs/run_1/gen_2/improvement.md
```

--------------------------------

### Expected Response Format for Knight Path

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/longcot-chess/data/public/task.md

Illustrates the required integer format for reporting the minimum knight moves.

```json
solution = 218
```

--------------------------------

### View Agent Execution Logs

Source: https://github.com/hexo-ai/sia/blob/main/docs/troubleshooting.md

Use this command to view the agent execution log file. Check for errors related to dataset paths, missing Python packages, or unset API keys.

```bash
cat runs/run_1/gen_1/agent_execution.json
```

--------------------------------

### Current Target Agent Implementation (Generation 2)

Source: https://github.com/hexo-ai/sia/blob/main/tests/golden/feedback_prompt.txt

This snippet represents the current implementation of the target agent for generation 2. It is a simple print statement indicating the generation.

```python
print('current target agent gen 2')
```

--------------------------------

### Expected Response Format for Best Moves

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/longcot-chess/data/public/task.md

Illustrates the required JSON array format for reporting the 3 best chess moves.

```json
solution = ["c4", "bxc4", "bxc4"]
```

--------------------------------

### Generate Profile Chips

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Creates a row of compact profile chips for a given run object. Each chip displays a label and value, with a tooltip showing full profile details. It uses the flattenRows utility for tooltip content.

```javascript
// Three compact profile chips (meta / target / task). Each shows a label + key
// value inline, with a hover/focus tooltip rendering the full profile JSON so the
// run header stays tiny. Falls back to context.md fields for pre-profiles.json runs.
function profileChips(run) {
  const p = run.profiles || {};
  const defs = [
    { label: "Meta agent", value: p.meta?.model || p.meta?.name || run.meta_model, obj: p.meta || (run.meta_model ? { model: run.meta_model, agent_impl: run.agent_impl } : null) },
    { label: "Target agent", value: p.target?.model || p.target?.name || run.task_model, obj: p.target || (run.task_model ? { model: run.task_model } : null) },
    { label: "Task", value: run.task, obj: { task: run.task, max_generations: run.max_generations } },
  ].filter(d => d.value != null && d.value !== "");
  const row = h("div", { class: "chip-row" });
  defs.forEach((d, i) => {
    const rows = flattenRows(d.obj);
    const tipClass = "pchip-tip" + (i === defs.length - 1 ? " pchip-tip--right" : "");
    const tip = h("div", { class: tipClass }, rows.length ? rows.map((\[k, v\]) => h("div", { class: "kv" }, h("span", { class: "kv-k" }, k), h("span", { class: "kv-v" }, v)))
      : h("div", { class: "muted" }, "No profile detail"));
    row.append(h("div", { class: "pchip", tabindex: "0" }, h("div", { class: "pchip-col" }, h("div", { class: "k" }, d.label), h("div", { class: "pchip-val" }, String(d.value))), tip));
  });
  return row;
}
```

--------------------------------

### Diff Successive Agent Versions

Source: https://github.com/hexo-ai/sia/blob/main/docs/walkthrough.md

Compares two versions of the target agent's Python script to highlight changes made between generations. Useful for tracking agent evolution.

```bash
diff runs/run_1/gen_1/target_agent.py runs/run_1/gen_2/target_agent.py
```

--------------------------------

### Handle Route Changes and Render Content

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Listens for hash changes in the URL and updates the main content area accordingly. It fetches run details if not cached and renders the appropriate view.

```javascript
const RUN_CACHE = {};
const VIEW = { pillsRow: null, genHost: null, tabsBar: null, genBody: null };
let MOUNTED = { run: null, gen: null, tab: null };
async function route() {
    const { run } = currentRoute();
    syncSidebarActive();
    if (!run) {
        MOUNTED = { run: null, gen: null, tab: null };
        $("#main").innerHTML = `<div class="empty">Select a run from the left.</div>`;
        return;
    }
    let detail = RUN_CACHE[run];
    if (!detail) {
        if (MOUNTED.run !== run) $("#main").innerHTML = `<div class="spinner">Loading ${esc(run)}…</div>`;
        try {
            detail = RUN_CACHE[run] = await api(`/api/runs/${run}`);
        } catch (e) {
            MOUNTED = { run: null, gen: null, tab: null };
            $("#main").innerHTML = `<div class="empty">Failed to load ${esc(run)}.<br>${esc(e.message)}</div>`;
            return;
        }
    }
    if (MOUNTED.run !== run) {
        renderRun(detail);
        MOUNTED = { run, gen: null, tab: null };
    }
    mountGen(detail);
}
```

--------------------------------

### Define Gemini Meta Agent Profile

Source: https://github.com/hexo-ai/sia/blob/main/docs/configuration.md

Create a JSON configuration file for a Gemini meta agent profile. This profile uses the 'openhands' agent implementation and specifies the Gemini model and provider.

```jsonc
// ./profiles/gemini-meta.json
{
  "profile_id": "gemini-meta", 
  "name": "Gemini meta agent", 
  "agent_impl": "openhands",
  "model": "gemini/gemini-3.1-pro-preview", 
  "provider_id": "gemini"
}
```

--------------------------------

### Mount Generation Details

Source: https://github.com/hexo-ai/sia/blob/main/sia/web/static/index.html

Updates the generation host element with generation details. If there are no generations, it displays an empty message. Otherwise, it determines the active generation and tab to render.

```javascript
// ---- Generation mount: only re-render the bits that changed ------------------
function mountGen(detail) {
  const gens = detail.generations;
  if (!gens.length) {
    VIEW.genHost.innerHTML = `<div class="empty">No generations yet.</div>`;
    return;
  }
  const { gen, tab } = currentRoute();
  const activeGen = gens.find(g => g.name === gen) || gens[gens.length - 1];
  const activeTab = tab || "overview";
  // Pills: cheap synchronous rebuild (preserves the active tab when switching gen).

```

--------------------------------

### Chess Best Moves Return Format

Source: https://github.com/hexo-ai/sia/blob/main/sia/tasks/longcot-chess/reference/SAMPLE_TASK_DESCRIPTIONS.md

This format specifies how to return the 3 best chess moves for a given position in Standard Algebraic Notation (SAN).

```text
solution = [move1, move2, move3]
```