### Customize Python Setup with `--install`

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

Override gantry's default Python setup steps using the `--install` option with a custom command or shell script. This allows for flexible dependency management beyond default pip installations.

```bash
gantry run --show-logs --install='pip install -r custom_requirements.txt' -- echo "Hello, World!"
```

--------------------------------

### YAML: GitHub Actions for GPU Compute with Beaker Gantry

Source: https://context7.com/allenai/beaker-gantry/llms.txt

An example GitHub Actions workflow file (`.github/workflows/gpu-tests.yml`) demonstrating integration with Beaker Gantry for running GPU-accelerated tests. It checks out code, sets up Python using `uv`, installs `beaker-gantry`, and executes Gantry commands with specified GPU resources and configurations.

```yaml
# .github/workflows/gpu-tests.yml
name: GPU Tests

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  gpu_tests:
    name: GPU Tests
    runs-on: ubuntu-latest
    timeout-minutes: 30
    env:
      BEAKER_TOKEN: ${{ secrets.BEAKER_TOKEN }}
      GANTRY_GITHUB_TESTING: 'true'
      BEAKER_WORKSPACE: 'ai2/my-workspace'
    steps:
      - uses: actions/checkout@v5
        with:
          ref: ${{ github.event.pull_request.head.sha }}

      - uses: astral-sh/setup-uv@v6
        with:
          python-version: '3.12'

      - name: Install gantry
        run: uv tool install 'beaker-gantry>=3.1,<4.0'

      - name: Set commit info
        run: |
          if [ "${{ github.event_name }}" == "pull_request" ]; then
            echo "COMMIT_SHA=${{ github.event.pull_request.head.sha }}" >> $GITHUB_ENV
            echo "BRANCH_NAME=${{ github.head_ref }}" >> $GITHUB_ENV
          else
            echo "COMMIT_SHA=$GITHUB_SHA" >> $GITHUB_ENV
            echo "BRANCH_NAME=${{ github.ref_name }}" >> $GITHUB_ENV
          fi

      - name: Run GPU tests
        run: |
          exec gantry run \
            --show-logs \
            --yes \
            --workspace ${{ env.BEAKER_WORKSPACE }} \
            --budget ai2/my-budget \
            --description 'GPU tests from PR' \
            --ref ${{ env.COMMIT_SHA }} \
            --branch ${{ env.BRANCH_NAME }} \
            --priority normal \
            --preemptible \
            --gpus 1 \
            --gpu-type h100 \
            -- pytest -v tests/gpu/

```

--------------------------------

### Launch Beaker Jobs from GitHub Actions (YAML)

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

Integrate Beaker Gantry with GitHub Actions to launch Beaker jobs. This workflow demonstrates setting up Beaker API tokens, checking out code, installing gantry, and executing a gantry run command with specified resources and parameters.

```yaml
name: Beaker

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

on:
  pull_request:
    branches:
      - main
  push:
    branches:
      - main

jobs:
  gpu_tests:
    name: GPU Tests
    runs-on: ubuntu-latest
    timeout-minutes: 15
    env:
      BEAKER_TOKEN: ${{ secrets.BEAKER_TOKEN }}
      GANTRY_GITHUB_TESTING: 'true'  # force better logging for CI
      BEAKER_WORKSPACE: 'ai2/your-workspace'  # TODO: change this to your Beaker workspace
    steps:
      - uses: actions/checkout@v5
        with:
          ref: ${{ github.event.pull_request.head.sha }}  # check out PR head commit instead of merge commit

      - uses: astral-sh/setup-uv@v6
        with:
          python-version: '3.12'

      - name: install gantry
        run:
          uv tool install 'beaker-gantry>=3.1,<4.0'

      - name: Determine current commit SHA (pull request)
        if: github.event_name == 'pull_request'
        run: |
          echo "COMMIT_SHA=${{ github.event.pull_request.head.sha }}" >> $GITHUB_ENV
          echo "BRANCH_NAME=${{ github.head_ref }}" >> $GITHUB_ENV

      - name: Determine current commit SHA (push)
        if: github.event_name != 'pull_request'
        run: |
          echo "COMMIT_SHA=$GITHUB_SHA" >> $GITHUB_ENV
          echo "BRANCH_NAME=${{ github.ref_name }}" >> $GITHUB_ENV

      - name: launch job
        run: |
          exec gantry run \
            --show-logs \
            --yes \
            --workspace ${{ env.BEAKER_WORKSPACE }} \
            --description 'GitHub Actions GPU tests' \
            --ref ${{ env.COMMIT_SHA }} \
            --branch ${{ env.BRANCH_NAME }} \
            --priority normal \
            --preemptible \
            --gpus 1 \
            --gpu-type h100 \
            --gpu-type a100 \
            -- pytest -v tests/cuda_tests/  # TODO: change to your own command

```

--------------------------------

### Create and Launch Multi-Node Torchrun Recipe

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Defines a distributed training recipe using torchrun for multi-node execution. It specifies the command, GPUs per node, number of nodes, GPU types, workspace, budget, and installation command. The recipe is then launched, and logs are shown.

```python
distributed_recipe = Recipe.multi_node_torchrun(
    cmd=["python", "-m", "my_module.train"],
    gpus_per_node=8,
    num_nodes=4,
    gpu_types=["h100"],
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    install="uv pip install . torch --torch-backend=cu129",
)
workload = distributed_recipe.launch(show_logs=True)
```

--------------------------------

### Add Replicas to an Existing Recipe

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Demonstrates how to add replicas to an existing recipe object. This allows for running multiple instances of the same experiment with options for leader selection, host networking, failure propagation, and synchronized start timeout.

```python
recipe_with_replicas = recipe.with_replicas(
    replicas=2,
    leader_selection=True,
    host_networking=True,
    propagate_failure=True,
    synchronized_start_timeout="5m",
)
```

--------------------------------

### Launch Experiments Directly with launch_experiment

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Provides a low-level function `launch_experiment` for launching experiments with full parameter control. Examples show launching a simple experiment and a more complex one with all available options, including clusters, memory, datasets, environment variables, and advanced replica settings.

```python
from gantry.api import launch_experiment

# Launch a simple experiment
workload = launch_experiment(
    args=["python", "evaluate.py"],
    name="evaluation-run",
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    gpus=1,
    show_logs=True,
    timeout=-1,  # Wait indefinitely
)

# Launch with all options
workload = launch_experiment(
    args=["python", "train.py"],
    name="full-training",
    description="Complete training pipeline",
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    clusters=["ai2/jupiter", "ai2/saturn"],
    gpu_types=["h100"],
    gpus=8,
    memory="64GiB",
    shared_memory="10GiB",
    datasets=["my-dataset:/input"],
    env_vars=[("DEBUG", "1")],
    env_secrets=[("API_KEY", "my-secret")],
    priority="normal",
    preemptible=False,
    replicas=2,
    leader_selection=True,
    host_networking=True,
    propagate_failure=True,
    torchrun=True,
    show_logs=True,
    timeout=3600,  # 1 hour timeout
    start_timeout=300,  # 5 minute start timeout
)
```

--------------------------------

### Follow Running Workloads with follow_workload

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Enables programmatic following and streaming of logs from a running workload. It uses the `Beaker` client to get an existing workload and then `follow_workload` to monitor its execution, with options for timeout, log display, and automatic cancellation.

```python
from beaker import Beaker
from gantry.api import follow_workload, launch_experiment

# Launch and follow a workload
with Beaker.from_env() as beaker:
    # Get an existing workload
    workload = beaker.workload.get("my-experiment-id")

    # Follow until completion
    job = follow_workload(
        beaker,
        workload,
        timeout=3600,
        show_logs=True,
        auto_cancel=True,  # Cancel on timeout
    )

    print(f"Job finished with status: {job.status}")
```

--------------------------------

### Install Project in Editable Mode - Pip

Source: https://github.com/allenai/beaker-gantry/blob/main/CONTRIBUTING.md

Installs the Beaker Gantry project in editable mode using pip, along with development dependencies. Editable mode links the installed package to your local source code, so changes are reflected immediately.

```shell
pip install -U pip setuptools wheel
pip install -e .[dev]
```

--------------------------------

### Python: Custom Callbacks for Gantry Job Events

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Demonstrates how to create and use custom Python callbacks to hook into Gantry's job lifecycle. These callbacks can be used for monitoring, logging, and sending notifications on events like job start, log updates, preemption, failure, and success. Requires the 'beaker' and 'gantry' libraries.

```python
from dataclasses import dataclass
from typing import Any
from beaker import BeakerDataset, BeakerJob
from gantry.api import Callback

@Callback.register("my-callback")
@dataclass(kw_only=True)
class MyCustomCallback(Callback):
    webhook_url: str

    def on_start(self, job: BeakerJob):
        """Called when the job starts running."""
        print(f"Job {job.id} started!")
        # Send notification, log to external system, etc.

    def on_log(self, job: BeakerJob, log_line: str, log_time: float):
        """Called for each log line received."""
        if "ERROR" in log_line:
            self.send_alert(f"Error detected: {log_line}")

    def on_no_new_logs(self, job: BeakerJob):
        """Called periodically when no new logs are received."""
        pass  # Could check for stalled jobs

    def on_preemption(self, job: BeakerJob):
        """Called when job is preempted."""
        print(f"Job {job.id} was preempted, will be rescheduled")

    def on_failure(self, job: BeakerJob, *, metrics: dict[str, Any] | None = None,
                   results_ds: BeakerDataset | None = None):
        """Called when job fails."""
        self.send_alert(f"Job {job.id} failed!")

    def on_success(self, job: BeakerJob, *, metrics: dict[str, Any] | None = None,
                   results_ds: BeakerDataset | None = None):
        """Called when job succeeds."""
        print(f"Job {job.id} succeeded! Metrics: {metrics}")

    def send_alert(self, message: str):
        import requests
        requests.post(self.webhook_url, json={"text": message})

# Use the callback with gantry run
# gantry run --callback '{"type": "my-callback", "webhook_url": "https://..."}' -- python train.py

# Or programmatically
from gantry.api import Recipe

recipe = Recipe(
    args=["python", "train.py"],
    callbacks=[MyCustomCallback(webhook_url="https://hooks.slack.com/...")],
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
)
recipe.launch(show_logs=True)

```

--------------------------------

### Run Workload on Beaker (CLI)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `gantry run` command submits experiments to Beaker. It handles Git cloning, Python environment setup (uv or conda), and executes the specified command. Supports GPU workloads, distributed training, dataset mounting, environment variables, and more. A dry run option previews the experiment spec.

```bash
# Basic experiment - prints Hello World
gantry run --show-logs -- python -c 'print("Hello, World!")'

# GPU workload with specific GPU type
gantry run --show-logs \
  --workspace ai2/my-workspace \
  --budget ai2/my-budget \
  --gpus 1 \
  --gpu-type h100 \
  -- python train.py

# With custom Python dependencies and dataset
gantry run --show-logs \
  --name "my-training-job" \
  --description "Fine-tuning experiment" \
  --cluster ai2/jupiter \
  --gpus 8 \
  --dataset 'petew/squad-train:/input-data' \
  --weka 'oe-training-default:/mount/weka' \
  --env 'WANDB_PROJECT=my-project' \
  --env-secret 'WANDB_API_KEY=wandb-key' \
  --install 'uv pip install . torch --torch-backend=cu129' \
  -- python train.py --data-dir /input-data

# Distributed multi-node training with torchrun
gantry run --show-logs \
  --gpus 8 \
  --gpu-type h100 \
  --replicas 4 \
  --torchrun \
  --install 'uv pip install . torch --torch-backend=cu129' \
  -- python -m my_module.train

# Dry run to preview the experiment spec
gantry run --dry-run --save-spec experiment.yml -- python train.py
```

--------------------------------

### gantry run - Run a Workload on Beaker

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The primary command for submitting experiments to Beaker. It automatically handles Git cloning, Python environment setup, and workload execution.

```APIDOC
## POST /gantry/run

### Description
Submits experiments to Beaker clusters by cloning your Git repository, reconstructing your Python environment (via `uv` or `conda`), and executing your specified command. Supports GPU workloads, distributed multi-node training, dataset mounting, environment variable injection, and more.

### Method
POST

### Endpoint
/gantry/run

### Parameters
#### Query Parameters
- **show-logs** (boolean) - Optional - Stream logs from the experiment in real-time.
- **workspace** (string) - Optional - The Beaker workspace to use (e.g., `ai2/my-workspace`).
- **budget** (string) - Optional - The Beaker budget to use (e.g., `ai2/my-budget`).
- **gpus** (integer) - Optional - The number of GPUs to allocate.
- **gpu-type** (string) - Optional - The type of GPU to allocate (e.g., `h100`).
- **replicas** (integer) - Optional - The number of replicas for distributed training.
- **torchrun** (boolean) - Optional - Enable distributed multi-node training with `torchrun`.
- **name** (string) - Optional - A custom name for the experiment.
- **description** (string) - Optional - A description for the experiment.
- **cluster** (string) - Optional - The Beaker cluster to use (e.g., `ai2/jupiter`).
- **dataset** (string) - Optional - Mount a dataset (e.g., `'petew/squad-train:/input-data'`).
- **weka** (string) - Optional - Mount a Weka volume (e.g., `'oe-training-default:/mount/weka'`).
- **env** (string) - Optional - Inject an environment variable (e.g., `'WANDB_PROJECT=my-project'`). Can be specified multiple times.
- **env-secret** (string) - Optional - Inject a secret as an environment variable (e.g., `'WANDB_API_KEY=wandb-key'`). Can be specified multiple times.
- **install** (string) - Optional - Commands to install Python dependencies (e.g., `'uv pip install . torch --torch-backend=cu129'`).
- **dry-run** (boolean) - Optional - Perform a dry run without submitting the experiment.
- **save-spec** (string) - Optional - Save the experiment specification to a file (e.g., `experiment.yml`).

#### Request Body
- **command** (string) - Required - The command to execute (e.g., `python train.py` or `python -m my_module.train`).

### Request Example
```json
{
  "command": "python train.py --data-dir /input-data"
}
```

### Response
#### Success Response (200)
- **experiment_id** (string) - The ID of the submitted experiment.
- **status** (string) - The status of the experiment.

#### Response Example
```json
{
  "experiment_id": "exp-abcdef123456",
  "status": "submitted"
}
```
```

--------------------------------

### Create and Launch Basic Training Recipe

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Defines a basic training recipe with specified arguments, name, description, workspace, budget, GPUs, GPU types, datasets, and environment variables. It then performs a dry run, launches the experiment, and prints the experiment name upon completion.

```python
recipe = Recipe(
    args=["python", "train.py", "--epochs", "10"],
    name="training-experiment",
    description="Training run with 10 epochs",
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    gpus=1,
    gpu_types=["h100", "a100"],
    datasets=["petew/my-dataset:/data"],
    env_vars=[("WANDB_PROJECT", "my-project")],
    env_secrets=[("WANDB_API_KEY", "wandb-key")],
)

# Validate with dry run
recipe.dry_run()

# Launch and wait for completion
workload = recipe.launch(show_logs=True, timeout=-1)
print(f"Experiment completed: {workload.experiment.name}")
```

--------------------------------

### Run a Python 'Hello, World!' experiment with Gantry

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This bash command demonstrates submitting a simple Python script to Beaker using Gantry. The `--show-logs` flag ensures logs are displayed, and the `--` separates Gantry options from the command to be executed on Beaker.

```bash
gantry run --show-logs -- python -c 'print("Hello, World!")'
```

--------------------------------

### Recipe Creation and Management

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Demonstrates how to create and configure basic and multi-node recipes, including setting arguments, resources, and environment variables.

```APIDOC
## Recipe Creation and Management

### Description
This section covers the creation and configuration of `Recipe` objects for running experiments. It includes examples for basic recipes, multi-node torchrun recipes, and adding replicas to existing recipes.

### Methods
- **Recipe(...)**: Constructor for creating a basic recipe.
- **Recipe.multi_node_torchrun(...)**: Class method for creating a multi-node torchrun recipe.
- **recipe.with_replicas(...)**: Method for adding replicas to an existing recipe.
- **recipe.dry_run()**: Simulates the execution of the recipe without actually running it.
- **recipe.launch(...)**: Launches the experiment defined by the recipe.

### Parameters for Recipe(...)

#### Arguments
- **args** (list[str]) - Required - Command-line arguments for the experiment.
- **name** (str) - Required - A unique name for the experiment.
- **description** (str) - Optional - A detailed description of the experiment.
- **workspace** (str) - Required - The workspace where the experiment will run.
- **budget** (str) - Required - The budget allocated for the experiment.
- **gpus** (int) - Optional - Number of GPUs to allocate.
- **gpu_types** (list[str]) - Optional - Preferred GPU types.
- **datasets** (list[str]) - Optional - List of datasets to mount.
- **env_vars** (list[tuple[str, str]]) - Optional - Environment variables to set.
- **env_secrets** (list[tuple[str, str]]) - Optional - Secrets to set as environment variables.

### Parameters for Recipe.multi_node_torchrun(...)

#### Arguments
- **cmd** (list[str]) - Required - The command to run on each node.
- **gpus_per_node** (int) - Required - Number of GPUs per node.
- **num_nodes** (int) - Required - Total number of nodes.
- **gpu_types** (list[str]) - Optional - Preferred GPU types.
- **workspace** (str) - Required - The workspace for the experiment.
- **budget** (str) - Required - The budget for the experiment.
- **install** (str) - Optional - Installation command to run before the main command.

### Parameters for recipe.with_replicas(...)

#### Arguments
- **replicas** (int) - Required - Number of replicas to add.
- **leader_selection** (bool) - Optional - Enable leader selection.
- **host_networking** (bool) - Optional - Enable host networking.
- **propagate_failure** (bool) - Optional - Propagate failure to all replicas.
- **synchronized_start_timeout** (str) - Optional - Timeout for synchronized start.

### Example Usage
```python
# Create a basic recipe
recipe = Recipe(
    args=["python", "train.py", "--epochs", "10"],
    name="training-experiment",
    description="Training run with 10 epochs",
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    gpus=1,
    gpu_types=["h100", "a100"],
    datasets=["petew/my-dataset:/data"],
    env_vars=[("WANDB_PROJECT", "my-project")],
    env_secrets=[("WANDB_API_KEY", "wandb-key")],
)

# Validate with dry run
recipe.dry_run()

# Launch and wait for completion
workload = recipe.launch(show_logs=True, timeout=-1)
print(f"Experiment completed: {workload.experiment.name}")

# Multi-node torchrun recipe
distributed_recipe = Recipe.multi_node_torchrun(
    cmd=["python", "-m", "my_module.train"],
    gpus_per_node=8,
    num_nodes=4,
    gpu_types=["h100"],
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    install="uv pip install . torch --torch-backend=cu129",
)
workload = distributed_recipe.launch(show_logs=True)

# Add replicas to an existing recipe
recipe_with_replicas = recipe.with_replicas(
    replicas=2,
    leader_selection=True,
    host_networking=True,
    propagate_failure=True,
    synchronized_start_timeout="5m",
)
```
```

--------------------------------

### Run Distributed Multi-Node Batch Jobs with Gantry and Torchrun

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This section details how to run distributed multi-node batch jobs using Gantry, particularly with `torchrun`. It covers automatic configuration with `--replicas` and `--torchrun`, as well as manual configuration using `--leader-selection`, `--host-networking`, and environment variable expansion via `bash -c`.

```bash
gantry run \
  --show-logs \
  --gpus=8 \
  --gpu-type='h100' \
  --replicas=2 \
  --torchrun \
  --install 'uv pip install . torch numpy --torch-backend=cu129' \
  -- python -m gantry.all_reduce_bench
```

```bash
gantry run \
  --show-logs \
  --gpus=8 \
  --gpu-type='h100' \
  --replicas=2 \
  --leader-selection \
  --host-networking \
  --propagate-failure \
  --propagate-preemption \
  --synchronized-start-timeout='5m' \
  --install 'uv pip install . torch numpy --torch-backend=cu129' \
  --exec-method=bash \
  -- torchrun \
    '--nnodes="$BEAKER_REPLICA_COUNT:$BEAKER_REPLICA_COUNT"' \
    '--nproc-per-node="$BEAKER_ASSIGNED_GPU_COUNT"' \
    '--rdzv-id=12347' \
    '--rdzv-backend=static' \
    '--rdzv-endpoint="$BEAKER_LEADER_REPLICA_HOSTNAME:29400"' \
    '--node-rank="$BEAKER_REPLICA_RANK"' \
    '--rdzv-conf="read_timeout=420"' \
    -m gantry.all_reduce_bench
```

```bash
gantry run \
  --show-logs \
  --gpus=8 \
  --gpu-type='h100' \
  --replicas=2 \
  --leader-selection \
  --host-networking \
  --propagate-failure \
  --propagate-preemption \
  --synchronized-start-timeout='5m' \
  --install 'uv pip install . torch numpy --torch-backend=cu129' \
  -- ./launch-torchrun.sh
```

--------------------------------

### Python API - Recipe

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `Recipe` class allows programmatic configuration and launching of Gantry workloads from Python.

```APIDOC
## Python API - Recipe Class

### Description
Programmatically define and launch Gantry workloads using the `Recipe` class in Python. This allows for dynamic experiment configuration and execution within your Python scripts.

### Class
`gantry.api.Recipe`

### Methods
#### `__init__`
Initializes a `Recipe` object. Accepts arguments similar to the `gantry run` CLI command.

#### `run`
Launches the experiment defined by the `Recipe`.

### Parameters (for `Recipe` constructor and `run` method)
- **command** (list[str]) - Required - The command and its arguments to execute (e.g., `['python', 'train.py']`).
- **show_logs** (bool) - Optional - Stream logs from the experiment.
- **workspace** (str) - Optional - The Beaker workspace.
- **budget** (str) - Optional - The Beaker budget.
- **gpus** (int) - Optional - Number of GPUs.
- **gpu_type** (str) - Optional - Type of GPU.
- **replicas** (int) - Optional - Number of replicas for distributed training.
- **torchrun** (bool) - Optional - Enable `torchrun`.
- **name** (str) - Optional - Experiment name.
- **description** (str) - Optional - Experiment description.
- **cluster** (str) - Optional - Beaker cluster.
- **datasets** (dict[str, str]) - Optional - Dictionary mapping dataset names to mount paths (e.g., `{{'petew/squad-train': '/input-data'}}`).
- **weka_mounts** (dict[str, str]) - Optional - Dictionary mapping Weka volume names to mount paths.
- **env_vars** (dict[str, str]) - Optional - Dictionary of environment variables.
- **env_secrets** (dict[str, str]) - Optional - Dictionary of environment variables sourced from secrets.
- **install_commands** (list[str]) - Optional - List of commands to install dependencies.
- **dry_run** (bool) - Optional - Perform a dry run.
- **save_spec_path** (str) - Optional - Path to save the experiment spec.

### Request Example
```python
from gantry.api import Recipe

# Define a recipe for a training job
training_recipe = Recipe(
    command=['python', 'train.py', '--data-dir', '/input-data'],
    name='my-python-training',
    gpus=4,
    gpu_type='a100',
    datasets={'petew/squad-train': '/input-data'},
    install_commands=['uv pip install . torch --torch-backend=cu129'],
    show_logs=True
)

# Launch the experiment
# experiment = training_recipe.run()

# For a dry run:
# spec = training_recipe.dry_run()
# print(spec)
```

### Response
#### Success Response (from `run()`)
- **experiment_id** (str) - The ID of the launched experiment.
- **status** (str) - The status of the experiment.

#### Response Example (from `run()`)
```json
{
  "experiment_id": "exp-pythonapi12345",
  "status": "submitted"
}
```
```

--------------------------------

### launch_experiment - Launch Experiments Directly

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Provides a low-level function to launch experiments with fine-grained control over all available parameters.

```APIDOC
## POST /launch_experiment

### Description
This function provides a low-level interface to launch experiments with full control over all parameters. It allows for detailed configuration of resources, environment, and execution behavior.

### Method
POST

### Endpoint
`/launch_experiment`

### Parameters

#### Query Parameters
None

#### Request Body
- **args** (list[str]) - Required - Command-line arguments for the experiment.
- **name** (str) - Required - A unique name for the experiment.
- **workspace** (str) - Required - The workspace where the experiment will run.
- **budget** (str) - Required - The budget allocated for the experiment.
- **gpus** (int) - Optional - Number of GPUs to allocate.
- **show_logs** (bool) - Optional - Whether to stream logs during execution.
- **timeout** (int) - Optional - Timeout in seconds. -1 means indefinite.
- **description** (str) - Optional - A detailed description of the experiment.
- **clusters** (list[str]) - Optional - List of clusters to use for the experiment.
- **gpu_types** (list[str]) - Optional - Preferred GPU types.
- **memory** (str) - Optional - Amount of memory to allocate (e.g., "64GiB").
- **shared_memory** (str) - Optional - Amount of shared memory to allocate (e.g., "10GiB").
- **datasets** (list[str]) - Optional - List of datasets to mount (e.g., "my-dataset:/input").
- **env_vars** (list[tuple[str, str]]) - Optional - Environment variables to set (e.g., [("DEBUG", "1")]).
- **env_secrets** (list[tuple[str, str]]) - Optional - Secrets to set as environment variables (e.g., [("API_KEY", "my-secret")]).
- **priority** (str) - Optional - Priority of the experiment (e.g., "normal").
- **preemptible** (bool) - Optional - Whether the experiment can be preemptible.
- **replicas** (int) - Optional - Number of replicas for the experiment.
- **leader_selection** (bool) - Optional - Enable leader selection for replicas.
- **host_networking** (bool) - Optional - Enable host networking for replicas.
- **propagate_failure** (bool) - Optional - Propagate failure to all replicas.
- **torchrun** (bool) - Optional - Enable torchrun for distributed training.
- **start_timeout** (int) - Optional - Timeout in seconds for the experiment to start.

### Request Example
```json
{
  "args": ["python", "evaluate.py"],
  "name": "evaluation-run",
  "workspace": "ai2/my-workspace",
  "budget": "ai2/my-budget",
  "gpus": 1,
  "show_logs": true,
  "timeout": -1
}
```

### Response
#### Success Response (200)
- **workload** (object) - An object representing the launched workload.

#### Response Example
```json
{
  "workload": {
    "experiment": {
      "name": "evaluation-run"
    }
  }
}
```
```

--------------------------------

### Run a Python experiment on a specific Beaker cluster with Gantry

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This bash command illustrates how to target a specific Beaker cluster for an experiment. It uses the `--cluster` option to specify the cluster name ('ai2/jupiter') and requests one GPU.

```bash
gantry run --show-logs --cluster=ai2/jupiter --gpus=1 -- python -c 'print("Hello, World!")'
```

--------------------------------

### Attach Beaker Datasets to Gantry Experiments

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This command shows how to attach existing Beaker datasets to a Gantry experiment. The `--dataset` option maps a dataset to a path within the experiment's filesystem.

```bash
gantry run --show-logs --dataset='petew/squad-train:/input-data' -- ls /input-data
```

--------------------------------

### Configure Gantry with pyproject.toml

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This TOML snippet shows how to configure Gantry settings within the `pyproject.toml` file. It allows specifying the default Beaker workspace, GitHub token secret, budget, log level, and quiet mode.

```toml
# pyproject.toml
[tool.gantry]
workspace = "ai2/my-default-workspace"
gh_token_secret = "GITHUB_TOKEN"
budget = "ai2/my-teams-budget"
log_level = "warning"
quiet = false
```

--------------------------------

### follow_workload - Follow Running Workloads

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Allows programmatic following and streaming of logs from a running workload until completion or timeout.

```APIDOC
## follow_workload

### Description
This function allows you to follow and stream logs from a running workload programmatically. It is useful for monitoring experiment progress and capturing output in real-time.

### Method
N/A (Function call)

### Endpoint
N/A

### Parameters

#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

#### Function Arguments
- **beaker** (Beaker) - Required - An initialized Beaker client instance.
- **workload** (object) - Required - The workload object to follow.
- **timeout** (int) - Optional - Timeout in seconds. If the workload does not complete within this time, the function will exit.
- **show_logs** (bool) - Optional - If true, logs from the workload will be streamed to the console.
- **auto_cancel** (bool) - Optional - If true, the workload will be automatically cancelled upon timeout.

### Request Example
```python
from beaker import Beaker
from gantry.api import follow_workload, launch_experiment

# Launch and follow a workload
with Beaker.from_env() as beaker:
    # Get an existing workload
    workload = beaker.workload.get("my-experiment-id")

    # Follow until completion
    job = follow_workload(
        beaker,
        workload,
        timeout=3600,
        show_logs=True,
        auto_cancel=True,  # Cancel on timeout
    )

    print(f"Job finished with status: {job.status}")
```

### Response
#### Success Response (200)
- **job** (object) - An object representing the finished job, containing its status.

#### Response Example
```json
{
  "job": {
    "status": "completed"
  }
}
```
```

--------------------------------

### Format Code with isort and black

Source: https://github.com/allenai/beaker-gantry/blob/main/CONTRIBUTING.md

Format Python code using the isort and black command-line tools. These tools help maintain consistent code style across the project. Run from the root of the project clone.

```shell
isort .
black .
```

--------------------------------

### Run a Python version check using a custom Docker image with Gantry

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This bash command demonstrates running a command within a specified Docker image using Gantry. The `--docker-image` option points to the image ('python:3.10'), and `--system-python` tells Gantry to use the Python interpreter present in the image.

```bash
gantry run --show-logs --docker-image='python:3.10' --system-python -- python --version
```

--------------------------------

### Run a GPU-accelerated Python experiment with Gantry

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This bash command shows how to request specific GPU resources for a Gantry experiment. It specifies the GPU type ('h100') and the number of GPUs (1) required for the Python command.

```bash
gantry run --show-logs --gpu-type=h100 --gpus=1 -- python -c 'print("Hello, World!")'
```

--------------------------------

### Follow Running Workload (CLI)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `gantry follow` command streams logs from a running experiment in real-time. It can target a specific experiment by name or ID, follow the latest running experiment, or use tail mode to only show new logs.

```bash
# Follow a specific experiment by ID or name
gantry follow my-experiment-name

# Follow the latest running experiment in workspace
gantry follow --latest

# Follow with tail mode (only new logs)
gantry follow --tail --latest -w ai2/my-workspace

# Follow latest experiment from a specific author
gantry follow --latest -a john.doe
```

--------------------------------

### gantry follow - Follow a Running Workload

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Streams logs from a running experiment in real-time.

```APIDOC
## GET /gantry/follow

### Description
Streams logs from a running experiment in real-time. Can follow a specific experiment, the latest running experiment, or filter by author.

### Method
GET

### Endpoint
/gantry/follow

### Parameters
#### Query Parameters
- **experiment_id_or_name** (string) - Optional - The ID or name of the experiment to follow.
- **latest** (boolean) - Optional - Follow the latest running experiment.
- **tail** (boolean) - Optional - Only show new logs (similar to `tail -f`).
- **w** (string) - Optional - Filter the latest running experiment by workspace.
- **a** (string) - Optional - Filter the latest running experiment by author.

### Response
#### Success Response (200)
- **logs** (string) - A stream of log output from the experiment.

#### Response Example
```
[2024-01-01 10:00:00] Starting training...
[2024-01-01 10:05:15] Epoch 1/10, Loss: 0.5
...
```
```

--------------------------------

### Attach WEKA Bucket to Gantry Experiments

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This command demonstrates attaching a WEKA bucket to a Gantry experiment. The `--weka` option mounts the WEKA bucket to a specified path within the experiment.

```bash
gantry run --show-logs --weka='oe-training-default:/mount/weka' -- ls -l /mount/weka
```

--------------------------------

### Programmatic Workload Configuration (Python API)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `Recipe` class from the `gantry.api` module allows programmatic configuration and launching of Gantry workloads within Python scripts. This provides an alternative to using the CLI for defining and submitting experiments.

```python
from gantry.api import Recipe

```

--------------------------------

### View Gantry Experiment Spec with Dry Run

Source: https://github.com/allenai/beaker-gantry/blob/main/README.md

This command demonstrates how to view the Beaker experiment specification that Gantry will use without actually submitting the experiment. The `--dry-run` flag shows the spec, and `--save-spec PATH` saves it to a YAML file.

```bash
gantry run --dry-run --save-spec ./experiment-spec.yaml -- <your_command>
```

--------------------------------

### Manage Gantry Configuration (CLI)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Gantry configuration commands manage settings, including GitHub tokens for private repositories. The `pyproject.toml` file can also specify default Gantry configurations like workspace, budget, and log level.

```bash
# Set or update GitHub token for private repositories
gantry config set-gh-token

# Example pyproject.toml configuration
# [tool.gantry]
# workspace = "ai2/my-default-workspace"
# gh_token_secret = "GITHUB_TOKEN"
# budget = "ai2/my-teams-budget"
# log_level = "warning"
# quiet = false
```

--------------------------------

### Download Workload Logs (CLI)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `gantry logs` command downloads logs from experiments. It allows specifying a particular experiment, replica, or downloading all logs to a directory. Options include tailing recent lines, continuous streaming, and retrieving logs since a specific timestamp.

```bash
# Download logs from a specific experiment
gantry logs my-experiment-id

# Download logs from a specific replica
gantry logs my-experiment-id --replica 0

# Download logs from all replicas to a directory
gantry logs my-experiment-id --all --output ~/Downloads/logs/

# Tail last 100 lines and save to file
gantry logs my-experiment-id --tail 100 --output logs.txt

# Stream logs continuously
gantry logs my-experiment-id --follow

# Get logs since a specific time
gantry logs my-experiment-id --since "2025-01-15T10:00:00Z"
```

--------------------------------

### Clone Repository - Git

Source: https://github.com/allenai/beaker-gantry/blob/main/CONTRIBUTING.md

Clones your fork of the Beaker Gantry repository locally. This is the first step in setting up your development environment. You can use either HTTPS or SSH URLs.

```git
git clone https://github.com/USERNAME/beaker-gantry.git
```

```git
git clone git@github.com:USERNAME/beaker-gantry.git
```

--------------------------------

### gantry logs - Download Workload Logs

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Downloads logs from completed or running experiments.

```APIDOC
## GET /gantry/logs

### Description
Downloads logs from specified experiments. Supports downloading logs from all replicas, a specific replica, tailing logs, or streaming logs continuously.

### Method
GET

### Endpoint
/gantry/logs

### Parameters
#### Query Parameters
- **experiment_id** (string) - Required - The ID of the experiment whose logs to download.
- **replica** (integer) - Optional - The specific replica number to download logs from.
- **all** (boolean) - Optional - Download logs from all replicas.
- **output** (string) - Optional - The directory or file path to save the logs to.
- **tail** (integer) - Optional - Download the last N lines of logs.
- **follow** (boolean) - Optional - Stream logs continuously.
- **since** (string) - Optional - Download logs generated since a specific timestamp (ISO 8601 format, e.g., `"2025-01-15T10:00:00Z"`).

### Response
#### Success Response (200)
- **logs** (string) - The downloaded log content or a stream of logs if `follow` is used.

#### Response Example
```
# If downloading to a file, this might be empty and logs are saved to the specified path.
# If streaming, this contains the log output.
[2024-01-01 10:00:00] Experiment started...
...
```
```

--------------------------------

### gantry config - Configuration Commands

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Manage Gantry configuration, including setting GitHub tokens and viewing configuration details.

```APIDOC
## GET /gantry/config and POST /gantry/config/set-gh-token

### Description
Manages Gantry's configuration settings. This includes setting a GitHub token for accessing private repositories and viewing current configuration.

### Method
GET (for viewing), POST (for setting GitHub token)

### Endpoint
/gantry/config (view)
/gantry/config/set-gh-token (set GitHub token)

### Parameters
#### Query Parameters (for GET /gantry/config)
- None

#### Request Body (for POST /gantry/config/set-gh-token)
- **token** (string) - Required - The GitHub Personal Access Token.

### Response
#### Success Response (200) - GET /gantry/config
- **config** (object) - An object containing the current Gantry configuration.
  - **workspace** (string) - Default workspace.
  - **gh_token_secret** (string) - Name of the secret storing the GitHub token.
  - **budget** (string) - Default budget.
  - **log_level** (string) - Logging level.
  - **quiet** (boolean) - Quiet mode enabled.

#### Response Example - GET /gantry/config
```json
{
  "config": {
    "workspace": "ai2/my-default-workspace",
    "gh_token_secret": "GITHUB_TOKEN",
    "budget": "ai2/my-teams-budget",
    "log_level": "warning",
    "quiet": false
  }
}
```

#### Success Response (200) - POST /gantry/config/set-gh-token
- **message** (string) - Confirmation message.

#### Response Example - POST /gantry/config/set-gh-token
```json
{
  "message": "GitHub token updated successfully."
}
```
```

--------------------------------

### Python: Slack Notifications with Gantry's SlackCallback

Source: https://context7.com/allenai/beaker-gantry/llms.txt

Shows how to use the built-in `SlackCallback` in Gantry for sending workload event notifications directly to Slack. This simplifies setting up alerts without custom callback implementations. Requires the 'gantry' library and a Slack webhook URL.

```python
from gantry.api import Recipe, SlackCallback

# Using the built-in Slack callback
recipe = Recipe(
    args=["python", "train.py"],
    workspace="ai2/my-workspace",
    budget="ai2/my-budget",
    callbacks=[
        SlackCallback(webhook_url="https://hooks.slack.com/services/T.../B.../...")
    ],
)
workload = recipe.launch(show_logs=True)

# Or via CLI
# gantry run --slack-webhook-url "https://hooks.slack.com/..." -- python train.py

# Or via callback config
# gantry run --callback '{"type": "slack", "webhook_url": "https://..."}' -- python train.py

```

--------------------------------

### Run Release Script

Source: https://github.com/allenai/beaker-gantry/blob/main/RELEASE_PROCESS.md

Executes the release script to automate the process of updating the CHANGELOG and version files, and creating a new git tag. This script triggers a GitHub Actions workflow for the full release.

```bash
./scripts/release.sh

```

--------------------------------

### List Workloads (CLI)

Source: https://context7.com/allenai/beaker-gantry/llms.txt

The `gantry list` command displays experiments submitted through Gantry. It supports filtering by status, author, age, and can list all experiments (not just Gantry-submitted ones) or those within a specific group.

```bash
# List recent experiments in current workspace
gantry list

# List your own experiments with status filter
gantry list --me --status running --limit 20

# List experiments from a specific author in a workspace
gantry list -w ai2/my-workspace -a john.doe --max-age 30d

# List all experiments (not just Gantry-submitted)
gantry list --all --text "training"

# List experiments in a specific group
gantry list -g my-experiment-group --status succeeded
```

--------------------------------

### Lint Code with ruff

Source: https://github.com/allenai/beaker-gantry/blob/main/CONTRIBUTING.md

Check the codebase for linting errors using the ruff tool. This helps identify and fix style and potential errors in the Python code. Run from the root of the project clone.

```shell
ruff check .
```