### Running Inspect Examples

Source: https://inspect.aisi.org.uk/tools-standard

Command-line examples to run Inspect AI examples demonstrating computer use.

```bash
inspect eval examples/computer
```

```bash
inspect eval examples/intervention -T mode=computer --display conversation
```

--------------------------------

### Configuring think() tool with custom instructions

Source: https://inspect.aisi.org.uk/llms-guide.txt

This example shows how to configure the think() tool with a custom instruction string. The provided text guides the model on when and how to use the think tool, particularly for complex reasoning, brainstorming fixes, or analyzing test results in a SWE-Bench context.

```python
from textwrap import dedent

from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, system_message, use_tools
from inspect_ai.tool import bash_session, text_editor, think

@task
def swe_bench():

    tools = [
        bash_session(timeout=180),
        text_editor(timeout=180),
        think(dedent("""
            Use the think tool to think about something. It will not obtain
            new information or make any changes to the repository, but just 
            log the thought. Use it when complex reasoning or brainstorming
            is needed. For example, if you explore the repo and discover
            the source of a bug, call this tool to brainstorm several unique
            ways of fixing the bug, and assess which change(s) are likely to 
            be simplest and most effective. Alternatively, if you receive
            some test results, call this tool to brainstorm ways to fix the
            failing tests.
        "))
    ])

    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools(tools),
            generate(),
        ),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Running Inspect Computer Example

Source: https://inspect.aisi.org.uk/llms-guide.txt

Bash command to evaluate the basic computer use example from the Inspect AI repository.

```bash
inspect eval examples/computer
```

--------------------------------

### Install and Configure Groq Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the groq package, set your GROQ_API_KEY, and specify a Groq model to use the provider.

```bash
pip install groq
export GROQ_API_KEY=your-groq-api-key
inspect eval arc.py --model groq/llama-3.1-70b-versatile
```

--------------------------------

### Install and Configure Google Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the google-genai package and set your API key. Then, specify a Google model to use with Inspect.

```bash
pip install google-genai
export GOOGLE_API_KEY=your-google-api-key
inspect eval arc.py --model google/gemini-2.5-pro
```

--------------------------------

### Install vLLM and Run Model

Source: https://inspect.aisi.org.uk/providers.html

Install the vLLM package and run an evaluation with a specified vLLM model.

```bash
pip install vllm
inspect eval arc.py --model vllm/openai-community/gpt2
```

--------------------------------

### Headless Mode Output Example

Source: https://inspect.aisi.org.uk/human-agent.html

Example output when running in headless mode, providing instructions to log into the system via Docker exec.

```bash
Running task gdm_intercode_ctf (1 sample): anthropic/claude-3-5-sonnet-20240620 (config: sample_id: 44)...

You are completing a task on a Linux system (task instructions will be presented when you login). Login to the system with the following command:

docker exec -it inspect-gdm_intercod-iebwzkg-default-1 bash -l
```

--------------------------------

### list_files Tool Example

Source: https://inspect.aisi.org.uk/tools-custom.html

An example of how to define a tool that uses the sandbox to list files in a directory. It demonstrates error handling for command execution.

```APIDOC
## Tool: list_files

### Description
Defines a tool that lists the files in a specified directory using the sandbox environment.

### Usage
```python
from inspect_ai.tool import ToolError, tool
from inspect_ai.util import sandbox

@tool
def list_files():
    async def execute(dir: str):
        """List the files in a directory.

        Args:
            dir: Directory

        Returns:
            File listing of the directory
        """
        result = await sandbox().exec(["ls", dir])
        if result.success:
            return result.stdout
        else:
            raise ToolError(result.stderr)

    return execute
```
```

--------------------------------

### Example .env File Configuration

Source: https://inspect.aisi.org.uk/options.html

This example shows how to set API keys and default options for Inspect evaluations using a .env file. It includes settings for logging directory, log level, and evaluation retries/connections.

```makefile
OPENAI_API_KEY=your-api-key
ANTHROPIC_API_KEY=your-api-key
GOOGLE_API_KEY=your-api-key

INSPECT_LOG_DIR=./logs-04-07-2024
INSPECT_LOG_LEVEL=warning

INSPECT_EVAL_MAX_RETRIES=5
INSPECT_EVAL_MAX_CONNECTIONS=20
INSPECT_EVAL_MODEL=anthropic/claude-3-5-sonnet-20240620
```

--------------------------------

### Install Inspect AI

Source: https://inspect.aisi.org.uk/

Install the Inspect AI library using pip. This is the first step to using the library.

```bash
pip install inspect-ai
```

--------------------------------

### Install and Run Hugging Face Model

Source: https://inspect.aisi.org.uk/providers.html

Install necessary packages and run an evaluation with a Hugging Face model.

```bash
pip install torch transformers accelerate
inspect eval arc.py --model hf/openai-community/gpt2
```

--------------------------------

### Install and Configure SambaNova Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set your SambaNova API key, and specify the model for evaluation.

```bash
pip install openai
export SAMBANOVA_API_KEY=your-sambanova-api-key
inspect eval arc.py --model sambanova/DeepSeek-V1-0324
```

--------------------------------

### Install and Configure Together AI Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set your TOGETHER_API_KEY, and specify a Together AI model to use the provider.

```bash
pip install openai
export TOGETHER_API_KEY=your-together-api-key
inspect eval arc.py --model together/MiniMaxAI/MiniMax-M2.7
```

--------------------------------

### Install Agent Skills into Sandbox

Source: https://inspect.aisi.org.uk/reference/inspect_ai.tool.html

Asynchronously installs agent skills into a specified sandbox environment. Allows setting the user and installation directory.

```python
async def install_skills(
    skills: Sequence[str | Path | Skill],
    sandbox: str | SandboxEnvironment | None = None,
    user: str | None = None,
    dir: str | None = None,
) -> list[SkillInfo]
```

--------------------------------

### Install and Use Mistral Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the Mistral package, set your API key, and specify a model to use the Mistral provider.

```bash
pip install mistral
export MISTRAL_API_KEY=your-mistral-api-key
inspect eval arc.py --model mistral/mistral-large-latest
```

--------------------------------

### Start External vLLM Server with Pre-loaded LoRA Adapter

Source: https://inspect.aisi.org.uk/providers.html

Start an external vLLM server with a pre-loaded LoRA adapter using the `--lora-modules` flag. Adapters are then referenced by name.

```bash
# Start server with pre-loaded adapter
vllm serve meta-llama/Llama-3-8B --enable-lora \
    --lora-modules my-adapter=path/to/adapter

# Use the adapter name directly (not the path)
inspect eval arc.py --model vllm/meta-llama/Llama-3-8B:my-adapter
```

--------------------------------

### Install aioboto3 and Set AWS Credentials

Source: https://inspect.aisi.org.uk/llms-guide.txt

Install the necessary package and set your AWS access key ID and secret access key as environment variables.

```bash
pip install aioboto3
export AWS_ACCESS_KEY_ID=access-key-id
export AWS_SECRET_ACCESS_KEY=secret-access-key
```

--------------------------------

### Import Service into Sandbox (sys.path)

Source: https://inspect.aisi.org.uk/reference/inspect_ai.util.html

Example of how to import a service into a sandbox by appending its path to sys.path.

```python
import sys
sys.path.append("/var/tmp/sandbox-services/foo")
import foo
```

--------------------------------

### Install Hugging Face Hub and Authenticate

Source: https://inspect.aisi.org.uk/eval-logs.html

Install the Hugging Face Hub package and log in to your Hugging Face account to manage storage buckets. This is required before creating or accessing buckets.

```bash
pip install "huggingface_hub>=1.6.0"
hf auth login
hf buckets create my-org/inspect-logs --private
```

--------------------------------

### Install and Authenticate OpenRouter

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package and set your OpenRouter API key as an environment variable. Then, specify an OpenRouter model for evaluation.

```bash
pip install openai
export OPENROUTER_API_KEY=your-openrouter-api-key
inspect eval arc.py --model openrouter/gryphe/mythomax-l2-13b
```

--------------------------------

### fetch_url (parallel execution example)

Source: https://inspect.aisi.org.uk/llms-guide.txt

An example of a tool that can opt-in to running concurrently with its siblings using `@tool(parallel=True)`. This is suitable for tools with no shared mutable state.

```APIDOC
## fetch_url

### Description
Fetch a URL and return its contents. This tool can be executed in parallel.

### Method
`@tool(parallel=True)
def fetch_url(url: str) -> str:`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
- **url** (str) - Required - The URL to fetch.
```

--------------------------------

### Install OpenAI and Set API Key

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package and set your API key as an environment variable before running Inspect.

```bash
pip install openai
export OPENAI_API_KEY=your-openai-api-key
inspect eval arc.py --model openai/gpt-4o-mini
```

--------------------------------

### Install SGLang and Run Inspect

Source: https://inspect.aisi.org.uk/providers.html

Install the SGLang package with specific dependencies and then use Inspect to evaluate a model served by SGLang. Ensure the correct version and CUDA compatibility for flashinfer.

```bash
pip install "sglang[all]>=0.4.4.post2" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer-python
inspect eval arc.py --model sglang/meta-llama/Meta-Llama-3-8B-Instruct
```

--------------------------------

### Install and Configure AWS SageMaker Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the aioboto3 package, set AWS credentials, and specify a SageMaker endpoint name to use the SageMaker provider.

```bash
pip install aioboto3
export AWS_ACCESS_KEY_ID=access-key-id
export AWS_SECRET_ACCESS_KEY=secret-access-key
inspect eval arc.py --model sagemaker/my-endpoint-name \
  -M region_name=us-west-2
```

--------------------------------

### Install and Use Ollama Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package and specify an Ollama model to use it with Inspect. Ensure Ollama is running locally.

```bash
pip install openai
inspect eval arc.py --model ollama/llama3.1
```

--------------------------------

### Install Anthropic and Set API Key

Source: https://inspect.aisi.org.uk/providers.html

Install the Anthropic package and set your API key as an environment variable before using the provider.

```bash
pip install anthropic
export ANTHROPIC_API_KEY=your-anthropic-api-key
inspect eval arc.py --model anthropic/claude-sonnet-4-0
```

--------------------------------

### Set Model Cost Data (CLI Example)

Source: https://inspect.aisi.org.uk/llms-guide.txt

Example of a YAML file to configure model cost data for CLI usage. Prices are in dollars per million tokens.

```yaml
openai/gpt-4o:
    input: 2.50
    output: 10.00
    input_cache_write: 0
    input_cache_read: 1.25
anthropic/claude-sonnet-4-5-20250514:
    input: 3.00
    output: 15.00
    input_cache_write: 3.75
    input_cache_read: 0.30
```

--------------------------------

### Install and Use Llama-cpp-python Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package and specify a llama-cpp-python model to use it with Inspect. Ensure the llama-cpp-python server is running.

```bash
pip install openai
inspect eval arc.py --model llama-cpp-python/llama3
```

--------------------------------

### Running Inspect Intervention Example Interactively

Source: https://inspect.aisi.org.uk/llms-guide.txt

Bash command to evaluate the intervention example, setting the mode to computer and displaying the conversation for interactive use.

```bash
inspect eval examples/intervention -T mode=computer --display conversation
```

--------------------------------

### Install Azure AI Inference Package

Source: https://inspect.aisi.org.uk/providers.html

Install the necessary package for using the Azure AI provider. This is the first step before setting credentials and base URL.

```bash
pip install azure-ai-inference
export AZUREAI_API_KEY=api-key
export AZUREAI_BASE_URL=https://your-url-at.azure.com/models
$ inspect eval math.py --model azureai/Llama-3.3-70B-Instruct
```

--------------------------------

### inspect view start

Source: https://inspect.aisi.org.uk/reference/inspect_view.html

Starts a web-based viewer for evaluation logs. It can be configured to include logs recursively, specify the host and port, set the log level, and more.

```APIDOC
## inspect view start

### Description
View evaluation logs.

### Usage
```text
inspect view start [OPTIONS]
```

### Options
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Options
- **`--recursive`** (boolean) - Optional - Include all logs in log_dir recursively. Default: `True`
- **`--host`** (text) - Optional - Tcp/Ip host. Note: you can use `0.0.0.0` to expose the viewer and connect remotely (e.g. SSH). Default: `127.0.0.1`
- **`--port`** (integer) - Optional - TCP/IP port. Default: `7575`
- **`--log-level`** (choice) - Optional - Set the log level. Choices: `debug`, `trace`, `http`, `info`, `warning`, `error`, `critical`, `notset`. Default: `warning`
- **`--log-dir`** (text) - Optional - Directory for log files. Default: `./logs`
- **`--display`** (choice) - Optional - Set the display type. Choices: `full`, `conversation`, `rich`, `plain`, `log`, `none`. Default: `full`
- **`--traceback-locals`** (boolean) - Optional - Include values of local variables in tracebacks. Default: `False`
- **`--env`** (text) - Optional - Define an environment variable e.g. –env NAME=value (–env can be specified multiple times). Default: None
- **`--debug`** (boolean) - Optional - Wait to attach debugger. Default: `False`
- **`--debug-port`** (integer) - Optional - Port number for debugger. Default: `5678`
- **`--debug-errors`** (boolean) - Optional - Raise task errors (rather than logging them) so they can be debugged. Default: `False`
- **`--help`** (boolean) - Optional - Show this message and exit. Default: `False`

### Request Example
None

### Response
None
```

--------------------------------

### Define a CTF task with a setup solver

Source: https://inspect.aisi.org.uk/tasks.html

This snippet defines a CTF task that includes a 'setup' solver ('ctf_prompt') which is always executed, alongside a main solver that can be substituted. This is useful for steps like dynamic prompt engineering that should not be skipped.

```python
# prompt solver which should always be run
@solver
def ctf_prompt():
    async def solve(state, generate):
        # TODO: dynamic prompt engineering
        return state

    return solve

@task
def ctf(solver: Solver | None = None):
    # use default tool loop solver if no solver specified
    if solver is None:
        solver = ctf_tool_loop() # Assuming ctf_tool_loop is defined elsewhere

    # return task
    return Task(
        dataset=read_dataset(),
        setup=ctf_prompt(),
        solver=solver,
        sandbox="docker",
        scorer=includes()
    )

```

--------------------------------

### Install Apprise for Notifications

Source: https://inspect.aisi.org.uk/intervention.html

Installs the Apprise library, which Inspect AI uses for routing notifications to various services like Slack, desktop, SMS, and email.

```bash
pip install apprise
```

--------------------------------

### Get Event Start Time

Source: https://inspect.aisi.org.uk/reference/inspect_ai.event.html

Retrieves the timestamp of the event. This is a required field for all events.

```python
def start_time(self) -> datetime
```

--------------------------------

### Registering Entry Point with setuptools (pyproject.toml)

Source: https://inspect.aisi.org.uk/extensions-components.html

Configuration in pyproject.toml to register custom Inspect components as a setuptools entry point.

```toml
[project.entry-points.inspect_ai]
evals = "evals._registry"
```

--------------------------------

### Get Event End Time

Source: https://inspect.aisi.org.uk/reference/inspect_ai.event.html

Retrieves the completion time of the event. If not available, it returns the event's start time.

```python
def end_time(self) -> datetime
```

--------------------------------

### Computer Tool with Custom Options

Source: https://inspect.aisi.org.uk/tools-standard

Example of configuring the computer tool with custom max_screenshots and timeout options.

```python
solver=[
    use_tools([computer(max_screenshots=2, timeout=300)]),
    generate()
]
```

--------------------------------

### Get Model with Generation Configuration

Source: https://inspect.aisi.org.uk/custom-scorers.html

Retrieves a model and overrides its default generation options using the 'config' parameter. This example sets the temperature to 0.0.

```python
grader_model = get_model(
    "google/gemini-2.5-pro",
    config = GenerateConfig(
        temperature = 0.0
    )
)
```

--------------------------------

### Remote ACP Connection Setup

Source: https://inspect.aisi.org.uk/intervention.html

For remote evals, bind a TCP loopback port on the eval host and forward it over SSH. The ACP server has no built-in authentication, so the port should not be exposed directly. This example shows forwarding port 4545.

```bash
# eval listening for ACP connections on a loopback port
inspect eval terminal_bench_2 --acp-server 4545

# from your local machine, forward the port over SSH
ssh -L 4545:localhost:4545 user@eval-host

# in another local terminal, connect through the tunnel
inspect acp --server 127.0.0.1:4545
```

--------------------------------

### sample_init

Source: https://inspect.aisi.org.uk/reference/inspect_ai.util.html

Initializes sandbox environments for a specific sample within a task. It takes the task name, optional configuration, and metadata to set up the necessary environments.

```APIDOC
## sample_init

### Description
Initializes sandbox environments for a specific sample within a task. It takes the task name, optional configuration, and metadata to set up the necessary environments.

### Method
classmethod async

### Signature
`sample_init(task_name: str, config: SandboxEnvironmentConfigType | None, metadata: dict[str, str]) -> dict[str, "SandboxEnvironment"]`

### Parameters
#### Arguments
- **task_name** (str) - Name of task using the sandbox environment.
- **config** (SandboxEnvironmentConfigType | None) - Implementation defined configuration (optional).
- **metadata** (dict[str, str]) - Sample `metadata` field.
```

--------------------------------

### Weights & Biases Integration with Inspect AI Hooks

Source: https://inspect.aisi.org.uk/extensions-hooks.html

This Python code defines a custom Hooks class to integrate with Weights & Biases. It overrides run start, run end, and sample end events to log run information and sample scores. Ensure the 'wandb' library is installed.

```python
import wandb

from inspect_ai.hooks import Hooks, RunEnd, RunStart, SampleEnd, hooks

@hooks(name="w&b_hooks", description="Weights & Biases integration")
class WBHooks(Hooks):
    async def on_run_start(self, data: RunStart) -> None:
        wandb.init(name=data.run_id)

    async def on_run_end(self, data: RunEnd) -> None:
        wandb.finish()

    async def on_sample_end(self, data: SampleEnd) -> None:
        if data.sample.scores:
            scores = {k: v.value for k, v in data.sample.scores.items()}
            wandb.log({
                "sample_id": data.sample_id,
                "scores": scores,
            })
```

--------------------------------

### Example Run Configuration File

Source: https://inspect.aisi.org.uk/tasks.html

A sample YAML file defining task, model, generation, solver, and evaluation configurations for an Inspect AI run.

```yaml
task:
  task: inspect_evals/simpleqa
  args:
    split: test

model:
  model: anthropic/claude-sonnet-4-5
  args:
    max_retries: 3

model_roles:
  grader:
    model: openai/gpt-4o
    config:
      temperature: 0.0

generate_config:
  temperature: 0.5
  max_tokens: 4096
  seed: 42

solver:
  solver: my_solvers.py@chain_of_thought
  args:
    cot_template: detailed

eval_config:
  limit: 100
  epochs: 3
  message_limit: 50
```

--------------------------------

### Install TransformerLens Package

Source: https://inspect.aisi.org.uk/providers.html

Install the transformer_lens package using pip.

```bash
pip install transformer_lens
```

--------------------------------

### Install nnterp Package

Source: https://inspect.aisi.org.uk/providers.html

Install the nnterp package using pip.

```bash
pip install nnterp
```

--------------------------------

### Example Solver Implementation

Source: https://inspect.aisi.org.uk/reference/inspect_ai.solver.html

An example of how to define a solver function using the @solver decorator. This specific example shows a solver that inserts a chain of thought prompt but does not modify the state.

```python
@solver
def prompt_cot(template: str) -> Solver:
    def solve(state: TaskState, generate: Generate) -> TaskState:
        # insert chain of thought prompt
        return state

    return solve
```

--------------------------------

### Basic Code Execution Task Setup

Source: https://inspect.aisi.org.uk/tools-standard

Demonstrates how to set up a task that uses the code_execution() tool for a simple addition problem. This requires importing necessary components from the inspect_ai library.

```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.agent import react
from inspect_ai.tool import code_execution

@task
def code_execution_task():
    return Task(
        dataset=[Sample("Add 435678 + 23457")],
        solver=react(tools=[code_execution()])
    )
```

--------------------------------

### Run Security Guide Benchmark

Source: https://inspect.aisi.org.uk/llms-guide.txt

Command to evaluate the 'security_guide' task using a specified model. This command initiates the benchmark run and generates results.

```bash
inspect eval security_guide.py --model openai/gpt-5
```

--------------------------------

### Run Inspect View

Source: https://inspect.aisi.org.uk/log-viewer.html

Launches the Inspect View with default settings, using the configured log directory.

```bash
$ inspect view
```

--------------------------------

### Get Tools from MCP Server

Source: https://inspect.aisi.org.uk/reference/inspect_ai.tool.html

Retrieve a list of tools from an MCP server. You can specify which tools to get or retrieve all of them.

```python
def mcp_tools(
    server: MCPServer,
    *,
    tools: Literal["all"] | list[str] = "all",
) -> ToolSource
```

--------------------------------

### Get Model with Provider-Specific Parameters

Source: https://inspect.aisi.org.uk/models.html

Pass provider-specific parameters, such as 'device="cuda:0"', when getting a model.

```python
model = get_model("hf/openai-community/gpt2", device="cuda:0")
```

--------------------------------

### Task Setup with Memory Tool

Source: https://inspect.aisi.org.uk/tools-standard

Configure a task to use the memory tool by importing and including it in the solver's tools list.

```python
from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.agent import react
from inspect_ai.tool import memory

@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            react(tools=[memory()]),
        ],
        scorer=includes(),
    )
```

--------------------------------

### Install and Configure Fireworks AI Provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set your Fireworks API key, and specify the model for evaluation.

```bash
pip install openai
export FIREWORKS_API_KEY=your-firewrks-api-key
inspect eval arc.py --model fireworks/accounts/fireworks/models/deepseek-r1-0528
```

--------------------------------

### Evaluating Computer Task with Different Models

Source: https://inspect.aisi.org.uk/tools-standard

Command-line examples for evaluating a computer task using different AI models.

```bash
inspect eval computer.py --model anthropic/claude-sonnet-4-6
```

```bash
inspect eval computer.py --model openai/gpt-5.4
```

```bash
inspect eval computer.py --model google/gemini-3-flash-preview
```

--------------------------------

### Start Inspect View Log Viewer

Source: https://inspect.aisi.org.uk/reference/inspect_view.html

Launches a web-based viewer for evaluation logs. Configure host, port, and log directory for access. Useful for real-time log inspection.

```text
inspect view start [OPTIONS]
```

--------------------------------

### Configure OpenAI Compatible Provider (DeepSeek Example)

Source: https://inspect.aisi.org.uk/providers.html

Set environment variables for API key and base URL, then specify the model using the 'openai-api' provider. Hyphens in provider names are converted to underscores for environment variables.

```bash
export DEEPSEEK_API_KEY=your-deepseek-api-key
export DEEPSEEK_BASE_URL=https://api.deepseek.com
inspect eval arc.py --model openai-api/deepseek/deepseek-reasoner
```

--------------------------------

### Calculate Start Time of TimelineSpan

Source: https://inspect.aisi.org.uk/reference/inspect_ai.event.html

Calculates the earliest start time within a TimelineSpan. Optionally includes time from branched executions.

```python
def start_time(self, include_branches: bool = True) -> datetime
```

--------------------------------

### Configure Task with Todo Write Tool

Source: https://inspect.aisi.org.uk/tools-standard

This example shows how to set up a task that utilizes the todo_write() tool for managing long-horizon tasks. It's suitable for scenarios where tracking steps and progress is crucial.

```python
from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.agent import react
from inspect_ai.tool import bash, todo_write

@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=react(tools=[bash(timeout=180), todo_write()]),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Custom Scorer Example

Source: https://inspect.aisi.org.uk/reference/inspect_ai.scorer.html

Example of how to define a custom scorer function. This scorer compares model output state with a target to yield a score.

```python
@scorer
def custom_scorer() -> Scorer:
    async def score(state: TaskState, target: Target) -> Score:
        # Compare state / model output with target
        # to yield a score
        return Score(value=...)

    return score
```

--------------------------------

### task_init

Source: https://inspect.aisi.org.uk/reference/inspect_ai.util.html

Called at task startup to initialize resources for the sandbox environment.

```APIDOC
## task_init

### Description
Called at task startup initialize resources.

### Method
`task_init(cls, task_name: str, config: SandboxEnvironmentConfigType | None) -> None`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Parameters
* **task_name** (str) - Name of task using the sandbox environment.
* **config** (SandboxEnvironmentConfigType | None) - Implementation defined configuration (optional).
```

--------------------------------

### Initialize Sandbox Environment for Sample

Source: https://inspect.aisi.org.uk/reference/inspect_ai.util.html

Initializes sandbox environments for a given sample, using task name, configuration, and metadata. This is used when a sample requires its own isolated environment.

```python
async def sample_init(
    cls,
    task_name: str,
    config: SandboxEnvironmentConfigType | None,
    metadata: dict[str, str],
) -> dict[str, "SandboxEnvironment"]
```

--------------------------------

### Basic React Agent with Compaction

Source: https://inspect.aisi.org.uk/compaction.html.md

Demonstrates basic usage of the react() agent with different compaction strategies. Automatic compaction is the recommended default.

```python
from inspect_ai.agent import react
from inspect_ai.model import (
    CompactionAuto, CompactionEdit, CompactionNative
)
from inspect_ai.tool import bash, text_editor

# automatic compaction (recommended default)
react(
    tools=[bash(), text_editor()],
    compaction=CompactionAuto()
)

# edit compaction
react(
    tools=[bash(), text_editor()],
    compaction=CompactionEdit(keep_tool_uses=3)
)
```

--------------------------------

### Configured Think Tool for SWE-Bench

Source: https://inspect.aisi.org.uk/tools-standard

Shows how to configure the think() tool with a specific prompt to guide its usage for complex reasoning or brainstorming within the SWE-Bench context.

```python
from textwrap import dedent

from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, system_message, use_tools
from inspect_ai.tool import bash_session, text_editor, think

@task
def swe_bench():

    tools = [
        bash_session(timeout=180),
        text_editor(timeout=180),  
        think(dedent("""
            Use the think tool to think about something. It will not obtain
            new information or make any changes to the repository, but just 
            log the thought. Use it when complex reasoning or brainstorming
            is needed. For example, if you explore the repo and discover
            the source of a bug, call this tool to brainstorm several unique
            ways of fixing the bug, and assess which change(s) are likely to 
            be simplest and most effective. Alternatively, if you receive
            some test results, call this tool to brainstorm ways to fix the
            failing tests.
        """))
    ])

    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools(tools),
            generate(),
        ),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Install Development Version of Inspect AI

Source: https://inspect.aisi.org.uk/providers.html

Install the development version of Inspect AI from GitHub to enable features like model fallback.

```bash
pip install git+https://github.com/UKGovernmentBEIS/inspect_ai
```

--------------------------------

### Install Azure ADLFS Dependency

Source: https://inspect.aisi.org.uk/eval-logs.html

Install the 'adlfs' package, which is required for inspect to interact with Azure Blob Storage using fsspec schemes.

```bash
pip install "adlfs>=2025.8.0"
```

--------------------------------

### Configure Generation Options via Command Line

Source: https://inspect.aisi.org.uk/models.html

Specify generation configuration options like temperature and max_connections on the command line.

```bash
inspect eval arc.py --model openai/gpt-4 --temperature 0.9
inspect eval arc.py --model google/gemini-2.5-pro --max-connections 20
```

--------------------------------

### Install inspect-tool-support

Source: https://inspect.aisi.org.uk/tools-standard

Add these commands to your sandbox Dockerfile to install necessary dependencies for the web browser tool. This ensures the environment is set up correctly for Playwright.

```bash
RUN apt-get update && apt-get install -y pipx && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
ENV PATH="$PATH:/opt/inspect/bin"
RUN PIPX_HOME=/opt/inspect/pipx PIPX_BIN_DIR=/opt/inspect/bin PIPX_VENV_DIR=/opt/inspect/pipx/venvs \
    pipx install inspect-tool-support && \
    chmod -R 755 /opt/inspect && \
    inspect-tool-support post-install
```

--------------------------------

### Example Headless Output

Source: https://inspect.aisi.org.uk/llms-guide.txt

This is the expected output when running in headless mode with the `--display plain` option, showing task information and login instructions.

```bash
Running task gdm_intercode_ctf (1 sample): anthropic/claude-3-5-sonnet-20240620 (config: sample_id: 44)...

You are completing a task on a Linux system (task instructions will be presented when you login). Login to the system with the following command:

docker exec -it inspect-gdm_intercod-iebwzkg-default-1 bash -l
```

--------------------------------

### Override Default Column Definition

Source: https://inspect.aisi.org.uk/dataframe.html

Example of overriding a default column definition by redefining it later in the column list. This example keeps the 'metadata' column as JSON.

```python
samples_df(
    logs="logs",
    columns=SampleSummary + [SampleColumn("metadata", path="metadata")]
 )
```

--------------------------------

### Enable Beta Features with Custom Model Args

Source: https://inspect.aisi.org.uk/providers.html

Use the `-M` flag to pass custom arguments, such as beta identifiers, to the Anthropic provider. This example enables a 1M token context window for specific models.

```bash
inspect eval arc.py --model anthropic/claude-sonnet-4-0 -M betas=context-1m-2025-08-07
```

--------------------------------

### Sequential Tool Usage with `tool_choice`

Source: https://inspect.aisi.org.uk/tools-custom.html

This example demonstrates a sequence of operations where a tool is initially forced, followed by generation, and then subsequent tool usage is disabled.

```python
solver = [
  use_tools(addition(), tool_choice=ToolFunction(name="addition")),
  generate(),
  follow_up_prompt(),
  use_tools(tool_choice="none"),
  generate()
]
```

--------------------------------

### Get Cache Directory Path

Source: https://inspect.aisi.org.uk/reference/inspect_ai.model.html

Retrieve the file system path to the cache directory. Can specify a model name to get the path for a specific model's cache.

```python
def cache_path(model: str = "") -> Path

```

--------------------------------

### Implement RandomEarlyStopping Class

Source: https://inspect.aisi.org.uk/early-stopping.html

This example demonstrates a basic implementation of the `EarlyStopping` protocol, randomly deciding whether to skip a sample. It includes placeholders for tracking sample scores and returning custom log metadata.

```python
from pydantic import JsonValue
from typing_extensions import override

from inspect_ai.dataset import Sample
from inspect_ai.log import EvalSpec
from inspect_ai.scorer import SampleScore
from inspect_ai.util import EarlyStopping, EarlyStop

class RandomEarlyStopping(EarlyStopping):
    @override
    async def start_task(
        self, task: EvalSpec, samples: list[Sample], epochs: int
    ) -> str:
        """Task initialization."""

        # TODO: create a structure to track all of the samples/epochs
        # this will generally be updated w/ scores in complete_sample()

        # return task name
        return "random"

    @override
    async def schedule_sample(
        self, id: str | int, epoch: int
    ) -> EarlyStop | None:
        """Return EarlyStop to skip this sample, or None to run it."""

        # TODO: determine whether the given sample has been run based
        # on the previously accumulated samples scores.

        # randomly stop some samples
        if random() < 0.5:
            return EarlyStop(id=id, epoch=epoch, reason="random stop")

        return None

    @override
    async def complete_sample(
        self, id: str | int, epoch: int, scores: dict[str, SampleScore]
    ) -> None:
        """Process results from a completed sample."""

        # TODO: track scored samples and use this to determine the
        # appropriate return value for calls to schedule_sample()

        pass

    @override
    async def complete_task(self) -> dict[str, JsonValue]:
        """Return custom metadata to record in the eval log."""

        # TODO: return any custom data about the early stopping output
        # (will be written to the log and displayed in the viewer)

        return {}

```