Inspect AI

https://inspect.aisi.org.uk
Admin
Inspect AI is an open-source framework for large language model evaluations with support for coding,...
Tokens:838,392
Snippets:6,629
Trust Score:6
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Providing custom instructions to think() tool

Source: https://inspect.aisi.org.uk/llms-guide.txt

Demonstrates providing custom instructions to the think() tool via a dedent string, as used in the SWE-Bench example. The instructions guide the model on when to use the tool, such as for complex reasoning or brainstorming fixes.

```python
from textwrap import dedent

from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, system_message, use_tools
from inspect_ai.tool import bash_session, text_editor, think

@task
def swe_bench():

    tools = [
        bash_session(timeout=180),
        text_editor(timeout=180),  
        think(dedent("""
            Use the think tool to think about something. It will not obtain
            new information or make any changes to the repository, but just 
            log the thought. Use it when complex reasoning or brainstorming
            is needed. For example, if you explore the repo and discover
            the source of a bug, call this tool to brainstorm several unique
            ways of fixing the bug, and assess which change(s) are likely to 
            be simplest and most effective. Alternatively, if you receive
            some test results, call this tool to brainstorm ways to fix the
            failing tests.
        """))
    ])

    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools(tools),
            generate(),
        ),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Install Inspect from PyPI

Source: https://inspect.aisi.org.uk/

Install the inspect-ai package using pip.

```bash
pip install inspect-ai
```

--------------------------------

### Install azure-ai-inference and run model with API key

Source: https://inspect.aisi.org.uk/providers.html

Install the `azure-ai-inference` package, set `AZUREAI_API_KEY` and `AZUREAI_BASE_URL`, then run an evaluation with a deployed model (e.g., `Llama-3.3-70B-Instruct`).

```bash
pip install azure-ai-inference
export AZUREAI_API_KEY=api-key
export AZUREAI_BASE_URL=https://your-url-at.azure.com/models
$ inspect eval math.py --model azureai/Llama-3.3-70B-Instruct
```

--------------------------------

### compose.yaml with resource limits and network isolation

Source: https://inspect.aisi.org.uk/llms-guide.txt

Example compose.yaml that sets container resource limits (cpus, mem_limit) and isolates the container from all network interactions by setting network_mode: none. The init: true entry enables the container to respond to shutdown requests, and the command prevents the container from exiting after it starts.

```yaml
services:
  default: 
    build: .
    init: true
    command: tail -f /dev/null
    cpus: 1.0
    mem_limit: 0.5gb
    network_mode: none
```

--------------------------------

### Install and run Perplexity provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set the PERPLEXITY_API_KEY environment variable, and run an evaluation with the Perplexity provider using the perplexity/sonar model.

```bash
pip install openai
export PERPLEXITY_API_KEY=your-perplexity-api-key
inspect eval arc.py --model perplexity/sonar
```

--------------------------------

### Enable ACP and start interactive client

Source: https://inspect.aisi.org.uk/llms-guide.txt

Enable ACP for an eval with the --acp-server option, then run inspect acp in a separate terminal to attach to running sessions.

```bash
inspect eval terminal_bench_2 --acp-server
```

```bash
inspect acp
```

--------------------------------

### Define list_files() tool and file_probe() task with Docker sandbox

Source: https://inspect.aisi.org.uk/llms-guide.txt

These two Python blocks form a complete example: the first defines a list_files() tool that uses sandbox().exec() to run the ls command, and the second defines a file_probe() task that uses that tool with sandbox="docker". The task includes a sample with an inline file (bar.txt) and uses the includes() scorer.

```python
from inspect_ai.tool import ToolError, tool
from inspect_ai.util import sandbox

@tool
def list_files():
    async def execute(dir: str):
        """List the files in a directory.

        Args:
            dir: Directory

        Returns:
            File listing of the directory
        """
        result = await sandbox().exec(["ls", dir])
        if result.success:
            return result.stdout
        else:
            raise ToolError(result.stderr)

    return execute
```

```python
from inspect_ai import task, Task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, use_tools

dataset = [
    Sample(
        input='Is there a file named "bar.txt" ' 
               + 'in the current directory?',
        target="Yes",
        files={"bar.txt": "hello"},
    )
]

@task
def file_probe():
    return Task(
        dataset=dataset,
        solver=[
            use_tools([list_files()]), 
            generate()
        ],
        sandbox="docker",
        scorer=includes(),
    )
```

--------------------------------

### CheckpointConfig Python example and equivalent YAML

Source: https://inspect.aisi.org.uk/llms-guide.txt

Full Python configuration for checkpointing passed to the eval() API, with the equivalent YAML for use with --checkpoint=checkpoint.yaml. The YAML mirrors the Python object: a time-based trigger every 15 minutes, max_consecutive_failures of 3, and retention set to retain.

```python
from datetime import timedelta
from inspect_ai.util import CheckpointConfig, TimeInterval

eval(
    "arc.py", model="openai/gpt-4o",
    checkpoint=CheckpointConfig(
        trigger=TimeInterval(every=timedelta(minutes=15)),
        max_consecutive_failures=3,
        retention="retain",
    ),
)
```

```yaml
trigger:
  type: time
  every: 15m
max_consecutive_failures: 3
retention: retain
```

--------------------------------

### Configure safety_settings for Google models

Source: https://inspect.aisi.org.uk/llms-guide.txt

Sets safety settings for the Google provider. The Python example defines a dictionary with categories and thresholds, then passes it via model_args. The command-line equivalent uses the -M flag with a JSON-like string. Default is all categories set to 'none'.

```python
safety_settings = dict(
  dangerous_content = "medium_and_above",
  hate_speech = "low_and_above"
)
eval(
  "eval.py",
  model_args=dict(safety_settings=safety_settings)
)
```

```bash
inspect eval eval.py -M "safety_settings={'hate_speech': 'low_and_above'}"
```

--------------------------------

### Configure epoch reduction with Epochs

Source: https://inspect.aisi.org.uk/llms-guide.txt

Two examples of configuring epoch reduction in a task. The first uses a single reducer ("mode") with Epochs(5, "mode"). The second uses multiple reducers (["at_least_2", "at_least_5"]) to compute metrics with each reducer. Both are shown as task definitions for the gpqa task.

```python
@task
def gpqa():
    return Task(
        dataset=read_gpqa_dataset("gpqa_main.csv"),
        solver=[
            system_message(SYSTEM_MESSAGE),
            multiple_choice(),
        ],
        scorer=choice(),
        epochs=Epochs(5, "mode"),
    )
```

```python
@task
def gpqa():
    return Task(
        ...
        epochs=Epochs(5, ["at_least_2", "at_least_5"]),
    )
```

--------------------------------

### Set up and run Groq provider

Source: https://inspect.aisi.org.uk/providers.html

Install the groq package, set the GROQ_API_KEY environment variable, and run an evaluation with the Groq provider using the --model option.

```bash
pip install groq
export GROQ_API_KEY=your-groq-api-key
inspect eval arc.py --model groq/llama-3.1-70b-versatile
```

--------------------------------

### Get model for critique with get_model()

Source: https://inspect.aisi.org.uk/solvers.html

Use get_model() to retrieve the currently evaluated model or another model interface. The first example gets the model being evaluated, the second gets a specific model by name.

```python
# use the model being evaluated for critique
critique_model = get_model() 

# use another model for critique
critique_model = get_model("google/gemini-2.5-pro")
```

--------------------------------

### Task setup with web_browser()

Source: https://inspect.aisi.org.uk/tools-standard

Example task setup that uses the web_browser() tool. Note that web_browser() returns a list of tools, so it is concatenated with the list of other tools in the call to use_tools(). A separate web browser process is created within the sandbox for each instance of the tool.

```python
from inspect_ai import Task, task
from inspect_ai.scorer import match
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import bash, python, web_browser

@task
def browser_task():
    return Task(
        dataset=read_dataset(),
        solver=[
            use_tools([bash(), python()] + web_browser()),
            generate(),
        ],
        scorer=match(),
        sandbox=("docker", "compose.yaml"),
    )
```

--------------------------------

### sample_init

Source: https://inspect.aisi.org.uk/reference/inspect_ai.util.html

Initialize sandbox environments for a sample.

```APIDOC
## sample_init

### Description
Initialize sandbox environments for a sample.

### Method
`@classmethod async def sample_init(cls, task_name: str, config: SandboxEnvironmentConfigType | None, metadata: dict[str, str]) -> dict[str, "SandboxEnvironment"]`

### Parameters
- **task_name** (str) - Required - Name of task using the sandbox environment.
- **config** (SandboxEnvironmentConfigType | None) - Optional - Implementation defined configuration.
- **metadata** (dict[str, str]) - Required - Sample `metadata` field.

### Returns
- **dict[str, SandboxEnvironment]** - A dictionary mapping environment names to `SandboxEnvironment` instances.

```

--------------------------------

### SampleStart

Source: https://inspect.aisi.org.uk/reference/inspect_ai.hooks.html

Sample start hook event data.

```APIDOC
## SampleStart

### Description
Sample start hook event data.

### Attributes
- `eval_set_id` (str | None) - The globally unique identifier for the eval set (if any).
- `run_id` (str) - The globally unique identifier for the run.
- `eval_id` (str) - The globally unique identifier for the task execution.
- `sample_id` (str) - The globally unique identifier for the sample execution.
- `summary` (EvalSampleSummary) - Summary of the sample to be run.
```

--------------------------------

### Task setup with read-only tools

Source: https://inspect.aisi.org.uk/tools-standard

Example of configuring a task with read-only tools using the react solver. The task uses read_file(), list_files(), and grep() as tools, with a docker sandbox.

```python
from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.agent import react
from inspect_ai.tool import read_file, list_files, grep

@task
def code_analysis():
    return Task(
        dataset=read_dataset(),
        solver=react(tools=[read_file(), list_files(), grep()]),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Task setup using computer tool

Source: https://inspect.aisi.org.uk/tools-standard

Example task configuration that uses the computer tool. The task uses the computer tool with generate() and match() scorer, and specifies the Docker sandbox with compose.yaml.

```python
from inspect_ai import Task, task
from inspect_ai.scorer import match
from inspect_ai.solver import generate, use_tools
from inspect_ai.tool import computer

@task
def computer_task():
    return Task(
        dataset=read_dataset(),
        solver=[
            use_tools([computer()]),
            generate(),
        ],
        scorer=match(),
        sandbox=("docker", "compose.yaml"),
    )
```

--------------------------------

### Set up Google provider

Source: https://inspect.aisi.org.uk/

Install the Google GenAI Python package, set the API key, and run an eval with the Google Gemini model.

```bash
pip install google-genai
export GOOGLE_API_KEY=your-google-api-key
inspect eval simpleqa.py --model google/gemini-3.5-flash-lite
```

--------------------------------

### compose.yaml for remote registry image

Source: https://inspect.aisi.org.uk/sandboxing.html

Example compose.yaml for an image from a remote registry. Since the image is remote, x-local is not needed. The command tail -f /dev/null prevents the container from exiting after it starts.

```yaml
services:
  default:
    image: python:3.12-bookworm
    init: true
    command: tail -f /dev/null
    network_mode: none

```

--------------------------------

### SampleSource.initial_samples

Source: https://inspect.aisi.org.uk/reference/inspect_ai.html

Samples to run first (the seed). Called once, synchronously, when the Task is created — so it must return immediately (no awaiting / blocking). The returned samples drive the task’s up-front setup (validation, sandbox startup) and are the first batch.

```APIDOC
## SampleSource.initial_samples

### Description
Samples to run first (the seed). Called once, synchronously, when the Task is created — so it must return immediately (no awaiting / blocking). The returned samples drive the task’s up-front setup (validation, sandbox startup) and are the first batch. May be empty, in which case the task starts by calling `next_samples()`. The seed isn’t required for sandboxes: a sandbox config first seen in a later-added sample gets the same startup (image build/pull, validation, registered cleanup) before that sample runs.

### Method
initial_samples (instance method)

### Signature
```python
def initial_samples(self) -> list["Sample"]
```

### Parameters
None

### Returns
- **list['Sample']** - The initial samples to run.

```

--------------------------------

### Sample

Source: https://inspect.aisi.org.uk/reference/inspect_ai.dataset.html

Documentation for the Sample class, including its constructor, attributes, and metadata_as method.

```APIDOC
## Sample

### Description
Represents a single evaluation task sample, containing the input, target, and optional metadata.

### Constructor
```python
Sample(
    input: str | list[ChatMessage],
    choices: list[str] | None = None,
    target: str | list[str] = "",
    id: int | str | None = None,
    metadata: dict[str, Any] | None = None,
    sandbox: SandboxEnvironmentType | None = None,
    files: dict[str, str] | None = None,
    setup: str | None = None,
    checkpoint: CheckpointSampleConfig | None = None
) -> None
```

### Parameters
- **input** (str | list[ChatMessage]) - Required - The input to be submitted to the model.
- **choices** (list[str] | None) - Optional - List of available answer choices (used only for multiple-choice evals).
- **target** (str | list[str]) - Optional - Ideal target output. May be a literal value or narrative text to be used by a model grader.
- **id** (int | str | None) - Optional - Unique identifier for sample.
- **metadata** (dict[str, Any] | None) - Optional - Arbitrary metadata associated with the sample.
- **sandbox** (SandboxEnvironmentType | None) - Optional - Sandbox specification for this sample.
- **files** (dict[str, str] | None) - Optional - Files that go along with the sample (copied to SandboxEnvironment). Files can be paths, inline text, or inline binary (base64 encoded data URL).
- **setup** (str | None) - Optional - Setup script to run for sample (run within default SandboxEnvironment).
- **checkpoint** (CheckpointSampleConfig | None) - Optional - Checkpoint configuration for this sample. Customize-only — it does not enable checkpointing by itself (that is turned on at the task or eval level) and is ignored when nothing enabled it.

### Attributes
- **input** (str | list[ChatMessage]) - The input to be submitted to the model.
- **choices** (list[str] | None) - List of available answer choices (used only for multiple-choice evals).
- **target** (str | list[str]) - Ideal target output. May be a literal value or narrative text to be used by a model grader.
- **id** (int | str | None) - Unique identifier for sample.
- **metadata** (dict[str, Any] | None) - Arbitrary metadata associated with the sample.
- **sandbox** (SandboxEnvironmentSpec | None) - Sandbox environment type and optional config file.
- **files** (dict[str, str] | None) - Files that go along with the sample (copied to SandboxEnvironment).
- **setup** (str | None) - Setup script to run for sample (run within default SandboxEnvironment).
- **checkpoint** (CheckpointSampleConfig | None) - Checkpoint configuration for this sample: capture paths and snapshot-strategy selection per sandbox, trigger, and failure tolerance. The eval-wide storage policy (`checkpoints_location`, `retention`) lives only on CheckpointConfig at the task / eval layers. Customize-only: a sample config never enables checkpointing (that happens at the task or eval layer) and is ignored when nothing enabled it.

### Methods
#### metadata_as
```python
metadata_as(self, metadata_cls: Type[MT]) -> MT
```
Converts the sample's metadata to a Pydantic model of the specified type.

##### Parameters
- **metadata_cls** (Type[MT]) - Required - BaseModel derived class.

##### Returns
- **MT** - The metadata as a Pydantic model instance.
```

--------------------------------

### Launch record JSON output

Source: https://inspect.aisi.org.uk/control-channel.html

Example of the launch record printed on stdout when the detached eval starts. Contains run_id, eval_set_id, pid, log_dir, control socket path, and output_file path.

```json
{"event": "launch", "run_id": "Ngkz4viFYq…", "eval_set_id": null, "pid": 17146, "log_dir": "/…/logs", "control": {"socket_path": "/…/control/17146.sock"}, "output_file": "/…/detach/20260716-141530-ab12cd34.out"}
```

--------------------------------

### Set up OpenAI provider

Source: https://inspect.aisi.org.uk/

Install the OpenAI Python package, set the API key, and run an eval with the OpenAI GPT-4o model.

```bash
pip install openai
export OPENAI_API_KEY=your-openai-api-key
inspect eval simpleqa.py --model openai/gpt-4o
```

--------------------------------

### compose.yaml for local pre-built image

Source: https://inspect.aisi.org.uk/sandboxing.html

Example compose.yaml for a local pre-built image named ctf-agent-environment. The x-local: true indicates the image should not be pulled from a remote registry. The command tail -f /dev/null prevents the container from exiting after it starts.

```yaml
services:
  default: 
    image: ctf-agent-environment
    x-local: true
    init: true
    command: tail -f /dev/null
    network_mode: none

```

--------------------------------

### Install and set up Hugging Face storage bucket

Source: https://inspect.aisi.org.uk/eval-logs.html

Install the Hugging Face Hub package, authenticate, and create a private storage bucket for evaluation logs. The bucket must be created before use; Inspect does not create it automatically.

```bash
pip install "huggingface_hub>=1.6.0"
hf auth login
hf buckets create my-org/inspect-logs --private
```

--------------------------------

### Set up and run Together AI provider

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set the TOGETHER_API_KEY environment variable, and run an evaluation with the Together AI provider using the --model option.

```bash
pip install openai
export TOGETHER_API_KEY=your-together-api-key
inspect eval arc.py --model together/MiniMaxAI/MiniMax-M2.7
```

--------------------------------

### compose.yaml for tagged local image

Source: https://inspect.aisi.org.uk/sandboxing.html

Example compose.yaml for a local image tagged with a version. Since local images are tagged, they will not be pulled by default, so x-local: true is not required. The command tail -f /dev/null prevents the container from exiting after it starts.

```yaml
services:
  default: 
    image: ctf-agent-environment:1.0.0
    init: true
    command: tail -f /dev/null
    network_mode: none

```

--------------------------------

### SampleInit

Source: https://inspect.aisi.org.uk/reference/inspect_ai.hooks.html

Sample init hook event data.

```APIDOC
## SampleInit

### Description
Sample init hook event data.

### Attributes
- `eval_set_id` (str | None) - The globally unique identifier for the eval set (if any).
- `run_id` (str) - The globally unique identifier for the run.
- `eval_id` (str) - The globally unique identifier for the task execution.
- `sample_id` (str) - The globally unique identifier for the sample execution.
- `summary` (EvalSampleSummary) - Summary of the sample to be initialized.
```

--------------------------------

### Provide custom instructions to think() for SWE-Bench

Source: https://inspect.aisi.org.uk/tools-standard

Shows how to pass a custom prompt to the think() tool using textwrap.dedent. The prompt guides the model on when to use the tool, such as for complex reasoning or brainstorming fixes. This example is based on the original think tool article and is tailored for the SWE-Bench domain.

```python
from textwrap import dedent

from inspect_ai import Task, task
from inspect_ai.scorer import includes
from inspect_ai.solver import generate, system_message, use_tools
from inspect_ai.tool import bash_session, text_editor, think

@task
def swe_bench():

    tools = [
        bash_session(timeout=180),
        text_editor(timeout=180),  
        think(dedent("""
            Use the think tool to think about something. It will not obtain
            new information or make any changes to the repository, but just 
            log the thought. Use it when complex reasoning or brainstorming
            is needed. For example, if you explore the repo and discover
            the source of a bug, call this tool to brainstorm several unique
            ways of fixing the bug, and assess which change(s) are likely to 
            be simplest and most effective. Alternatively, if you receive
            some test results, call this tool to brainstorm ways to fix the
            failing tests.
        "))
    ])

    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools(tools),
            generate(),
        ),
        scorer=includes(),
        sandbox=("docker", "compose.yaml")
    )
```

--------------------------------

### Define a model-graded security guide task with Inspect

Source: https://inspect.aisi.org.uk/tutorial.html

Defines an Inspect task that loads the security_guide example dataset, uses a system message and generate() as the solver, and grades responses with model_graded_fact(). The @task decorator makes it runnable via `inspect eval security_guide.py --model openai/gpt-5`.

```python
from inspect_ai import Task, task
from inspect_ai.dataset import example_dataset
from inspect_ai.scorer import model_graded_fact
from inspect_ai.solver import generate, system_message

SYSTEM_MESSAGE = """
You are a computer security expert tasked with providing
concise responses to the following questions. Provide a
short response in a few words, assuming the reader is 
also well versed in security.
"""

@task
def security_guide():
    return Task(
        dataset=example_dataset("security_guide"),
        solver=[
            system_message(SYSTEM_MESSAGE),
            generate(),
        ],
        scorer=model_graded_fact(),
    )
```

--------------------------------

### Create an agent with per-instance store using store_as()

Source: https://inspect.aisi.org.uk/agent-custom.html

Shows how to give an agent a store-per-instance by default. Add an `instance` parameter to the @agent function, pass a unique value, and forward it to store_as() and any stateful tools like bash_session().

```python
from pydantic import Field
from shortuuid import uuid

from inspect_ai.agent import Agent, agent
from inspect_ai.model import ChatMessage
from inspect_ai.tool import bash_session
from inspect_ai.util import StoreModel, store_as

class BashExplorerState(StoreModel):
    messages: list[ChatMessage] = Field(default_factory=list)

@agent
def bash_explorer(instance: str | None = None) -> Agent:
    
    async def execute(state: AgentState) -> AgentState:

        # get state for this instance
        explorer_state = store_as(BashExplorerState, instance=instance)

        ...

        # pass the instance on to bash_session 
        messages, state.output = await get_model().generate_loop(
            state.messages, tools=[bash_session(instance=instance)]
        )
```

--------------------------------

### Install and run Meta provider with muse-spark-1.3

Source: https://inspect.aisi.org.uk/providers.html

Install the openai package, set the META_API_KEY environment variable, and run an inspection eval with the Meta provider using the muse-spark-1.3 model.

```bash
pip install openai
export META_API_KEY=your-meta-api-key
inspect eval arc.py --model meta/muse-spark-1.3
```

--------------------------------

### Run task from installed package

Source: https://inspect.aisi.org.uk/tasks.html

Run a task from an installed package by name. This works after the package is installed and its entry point is configured.

```bash
inspect eval evals/mytask
```
Tokens:
Raw