### Development Environment Setup for τ-Bench

Source: https://github.com/sierra-research/tau2-bench/blob/main/CONTRIBUTING.md

Instructions for setting up the development environment, including cloning the repository, installing development dependencies with uv, and verifying the installation.

```bash
# Clone your fork
git clone https://github.com/your-username/tau2-bench.git
cd tau2-bench

# Install with dev tools (pytest, ruff, pre-commit — required for committing)
uv sync --extra dev

# Verify installation
uv run tau2 check-data
```

--------------------------------

### Copy Example .env File

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Copies the example environment file to be configured with API keys.

```bash
cp .env.example .env
```

--------------------------------

### Install τ-bench with Editable Mode

Source: https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md

Use this command for an editable installation, recommended for development.

```bash
pip install -e .
```

--------------------------------

### Install Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/web/leaderboard/README.md

Install all necessary project dependencies using npm.

```bash
npm install
```

--------------------------------

### Install All Tau2 Bench Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Installs Tau2 Bench with all available extra features.

```bash
uv sync --all-extras           # everything
```

--------------------------------

### Install Tau2 Bench with Knowledge Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Installs Tau2 Bench with the banking_knowledge domain and its retrieval pipeline.

```bash
uv sync --extra knowledge      # + banking_knowledge domain (retrieval pipeline)
```

--------------------------------

### Environment CLI Interaction Example

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/cli-reference.md

An example demonstrating interaction with the Environment CLI, including connecting to a domain and posing a query.

```bash
$ make env-cli

Welcome to the Environment CLI!
Connected to airline domain.

Query (:n new session, :d change domain, :q quit)> What flights are available from SF to LA tomorrow?
Assistant: Let me check the flight availability for you...

```

--------------------------------

### Start Interactive Play Mode with τ-bench

Source: https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md

Use this command to manually control an agent or user in interactive play mode. No setup is required beyond installing τ-bench.

```bash
tau2 play
```

--------------------------------

### Verify τ-bench Installation

Source: https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md

Run this command after installation to ensure the data directory is correctly set up.

```bash
tau2 check-data
```

--------------------------------

### Install Tau2 Bench Core

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Clones the repository and installs the core Tau2 Bench framework. This is for text-mode evaluation.

```bash
git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
uv sync                        # core only (text-mode: airline, retail, telecom, mock)
```

--------------------------------

### Install All τ-bench Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs all available dependencies for τ-bench, including core features and all extra options.

```bash
uv sync --all-extras
```

--------------------------------

### Start Development Server

Source: https://github.com/sierra-research/tau2-bench/blob/main/web/leaderboard/README.md

Launch the development server to run the web interface locally.

```bash
npm run dev
```

--------------------------------

### Run ReAct Agent Example

Source: https://github.com/sierra-research/tau2-bench/blob/main/examples/agents/README.md

Run the ReAct agent example to observe an agent that explicitly reasons before acting, demonstrating improved tool-use accuracy.

```bash
python examples/agents/react_agent.py
```

--------------------------------

### Install τ-bench with Optional Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/github-release-body.md

Install τ-bench using uv, with options to include extra dependencies for voice, knowledge, or all features.

```bash
uv sync                    # core text-mode evaluation  
uv sync --extra voice      # + voice/audio-native
uv sync --extra knowledge  # + banking_knowledge domain
uv sync --all-extras       # everything
```

--------------------------------

### Install Tau2 Bench with Gymnasium RL Interface

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Installs Tau2 Bench with the gymnasium RL interface.

```bash
uv sync --extra gym            # + gymnasium RL interface
```

--------------------------------

### Set up API Keys

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Copies the example environment file and prompts the user to edit it with their API keys. Supports any LiteLLM-compatible provider.

```bash
cp .env.example .env
# Edit .env with your API keys (uses LiteLLM — any supported provider works)
```

--------------------------------

### Run Demo Script

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/hyperparam/README.md

Execute the demo script to test the system with a small experiment. This is the recommended way to start.

```bash
src/experiments/hyperparam/demo.sh
```

--------------------------------

### Navigate to Leaderboard Directory

Source: https://github.com/sierra-research/tau2-bench/blob/main/web/leaderboard/README.md

Change the current directory to the web/leaderboard folder to begin setup.

```bash
cd web/leaderboard
```

--------------------------------

### Example Usage of tau2 new-command

Source: https://github.com/sierra-research/tau2-bench/blob/main/.release-template.md

Demonstrates how to use the 'tau2 new-command' with an example flag. This is typically found within release notes to show new functionality.

```bash
tau2 new-command --example
```

--------------------------------

### Install τ-bench Core Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs the core dependencies for τ-bench, supporting mock, airline, retail, and telecom domains.

```bash
uv sync
```

--------------------------------

### Install τ-bench with Gymnasium RL Interface

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs dependencies for τ-bench, including core features and the gymnasium RL interface.

```bash
uv sync --extra gym
```

--------------------------------

### Install τ-bench with Voice Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs dependencies for τ-bench, including core features and voice/audio-native capabilities.

```bash
uv sync --extra voice
```

--------------------------------

### Install τ-bench with Knowledge Domain Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs dependencies for τ-bench, including core features and the banking_knowledge domain with its retrieval pipeline.

```bash
uv sync --extra knowledge
```

--------------------------------

### Install Optional Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Manually installs 'langfuse' and 'redis' if 'USE_LANGFUSE=True' or 'LLM_CACHE_ENABLED=True' with redis cache is intended.

```bash
uv pip install langfuse redis
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/sierra-research/tau2-bench/blob/main/web/leaderboard/README.md

Copy the example environment file to create a local configuration. Uncomment VITE_SUBMISSIONS_BASE_URL to fetch data from S3.

```bash
cp .env.example .env.local
```

--------------------------------

### Install Tau2 Bench with Development Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Installs Tau2 Bench with development tools like pytest, ruff, and pre-commit.

```bash
uv sync --extra dev            # + pytest, ruff, pre-commit (required for contributing)
```

--------------------------------

### Install Core τ-bench Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Install the core dependencies for τ-bench, including support for text-mode domains. This command also sets up the virtual environment and enables the `tau2` command.

```bash
uv sync                        # core only (text-mode: airline, retail, telecom, mock)
```

--------------------------------

### Install τ-bench with Experiment Plotting Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs dependencies for τ-bench, including core features and plotting libraries for the src/experiments/ directory.

```bash
uv sync --extra experiments
```

--------------------------------

### Test Tier Installation Requirements

Source: https://github.com/sierra-research/tau2-bench/blob/main/tests/AGENTS.md

Table showing test tiers, their corresponding directories, and the required 'uv sync' command for installation.

```markdown
| Tier | Directories | Required install |
|------|-------------|-----------------|
| Core (`make test`) | `test_agent.py`, `test_environment.py`, `test_orchestrator.py`, `test_run.py`, `test_tasks.py`, `test_user.py`, `test_utils.py`, `test_llm_utils.py`, `test_checkpoint.py`, `test_results_format.py`, `test_domains/test_airline/`, `test_domains/test_mock/`, `test_domains/test_retail/`, `test_domains/test_telecom/` | `uv sync --extra dev` |
| Voice (`make test-voice`) | `test_voice/`, `test_streaming/` | `uv sync --extra voice --extra dev` |
| Knowledge (`make test-knowledge`) | `test_domains/test_banking_knowledge/` | `uv sync --extra knowledge --extra dev` |
| Gym (`make test-gym`) | `test_gym/` | `uv sync --extra gym --extra dev` |
| All (`make test-all`) | Everything above | `uv sync --all-extras` |

```

--------------------------------

### Initialization Function

Source: https://github.com/sierra-research/tau2-bench/blob/main/web/leaderboard/public/blog/tau-voice-examples.html

Clears previous renders and performs initial setup for the audio visualization components. This function is called when metadata is loaded.

```javascript
/* ── Init ─────────────────────────────────────── */
function init() {
  // Clear previous renders
  trackUser
```

--------------------------------

### Start All Tau2 Servers

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/cli-reference.md

Use this command to initiate all domain servers within the Tau2 environment.

```bash
tau2 start
```

--------------------------------

### Installing Optional τ-Bench Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/CONTRIBUTING.md

Commands to install additional optional dependencies for specific τ-Bench features, such as voice, knowledge, gym, or experiments.

```bash
uv sync --extra voice          # voice/audio-native features
uv sync --extra knowledge      # banking_knowledge domain (retrieval pipeline)
uv sync --extra gym            # gymnasium RL interface
uv sync --extra experiments    # plotting libs for src/experiments/
uv sync --all-extras           # everything
```

--------------------------------

### Run Minimal Text Agent

Source: https://github.com/sierra-research/tau2-bench/blob/main/examples/agents/README.md

Execute the minimal text agent example to see basic agent registration and execution against a mock domain.

```bash
python examples/agents/minimal_text_agent.py
```

--------------------------------

### Install τ-bench with Development Dependencies

Source: https://github.com/sierra-research/tau2-bench/blob/main/AGENTS.md

Installs dependencies for τ-bench, including core features and development tools like pytest, ruff, and pre-commit for code committing.

```bash
uv sync --extra dev
```

--------------------------------

### Example Filenames

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/hyperparam/README.md

Illustrates the naming convention for experiment result files, which includes LLM, domain, mode, and trial information.

```text
gpt-4.1-2025-04-14_retail_default_gpt-4.1-2025-04-14_4trials.json
claude-3-7-sonnet-20250219_telecom_oracle-plan_gpt-4.1-2025-04-14_3trials_50tasks.json
```

--------------------------------

### Install Optional τ-bench Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Install additional feature sets for τ-bench based on your needs. Use `--extra` flags to include specific functionalities like voice, knowledge, RL interfaces, development tools, or plotting libraries.

```bash
uv sync --extra voice          # + voice/audio-native features
```

```bash
uv sync --extra knowledge      # + banking_knowledge domain (retrieval pipeline)
```

```bash
uv sync --extra gym            # + gymnasium RL interface
```

```bash
uv sync --extra dev            # + pytest, ruff, pre-commit (required for contributing)
```

```bash
uv sync --extra experiments    # + plotting libs for src/experiments/
```

```bash
uv sync --all-extras           # everything
```

--------------------------------

### Provider Test Execution Example

Source: https://github.com/sierra-research/tau2-bench/blob/main/tests/AGENTS.md

Command to run audio native provider tests for a specific provider, enabling them via an environment variable.

```bash
{PROVIDER}_TEST_ENABLED=1 pytest tests/test_voice/test_audio_native/test_<provider>/ -v

```

--------------------------------

### Get CLI Help

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/hyperparam/README.md

Display help information for the main CLI or specific subcommands. Useful for understanding available options and arguments.

```bash
python -m experiments.hyperparam.cli --help
```

```bash
python -m experiments.hyperparam.cli run-evals --help
```

--------------------------------

### Run τ-bench with Mock Domain

Source: https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md

Example of running an evaluation using the mock domain, specifying agent and user LLMs, and trial count.

```bash
tau2 run --domain mock --agent-llm gpt-4 --user-llm gpt-4 --num-trials 1
```

--------------------------------

### Verifying Sandbox Runtime Setup on macOS

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/knowledge/README.md

Checks if all required sandbox runtime binaries (srt, rg) are installed and accessible in the system's PATH on macOS.

```bash
which srt rg
```

--------------------------------

### Verifying Sandbox Runtime Setup on Linux

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/knowledge/README.md

Checks if all required sandbox runtime binaries (srt, rg, bwrap, socat) are installed and accessible in the system's PATH on Linux.

```bash
which srt rg bwrap socat
```

--------------------------------

### Start Environment CLI

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/cli-reference.md

Launches an interactive command-line interface for querying and testing domain environments. Useful for debugging and exploring domain functionality.

```bash
make env-cli
```

--------------------------------

### LLMAgent Initialization and Usage

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/agent/README.md

Demonstrates how to create and use an LLMAgent for text-based evaluations. Requires environment tools and a domain policy. The agent is initialized with an LLM and its arguments, then used in a turn-based loop.

```python
from tau2.agent import LLMAgent

# Create the agent
agent = LLMAgent(
    tools=env.get_tools(),
    domain_policy=env.get_policy(),
    llm="gpt-4o",
    llm_args={"temperature": 0.7},
)

# Initialize state
state = agent.get_init_state()

# Turn-based loop (simplified)
response, state = agent.generate_next_message(user_msg, state)
```

--------------------------------

### Install Tau2 Bench with Voice Extras

Source: https://github.com/sierra-research/tau2-bench/blob/main/README.md

Installs Tau2 Bench with additional support for voice and audio-native features.

```bash
uv sync --extra voice          # + voice/audio-native features
```

--------------------------------

### Run Full Test Suite

Source: https://github.com/sierra-research/tau2-bench/blob/main/VERSIONING.md

Execute the complete test suite before a manual release. This command requires specific environment setup via 'uv sync --all-extras'.

```bash
make test-all  # Run full test suite (requires uv sync --all-extras)
```

--------------------------------

### Start Green Agent

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/agentify_tau_bench/README.md

Start the green agent (assessment manager) in a separate terminal. It runs on port 9001.

```bash
uv run tau-bench-agent green
```

--------------------------------

### Run Initial τ-bench Evaluation

Source: https://github.com/sierra-research/tau2-bench/blob/main/RELEASE_NOTES.md

This command initiates your first evaluation with specified domains, agent and user LLMs, and trial/task counts.

```bash
tau2 run --domain airline --agent-llm gpt-4 --user-llm gpt-4 --num-trials 1 --num-tasks 5
```

--------------------------------

### Start White Agent

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/agentify_tau_bench/README.md

Start the white agent (target agent being tested) in a separate terminal. It runs on port 9002.

```bash
uv run tau-bench-agent white
```

--------------------------------

### Example Model Release Configuration

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/leaderboard-submission.md

Use this configuration to provide metadata about the model itself, independent of the evaluation submission date. Ensure `release_date` is included if `announcement_url` is provided.

```json
{
  "model_release": {
    "release_date": "2025-08-05",
    "announcement_url": "https://www.anthropic.com/news/claude-opus-4-1",
    "announcement_title": "Claude Opus 4.1"
  }
}
```

--------------------------------

### Installing Sandbox Runtime Dependencies on macOS

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/knowledge/README.md

Installs ripgrep on macOS, which is a required dependency for the sandbox runtime when using shell tools.

```bash
brew install ripgrep
```

--------------------------------

### Run Basic Experiment

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/experiments/hyperparam/README.md

Initiate a basic experiment with multiple LLMs and domains. Specify the experiment directory, LLMs, domains, and number of trials.

```python
python -m experiments.hyperparam.cli run-evals \
    --exp-dir my-experiment \
    --llms gpt-4.1-2025-04-14 claude-3-7-sonnet-20250219 \
    --domains retail airline telecom \
    --num-trials 3
```

--------------------------------

### Commit Message Examples

Source: https://github.com/sierra-research/tau2-bench/blob/main/CONTRIBUTING.md

Examples of well-formatted commit messages following conventional commit standards. Use these to clearly describe the purpose of your changes.

```bash
# Good commit messages
feat: add memory system to agent base class
fix: resolve environment tool timeout issues
docs: update domain contribution guidelines
test: add integration tests for retail domain

# Avoid
fixed stuff
updates
wip
```

--------------------------------

### Install System Dependencies for Voice Features

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Install necessary system dependencies for enabling voice features in τ-bench. This includes `portaudio` and `ffmpeg` on macOS.

```bash
brew install portaudio ffmpeg
```

--------------------------------

### Installing Sandbox Runtime

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/knowledge/README.md

Installs the necessary sandbox runtime package for secure filesystem isolation. This is a prerequisite for retrieval configs that include shell access.

```bash
npm install -g @anthropic-ai/sandbox-runtime@0.0.23
```

--------------------------------

### Basic Setup for Tau2 Gym Environments

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/gym/README.md

Register and create AgentGymEnv or UserGymEnv instances. Use TAU_BENCH_ENV_ID for the agent and TAU_BENCH_USER_ENV_ID for the user. Additional configurations like solo_mode, user_llm, and user_llm_args can be passed during creation.

```python
import gymnasium as gym
from tau2.gym import register_gym_agent, TAU_BENCH_ENV_ID, TAU_BENCH_USER_ENV_ID

# Register the environments (only needed once)
register_gym_agent()

# Create AgentGymEnv - play as the agent
domain = "mock"
task_id = "create_task_1"

agent_env = gym.make(TAU_BENCH_ENV_ID, domain=domain, task_id=task_id)

# Or create UserGymEnv - play as the user
user_env = gym.make(TAU_BENCH_USER_ENV_ID, domain=domain, task_id=task_id)

# With additional configuration for AgentGymEnv
agent_env = gym.make(
    TAU_BENCH_ENV_ID, 
    domain=domain, 
    task_id=task_id,
    solo_mode=True,
    user_llm="gpt-4",
    user_llm_args={"temperature": 0.7}
)

# With additional configuration for UserGymEnv
user_env = gym.make(
    TAU_BENCH_USER_ENV_ID,
    domain=domain,
    task_id=task_id,
    agent_llm="gpt-4o",
    agent_llm_args={"temperature": 0.7}
)
```

--------------------------------

### Use Older ElevenLabs TTV v2 Model

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/voice-personas.md

Configure the setup script to use the older ElevenLabs TTV v2 model instead of the default v3 model. This is useful for compatibility or specific testing needs.

```bash
python -m tau2.voice.scripts.setup_voices --model eleven_multilingual_ttv_v2
```

--------------------------------

### Changelog Entry Format Example

Source: https://github.com/sierra-research/tau2-bench/blob/main/VERSIONING.md

Example of a manually formatted changelog entry, including version, date, and categorized changes (Added, Changed, Fixed).

```markdown
## [1.1.0] - 2025-02-15

### Added
- New healthcare domain with 50 evaluation tasks
- Support for streaming responses in CLI
- Agent performance visualization dashboard

### Changed
- Improved error messages in submission validation
- Updated default LLM timeout from 30s to 60s

### Fixed
- Fixed memory leak in concurrent evaluations
- Resolved issue with Unicode characters in task descriptions
```

--------------------------------

### Recommended Knowledge Base Workflow (shell)

Source: https://github.com/sierra-research/tau2-bench/blob/main/data/tau2/domains/banking_knowledge/prompts/components/shell_instructions.md

A step-by-step workflow for exploring and searching the knowledge base using shell commands.

```shell
ls
```

```shell
cat INDEX.md
```

```shell
grep -ri "<keyword>" .
```

```shell
cat <filename>
```

--------------------------------

### Installing Sandbox Runtime Dependencies on Linux

Source: https://github.com/sierra-research/tau2-bench/blob/main/src/tau2/knowledge/README.md

Installs ripgrep, bubblewrap, and socat on Debian/Ubuntu systems, which are required dependencies for the sandbox runtime when using shell tools.

```bash
# Ubuntu/Debian
sudo apt-get install ripgrep bubblewrap socat
```

--------------------------------

### Verify τ-bench Installation and Data Configuration

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Run this command to check if your data directory is correctly configured and all required files are present. This ensures τ-bench is ready for use.

```bash
uv run tau2 check-data
```

--------------------------------

### Conventional Commits: Good Examples

Source: https://github.com/sierra-research/tau2-bench/blob/main/AUTOMATION_GUIDE.md

Illustrative examples of well-formed conventional commit messages that adhere to the specified format and provide clear context for the changes made.

```bash
feat(domains): add telecom workflow policy support
fix(cli): handle missing config file gracefully  
docs(readme): update installation instructions
perf(evaluation): reduce memory usage in concurrent runs
test(domains): add integration tests for airline domain
```

--------------------------------

### Conventional Commits: Examples

Source: https://github.com/sierra-research/tau2-bench/blob/main/AUTOMATION_GUIDE.md

Examples of valid conventional commit messages for various types of changes, including features, fixes, performance improvements, documentation updates, and breaking changes.

```bash
# New feature
git commit -m "feat(domains): add healthcare domain with 50 tasks"

# Bug fix
git commit -m "fix(cli): resolve Unicode handling in task descriptions"

# Performance improvement
git commit -m "perf(evaluation): optimize concurrent task execution"

# Documentation
git commit -m "docs: update installation instructions for Windows"

# Breaking change
git commit -m "feat!: redesign agent configuration API"
```

--------------------------------

### Clone τ-bench Repository

Source: https://github.com/sierra-research/tau2-bench/blob/main/docs/getting-started.md

Clone the τ-bench repository and navigate into the project directory. This is the first step to setting up the project locally.

```bash
git clone https://github.com/sierra-research/tau2-bench
cd tau2-bench
```