### Run Osmosis Rollout Server (Bash)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Starts the Osmosis Rollout server using the SDK CLI, specifying the agent loop module and function to execute. It shows example output indicating the server is running.

```bash
uv run osmosis serve -m server:agent_loop
```

--------------------------------

### Install Osmosis AI Server Dependencies (Bash)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Installs the Osmosis AI server and its dependencies using the `uv` package manager. If `uv` is not available, it provides an alternative using `pip`.

```bash
uv sync
```

```bash
pip install osmosis-ai[server]
```

--------------------------------

### Complete Agent Example with Multiple Tools

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This comprehensive Python example illustrates a full agent implementation capable of using multiple tools ('add' and 'multiply'). It sets up a tool registry, defines the tool implementations, and includes the necessary imports for the Osmosis AI rollout framework. This serves as a template for building more complex agents.

```python
import time
from typing import List, Dict, Any, Optional

from osmosis_ai.rollout import (
    RolloutAgentLoop,
    RolloutContext,
    RolloutResult,
    RolloutRequest,
    create_app,
)
from osmosis_ai.rollout.tools import (
    get_tool_call_info,
    create_tool_result,
    create_tool_error_result,
    execute_tool_calls,
)

# Tool implementations
async def add(a: float, b: float) -> float:
    return a + b

async def multiply(a: float, b: float) -> float:
    return round(a * b, 4)

TOOL_REGISTRY = {
    "add": add,
    "multiply": multiply,
}
```

--------------------------------

### Osmosis Eval Command Examples

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Examples demonstrating how to use the 'osmosis eval' command for various benchmarking and evaluation tasks. These examples showcase setting model endpoints, using multiple evaluation functions, performing pass@k analysis, comparing with external LLMs, and enabling concurrent execution.

```bash
# Benchmark trained model at an endpoint
osmosis eval -m server:agent_loop -d data.jsonl \
    --eval-fn rewards:compute_reward \
    --model my-finetuned-model --base-url http://localhost:8000/v1

# Multiple eval functions
osmosis eval -m server:agent_loop -d data.jsonl \
    --eval-fn rewards:exact_match \
    --eval-fn rewards:semantic_similarity \
    --model my-finetuned-model --base-url http://localhost:8000/v1

# pass@5 analysis
osmosis eval -m server:agent_loop -d data.jsonl \
    --eval-fn rewards:compute_reward --n 5 \
    --model my-finetuned-model --base-url http://localhost:8000/v1

# Baseline comparison with external LLM
osmosis eval -m server:agent_loop -d data.jsonl \
    --eval-fn rewards:compute_reward --model openai/gpt-5-mini -o results.json

# Concurrent execution
osmosis eval -m server:agent_loop -d data.jsonl \
    --eval-fn rewards:compute_reward --batch-size 5 \
    --model my-finetuned-model --base-url http://localhost:8000/v1
```

--------------------------------

### Starting Osmosis Rollout Server

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Start a RolloutServer for an agent loop implementation using the `osmosis serve` command. This command requires specifying the module path to the agent loop and offers optional parameters for port, host, logging, and development modes.

```bash
# Start server with validation (default port 9000)
osmosis serve -m server:agent_loop

# Specify port
osmosis serve -m server:agent_loop -p 8080

# Enable debug logging
osmosis serve -m server:agent_loop --log ./rollout_logs

# Enable auto-reload for development
osmosis serve -m server:agent_loop --reload

# Local debug mode (no auth)
osmosis serve -m server:agent_loop --local
```

--------------------------------

### Install Osmosis AI Python SDK

Source: https://docs.osmosis.ai/python-sdk/installation

Installs the core Osmosis AI Python library and CLI tool using pip. This is the fundamental step for using the SDK.

```bash
pip install osmosis-ai
```

--------------------------------

### Run Osmosis in Local Mode

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Start the Osmosis server in local mode, bypassing the need for login and platform registration. This is suitable for local development and testing without network connectivity.

```bash
uv run osmosis serve -m server:agent_loop --local
```

--------------------------------

### Install Osmosis AI Python SDK with Optional Dependencies

Source: https://docs.osmosis.ai/python-sdk/installation

Installs the Osmosis AI Python SDK with additional features. Use `[server]` for remote rollout servers, `[dev]` for development tools, and `[full]` for all available features.

```bash
# For running remote rollout servers (FastAPI + Uvicorn)
pip install "osmosis-ai[server]"

# For development (pytest, formatting tools)
pip install "osmosis-ai[dev]"

# Install everything
pip install "osmosis-ai[full]"
```

--------------------------------

### Define Calculator Agent Loop (Python)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Defines a `CalculatorAgent` that inherits from `RolloutAgentLoop`. It includes methods to get available tools and run the agent loop, with placeholders for implementation details.

```python
from osmosis_ai.rollout import RolloutAgentLoop, RolloutContext, RolloutResult, RolloutRequest, create_app

class CalculatorAgent(RolloutAgentLoop):
    name = "calculator"

    def get_tools(self, request: RolloutRequest):
        """Return available tools."""
        return CALCULATOR_TOOLS  # Defined in tools.py

    async def run(self, ctx: RolloutContext) -> RolloutResult:
        """Execute the agent loop."""
        ...  # See template repo for full implementation

agent_loop = CalculatorAgent()
app = create_app(agent_loop)
```

--------------------------------

### Create Osmosis AI App with Parameters

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

Demonstrates how to initialize an osmosis_ai application using the `create_app` function. It shows how to configure parameters like maximum concurrent rollouts, record time-to-live, debug directory, and startup/shutdown callbacks.

```python
from osmosis_ai.rollout import create_app, RolloutSettings

app = create_app(
    agent_loop,
    max_concurrent=10,
    record_ttl_seconds=3600.0,
    settings=RolloutSettings(...),
    debug_dir="./logs",
    on_startup=async_startup_fn,
    on_shutdown=async_shutdown_fn,
)
```

--------------------------------

### Enable Osmosis Debug Logging

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Run the Osmosis server with debug logging enabled, directing execution traces to a specified directory. This is crucial for diagnosing issues during development.

```bash
uv run osmosis serve -m server:agent_loop --log ./logs
```

--------------------------------

### Interactive Session Example Output

Source: https://docs.osmosis.ai/remote-rollout/test-mode

An example of the output during an interactive test session. It shows the system and user prompts, tool calls, tool results, and LLM responses, allowing for granular debugging.

```text
=== Interactive Mode ===
Dataset: test_data.jsonl (3 rows)
Starting at row 0

--- Row 0 ---
System: You are a calculator...
User: What is 15 * 7?

[turn 0] > n
Calling LLM...
Assistant: I'll calculate 15 * 7 using the multiply tool.
Tool calls: multiply(a=15, b=7)

Executing tools...
Tool result: 105

[turn 1] > n
Calling LLM...
Assistant: The answer is:

#### 105

[turn 1] Rollout complete
  Finish reason: stop
  Reward: 1.0

Continue to next row? (y/n/q) >
```

--------------------------------

### Example JSONL Dataset for Testing

Source: https://docs.osmosis.ai/remote-rollout/test-mode

An example of a JSONL formatted dataset suitable for Osmosis AI's test mode. Each line is a JSON object containing system prompt, user prompt, and optionally, the ground truth for reward computation.

```jsonl
{"system_prompt": "You are a calculator. Use tools to solve math problems. Format final answer as: #### <number>", "user_prompt": "What is 15 * 7?", "ground_truth": "105"}
{"system_prompt": "You are a calculator. Use tools to solve math problems. Format final answer as: #### <number>", "user_prompt": "What is 100 / 4?", "ground_truth": "25"}
{"system_prompt": "You are a calculator. Use tools to solve math problems. Format final answer as: #### <number>", "user_prompt": "What is 8 + 13?", "ground_truth": "21"}
```

--------------------------------

### Authenticate with Osmosis Platform (Bash)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Logs into the Osmosis Platform using the `uv osmosis login` command, which opens a browser for authentication. It also shows how to verify the login status.

```bash
uv run osmosis login
```

```bash
uv run osmosis whoami
```

--------------------------------

### Enable Osmosis Hot Reload for Development

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Start the Osmosis server with hot reloading enabled. This automatically restarts the server whenever code changes are detected, speeding up the development feedback loop.

```bash
uv run osmosis serve -m server:agent_loop --reload
```

--------------------------------

### Run Osmosis Test with Specific Model

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Execute the Osmosis test suite using a specific language model. This command requires the 'uv' tool and specifies the test module and data file.

```bash
uv run osmosis test -m server:agent_loop -d test_data.jsonl --model anthropic/claude-sonnet-4-5
```

--------------------------------

### Remote Rollout Commands: serve

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Start a RolloutServer for an agent loop implementation. This command allows you to serve an agent loop, with options to configure port, host, logging, and development settings.

```APIDOC
## Remote Rollout Commands: serve

Start a RolloutServer for an agent loop implementation.

### Usage

```bash
osmosis serve -m <module:attribute> [options]
```

### Required Options

| Option     | Short | Type   | Description                                                 |
| ---------- | ----- | ------ | ----------------------------------------------------------- |
| `--module` | `-m`  | string | Module path to the agent loop (e.g., `my_agent:agent_loop`) |

### Optional Parameters

| Option            | Short | Type    | Default        | Description                                           |
| ----------------- | ----- | ------- | -------------- | ----------------------------------------------------- |
| `--port`          | `-p`  | integer | `9000`         | Port to bind to                                       |
| `--host`          | `-H`  | string  | `0.0.0.0`      | Host to bind to                                       |
| `--no-validate`   |       | flag    | `false`        | Skip agent loop validation                            |
| `--reload`        |       | flag    | `false`        | Enable auto-reload for development                    |
| `--log-level`     |       | string  | `info`         | Uvicorn log level (debug/info/warning/error/critical) |
| `--log`           |       | string  |                | Enable logging to specified directory                 |
| `--api-key`       |       | string  | auto-generated | API key for TrainGate authentication                  |
| `--local`         |       | flag    | `false`        | Local debug mode (disable auth & registration)        |
| `--skip-register` |       | flag    | `false`        | Skip Osmosis Platform registration                    |

### Examples

```bash
# Start server with validation (default port 9000)
osmosis serve -m server:agent_loop

# Specify port
osmosis serve -m server:agent_loop -p 8080

# Enable debug logging
osmosis serve -m server:agent_loop --log ./rollout_logs

# Enable auto-reload for development
osmosis serve -m server:agent_loop --reload

# Local debug mode (no auth)
osmosis serve -m server:agent_loop --local
```
```

--------------------------------

### Check osmosis-ai CLI Version

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Displays the currently installed version of the osmosis-ai SDK. This is useful for verifying the installation and checking for updates.

```bash
osmosis --version
osmosis -V
```

--------------------------------

### Python: Usage Example for Reward Functions

Source: https://docs.osmosis.ai/python-sdk/api-reference

Demonstrates the usage of the 'exact_match' and 'semantic_eval' reward functions with sample solution and ground truth strings. It prints the scores obtained from both functions, illustrating their application.

```python
solution = "The capital of France is Paris"
truth = "Paris is France's capital"

local_score = exact_match(solution, truth)
semantic_score = semantic_eval(solution, truth, {})

print(f"Exact match: {local_score}")      # 0.0
print(f"Semantic: {semantic_score}")      # ~1.0
```

--------------------------------

### Provide Examples in Reward Rubrics for LLM Evaluation (Python)

Source: https://docs.osmosis.ai/git-sync/reward-rubrics

This Python code snippet shows how to enhance reward rubrics by including concrete examples of correct and incorrect outputs with their corresponding scores. This technique guides the LLM in understanding the desired evaluation outcomes and improves the accuracy of its judgments.

```python
rubric = """
Evaluate if the SQL query correctly answers the question.

Examples:
- "SELECT * FROM users WHERE age > 18" for "users over 18" → 1.0
- "SELECT name FROM users WHERE age >= 18" for "users over 18" → 0.8 (missing users exactly 18)
- "SELECT * FROM products" for "users over 18" → 0.0 (wrong table)

Score from 0.0 (completely wrong) to 1.0 (perfect).
"""
```

--------------------------------

### Verify Osmosis AI CLI Installation

Source: https://docs.osmosis.ai/python-sdk/installation

Checks if the Osmosis AI command-line interface (CLI) is installed and accessible by running the help command. This verifies the CLI component of the SDK.

```bash
osmosis --help
```

--------------------------------

### Verify Osmosis AI Python SDK Installation

Source: https://docs.osmosis.ai/python-sdk/installation

Tests the functionality of the installed Osmosis AI Python SDK by defining and calling a simple reward function. This confirms the SDK is correctly set up and operational.

```python
from osmosis_ai import osmosis_reward

@osmosis_reward
def test_fn(solution_str: str, ground_truth: str, extra_info: dict = None, **kwargs) -> float:
    return 1.0

print(test_fn("hello", "hello"))  # Output: 1.0
```

--------------------------------

### Run Local Agent Tests (Bash)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Runs batch tests for the agent using a local test dataset. It requires setting the `OPENAI_API_KEY` environment variable and provides options to limit the number of tests executed.

```bash
export OPENAI_API_KEY="your-key-here"

# Run batch tests
uv run osmosis test -m server:agent_loop -d test_data.jsonl

# Limit number of tests
uv run osmosis test -m server:agent_loop -d test_data.jsonl --limit 10
```

--------------------------------

### Run Tests with Different Models and APIs

Source: https://docs.osmosis.ai/remote-rollout/test-mode

Examples demonstrating how to run tests using different LLM providers and custom API endpoints. This includes specifying models like Claude and configuring a local OpenAI-compatible API.

```bash
# Default: GPT-5-mini
osmosis test -m server:agent_loop -d test_data.jsonl

# Use Claude
osmosis test -m server:agent_loop -d test_data.jsonl --model anthropic/claude-sonnet-4-5

# Use custom OpenAI-compatible API
osmosis test -m server:agent_loop -d test_data.jsonl --base-url http://localhost:8000/v1
```

--------------------------------

### create_app() Function

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This section details the parameters available for the `create_app` function, which is used to initialize a rollout server. It covers required and optional parameters for configuring agent loops, concurrency limits, record time-to-live, custom settings, debug directories, and startup/shutdown callbacks.

```APIDOC
## POST /create_app

### Description
Initializes and configures a rollout server with specified parameters.

### Method
POST

### Endpoint
/create_app

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **agent_loop** (RolloutAgentLoop) - Required - Your agent loop implementation.
- **max_concurrent** (int | None) - Optional - Maximum concurrent rollouts (None = unlimited). Default: None.
- **record_ttl_seconds** (float | None) - Optional - Time-to-live for rollout records in seconds. Default: 3600.0.
- **settings** (RolloutSettings | None) - Optional - Custom configuration settings.
- **debug_dir** (str | None) - Optional - Directory for debug logging. Default: None.
- **on_startup** (Callable[[], Awaitable[None]] | None) - Optional - Async function called on server start.
- **on_shutdown** (Callable[[], Awaitable[None]] | None) - Optional - Async function called on server stop.
- **credentials** (WorkspaceCredentials | None) - Optional - Platform credentials (auto-loaded from `osmosis login`).
- **server_host** (str | None) - Optional - Host for platform registration.
- **server_port** (int | None) - Optional - Port for platform registration.
- **api_key** (str | None) - Optional - API key for TrainGate authentication (auto-generated if omitted).

### Request Example
```json
{
  "agent_loop": "your_agent_loop_implementation",
  "max_concurrent": 10,
  "record_ttl_seconds": 3600.0,
  "settings": { ... },
  "debug_dir": "./logs",
  "on_startup": "async_startup_fn",
  "on_shutdown": "async_shutdown_fn"
}
```

### Response
#### Success Response (200)
- **app** (object) - The created application instance.

#### Response Example
```json
{
  "app": "<app_instance>"
}
```
```

--------------------------------

### RolloutSettings Configuration

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

Details on how to configure advanced settings using `RolloutSettings` and `RolloutClientSettings`, including client-side parameters like timeouts, retries, and connection pooling.

```APIDOC
## POST /create_app/settings

### Description
Configures advanced client settings for the rollout server using `RolloutSettings` and `RolloutClientSettings`.

### Method
POST

### Endpoint
/create_app/settings

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **settings** (RolloutSettings) - Required - Object containing `RolloutClientSettings`.
  - **client** (RolloutClientSettings) - Required - Client-specific settings.
    - **timeout_seconds** (float | None) - Optional - HTTP client timeout. Default: 300.0.
    - **max_retries** (int | None) - Optional - Maximum retry attempts. Default: 3.
    - **complete_rollout_retries** (int | None) - Optional - Retries for the `/completed` endpoint. Default: 2.
    - **retry_base_delay** (float | None) - Optional - Initial retry delay in seconds. Default: 1.0.
    - **retry_max_delay** (float | None) - Optional - Maximum retry delay in seconds. Default: 30.0.
    - **max_connections** (int | None) - Optional - HTTP connection pool size. Default: 100.
    - **max_keepalive_connections** (int | None) - Optional - Keep-alive connections. Default: 20.

### Request Example
```json
{
  "settings": {
    "client": {
      "timeout_seconds": 300.0,
      "max_retries": 3,
      "complete_rollout_retries": 2,
      "retry_base_delay": 1.0,
      "retry_max_delay": 30.0,
      "max_connections": 100,
      "max_keepalive_connections": 20
    }
  }
}
```

### Response
#### Success Response (200)
- **message** (str) - Confirmation message.

#### Response Example
```json
{
  "message": "Settings updated successfully."
}
```
```

--------------------------------

### Set LLM API Keys

Source: https://docs.osmosis.ai/remote-rollout/test-mode

Environment variables to set for authenticating with different LLM providers. This example shows how to set keys for OpenAI and Anthropic.

```bash
# OpenAI (default)
export OPENAI_API_KEY="your-key"

# Anthropic
export ANTHROPIC_API_KEY="your-key"
```

--------------------------------

### Configure RolloutSettings for Osmosis AI App

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

Illustrates advanced configuration of an osmosis_ai application using `RolloutSettings` and `RolloutClientSettings`. This allows fine-tuning of HTTP client behavior, including timeouts, retry mechanisms, and connection pooling.

```python
from osmosis_ai.rollout.config import RolloutSettings, RolloutClientSettings

settings = RolloutSettings(
    client=RolloutClientSettings(
        timeout_seconds=300.0,
        max_retries=3,
        complete_rollout_retries=2,
        retry_base_delay=1.0,
        retry_max_delay=30.0,
        max_connections=100,
        max_keepalive_connections=20,
    )
)

app = create_app(agent_loop, settings=settings)
```

--------------------------------

### Define Tool Schema for Multiplication (Python)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Defines a tool schema for multiplication using the OpenAI function calling format. It specifies the tool type, function name, description, parameters, and required arguments.

```python
CALCULATOR_TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "multiply",
            "description": "Multiply two numbers",
            "parameters": {
                "type": "object",
                "properties": {
                    "a": {"type": "number", "description": "First number"},
                    "b": {"type": "number", "description": "Second number"}
                },
                "required": ["a", "b"]
            }
        }
    }
]
```

--------------------------------

### Validate Agent Configuration (Bash)

Source: https://docs.osmosis.ai/remote-rollout/quickstart

Validates the agent implementation using the `uv osmosis validate` command. It shows the expected output indicating successful validation and lists the agent's name and available tools.

```bash
uv run osmosis validate -m server:agent_loop
```

--------------------------------

### Enable Debug Logging (Bash)

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This bash command enables debug logging for the `osmosis serve` command. The logs are directed to the specified directory, allowing for detailed tracing of agent execution.

```bash
osmosis serve -m server:agent_loop --log ./logs
```

--------------------------------

### Basic rubric evaluation example

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Performs a basic evaluation of a dataset against a rubric. This is the simplest form of the eval-rubric command.

```bash
osmosis eval-rubric --rubric helpfulness --data responses.jsonl
```

--------------------------------

### MCP Server Entry Point Script

Source: https://docs.osmosis.ai/git-sync/mcp-tools

This Python script serves as the entry point for running the MCP server. It uses `argparse` to handle command-line arguments for host and port, imports all defined tools, and then starts the FastMCP server.

```python
import argparse
from server.mcp_server import mcp
from tools import *  # Import all tools

if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--host", default="0.0.0.0")
    parser.add_argument("--port", type=int, default=8080)
    args = parser.parse_args()

    mcp.run(transport="http", host=args.host, port=args.port)
```

--------------------------------

### Complex MCP Tool for Fetching User Profile

Source: https://docs.osmosis.ai/git-sync/mcp-tools

This Python example defines a sophisticated MCP tool, `fetch_user_profile`, which retrieves user data and optionally includes activity history. It demonstrates optional parameters, conditional logic, and detailed docstrings explaining arguments, return values, and potential exceptions.

```python
from server import mcp
from typing import Optional
import json

@mcp.tool()
def fetch_user_profile(
    user_id: str,
    include_history: bool = False,
    max_items: Optional[int] = None
) -> dict:
    """
    Fetch a user's profile information from the database

    This tool retrieves user profile data and optionally includes
    their activity history. Use this when you need to look up
    information about a specific user.

    Args:
        user_id: Unique identifier for the user
        include_history: Whether to include activity history (default: False)
        max_items: Maximum number of history items to return (optional)

    Returns:
        Dictionary containing user profile data with keys:
        - id: User ID
        - name: User's full name
        - email: User's email address
        - history: Activity history (if include_history=True)

    Raises:
        ValueError: If user_id is not found
    """
    # Your implementation here
    profile = {
        "id": user_id,
        "name": "John Doe",
        "email": "john@example.com"
    }

    if include_history:
        history = get_user_history(user_id, max_items)
        profile["history"] = history

    return profile
```

--------------------------------

### Configure Osmosis AI Client Settings via Environment Variables

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

Shows how to override osmosis_ai client settings using environment variables. This provides a flexible way to manage configurations without modifying code, especially useful in different deployment environments.

```bash
export OSMOSIS_ROLLOUT_CLIENT_TIMEOUT_SECONDS=120
export OSMOSIS_ROLLOUT_CLIENT_MAX_RETRIES=5
osmosis serve -m server:agent_loop
```

--------------------------------

### Test Reward Function (Python)

Source: https://docs.osmosis.ai/git-sync/overview

This Python snippet demonstrates how to locally test a reward function. It imports a specific reward function and calls it with example string inputs to verify its output. This is useful for quick validation before pushing changes.

```python
from reward_fn.compute_reward import numbers_match_reward
print(numbers_match_reward('#### 42', '42'))
```

--------------------------------

### FastMCP Server Configuration

Source: https://docs.osmosis.ai/git-sync/mcp-tools

This Python snippet shows the basic setup for a FastMCP server, which is used to host Osmosis AI tools. It initializes the `FastMCP` application and defines a simple `/health` endpoint for checking server status.

```python
from fastmcp import FastMCP
from starlette.requests import Request
from starlette.responses import JSONResponse

mcp = FastMCP("OsmosisTools")

@mcp.custom_route("/health", methods=["GET"])
async def health(request: Request) -> JSONResponse:
    return JSONResponse({"status": "healthy"})
```

--------------------------------

### Benchmark Trained Model with Eval Mode

Source: https://docs.osmosis.ai/remote-rollout/eval-mode

Example commands to benchmark a trained model served at an endpoint using Eval Mode. This involves specifying the agent loop, dataset, evaluation function, model name, and the base URL of the serving endpoint.

```bash
# Benchmark a trained model served at an endpoint
# Replace <your-model> with the model name registered in your serving endpoint
osmosis eval -m server:agent_loop -d test_data.jsonl \
    --eval-fn rewards:compute_reward \
    --model <your-model> \
    --base-url http://localhost:8000/v1

# Multiple eval functions
osmosis eval -m server:agent_loop -d test_data.jsonl \
    --eval-fn rewards:exact_match \
    --eval-fn rewards:partial_match \
    --model <your-model> \
    --base-url http://localhost:8000/v1
```

--------------------------------

### Optional Helper Functions for Reward Calculation (Python)

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

These optional helper functions, `get_last_assistant_content` and `compute_reward_from_messages`, are provided for logging, debugging, and reward computation. They are not strictly required by the SDK but can be useful when rewards are computed on the remote rollout server.

```python
from typing import List, Dict, Any, Optional

def get_last_assistant_content(messages: List[Dict[str, Any]]) -> Optional[str]:
    """Get the content of the last assistant message.
    
    NOTE: This is a helper function for logging/debugging purposes only.
    It is NOT required by the SDK.
    """
    for message in reversed(messages):
        if message.get("role") == "assistant":
            return message.get("content", "")
    return None


def compute_reward_from_messages(
    messages: List[Dict[str, Any]], 
    ground_truth: Optional[str]
) -> Optional[float]:
    """Compute reward from messages if ground_truth is available.
    
    NOTE: This function is only needed if the platform is configured to 
    compute rewards in the remote rollout server. If not, you can skip 
    this entirely and return None for reward.
    """
    if not ground_truth:
        return None

    solution_str = get_last_assistant_content(messages)
    if not solution_str:
        return 0.0

    # Your reward logic here
    return compute_reward(solution_str, ground_truth)
```

--------------------------------

### Tool Function Signature with Type Hints

Source: https://docs.osmosis.ai/git-sync/mcp-tools

This Python example illustrates the required signature for an MCP tool function, emphasizing the use of type hints for all parameters and the return value. This ensures that the AI can correctly understand and utilize the tool's inputs and outputs.

```python
def my_tool(input_text: str, count: int) -> list[str]:
    # Implementation
    pass
```

--------------------------------

### Implement Custom Agent Loop with RolloutAgentLoop (Python)

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This Python code demonstrates how to create a custom agent by inheriting from the RolloutAgentLoop base class. It includes the required 'name' attribute and placeholder implementations for the 'get_tools' and 'run' methods, which are essential for agent functionality within the Osmosis training cluster.

```python
from osmosis_ai.rollout import (
    RolloutAgentLoop,
    RolloutContext,
    RolloutResult,
    RolloutRequest,
)

class MyAgent(RolloutAgentLoop):
    name = "my_agent"  # Required: unique identifier

    def get_tools(self, request: RolloutRequest) -> list:
        """Return tools available for this rollout."""
        pass

    async def run(self, ctx: RolloutContext) -> RolloutResult:
        """Execute the agent loop."""
        pass

```

--------------------------------

### Debug Logging with ctx.log_event (Python)

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This Python snippet illustrates how to use `ctx.log_event()` within the `run` method to trace the execution flow of an agent. It shows logging events at the start, after an LLM response, and upon agent completion, including relevant metrics.

```python
async def run(self, ctx: RolloutContext) -> RolloutResult:
    ctx.log_event("agent_start", 
                  num_messages=len(ctx.request.messages))
    
    # ... agent logic ...
    
    ctx.log_event("llm_response",
                  has_tools=result.has_tool_calls,
                  tokens=result.usage.get("total_tokens"))
    
    ctx.log_event("agent_complete",
                  finish_reason=finish_reason,
                  reward=reward)
```

--------------------------------

### Initialize and Push to GitHub (Bash)

Source: https://docs.osmosis.ai/git-sync/overview

These bash commands initialize a new Git repository, stage all current files, commit them with an initial message, set up a remote origin pointing to a GitHub repository, and push the changes to the main branch. This is the standard workflow for versioning and sharing code via GitHub.

```bash
git init
git add .
git commit -m "Initial Osmosis sync setup"
git remote add origin https://github.com/your-username/my-osmosis-repo.git
git push -u origin main
```

--------------------------------

### Remote Rollout Commands: validate

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Validate a RolloutAgentLoop implementation without starting the server. This command checks the agent loop for correctness.

```APIDOC
## Remote Rollout Commands: validate

Validate a RolloutAgentLoop implementation without starting the server.

### Usage

```bash
osmosis validate -m <module:attribute> [options]
```

### Options

| Option      | Short | Type   | Description                     |
| ----------- | ----- | ------ | ------------------------------- |
| `--module`  | `-m`  | string | Module path to the agent loop   |
| `--verbose` | `-v`  | flag   | Show detailed validation output |

### Examples

```bash
# Validate agent loop
osmosis validate -m server:agent_loop
```
```

--------------------------------

### Osmosis AI CLI Global Usage

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Demonstrates the general syntax for using the osmosis-ai CLI. It shows how to specify a command and its options.

```bash
osmosis [command] [options]
```

--------------------------------

### Error Handling: Model Not Found

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Resolves the 'Model not found' error by guiding the user to use a valid model identifier for their chosen provider.

```APIDOC
## Error Handling: Model Not Found

**Error:**
```
Error: Model 'gpt-5.2' not available for provider 'openai'
```

**Solution:** Use a valid model identifier for your chosen provider.
```

--------------------------------

### Set Up LLM Provider API Keys via Environment Variables

Source: https://docs.osmosis.ai/python-sdk/installation

Configures API keys for various LLM providers (OpenAI, Anthropic, Google Gemini) by setting environment variables. This is necessary for the SDK to authenticate with these services.

```bash
# OpenAI
export OPENAI_API_KEY="sk-..."

# Anthropic (Claude)
export ANTHROPIC_API_KEY="sk-ant-..."

# Google Gemini
export GOOGLE_API_KEY="..."
```

--------------------------------

### Example Test Results JSON Structure

Source: https://docs.osmosis.ai/remote-rollout/test-mode

The JSON structure representing the output when test results are saved. It includes overall summary metrics and a list of results for each processed row.

```json
{
  "summary": {
    "total": 3,
    "passed": 3,
    "failed": 0,
    "total_duration_ms": 5700,
    "total_tokens": 434
  },
  "results": [
    {
      "row_index": 0,
      "success": true,
      "error": null,
      "duration_ms": 2100,
      "token_usage": {"total_tokens": 148},
      "reward": 1.0,
      "finish_reason": "stop"
    }
  ]
}
```

--------------------------------

### Environment Variables for Client Settings

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

Lists environment variables that can be used to override default client settings for the rollout server, providing flexibility in configuration without code changes.

```APIDOC
## Environment Variables

### Description
Environment variables can be used to configure client settings for the Osmosis AI rollout server. These variables allow for runtime configuration without modifying the code.

### Variables
- **OSMOSIS_ROLLOUT_CLIENT_TIMEOUT_SECONDS** (float) - HTTP client timeout in seconds. Default: `300.0`.
- **OSMOSIS_ROLLOUT_CLIENT_MAX_RETRIES** (int) - Maximum number of retry attempts for requests. Default: `3`.
- **OSMOSIS_ROLLOUT_CLIENT_COMPLETE_ROLLOUT_RETRIES** (int) - Number of retries specifically for the `/completed` endpoint. Default: `2`.
- **OSMOSIS_ROLLOUT_CLIENT_RETRY_BASE_DELAY** (float) - The initial delay in seconds before the first retry. Default: `1.0`.
- **OSMOSIS_ROLLOUT_CLIENT_RETRY_MAX_DELAY** (float) - The maximum delay in seconds between retries. Default: `30.0`.
- **OSMOSIS_ROLLOUT_CLIENT_MAX_CONNECTIONS** (int) - The maximum number of concurrent connections allowed in the HTTP connection pool. Default: `100`.
- **OSMOSIS_ROLLOUT_CLIENT_MAX_KEEPALIVE_CONNECTIONS** (int) - The maximum number of keep-alive connections allowed. Default: `20`.

### Example Usage
```bash
export OSMOSIS_ROLLOUT_CLIENT_TIMEOUT_SECONDS=120
export OSMOSIS_ROLLOUT_CLIENT_MAX_RETRIES=5
osmosis serve -m server:agent_loop
```
```

--------------------------------

### Troubleshoot CLI Not Found Error

Source: https://docs.osmosis.ai/python-sdk/installation

Guides users to reinstall the Osmosis AI SDK forcefully and check their system's PATH environment variable if the CLI command is not found.

```bash
pip install --force-reinstall osmosis-ai
which osmosis
```

--------------------------------

### Skip Reward Computation in Python

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This snippet shows how to return `None` for the reward when reward computation is handled elsewhere, such as on the platform side. This is used when no reward computation is needed in the remote rollout.

```python
async def run(self, ctx: RolloutContext) -> RolloutResult:
    # ... agent loop logic ...
    
    # No reward computation needed
    return ctx.complete(messages, finish_reason=finish_reason, reward=None)
```

--------------------------------

### Define Tools using Plain Dictionaries

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This snippet shows an alternative way to define tools using plain Python dictionaries, which also follows OpenAI's function calling format. This method is less verbose than using schema classes and can be convenient for simpler tool definitions. It outlines the tool's type, function name, description, and parameters.

```python
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "search",
            "description": "Search for information",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string", "description": "Search query"}
                },
                "required": ["query"]
            }
        }
    }
]
```

--------------------------------

### Create Repository Structure (Bash)

Source: https://docs.osmosis.ai/git-sync/overview

This command creates the necessary directory structure for an Osmosis AI project, including folders for MCP tools, reward functions, and reward rubrics. It's a foundational step for organizing your AI components within a Git repository.

```bash
mkdir -p my-osmosis-repo/mcp/server my-osmosis-repo/mcp/tools my-osmosis-repo/mcp/test my-osmosis-repo/reward_fn my-osmosis-repo/reward_rubric
cd my-osmosis-repo
```

--------------------------------

### Full Mode Eval Function (Python)

Source: https://docs.osmosis.ai/remote-rollout/eval-mode

An example of a full mode evaluation function in Python, `conversation_quality`. This function receives the complete conversation history (`messages`) and metadata, allowing for more complex scoring logic.

```python
def conversation_quality(messages: list, ground_truth: str, metadata: dict, **kwargs) -> float:
    """Score based on the full conversation."""
    assistant_messages = [m for m in messages if m["role"] == "assistant"]
    return min(1.0, len(assistant_messages) / 3)
```

--------------------------------

### Extract Answer using Regex (Python)

Source: https://docs.osmosis.ai/remote-rollout/agent-loop

This Python function uses regular expressions to extract an answer from text, specifically looking for content following the '####' marker. This is a common pattern for parsing structured output.

```python
import re

def extract_answer(text: str) -> str | None:
    """Extract answer after #### marker."""
    match = re.search(r"####\s*(.+)", text)
    return match.group(1).strip() if match else None
```

--------------------------------

### Compare Baselines with LiteLLM in Eval Mode

Source: https://docs.osmosis.ai/remote-rollout/eval-mode

Demonstrates how to use Eval Mode to compare trained models against external LLM providers via LiteLLM. This requires setting API keys and specifying LiteLLM model identifiers.

```bash
export OPENAI_API_KEY="your-key"

# Compare against GPT-5-mini as a baseline
osmosis eval -m server:agent_loop -d test_data.jsonl \
    --eval-fn rewards:compute_reward --model openai/gpt-5-mini

# Compare against Claude
export ANTHROPIC_API_KEY="your-key"
osmosis eval -m server:agent_loop -d test_data.jsonl \
    --eval-fn rewards:compute_reward --model anthropic/claude-sonnet-4-5
```

--------------------------------

### Validating Osmosis Rollout Agent Loop

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Validate a RolloutAgentLoop implementation without starting the server using the `osmosis validate` command. This command requires the module path to the agent loop and can optionally show detailed validation output.

```bash
# Validate agent loop
osmosis validate -m server:agent_loop
```

--------------------------------

### Test Agent Loop with Cloud LLM Providers

Source: https://docs.osmosis.ai/python-sdk/cli-reference

Tests a RolloutAgentLoop against a dataset using cloud LLM providers. It requires specifying the module path to the agent loop and the dataset path. Optional parameters allow customization of the model, testing parameters, and output.

```bash
osmosis test -m <module:attribute> -d <dataset> [options]
```

```bash
# Batch test with default model
osmosis test -m server:agent_loop -d multiply.parquet

# Use Claude
osmosis test -m server:agent_loop -d data.jsonl --model anthropic/claude-sonnet-4-5

# Test subset of data
osmosis test -m server:agent_loop -d data.jsonl --limit 10

# Save results
osmosis test -m server:agent_loop -d data.jsonl -o results.json

# Interactive debugging
osmosis test -m server:agent_loop -d data.jsonl --interactive
```