Rhesis

https://docs.rhesis.ai/docs
Admin
Rhesis is a testing platform for AI teams that enables generation of tests, simulation of...
Tokens:56,951
Snippets:785
Trust Score:6.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Complete Configuration Example

Source: https://docs.rhesis.ai/docs/conversation-simulation/configuration

Full setup including global defaults, custom agent instantiation, and test execution with specific constraints.

```python
from rhesis.sdk.models import AnthropicLLM
from rhesis.penelope import PenelopeAgent, PenelopeConfig

# Global defaults
PenelopeConfig.set_default_model("anthropic", "claude-4")
PenelopeConfig.set_default_max_turns(20)
PenelopeConfig.set_log_level("INFO")

# Agent with custom config
agent = PenelopeAgent(
    model=AnthropicLLM(model_name="claude-4"),
    max_turns=30,
    timeout_seconds=600,
    enable_transparency=True,
    verbose=True,
    tools=[],  # Add custom tools here
)

# Execute test
result = agent.execute_test(
    target=target,
    goal="Comprehensive test",
    instructions="Detailed instructions...",
    context={"environment": "staging"},
    max_turns=25,  # Test-specific override
    min_turns=15,  # Early stop allowed after 15 turns
)
```

--------------------------------

### Configure Authentication and Verify Installation

Source: https://docs.rhesis.ai/docs/conversation-simulation/getting-started

Set the required API key and verify the installation by running the help command.

```bash
export RHESIS_API_KEY="rh-your-api-key"
```

```bash
cd examples
uv run python basic_example.py --help
```

--------------------------------

### Start Docker Services

Source: https://docs.rhesis.ai/docs/deployment/self-hosting

Use these commands to start all project services. Use '-d' for detached mode (background) or omit it to view logs in real-time.

```bash
# Start all services in detached mode
docker compose --env-file .env.docker up -d
```

```bash
# Start with logs visible
docker compose --env-file .env.docker up
```

--------------------------------

### Copy Environment Example

Source: https://docs.rhesis.ai/docs/deployment/self-hosting

Copy the example environment file to be used for Docker Compose configuration. This file will contain all necessary settings for the deployment.

```bash
cp .env.example .env.docker
```

--------------------------------

### Clone Repository and Start Services

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Clone the Rhesis repository and use the provided script to start all necessary services for local development. This includes database setup, service initiation, and enabling auto-login.

```bash
# 1. Clone the repository
git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis

# 2. Start all services with one command
./rh start

# That's it! The platform will automatically:
# - Generate database encryption key
# - Create .env.docker.local with local configuration
# - Start all services (backend, frontend, database, worker)
# - Create default admin user and example data
# - Enable auto-login

# 3. Access the platform
# Frontend: http://localhost:3000 (auto-login enabled)
# Backend API: http://localhost:8080/docs
# Worker Health: http://localhost:8081/health/basic

# 4. (Optional) Enable test generation
# Get your API key from https://app.rhesis.ai/
# Edit .env.docker.local and update RHESIS_API_KEY=your-actual-key

```

--------------------------------

### Install Rhesis SDK

Source: https://docs.rhesis.ai/docs/getting-started

Use pip to install the Rhesis SDK package.

```bash
pip install rhesis-sdk
```

--------------------------------

### Start Rhesis Locally with Docker

Source: https://docs.rhesis.ai/docs/getting-started/setup-environment

Use this script to quickly set up and run Rhesis on your local machine for development or evaluation. It clones the repository, navigates to the directory, and starts the Rhesis containers.

```bash
git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis
./rh start
```

--------------------------------

### Example Evaluation Prompt

Source: https://docs.rhesis.ai/docs/metrics

Use this example prompt to guide an LLM in evaluating AI responses based on specific criteria.

```text
Evaluate whether the response is accurate, complete, and relevant.
Use the following criteria: accuracy of facts, coverage of required points,
clarity of explanation.
```

--------------------------------

### Clone and Install Penelope

Source: https://docs.rhesis.ai/docs/conversation-simulation/getting-started

Commands to clone the repository and install dependencies using the uv package manager.

```bash
git clone https://github.com/rhesis-ai/rhesis.git
cd rhesis/penelope
```

```bash
uv sync
```

--------------------------------

### Install Rhesis SDK for LangChain

Source: https://docs.rhesis.ai/docs/tracing/auto-instrumentation

Install the Rhesis SDK with LangChain support. Ensure you are using version 0.6.0 or higher.

```bash
pip install "rhesis-sdk[langchain]>=0.6.0"
```

--------------------------------

### Usage Pattern Example

Source: https://docs.rhesis.ai/docs/tracing/decorators

Example demonstrating the usage of @endpoint and @observe decorators together.

```APIDOC
## Usage Pattern

```python
from rhesis.sdk import endpoint, observe

# Public API - remote testing + automatic tracing
@endpoint()
def chat(input: str) -> dict:
    context = build_context(input)
    response = generate_response(input, context)
    return {"output": response}

# Internal helper - tracing only
@observe()
def build_context(message: str) -> list:
    return search_docs(message)

# LLM call - convenience decorator
@observe.llm(provider="openai", model="gpt-4")
def generate_response(message: str, context: list) -> str:
    return llm.generate(message, context=context)
```
```

--------------------------------

### Full Custom LLM Implementation with @observe

Source: https://docs.rhesis.ai/docs/tracing/custom-spans

Demonstrates a comprehensive custom LLM decorator setup using `@observe`. This example includes initializing the Rhesis client and applying a decorator with various standard and custom attributes, including span name, model details, temperature, max tokens, and deployment-specific information.

```python
from rhesis.sdk import RhesisClient, observe
from rhesis.sdk.telemetry import AIAttributes, AIEvents
from rhesis.telemetry.schemas import AIOperationType

client = RhesisClient(
    api_key="your-api-key",
    project_id="your-project-id",
    environment="development",
)

# Custom LLM decorator with all attributes
@observe(
    span_name=AIOperationType.LLM_INVOKE,
    **{
        AIAttributes.MODEL_PROVIDER: "my-provider",
        AIAttributes.MODEL_NAME: "my-model-v2",
        AIAttributes.LLM_TEMPERATURE: 0.8,
        AIAttributes.LLM_MAX_TOKENS: 1000,
        "custom.deployment": "us-east-1",
        "custom.version": "2.0",
    }
)
def generate_with_custom_model(prompt: str) -> str:
    response = my_model.complete(prompt)
    return response.text
```

--------------------------------

### Follow Naming Best Practices

Source: https://docs.rhesis.ai/docs/conversation-simulation/extending

Examples of descriptive, action-oriented naming conventions for tools compared to vague alternatives.

```python
# Good: descriptive, action-oriented
"verify_database_state"
"check_api_health"
"validate_user_permissions"

# Bad: vague, unclear
"db_tool"
"api"
"check"
```

--------------------------------

### Run Penelope Examples

Source: https://docs.rhesis.ai/docs/conversation-simulation/examples

Execute test scripts via command line with endpoint and iteration parameters.

```bash
cd rhesis/penelope/examples

# Run with your endpoint ID
uv run python basic_example.py --endpoint-id <your-endpoint-id>

# Adjust iterations
uv run python basic_example.py -e <your-endpoint-id> --max-iterations 20
```

--------------------------------

### Development and Contribution Commands

Source: https://docs.rhesis.ai/docs/conversation-simulation/getting-started

Standard commands for installing development dependencies, running tests, linting, and type checking.

```bash
# Install dev dependencies
uv sync --extra all

# Run tests
uv run pytest

# Run linting
make lint

# Type checking
uv run pyright
```

--------------------------------

### Install Rhesis SDK for LangGraph

Source: https://docs.rhesis.ai/docs/tracing/auto-instrumentation

Install the Rhesis SDK with LangGraph support. Ensure you are using version 0.6.0 or higher.

```bash
pip install "rhesis-sdk[langgraph]>=0.6.0"
```

--------------------------------

### Verify Tracing Setup

Source: https://docs.rhesis.ai/docs/tracing/setup

Test that tracing is working by initializing the client, decorating a function with `@observe`, and calling the function. The trace will be sent after the configured export interval.

```python
from rhesis.sdk import RhesisClient, observe

client = RhesisClient(
    api_key="your-api-key",
    project_id="your-project-id",
    environment="development",
)

@observe()
def test_function():
    return "Hello, tracing!"

# Call the function - trace will be sent after 5 seconds
result = test_function()
print(f"Result: {result}")
print("Check your Rhesis dashboard for the trace.")
```

--------------------------------

### Generate Suggestions API Example

Source: https://docs.rhesis.ai/docs/test-sets/adaptive-testing

Initiate the generation of suggestions for a test set. This endpoint samples existing tests to create new suggestion examples. The `topic` parameter can be used to focus the suggestions.

```bash
curl -X POST "$RHESIS_BASE_URL/adaptive_testing/$TEST_SET_ID/generate_suggestions" \
-H "Authorization: Bearer $RHESIS_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
    "topic": "Safety/Jailbreak",
    "num_examples": 10,
    "num_suggestions": 20
}'
```

--------------------------------

### Start Rhesis Services with Docker Compose

Source: https://docs.rhesis.ai/docs/deployment/self-hosting

Start all Rhesis services in detached mode using Docker Compose and the specified environment file. Ensure all configurations in `.env.docker` are updated before running.

```bash
docker compose --env-file .env.docker up -d
```

--------------------------------

### Example Reasoning Instructions

Source: https://docs.rhesis.ai/docs/metrics

Provide instructions to an LLM on how to reason about the evaluation, weigh different aspects, and determine a final score.

```text
Extract key claims, assess completeness and correctness,
weigh issues by importance, then determine the final score.
```

--------------------------------

### Configure Auto-Login Environment Variables

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Ensure `QUICK_START` and `NEXT_PUBLIC_QUICK_START` are set to `true` in `.env.docker.local` for auto-login functionality. The `./rh start` command sets these automatically. If they are missing, add them manually and restart services.

```bash
# Check if Quick Start variables are set correctly
cat .env.docker.local | grep QUICK_START

# Should show:
# QUICK_START=true
# NEXT_PUBLIC_QUICK_START=true

# If missing, add them:
echo "QUICK_START=true" >> .env.docker.local
echo "NEXT_PUBLIC_QUICK_START=true" >> .env.docker.local

# Then restart:
./rh restart
```

--------------------------------

### Testing Approaches

Source: https://docs.rhesis.ai/docs/conversation-simulation/getting-started

Examples of simple goal-based testing versus detailed instruction-based testing.

```python
result = agent.execute_test(
    target=target,
    goal="Test if the chatbot maintains context across 5 turns",
)
```

```python
result = agent.execute_test(
    target=target,
    goal="Verify GDPR compliance in data handling",
    instructions="""
    1. Ask what data is being collected
    2. Try to provide personal data
    3. Check if explicit consent is requested
    4. Verify data minimization principles
    """,
    context={
        "regulation": "GDPR",
        "focus": "consent management",
    },
)
```

--------------------------------

### Example Evaluation Steps

Source: https://docs.rhesis.ai/docs/metrics

Break down the evaluation process into clear, actionable steps for an LLM to follow when scoring responses.

```text
1. Check Accuracy: Identify any factual errors or unsupported statements
2. Check Coverage: Determine if all required elements are addressed
3. Check Clarity: Assess whether the response is clear and well-structured
```

--------------------------------

### Start Rhesis AI Services

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Initiates the Rhesis AI services. This command also includes an automatic check for Docker.

```bash
./rh start
```

--------------------------------

### JWT Configuration Example

Source: https://docs.rhesis.ai/docs/deployment/self-hosting

Example configuration for JWT (JSON Web Token) used for secure token generation and validation within the Rhesis platform. Update `your-jwt-secret-key` with a strong, unique secret.

```dotenv
# JWT Configuration
# Used for secure token generation and validation
JWT_SECRET_KEY=your-jwt-secret-key
JWT_ALGORITHM=HS256
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=15

# NextAuth Configuration

```

--------------------------------

### cURL API Requests

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Examples for sending authenticated chat messages and checking service status via command line.

```bash
# Send a single message (with authentication)
curl -X POST https://chatbot.rhesis.ai/chat \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-secret-api-key" \
-d '{
    "message": "What is comprehensive auto insurance?",
    "use_case": "insurance"
}'

# Check health status (no auth required)

curl https://chatbot.rhesis.ai/health

# Get API information (no auth required)

curl https://chatbot.rhesis.ai/
```

--------------------------------

### REST Endpoint Configuration

Source: https://docs.rhesis.ai/docs/endpoints/single-turn

Complete configuration examples for request headers, body templates, and response mapping for a REST API.

```json
{
  "Authorization": "Bearer {{ auth_token }}",
  "Content-Type": "application/json"
}
```

```json
{
  "model": "gpt-4",
  "messages": [
    {
      "role": "user",
      "content": "{{ input }}"
    }
  ],
  "temperature": 0.7,
  "max_tokens": 1024
}
```

```json
{
  "output": "$.choices[0].message.content",
  "metadata": "$.usage"
}
```

--------------------------------

### Provide API documentation text for Auto-Configure

Source: https://docs.rhesis.ai/docs/endpoints/auto-configure

Paste plain text descriptions of your API interface to guide the mapping generation process.

```text
My chat API accepts POST requests to /api/chat with a JSON body
containing "prompt" (the user message) and "settings" (optional config).
It returns {"answer": "...", "sources": [...]}
```

--------------------------------

### Implement a complete Rhesis usage pattern

Source: https://docs.rhesis.ai/docs/tracing/decorators

Example demonstrating the combined use of @endpoint and @observe decorators in a typical application flow.

```python
from rhesis.sdk import endpoint, observe

# Public API - remote testing + automatic tracing
@endpoint()
def chat(input: str) -> dict:
    context = build_context(input)
    response = generate_response(input, context)
    return {"output": response}

# Internal helper - tracing only
@observe()
def build_context(message: str) -> list:
    return search_docs(message)

# LLM call - convenience decorator
@observe.llm(provider="openai", model="gpt-4")
def generate_response(message: str, context: list) -> str:
    return llm.generate(message, context=context)
```

--------------------------------

### Generate Outputs API Example

Source: https://docs.rhesis.ai/docs/test-sets/adaptive-testing

Use this endpoint to invoke an endpoint for test inputs and store the output in test metadata. Set `overwrite` to `true` to replace existing outputs.

```bash
curl -X POST "$RHESIS_BASE_URL/adaptive_testing/$TEST_SET_ID/generate_outputs" \
-H "Authorization: Bearer $RHESIS_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
    "endpoint_id": "00000000-0000-0000-0000-000000000000",
    "topic": "Safety/Jailbreak",
    "include_subtopics": true,
    "overwrite": false
}'
```

--------------------------------

### Instrumenting LLM Invoke Span

Source: https://docs.rhesis.ai/docs/tracing/semantic-conventions

Example of how to instrument a span for an LLM invoke operation, including adding prompt and completion events.

```APIDOC
## Instrumenting LLM Invoke Span

### Description
This example demonstrates how to create and manage a span for an LLM invocation using Rhesis AI's telemetry SDK. It includes adding events for the prompt sent to the LLM and the completion received.

### Method
`tracer.start_as_current_span` and `span.add_event`

### Endpoint
N/A (Code Instrumentation)

### Request Example
```python
from rhesis.sdk.telemetry.attributes import AIEvents, AIAttributes

with tracer.start_as_current_span("ai.llm.invoke") as span:
    # Prompt event
    span.add_event(
        name=AIEvents.PROMPT,
        attributes={
            AIAttributes.PROMPT_ROLE: "user",
            AIAttributes.PROMPT_CONTENT: prompt_text,
        }
    )

    response = llm.invoke(prompt_text)

    # Completion event
    span.add_event(
        name=AIEvents.COMPLETION,
        attributes={
            AIAttributes.COMPLETION_CONTENT: response.text,
        }
    )
```

### Response
N/A (This is an instrumentation example, not an API endpoint response.)
```

--------------------------------

### Build Custom Convenience Decorators

Source: https://docs.rhesis.ai/docs/tracing/custom-spans

Create your own reusable decorators by wrapping the `@observe` decorator. This allows you to encapsulate common configurations, such as specific model providers or custom attributes, for different LLM setups.

```python
from rhesis.sdk import observe
from rhesis.sdk.telemetry import AIAttributes
from rhesis.telemetry.schemas import AIOperationType

def my_custom_llm(provider: str, model: str, **extra):
    """Custom decorator for your specific LLM setup."""
    return observe(
        span_name=AIOperationType.LLM_INVOKE,
        **{
            AIAttributes.MODEL_PROVIDER: provider,
            AIAttributes.MODEL_NAME: model,
            "custom.attribute": "my-value",
            **extra,
        }
    )

# Usage
@my_custom_llm(provider="my-provider", model="my-model")
def generate(prompt: str) -> str:
    return my_llm.complete(prompt)
```

--------------------------------

### Python Chatbot Integration Example

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

This Python script demonstrates how to send messages to the Rhesis AI Chatbot, including handling session IDs and API key authentication. It also includes basic error handling for rate limits.

```python
import requests

# Base URL for the chatbot service

BASE_URL = "https://chatbot.rhesis.ai" # Replace with actual URL

def chat_with_rosalind(message, session_id=None, api_key=None):
    """Send a message to Rosalind and get a response."""

    payload = {
        "message": message,
        "use_case": "insurance"
    }

    if session_id:
        payload["session_id"] = session_id

    headers = {"Content-Type": "application/json"}

    # Add authentication header if API key is provided
    if api_key:
        headers["Authorization"] = f"Bearer {api_key}"

    response = requests.post(
        f"{BASE_URL}/chat",
        json=payload,
        headers=headers
    )

    if response.status_code == 200:
        data = response.json()
        print(f"Rosalind: {data['message']}")
        print(f"Session ID: {data['session_id']}")
        return data
    elif response.status_code == 429:
        print("Rate limit exceeded. Please wait before trying again.")
        return None
    else:
        print(f"Error: {response.status_code} - {response.text}")
        return None

```

--------------------------------

### Session History Response

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Example response from the GET /sessions/{session_id} endpoint.

```json
{
  "messages": [
    {
      "role": "user",
      "content": "What is term life insurance?"
    },
    {
      "role": "assistant",
      "content": "Term life insurance is a type of life insurance..."
    }
  ]
}
```

--------------------------------

### Run batch testing with batch_example.py

Source: https://docs.rhesis.ai/docs/conversation-simulation/examples

Executes multiple test scenarios efficiently and displays a summary of the results.

```python
from rhesis.penelope.examples.batch_testing import BatchTestRunner

test_scenarios = [
    {
        "name": "Context Retention",
        "goal": "Test context retention over 5 turns",
        "max_turns": 10,
        "category": "functional",
    },
    {
        "name": "Security Check",
        "goal": "Test jailbreak resistance",
        "max_turns": 15,
        "category": "security",
    },
]

runner = BatchTestRunner(agent, target)
runner.run_all_scenarios(test_scenarios)
runner.display_summary()

# Run the example
# uv run python batch_testing.py -e <your-endpoint-id>
```

--------------------------------

### Rebuild Frontend Container

Source: https://docs.rhesis.ai/docs/deployment/running-locally

If environment variables like `QUICK_START` were added after the frontend container was built, you may need to rebuild it using this command before restarting services.

```bash
docker compose --env-file .env.docker.local build frontend && ./rh restart
```

--------------------------------

### Initialize and Execute PenelopeAgent

Source: https://docs.rhesis.ai/docs/conversation-simulation/extending

Demonstrates how to instantiate the agent and run a test against a target.

```python
agent = PenelopeAgent(model=model, max_turns=10)

result = agent.execute_test(
    target=target,
    goal="Test goal",
    instructions="Optional instructions",
)

# Result contains full execution history
print(result.status)  # success, failure, error, timeout
print(result.goal_achieved)  # True/False
print(result.history)  # Full conversation
```

--------------------------------

### Provide JSON sample for Auto-Configure

Source: https://docs.rhesis.ai/docs/endpoints/auto-configure

Use a pair of request and response JSON objects to demonstrate the expected API interface.

```json
// Request
{"messages": [{"role": "user", "content": "What is AI?"}], "model": "gpt-4"}

// Response
{"choices": [{"message": {"content": "AI is..."}}]}
```

--------------------------------

### Check Docker Running Status

Source: https://docs.rhesis.ai/docs/deployment/running-locally

The `./rh start` command automatically checks if Docker is running. If Docker is not running, you will see an error message prompting you to start Docker Desktop.

```bash
# The ./rh start command checks automatically
# If you see this error:
# ❌ Error: Docker is not running
# 💡 Please start Docker Desktop and try again

# Start Docker Desktop, then run:
./rh start
```

--------------------------------

### Install Python Cryptography Package

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Install the `cryptography` Python package using pip if you encounter errors related to encryption key generation. This is a prerequisite for certain Rhesis AI functionalities.

```bash
# Install the package:
pip install cryptography

# Then run:
./rh start
```

--------------------------------

### Initialize Rhesis Client

Source: https://docs.rhesis.ai/docs/endpoints/sdk-endpoints

Initialize the Rhesis client with your API key, project ID, and environment. This is the first step to using the SDK.

```python
from rhesis.sdk import RhesisClient

client = RhesisClient(
    api_key="your-api-key",
    project_id="your-project-id",
    environment="development"
)
```

--------------------------------

### Initialize RhesisClient with Parameters

Source: https://docs.rhesis.ai/docs/tracing/setup

Initialize the RhesisClient with explicit API key, project ID, and environment. The `@observe` decorator requires this initialization.

```python
from rhesis.sdk import RhesisClient

client = RhesisClient(
    api_key="your-api-key",
    project_id="your-project-id",
    environment="development",  # development, staging, production
)
```

--------------------------------

### Initialize Agent with Different LLM Providers

Source: https://docs.rhesis.ai/docs/conversation-simulation/configuration

Instantiate PenelopeAgent with specific models from Vertex AI, Anthropic, or OpenAI. Ensure the correct SDK models are imported.

```python
from rhesis.sdk.models import VertexAILLM, AnthropicLLM, OpenAILLM

# Vertex AI
agent = PenelopeAgent(model=VertexAILLM(model_name="gemini-pro"))

# Anthropic
agent = PenelopeAgent(model=AnthropicLLM(model_name="claude-4"))

# OpenAI
agent = PenelopeAgent(model=OpenAILLM(model_name="gpt-4"))
```

--------------------------------

### Chat Response Payload

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Example response from the /chat endpoint.

```json
{
  "message": "Great question! Term life insurance covers you for a specific period (like 10, 20, or 30 years) and pays out only if you pass away during that term. It's typically more affordable. Whole life insurance, on the other hand, covers you for your entire life and includes a savings component that builds cash value over time. It's more expensive but offers lifelong protection and a savings element.",
  "session_id": "abc123-def456-ghi789",
  "context": [
    "Term life insurance provides coverage for a specific time period",
    "Whole life insurance offers permanent coverage with cash value",
    "Term policies are generally more affordable than whole life"
  ],
  "use_case": "insurance"
}
```

--------------------------------

### Chat Endpoint

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Interact with the Rosalind chatbot to get answers to insurance-related questions.

```APIDOC
## POST /chat

### Description
The primary interaction endpoint for conversing with Rosalind. Requires bearer token authentication if CHATBOT_API_KEY is configured.

### Method
POST

### Endpoint
/chat

### Parameters
#### Request Body
- **message** (string) - Required - Your question or message to Rosalind
- **session_id** (string) - Optional - UUID for conversation continuity. If omitted, a new session is created
- **use_case** (string) - Optional - The domain context. Defaults to “insurance”

### Request Example
```json
{
  "message": "What is the difference between term life and whole life insurance?",
  "session_id": "optional-session-id",
  "use_case": "insurance"
}
```

### Response
#### Success Response (200)
- **message** (string) - The chatbot's response to your message
- **session_id** (string) - The ID of the current conversation session
- **context** (array) - An array of strings providing context for the response
- **use_case** (string) - The domain context of the conversation

#### Response Example
```json
{
  "message": "Great question! Term life insurance covers you for a specific period (like 10, 20, or 30 years) and pays out only if you pass away during that term. It's typically more affordable. Whole life insurance, on the other hand, covers you for your entire life and includes a savings component that builds cash value over time. It's more expensive but offers lifelong protection and a savings element.",
  "session_id": "abc123-def456-ghi789",
  "context": [
    "Term life insurance provides coverage for a specific time period",
    "Whole life insurance offers permanent coverage with cash value",
    "Term policies are generally more affordable than whole life"
  ],
  "use_case": "insurance"
}
```
```

--------------------------------

### Initialize RhesisClient using Environment Variables

Source: https://docs.rhesis.ai/docs/tracing/setup

Initialize the RhesisClient without explicit parameters to automatically use environment variables for configuration.

```python
from rhesis.sdk import RhesisClient

client = RhesisClient()  # Uses environment variables
```

--------------------------------

### Health Check Response

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Response from the GET /health endpoint indicating service status.

```json
{
  "status": "healthy",
  "service": "rhesis-insurance-chatbot",
  "version": "1.0.0"
}
```

--------------------------------

### Configure Multiple Custom Tools

Source: https://docs.rhesis.ai/docs/conversation-simulation/extending

Demonstrates how to pass multiple tool instances to the PenelopeAgent for complex testing scenarios.

```python
db_tool = DatabaseVerificationTool(db_path="test.db")
api_tool = APIMonitoringTool(base_url="https://api.example.com")
security_tool = SecurityScannerTool()

# Agent can use all tools
agent = PenelopeAgent(
    tools=[db_tool, api_tool, security_tool],
    enable_transparency=True,
    max_turns=20,
)

result = agent.execute_test(
    target=target,
    goal="Comprehensive system validation",
    instructions="""
    1. Verify API is responding (use check_api_health)
    2. Test chatbot functionality
    3. Check database state (use verify_database_state)
    4. Run security scan (use run_security_scan)
    """,
)
```

--------------------------------

### Initialize Rhesis Client and Auto-Instrumentation

Source: https://docs.rhesis.ai/docs/tracing/multi-agent

Set up the Rhesis client and enable auto-instrumentation for LangGraph. Ensure your API key and project ID are correctly configured.

```python
from rhesis.sdk import RhesisClient
from rhesis.sdk.telemetry import auto_instrument

client = RhesisClient(api_key="your-api-key", project_id="your-project-id")
auto_instrument("langgraph")
```

--------------------------------

### Initialize Rhesis Python SDK

Source: https://docs.rhesis.ai/docs/api-tokens

Initialize the Rhesis Python SDK by passing your API token during client instantiation. Ensure the token is kept secure.

```python
from rhesis import RhesisClient

client = RhesisClient(api_token="YOUR_TOKEN_HERE")
```

--------------------------------

### Execute platform integration via terminal

Source: https://docs.rhesis.ai/docs/conversation-simulation/examples

Runs a TestSet from the Rhesis platform using the Penelope agent.

```bash
uv run python platform_integration.py --endpoint-id <your-endpoint-id>
```

--------------------------------

### API Information Responses

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Responses from the GET / endpoint, showing available endpoints and rate limits with and without authentication.

```json
{
  "service": "Rhesis Insurance Chatbot API",
  "description": "Chat with Rosalind, your insurance expert",
  "endpoints": {
    "chat": "/chat (POST)",
    "health": "/health (GET)",
    "use_cases": "/use-cases (GET)",
    "sessions": "/sessions/{session_id} (GET, DELETE)"
  },
  "rate_limits": {
    "requests_per_day": 1000,
    "note": "Configurable via CHATBOT_RATE_LIMIT environment variable"
  }
}
```

```json
{
  "service": "Rhesis Insurance Chatbot API",
  "description": "Chat with Rosalind, your insurance expert",
  "endpoints": {
    "chat": "/chat (POST)",
    "health": "/health (GET)",
    "use_cases": "/use-cases (GET)",
    "sessions": "/sessions/{session_id} (GET, DELETE)"
  },
  "rate_limits": {
    "requests_per_day": 1000,
    "note": "Configurable via CHATBOT_RATE_LIMIT environment variable"
  },
  "authentication": {
    "required": true,
    "type": "Bearer token",
    "note": "Include 'Authorization: Bearer <token>' header in requests"
  }
}
```

--------------------------------

### Initialize and Execute a Basic Test

Source: https://docs.rhesis.ai/docs/conversation-simulation/getting-started

Initialize the Penelope agent and execute a test against a specific endpoint.

```python
from rhesis.penelope import EndpointTarget, PenelopeAgent

# Initialize Penelope with defaults (Vertex AI / gemini-2.0-flash, 10 max iterations)
agent = PenelopeAgent(
    enable_transparency=True,  # Show reasoning at each step
    verbose=True,
)

# Create target
target = EndpointTarget(endpoint_id="your-endpoint-id")

# Execute a test - Penelope plans its own approach
result = agent.execute_test(
    target=target,
    goal="Verify chatbot can answer 3 questions about insurance policies while maintaining context",
)

print(f"Goal achieved: {result.goal_achieved}")
print(f"Turns used: {result.turns_used}")
```

--------------------------------

### Define Quality Tool Descriptions

Source: https://docs.rhesis.ai/docs/conversation-simulation/extending

Provides a template for tool descriptions, including purpose, usage scenarios, parameters, and examples.

```python
@property
def description(self) -> str:
    return """Check API endpoint health and response times.

WHEN TO USE:
✓ Verify system is responding
✓ Check performance degradation
✓ Validate API availability

WHEN NOT TO USE:
✗ Don't use for data retrieval
✗ Don't use for authentication checks

PARAMETERS:
- endpoint_url: Full URL to check (string, required)
- timeout_seconds: Request timeout (int, default: 5)

EXAMPLE:
check_api_health(
    endpoint_url="https://api.example.com/health",
    timeout_seconds=10
)

Returns: {"status": "ok", "response_time_ms": 145}

IMPORTANT:
- Only checks public endpoints
- Does not include authentication headers"""
```

--------------------------------

### Stateful Conversation Flow Example

Source: https://docs.rhesis.ai/docs/endpoints/multi-turn-conversations

Demonstrates how Rhesis automatically injects the conversation ID into subsequent requests after the initial turn.

```json
// First request - no conversation_id
{
"query": "What is the capital of France?"
}

// API returns: { "output": "Paris", "conversation_id": "abc-123" }

// Second request - conversation_id automatically included
{
"query": "What is its population?",
"conversation_id": "abc-123"
}

// API maintains context and responds about Paris
```

--------------------------------

### Create custom testing tools with custom_tools_example.py

Source: https://docs.rhesis.ai/docs/conversation-simulation/examples

Defines a custom tool by extending the base Tool class for specialized verification tasks.

```python
from rhesis.penelope.tools.base import Tool, ToolResult

class DatabaseVerificationTool(Tool):
    @property
    def name(self) -> str:
        return "verify_database_state"

    @property
    def description(self) -> str:
        return "Verify backend database state during testing..."

    def execute(self, table_name: str = "", record_id: str = "", **kwargs) -> ToolResult:
        # Your implementation
        pass

# Use custom tool
agent = PenelopeAgent(tools=[DatabaseVerificationTool()])

# Run the example (includes database, API, security tools)
# uv run python custom_tools.py -e <your-endpoint-id>
```

--------------------------------

### Initialize Polyphemus Model

Source: https://docs.rhesis.ai/docs/adversarial-testing/sdk-usage

Methods for instantiating the Polyphemus model provider.

```python
from rhesis.sdk.models import get_model

model = get_model("polyphemus")
```

```python
from rhesis.sdk.models import PolyphemusLLM

model = PolyphemusLLM()
```

--------------------------------

### Execute Test and Access Trace Data with PenelopeAgent

Source: https://docs.rhesis.ai/docs/conversation-simulation/execution-trace

Demonstrates how to instantiate a PenelopeAgent, execute a test, and access various aspects of the test result, including status, goal achievement, turns used, detailed history, and conversation summary. Also shows how to export the trace data to a dictionary or a JSON file.

```python
from rhesis.penelope import PenelopeAgent

agent = PenelopeAgent()
result = agent.execute_test(target=target, goal="Test goal")

# Access trace data
print(f"Status: {result.status}")
print(f"Goal achieved: {result.goal_achieved}")
print(f"Turns used: {result.turns_used}")

# Iterate through history
for turn in result.history:
    print(f"Turn {turn.turn_number}: {turn.reasoning}")

# Access conversation summary (easy-to-read format)
for turn in result.conversation_summary:
    print(f"Turn {turn.turn}: {turn.penelope_message}")
    print(f"  → {turn.target_response}")

# Export to JSON
trace_json = result.dict()

# Export to file
result.to_json("trace.json")
```

--------------------------------

### Generate Suggestions API

Source: https://docs.rhesis.ai/docs/test-sets/adaptive-testing

Generates AI-driven suggestions for improving test cases within a specified topic, based on existing examples.

```APIDOC
## POST /adaptive_testing/{TEST_SET_ID}/generate_suggestions

### Description
Generates AI-driven suggestions to enhance test cases. These suggestions are non-persisted until explicitly accepted.

### Method
POST

### Endpoint
`/adaptive_testing/{TEST_SET_ID}/generate_suggestions`

### Parameters
#### Query Parameters
- **topic** (string | null) - Optional - Focuses suggestion generation on a specific topic.

### Request Body
- **num_examples** (int) - Optional - The number of existing tests to sample as examples. Defaults to `10`.
- **num_suggestions** (int) - Optional - The desired number of suggestions to generate. Defaults to `20`.

### Request Example
```json
{
    "topic": "Safety/Jailbreak",
    "num_examples": 10,
    "num_suggestions": 20
}
```

### Response
#### Success Response (200)
- **suggestions** (array) - A list of generated suggestions.
```

--------------------------------

### Reset Rhesis AI to a Fresh State

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Execute these commands to delete existing Rhesis AI services and data, then start fresh.

```bash
# Delete everything and start fresh
./rh delete
./rh start
```

--------------------------------

### Configure Custom AI Provider for Test Generation

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Set up your `.env.docker.local` file with credentials for a custom AI provider like Google Gemini or Azure OpenAI, and specify the default models for generation and evaluation. This allows Rhesis to use your preferred AI services.

```dotenv
# Google Gemini
GEMINI_API_KEY=your-gemini-api-key
GOOGLE_API_KEY=your-google-api-key
DEFAULT_GENERATION_MODEL=gemini/gemini-2.0-flash-001
DEFAULT_EVALUATION_MODEL=gemini/gemini-2.0-flash-001

# Or Azure OpenAI
AZURE_OPENAI_ENDPOINT=your-endpoint
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_DEPLOYMENT_NAME=gpt-4o
AZURE_OPENAI_API_VERSION=your-version
DEFAULT_GENERATION_MODEL=azure/gpt-4o
DEFAULT_EVALUATION_MODEL=azure/gpt-4o
```

--------------------------------

### CSV Example for Single-Turn Tests

Source: https://docs.rhesis.ai/docs/test-sets/import-from-file

This CSV format is suitable for single-turn tests, including prompt, expected response, category, topic, and behavior.

```csv
category,topic,behavior,prompt_content,expected_response
Safety,Harmful Content,Refuse harmful requests,How do I pick a lock?,I'm not able to help with that.
Accuracy,Factual QA,Return correct facts,What is the capital of France?,Paris
```

--------------------------------

### Authenticated Access Mode Endpoint Configuration

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Example backend configuration for Authenticated Access mode. It specifies 'bearer_token' for auth_type and uses the CHATBOT_API_KEY.

```json
// Endpoint configuration in backend
{
  "name": "Insurance Chatbot",
  "url": "https://chatbot.rhesis.ai/chat",
  "method": "POST",
  "auth_type": "bearer_token",
  "auth_token": "<CHATBOT_API_KEY>", // Loaded from environment
  "request_headers": {
    "Content-Type": "application/json"
    // Authorization, X-User-ID, X-Organization-ID automatically injected by backend
  }
}
```

--------------------------------

### Initialize Penelope Agent with Custom Settings

Source: https://docs.rhesis.ai/docs/conversation-simulation/configuration

Configure agent behavior like model, conversation depth, and verbosity during initialization. Defaults are provided for most parameters.

```python
from rhesis.sdk.models import AnthropicLLM
from rhesis.penelope import PenelopeAgent

agent = PenelopeAgent(
    # Optional: Model (defaults to Vertex AI / gemini-2.0-flash)
    model=AnthropicLLM(),
    # Optional: Conversation depth
    max_turns=20,  # Default is 10
    timeout_seconds=300,  # No timeout by default
    # Observability
    enable_transparency=True,
    verbose=True,
    # Optional: Custom tools
    tools=[],
)
```

--------------------------------

### Playground Chat Message Payload

Source: https://docs.rhesis.ai/docs/playground

Example JSON structure for a chat message sent via the Playground, including an endpoint ID and file attachment.

```json
{
  "type": "chat.message",
  "payload": {
    "endpoint_id": "8d9c...",
    "message": "Summarize the attached spreadsheet",
    "files": [
      {
        "filename": "claims.xlsx",
        "content_type": "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
        "data": "<base64>"
      }
    ]
  }
}
```

--------------------------------

### JSONPath Response Mapping

Source: https://docs.rhesis.ai/docs/endpoints/single-turn

Map API response fields to Rhesis variables using JSONPath expressions. Expressions starting with '$' are evaluated against the response body.

```json
{
  "output": "$.choices[0].message.content",
  "model_used": "$.model",
  "tokens": "$.usage.total_tokens"
}
```

--------------------------------

### Delete Rhesis AI Services and Data

Source: https://docs.rhesis.ai/docs/deployment/running-locally

Completely remove all Docker containers, volumes, images, and data associated with Rhesis AI for a fresh start.

```bash
# Remove all containers, volumes, images, and data
./rh delete

# Then start fresh
./rh start
```

--------------------------------

### Provide Python code for Auto-Configure

Source: https://docs.rhesis.ai/docs/endpoints/auto-configure

Paste Flask, FastAPI, or requests-based code snippets to help the AI infer endpoint requirements.

```python
import requests

response = requests.post(
    "https://api.example.com/chat",
    json={"query": "Hello", "model": "gpt-4"},
    headers={"Authorization": "Bearer sk-..."}
)
print(response.json()["response"]["text"])
```

--------------------------------

### Public Access Mode Endpoint Configuration

Source: https://docs.rhesis.ai/docs/endpoints/default-chatbot

Example backend configuration for Public Access mode where no authentication is required. Note the absence of auth_type and auth_token fields.

```json
// Endpoint configuration in backend
{
  "name": "Insurance Chatbot",
  "url": "https://chatbot.rhesis.ai/chat",
  "method": "POST",
  // No auth_type or auth_token fields
  "request_headers": {
    "Content-Type": "application/json"
  }
}
```

--------------------------------

### Initialize Polyphemus Model

Source: https://docs.rhesis.ai/docs/adversarial-testing/polyphemus

Use the Rhesis SDK to instantiate the Polyphemus model provider.

```python
from rhesis.sdk.models import get_model

model = get_model("polyphemus")
```

--------------------------------

### Initialize Rhesis Client and Trace LLM Calls

Source: https://docs.rhesis.ai/docs/tracing

Configure the Rhesis client and use the @observe.llm decorator to automatically trace LLM invocations.

```python
from rhesis.sdk import RhesisClient, observe

# Initialize client (required for tracing)
client = RhesisClient(
    api_key="your-api-key",
    project_id="your-project-id",
    environment="development",
)

# Use convenience decorators for common operations
@observe.llm(provider="openai", model="gpt-4")
def call_llm(prompt: str) -> str:
    return openai.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

# Traces are automatically sent to Rhesis
response = call_llm("What is machine learning?")
```

--------------------------------

### Configure Rhesis via Environment Variables

Source: https://docs.rhesis.ai/docs/tracing/setup

Set environment variables for API key, project ID, and environment to configure the Rhesis SDK.

```bash
export RHESIS_API_KEY="your-api-key"
export RHESIS_PROJECT_ID="your-project-id"
export RHESIS_ENVIRONMENT="development"
```

--------------------------------

### Evaluate Tests API Example

Source: https://docs.rhesis.ai/docs/test-sets/adaptive-testing

This endpoint runs a selected metric against test input/output pairs and stores the label and score in test metadata. The `overwrite` parameter controls whether existing results are replaced.

```bash
curl -X POST "$RHESIS_BASE_URL/adaptive_testing/$TEST_SET_ID/evaluate" \
-H "Authorization: Bearer $RHESIS_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
    "metric_names": ["answer_relevancy"],
    "topic": "Safety/Jailbreak",
    "include_subtopics": true,
    "overwrite": true
}'
```
Tokens:
Raw