Galtea

https://docs.galtea.ai
Admin
Galtea is a comprehensive AI evaluation platform that empowers enterprises to improve AI...
Tokens:100,340
Snippets:824
Trust Score:5.7
Update:9 months ago
Show doc for...
Tokens:
Raw
### Install Galtea Python SDK

Source: https://docs.galtea.ai/quickstart

Installs the Galtea Python SDK using pip. This is the first step to programmatically interact with the Galtea platform.

```shell
pip install galtea
```

--------------------------------

### Construct Metric Input Format using Python

Source: https://docs.galtea.ai/quickstart

This example illustrates the recommended dictionary format for specifying metrics in Galtea evaluations. It highlights the flexibility of passing pre-computed scores.

```python
# Recommended format:
metrics=[{"name": "metric_name"}]

# Legacy format (will be deprecated):
# metrics=["metric_name"]

# With pre-computed score:
# metrics=[{"name": "metric_name", "score": 0.85}]
```

--------------------------------

### Connect to Galtea Platform with Python SDK

Source: https://docs.galtea.ai/quickstart

Initializes the Galtea client using an API key and lists available products. This code snippet demonstrates how to establish a connection to the Galtea platform and retrieve a list of your products.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

products = galtea.products.list() 
print(f"Found {len(products)} products.")
# Choose a product ID for the next steps
YOUR_PRODUCT_ID = products[0].id if products else "your_product_id"
```

--------------------------------

### Create a Product Version with Python SDK

Source: https://docs.galtea.ai/quickstart

Creates a new version for a product using the Galtea Python SDK. This allows for tracking and comparing different implementations of your AI product over time.

```python
version = galtea.versions.create(
    name="v0.1-quickstart",
    product_id=YOUR_PRODUCT_ID,
    description="Initial version for quickstart evaluation"
)
print(f"Created Version with ID: {version.id}")
```

--------------------------------

### Install Galtea Python SDK

Source: https://docs.galtea.ai/sdk/introduction

Installs the Galtea Python SDK using pip. Ensure you have Python 3.9+ and pip installed. This command fetches and installs the latest stable version of the SDK from PyPI.

```bash
pip install galtea

```

--------------------------------

### Verify Galtea SDK Installation

Source: https://docs.galtea.ai/sdk/installation

Verifies that the Galtea Python SDK has been installed correctly by importing the Galtea class and initializing it. Replace 'YOUR_API_KEY' with your actual API key. This code snippet requires the SDK to be installed.

```python
from galtea import Galtea

# Verify the SDK is properly installed by initializing it
# Note: Replace "YOUR_API_KEY" with your actual API key from the Galtea dashboard
galtea = Galtea(api_key="YOUR_API_KEY")
print("Galtea SDK installed successfully!")
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://docs.galtea.ai/sdk/introduction

Creates a Python virtual environment named 'venv' and activates it. This is recommended to isolate project dependencies. The activation command differs between Unix-based systems and Windows.

```bash
python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`

```

--------------------------------

### Retrieve a Test and its Test Cases using Python

Source: https://docs.galtea.ai/quickstart

This snippet demonstrates how to fetch a specific test by its name and then list all associated test cases using the Galtea Python SDK. It's useful for preparing data for evaluation.

```python
test = galtea.tests.get_by_name(product_id=YOUR_PRODUCT_ID, test_name="Jailbreak")
test_cases = galtea.test_cases.list(test_id=test.id)
print(f"Using test '{test.name}' with {len(test_cases)} test cases.")
```

--------------------------------

### Run an Evaluation with Python

Source: https://docs.galtea.ai/quickstart

This Python snippet defines a placeholder function for your AI model's inference and then iterates through test cases to create single-turn evaluations. It uses the Galtea SDK to submit the evaluation data, including the actual output and specified metrics.

```python
# Placeholder for your actual product/model inference function
def your_product_function(input_prompt):
    if "ignore" in input_prompt.lower():
        return "I am programmed to follow safety guidelines and cannot fulfill this request."
    return f"Of course! I will now {input_prompt}"

# An evaluation is created implicitly with the first evaluation.
# Loop through test cases and create evaluations.
for test_case in test_cases:
    actual_output = your_product_function(test_case.input)
    
    galtea.evaluations.create_single_turn(
        version_id=version.id,
        test_case_id=test_case.id,
        metrics=[{"name": metric.name}],
        actual_output=actual_output
    )

print(f"Submitted evaluations for version {version.id} using test '{test.name}'.")
```

--------------------------------

### Create and Activate Virtual Environment

Source: https://docs.galtea.ai/sdk/installation

Creates a Python virtual environment named 'venv' and activates it. This is recommended to isolate project dependencies. The activation command differs between Unix-based systems and Windows.

```bash
python -m venv venv
source venv/bin/activate  # On Windows use `venv\Scripts\activate`
```

--------------------------------

### Basic Pagination Example for Test Cases

Source: https://docs.galtea.ai/sdk/usage

This example demonstrates how to fetch all test cases for a specific test using pagination. It iteratively requests batches of test cases, extending a list until no more results are returned, ensuring complete data retrieval.

```python
# Get all test cases for a test using pagination
test_id = "your_test_id"
all_test_cases = []
offset = 0
limit = 100  # Fetch 100 at a time for better performance

while True:
    batch = galtea.test_cases.list(
        test_id=test_id,
        offset=offset,
        limit=limit
    )
    
    if not batch:
        break
        
    all_test_cases.extend(batch)
    
    # If we got fewer results than the limit, we've reached the end
    if len(batch) < limit:
        break
        
    offset += limit

print(f"Retrieved {len(all_test_cases)} test cases total")
```

--------------------------------

### Example Few-Shot Examples for Quality Tests

Source: https://docs.galtea.ai/sdk/api/test/create

Provides an example of how to format few-shot examples for 'QUALITY' type tests. These examples help the system understand the desired format and style for generated test cases, offering context for question-answer pairs.

```text
Q: What is the capital of France?
A: The capital of France is Paris.
Q: What is the capital of Germany?
A: The capital of Germany is Berlin.
```

--------------------------------

### Galtea SDK Test Service Quick Example (Python)

Source: https://docs.galtea.ai/sdk/api/test/service

Demonstrates creating, listing, and retrieving tests using the Galtea SDK's Test Service in Python. It requires the `galtea` library and an API key. The example shows how to instantiate the Galtea client, create a new test with specified parameters, list all tests for a product, and retrieve a specific test by its name.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

# Create a test
test = galtea.tests.create(
    name="my-quality-test",
    type="QUALITY",
    product_id="YOUR_PRODUCT_ID",
    ground_truth_file_path="path/to/knowledge.pdf",
    max_test_cases=50
)

print(f"Created test: {test.name} (ID: {test.id})")

# List all tests for a product
tests = galtea.tests.list(product_id="YOUR_PRODUCT_ID")
print(f"Total tests: {len(tests)}")

# Retrieve test by name
retrieved_test = galtea.tests.get_by_name(
    product_id="YOUR_PRODUCT_ID",
    test_name="my-quality-test"
)
print(f"Found test: {retrieved_test.name}")
```

--------------------------------

### Formatted conversation_turns Example

Source: https://docs.galtea.ai/concepts/metric

This example shows the formatted string that the {conversation_turns} placeholder will be replaced with when a SCENARIOS test is evaluated. It represents a multi-turn dialogue between a user and an assistant, suitable for consistency checks.

```text
user: Hello, what can you do?
assistant: I can help with your queries and provide information.
user: What's your return policy?
assistant: Returns are accepted within 30 days of purchase.
user: Can I return items after a month?
assistant: No, our return window is 30 days.
```

--------------------------------

### Quick Example: Manual Trace Collection with @trace Decorator

Source: https://docs.galtea.ai/sdk/api/trace/service

Demonstrates manual trace collection using the `galtea` object, including starting a collection context, defining traced functions with the `@trace` decorator, executing these functions, and saving the collected traces.

```python
from galtea import Galtea, trace, NodeType

galtea = Galtea(api_key="YOUR_API_KEY")

# Manual trace collection
galtea.traces.start_collection_context()

# Your traced functions (using @trace decorator)
@trace(name="search", node_type=NodeType.RETRIEVER)
def search_docs(query: str):
    return [{"doc": "relevant content"}]

@trace(name="generate", node_type=NodeType.LLM)
def generate(context):
    return "Generated response"

# Execute your traced operations
context = search_docs("What is pricing?")
response = generate(context)

# Save all collected traces
galtea.traces.save_context(
    session_id="YOUR_SESSION_ID",
    inference_result_id="YOUR_INFERENCE_RESULT_ID"
)

print("Traces saved successfully!")
# Context is automatically cleared after save
```

--------------------------------

### List Products using Galtea SDK (Python)

Source: https://docs.galtea.ai/sdk/api/galtea

Demonstrates how to initialize the Galtea SDK and then use the `products.list()` method to retrieve a list of products. This example assumes the SDK has already been initialized with a valid API key.

```python
from galtea import Galtea

# Initialize Galtea SDK
galtea = Galtea(api_key="YOUR_API_KEY", suppress_updatable_version_message=False)

# List products
products = galtea.products.list()
print(products)
```

--------------------------------

### Pagination Across Different Resources

Source: https://docs.galtea.ai/sdk/usage

This example shows the consistent application of pagination parameters (`offset` and `limit`) across various Galtea SDK list operations, including products, versions, sessions, and evaluations.

```python
# Paginate through products
products = galtea.products.list(offset=0, limit=50)

# Paginate through versions
versions = galtea.versions.list(product_id="YOUR_PRODUCT_ID", offset=0, limit=50)

# Paginate through sessions
sessions = galtea.sessions.list(version_id="YOUR_VERSION_ID", offset=0, limit=50)

# Paginate through evaluations
evaluations = galtea.evaluations.list(session_id="YOUR_SESSION_ID", offset=0, limit=50)
```

--------------------------------

### Construct Result URL using Python

Source: https://docs.galtea.ai/quickstart

This Python snippet generates a direct URL to view evaluation results on the Galtea platform. It includes placeholders for the product ID, allowing users to easily navigate to their specific analytics dashboard.

```python
print(f"View results at: https://platform.galtea.ai/product/{YOUR_PRODUCT_ID}?tab=1")
```

--------------------------------

### Upgrade Galtea Python SDK

Source: https://docs.galtea.ai/sdk/installation

Upgrades the Galtea Python SDK to the latest available version using pip. This command ensures you have the most recent features and bug fixes.

```bash
pip install --upgrade galtea
```

--------------------------------

### New Session-Based Workflow Example (Python)

Source: https://docs.galtea.ai/migration-to-v3

Provides an example of the new session-based workflow in Galtea SDK v3.0. This workflow is designed for logging and evaluating multi-turn conversations. It involves creating a session, logging inference results for each turn in a batch, and then running evaluations on the entire session.

```python
# SDK v3.0 New Workflow Example
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

YOUR_PRODUCT_ID = "your_product_id"
YOUR_VERSION_ID = "your_version_id"
YOUR_TEST_CASE_ID = "your_test_case_id"
YOUR_METRIC_NAME = "your_metric_name"

# 1. Create a Session to group a conversation
session = galtea.sessions.create(
    version_id=YOUR_VERSION_ID,
    test_case_id=YOUR_TEST_CASE_ID,
)
print(f"Created Session: {session.id}")

# 2. Log all inference results in a single efficient batch call
conversation_turns = [
    {
        "input": "What's your return policy?",
        "output": "Our return policy allows returns within 30 days of purchase."
    },
    {
        "input": "What if I lost the receipt?",
        "output": "A proof of purchase is required for all returns."
    },
    {
        "input": "Can I return items bought online to a physical store?",
        "output": "Yes, you can return online purchases to any of our physical store locations."
    },
    {
        "input": "What about exchanges?",
        "output": "Exchanges follow the same 30-day policy and can be done in-store or online."
    }
]

```

--------------------------------

### Get or Create Session using Python

Source: https://docs.galtea.ai/sdk/api/session/get-or-create

This Python snippet demonstrates how to use the `get_or_create` function to fetch an existing session by its `custom_id` or create a new one if it doesn't exist. It requires `version_id` and optionally accepts `test_case_id`, `context`, and `is_production` parameters.

```python
session = galtea.sessions.get_or_create(
    custom_id="existing-session-id",
    version_id="YOUR_VERSION_ID",
    test_case_id="YOUR_TEST_CASE_ID"
)
```

--------------------------------

### Start Collection Context

Source: https://docs.galtea.ai/sdk/api/trace/start-collection-context

Initializes trace collection for the current context. This is automatically called when using `galtea.inference_results.generate()`, but can be manually called for manual trace collection.

```APIDOC
## POST /traces/start_collection_context

### Description
Initializes trace collection for the current context. This is automatically called when using `galtea.inference_results.generate()`, but can be manually called for manual trace collection.

### Method
POST

### Endpoint
/traces/start_collection_context

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
```python
# No request body needed
pass
```

### Response
#### Success Response (200)
- **None** (NoneType) - This method returns None upon successful execution.

#### Response Example
```json
null
```

### Notes
If you're using `galtea.inference_results.generate()`, this is called automatically. You only need to call this method when using manual trace collection.
```

--------------------------------

### Install Galtea and Python Dependencies

Source: https://docs.galtea.ai/sdk/integrations/github-actions

Specifies the necessary Python libraries to be installed for the project, including the 'galtea' library for AI evaluation and 'python-dotenv' for managing environment variables. This file is crucial for setting up the project's dependencies.

```txt
galtea
python-dotenv
```

--------------------------------

### Example Strategies for Scenario Tests

Source: https://docs.galtea.ai/sdk/api/test/create

Presents the 'strategies' parameter for 'SCENARIOS' tests, specifying conversation styles like 'written' or 'spoken'. This influences the tone and formality of generated dialogues, with 'written' being the default.

```json
["written", "spoken"]
```

--------------------------------

### Python: New Session-Based Workflow Example for Galtea AI SDK

Source: https://docs.galtea.ai/migration-to-v2

This Python code demonstrates the new session-based workflow for the Galtea AI SDK v2.0. It shows how to create a session, log multiple inference results in a batch, and then create evaluations for the entire session. It requires the 'galtea' library and an API key.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

YOUR_PRODUCT_ID = "your_product_id"
YOUR_VERSION_ID = "your_version_id"
YOUR_TEST_CASE_ID = "your_test_case_id"
YOUR_METRIC_NAME = "your_metric_name"

# 1. Create a Session to group a conversation
session = galtea.sessions.create(
    version_id=YOUR_VERSION_ID,
    test_case_id=YOUR_TEST_CASE_ID,
)
print(f"Created Session: {session.id}")

# 2. Log all inference results in a single efficient batch call
conversation_turns = [
    {
        "input": "What's your return policy?",
        "output": "Our return policy allows returns within 30 days of purchase."
    },
    {
        "input": "What if I lost the receipt?",
        "output": "A proof of purchase is required for all returns."
    },
    {
        "input": "Can I return items bought online to a physical store?",
        "output": "Yes, you can return online purchases to any of our physical store locations."
    },
    {
        "input": "What about exchanges?",
        "output": "Exchanges follow the same 30-day policy and can be done in-store or online."
    }
]

# Use create_batch for efficient bulk processing
galtea.inference_results.create_batch(
    session_id=session.id,
    conversation_turns=conversation_turns
)

print(f"Stored {len(conversation_turns)} inference results in session {session.id} with a single call")

# 3. When ready, create evaluations for the entire session
evaluations = galtea.evaluations.create(
    session_id=session.id,
    metrics=[YOUR_METRIC_NAME]
)
print(f"Submitted {len(evaluations)} evaluations for session {session.id}")
```

--------------------------------

### Example Strategies for Red Teaming Tests

Source: https://docs.galtea.ai/sdk/api/test/create

Illustrates the 'strategies' parameter for 'RED_TEAMING' tests, including the mandatory 'original' strategy and potentially others for prompt modification. These strategies are techniques applied to generated prompts for each threat category.

```json
["original", "obfuscation_technique_1", "obfuscation_technique_2"]
```

--------------------------------

### Create Single-Turn Evaluation for Production Monitoring (Python)

Source: https://docs.galtea.ai/sdk/api/evaluation/create-single-turn

Demonstrates creating a single-turn evaluation for production monitoring. This example uses the recommended MetricInput dictionary format and includes optional parameters like `is_production`, `input`, `retrieval_context`.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

# Using the recommended MetricInput dictionary format
evaluations = galtea.evaluations.create_single_turn(
    version_id="YOUR_VERSION_ID",
    is_production=True,
    input="A real user's query from production",
    actual_output="The model's live response.",
    metrics=[
        {"name": "Contextual Relevancy"}, 
        {"name": "Non-Toxic"}
    ],
    retrieval_context="Context retrieved by RAG for this query."
)
```

--------------------------------

### Manage Sessions using Galtea SDK (Python)

Source: https://docs.galtea.ai/sdk/api/session

This snippet demonstrates how to use the Galtea SDK's Session Service to manage evaluation sessions. It covers creating test and production sessions, using the idempotent `get_or_create` method, and listing existing sessions for a given version. Ensure you have the `galtea` library installed and replace placeholders with your actual API key, version ID, and test case ID.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

# Create a test-based session
session = galtea.sessions.create(
    version_id="YOUR_VERSION_ID",
    test_case_id="YOUR_TEST_CASE_ID",
    is_production=False
)

print(f"Created session: {session.id}")

# Create a production session
prod_session = galtea.sessions.create(
    version_id="YOUR_VERSION_ID",
    custom_id="user-session-123",
    is_production=True
)

# Get or create session (idempotent operation)
session = galtea.sessions.get_or_create(
    version_id="YOUR_VERSION_ID",
    custom_id="user-session-123"
)

# List all sessions for a version
sessions = galtea.sessions.list(version_id="YOUR_VERSION_ID")
print(f"Total sessions: {len(sessions)}")

```

--------------------------------

### Run Evaluations for Test Cases

Source: https://docs.galtea.ai/sdk/usage

This snippet demonstrates how to iterate through test cases, retrieve retrieval context, get the product's actual output, and create single-turn evaluations using Galtea. It supports standard Galtea-hosted metrics and self-hosted metrics with dynamic scoring.

```python
for test_case in test_cases:
    # Retrieve relevant context for RAG. This may not apply to all products.
    retrieval_context = your_retriever_function(test_case.input)
    
    # Your product's actual response to the input
    actual_output = your_product_function(test_case.input, test_case.context, retrieval_context)
    
    # Run evaluation
    galtea.evaluations.create_single_turn(
        version_id=version.id,
        test_case_id=test_case.id,
        metrics=[
            # Standard Galtea-hosted metric, referenced by name
            {"name": metric_from_api.name},
            
            # Self-hosted metric with dynamic scoring
            {"score": keyword_metric} 
        ],
        actual_output=actual_output,
        retrieval_context=retrieval_context, 
    )
```

--------------------------------

### Python Example Judge Prompt for Full Prompt (LLM) Evaluation

Source: https://docs.galtea.ai/concepts/metric

This Python code snippet demonstrates a judge prompt for evaluating LLM outputs using a full prompt approach. It includes placeholders like {input}, {expected_output}, and {actual_output} to dynamically insert relevant data during evaluation, with a detailed scoring rubric ranging from 0 to 1.

```python
judge_prompt = """
You are an expert evaluator. Evaluate the factual accuracy of the given response by comparing it to the reference answer and considering how well it addresses the user's input.

**User Input:** {input}
**Reference Answer:** {expected_output}
**Response to Evaluate:** {actual_output}

Scoring Guidelines:
- Score 0: The response contains factual errors, omits essential information needed for the input, or includes unsupported content.
- Score 0.5: The response is partially accurate. It is generally relevant but includes minor inaccuracies, missing key details, or some unsupported claims.
- Score 1: The response fully aligns with the reference answer with no errors, omissions, or unsupported additions.
"""
```

--------------------------------

### Simulate Conversation with Galtea SDK

Source: https://docs.galtea.ai/concepts/product/test/case/conversation-simulator

This snippet demonstrates how to use the Galtea SDK to programmatically run a conversation simulation. It requires a scenario and a test case to guide the simulation. The output includes the session ID, total turns, message history, and simulation status.

```python
from galtea.simulator import simulate_conversation

# Assuming you have a scenario_id and test_case_id
session_id, result = simulate_conversation(
    scenario_id="your_scenario_id",
    test_case_id="your_test_case_id"
)

print(f"Session ID: {session_id}")
print(f"Simulation Result: {result}")
```

--------------------------------

### Python Decorator Usage for Tracing

Source: https://docs.galtea.ai/concepts/product/version/session/trace

Demonstrates different ways to apply the @trace decorator in Python. It shows how to provide custom names, use default naming, and apply decorators with empty parentheses. These examples are for instrumenting functions to capture execution traces.

```python
@trace(name="custom_name")
def my_function(): ...

# Bare decorator (uses function name)
@trace
def my_function(): ...

# Empty parentheses
@trace()
def my_function(): ...
```

--------------------------------

### Python Best Practice: Limiting Input/Output Data in Traces

Source: https://docs.galtea.ai/concepts/product/version/session/trace

Shows how to manage the amount of data captured by the trace decorator for function arguments and return values. This example limits the input to `doc_id` and returns a dictionary with summary and status, ensuring only relevant debugging information is stored.

```python
@trace(name="process_document", node_type=NodeType.TOOL)
def process(self, doc_id: str) -> dict:
    # Only doc_id is captured as input, not the full document
    doc = self.fetch_document(doc_id)
    return {"summary": doc.summary, "status": "processed"}
```

--------------------------------

### Python Best Practice: Tracing Meaningful Boundaries

Source: https://docs.galtea.ai/concepts/product/version/session/trace

Illustrates the best practice of tracing logical units of work rather than every single function call. This example shows tracing a `search_products` function that encapsulates internal calls to a vector database and result formatting, focusing on the overall operation.

```python
# ✅ Good - meaningful operation
@trace(name="search_products", node_type=NodeType.RETRIEVER)
def search_products(self, query):
    results = self._query_vector_db(query)  # Internal, not traced
    return self._format_results(results)     # Internal, not traced
```

--------------------------------

### Retrieve a Metric by Name using Python

Source: https://docs.galtea.ai/quickstart

This code snippet shows how to retrieve a specific metric object by its name using the Galtea Python SDK. This is a prerequisite for configuring evaluations.

```python
metric = galtea.metrics.get_by_name(name="Jailbreak Resilience")
```

--------------------------------

### Test Case Generation Parameters

Source: https://docs.galtea.ai/sdk/api/test/create

This section details the parameters used for generating test cases, including test name, type, product ID, ground truth file path, few-shot examples, custom user focus, language, variants, strategies, custom variant description, and maximum test cases.

```APIDOC
## POST /api/tests (Example Endpoint)

### Description
This endpoint allows for the generation of test cases based on specified parameters. It supports various test types like QUALITY, RED_TEAMING, and SCENARIOS, with options for customizing the generation process.

### Method
POST

### Endpoint
/api/tests

### Parameters
#### Request Body
- **name** (string) - Required - The name of the test.
- **type** (string) - Required - The type of test. Possible values: `QUALITY`, `RED_TEAMING`, `SCENARIOS`.
- **product_id** (string) - Required - The ID of the product you want to evaluate.
- **ground_truth_file_path** (string) - Optional - Path to a local file containing the knowledge base. Supported formats: `.pdf`, `.txt`, `.json`, `.html`, `.md`, `.yaml`, `.yml`, `.xml`, `.docx`, `.zip`. Maximum file size: 100 MB. Maximum ZIP contents: 1,000 files.
- **few_shot_examples** (string) - Optional - Optional few-shot examples to provide more context for test case generation. Only applies if tests are generated by Galtea and are of type `QUALITY`.
- **custom_user_focus** (string) - Optional - Narrow down the scope of generated scenarios by describing a specific type of user, context, or situation. Only applies if tests are generated by Galtea and are of type `SCENARIOS`.
- **language** (string) - Optional - The language for generating synthetic test cases if `ground_truth_file_path` is provided. Defaults to the language detected in the ground truth file. Should be a valid ISO 639 language name (e.g., "english").
- **variants** (list[string]) - Optional - Specifies how to generate test cases. For `RED_TEAMING` tests, specifies threat categories (e.g., `["data_leakage", "Misuse"]`). For `QUALITY` tests, specifies evolutions (e.g., `["paraphrased", "typos"]`).
- **strategies** (list[string]) - Optional - Specifies how to generate test cases related to its style. For `RED_TEAMING` tests, strategies are techniques for modifying or obfuscating prompts. For `SCENARIOS` tests, strategies define the conversation style (`written`, `spoken`).
- **custom_variant_description** (str) - Optional - Description for guiding synthetic data generation for security tests. Used when `Custom` threat is included in `variants` for `RED_TEAMING` tests.
- **max_test_cases** (int) - Optional - Maximum number of test cases to generate. Default values: 50 for QUALITY/RED_TEAMING, 20 for SCENARIOS. Maximum limits: 1000 for QUALITY/RED_TEAMING, 100 for SCENARIOS.

### Request Example
```json
{
  "name": "Website Quality Test",
  "type": "QUALITY",
  "product_id": "prod_12345",
  "ground_truth_file_path": "/path/to/knowledge_base.pdf",
  "few_shot_examples": "Q: What is the capital of France?\nA: The capital of France is Paris.",
  "language": "english",
  "variants": ["paraphrased", "typos"],
  "strategies": ["written"],
  "max_test_cases": 100
}
```

### Response
#### Success Response (200)
- **test_id** (string) - The unique identifier for the generated test.
- **status** (string) - The current status of the test generation process.

#### Response Example
```json
{
  "test_id": "test_abcde12345",
  "status": "pending"
}
```
```

--------------------------------

### Create a Product Version using Galtea SDK

Source: https://docs.galtea.ai/sdk/usage

This code demonstrates how to create a new version for a product using the Galtea SDK. A version helps in tracking changes over time and comparing different product implementations. It requires the product ID and basic information like name and description.

```python
# Create a version
version = galtea.versions.create(
    name="v1.0",
    product_id=product.id,
    description="Initial version with basic summarization capabilities"
)
```

--------------------------------

### Import and Initialize Galtea SDK

Source: https://docs.galtea.ai/sdk/usage

This snippet shows how to import the Galtea SDK and initialize the Galtea client with your API key. The API key is essential for authenticating with the Galtea platform and can be obtained from the platform's settings page.

```python
from galtea import Galtea

galtea = Galtea(api_key="<YOUR_GALTEA_PLATFORM_API_KEY>")
```

--------------------------------

### Create Test and Sessions with Galtea Client in Python

Source: https://docs.galtea.ai/sdk/tutorials/simulating-conversations

Initializes the Galtea client and interacts with products and versions to set up a simulation test. It demonstrates creating a new version if it doesn't exist and then creating a scenario-based test suite.

```python
# Initialize the Galtea client
galtea_client = galtea.Galtea(api_key="YOUR_API_KEY")

# Call the product and version you want to evaluate
product = galtea_client.products.get_by_name("Financial advisor agent")
version_name = "v1.0"
try:
    # Get the version if it exists
    print(f'Getting version {version_name}')
    version = galtea_client.versions.get_by_name(
        product_id=product.id,
        version_name=version_name
    )
except:
    # Create the version if it doesn't exist
    print(f'Version doesn\'t exist, creating version {version_name}')
    version = galtea_client.versions.create(
        name=version_name,
        product_id=product.id,
        description="Version created from the tutorial"
    )

# Create a test suite using the scenarios options
test = galtea_client.tests.create(
    product_id=product.id,
    name="Multi-turn Conversation Test",
    type="SCENARIOS",
    max_test_cases=5
)

# Get your test scenarios
test_cases = galtea_client.test_cases.list(test_id=test.id)

```

--------------------------------

### Get All Traces From Context

Source: https://docs.galtea.ai/sdk/api/trace/get-all-from-context

Retrieves all traces collected in the current context. This is useful for inspecting the execution flow and debugging.

```APIDOC
## GET /traces/context

### Description
Retrieve all traces collected in the current context. Returns a list of trace dictionaries.

### Method
GET

### Endpoint
/traces/context

### Parameters
#### Query Parameters
None

#### Request Body
None

### Request Example
```json
{
  "message": "No request body needed for this GET request."
}
```

### Response
#### Success Response (200)
- **traces** (list) - A list of trace dictionaries. Each dictionary contains:
  - **id** (string) - The unique identifier for the trace.
  - **name** (string) - The name of the trace node.
  - **node_type** (string) - The type of the node (e.g., 'llm', 'tool').
  - **input_data** (any) - The input data for the trace node.
  - **output_data** (any) - The output data from the trace node.
  - **error** (string) - Any error message associated with the trace node.
  - **latency_ms** (integer) - The latency in milliseconds for the trace node execution.
  - **start_time** (string) - The start time of the trace node execution (ISO format).
  - **end_time** (string) - The end time of the trace node execution (ISO format).
  - **parent_trace_id** (string) - The ID of the parent trace, if any.

#### Response Example
```json
{
  "traces": [
    {
      "id": "trace_123",
      "name": "llm_call_1",
      "node_type": "llm",
      "input_data": {"prompt": "What is Galtea AI?"},
      "output_data": {"response": "Galtea AI is a platform..."},
      "error": null,
      "latency_ms": 1500,
      "start_time": "2023-10-27T10:00:00Z",
      "end_time": "2023-10-27T10:00:01.500Z",
      "parent_trace_id": null
    },
    {
      "id": "trace_456",
      "name": "tool_use_1",
      "node_type": "tool",
      "input_data": {"query": "Galtea AI funding"},
      "output_data": {"result": "Information about funding..."},
      "error": null,
      "latency_ms": 500,
      "start_time": "2023-10-27T10:00:01.500Z",
      "end_time": "2023-10-27T10:00:02Z",
      "parent_trace_id": "trace_123"
    }
  ]
}
```

### Notes
- If `start_collection_context()` has not been called, this method returns an empty list.
```

--------------------------------

### Get Current ID From Context

Source: https://docs.galtea.ai/sdk/api/trace/get-current-id-from-context

Retrieves the current active trace ID from the context. This is useful for debugging purposes.

```APIDOC
## GET /traces/current_id

### Description
Get the current active trace_id from the context.

### Method
GET

### Endpoint
/traces/current_id

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **trace_id** (string) - The current trace ID, or null if no trace is active.

#### Response Example
{
  "trace_id": "a1b2c3d4-e5f6-7890-1234-567890abcdef"
}
```

--------------------------------

### Get Session By Custom ID

Source: https://docs.galtea.ai/sdk/api/session/get-by-custom-id

Retrieves a specific session using its custom ID and the associated version ID.

```APIDOC
## GET /websites/galtea_ai/sessions/custom_id

### Description
Retrieve a specific session by its Custom ID.

### Method
GET

### Endpoint
/websites/galtea_ai/sessions/custom_id

### Parameters
#### Query Parameters
- **version_id** (string) - Required - The ID of the version associated with the session to retrieve.
- **custom_id** (string) - Required - The Custom ID associated with the session to retrieve.

### Response
#### Success Response (200)
- **session** (object) - A Session object containing details of the retrieved session.

#### Response Example
```json
{
  "session_id": "sess_abc123",
  "custom_id": "YOUR_CUSTOM_ID",
  "version_id": "YOUR_VERSION_ID",
  "created_at": "2023-10-27T10:00:00Z",
  "metadata": {}
}
```
```

--------------------------------

### Manage Product Versions using Galtea SDK (Python)

Source: https://docs.galtea.ai/sdk/api/version/service

This snippet demonstrates how to use the Galtea SDK to interact with the Version Service. It covers creating a new version, listing all versions for a product, and retrieving a specific version by its name. Requires the 'galtea' library and an API key.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

# Create a version
version = galtea.versions.create(
    name="v1.0",
    product_id="YOUR_PRODUCT_ID",
    description="Initial version with basic capabilities",
    model_id="gpt-4"
)

print(f"Created version: {version.name} (ID: {version.id})")

# List all versions for a product
versions = galtea.versions.list(product_id="YOUR_PRODUCT_ID")
print(f"Total versions: {len(versions)}")

for v in versions:
    print(f"- {v.name} created at {v.created_at}")

# Retrieve version by name
version = galtea.versions.get_by_name(
    product_id="YOUR_PRODUCT_ID",
    version_name="v1.0"
)
print(f"Found version: {version.name}")
```

--------------------------------

### Production Data Logging with Single-Turn Evaluations (Python)

Source: https://docs.galtea.ai/migration-to-v3

Demonstrates how to log production data using `create_single_turn()` without a test case. It showcases the new recommended `MetricInput` dictionary format for specifying metrics, including pre-computed scores for self-hosted metrics, and sets `is_production=True` for production monitoring.

```python
# Production data logging (no test case required)
# Using the recommended MetricInput dictionary format
galtea.evaluations.create_single_turn(
    version_id=version.id,
    input="User's actual input",
    actual_output="Model's actual output",
    metrics=[
        {"name": "your_metric"},
        {"name": "custom-metric", "score": 0.95}  # Pre-computed score for self-hosted metric
    ],
    is_production=True
)
```

--------------------------------

### Get Trace by ID

Source: https://docs.galtea.ai/sdk/api/trace/get

Retrieves a single trace object using its unique identifier. This is useful for inspecting the details of a specific trace.

```APIDOC
## GET /websites/galtea_ai/traces/{trace_id}

### Description
Retrieves a single trace by its unique identifier.

### Method
GET

### Endpoint
/websites/galtea_ai/traces/{trace_id}

### Parameters
#### Path Parameters
- **trace_id** (string) - Required - The unique identifier of the trace to retrieve.

### Request Example
```python
trace = galtea.traces.get(trace_id="YOUR_TRACE_ID")
```

### Response
#### Success Response (200)
- **Trace** (object) - A Trace object containing details of the retrieved trace.

#### Response Example
```json
{
  "id": "trace_abc123",
  "timestamp": "2023-10-27T10:00:00Z",
  "session_id": "session_xyz789",
  "user_id": "user_pqr456",
  "data": {
    "query": "What is the capital of France?",
    "response": "The capital of France is Paris."
  }
}
```
```

--------------------------------

### Get Session by ID

Source: https://docs.galtea.ai/sdk/api/session/get

Retrieves a specific session using its unique identifier. This endpoint is useful for fetching detailed information about a past or active session.

```APIDOC
## GET /sessions/{session_id}

### Description
Retrieves a specific session by its ID.

### Method
GET

### Endpoint
/sessions/{session_id}

### Parameters
#### Path Parameters
- **session_id** (string) - Required - The ID of the session to retrieve.

### Request Example
```python
session = galtea.sessions.get(session_id="YOUR_SESSION_ID")
```

### Response
#### Success Response (200)
- **session** (Session object) - The retrieved session object, containing details about the session.

#### Response Example
```json
{
  "session_id": "YOUR_SESSION_ID",
  "created_at": "2023-10-27T10:00:00Z",
  "status": "completed",
  "metadata": {}
}
```
```

--------------------------------

### Manage Products using Galtea SDK (Python)

Source: https://docs.galtea.ai/sdk/api/product/service

Demonstrates how to interact with the Product Service API using the Galtea SDK in Python. This includes listing all products, retrieving a specific product by its ID, and fetching a product by its name. Note that product creation is not supported via the SDK.

```python
from galtea import Galtea

galtea = Galtea(api_key="YOUR_API_KEY")

# List all products
products = galtea.products.list()
print(f"Total products: {len(products)}")

for product in products:
    print(f"- {product.name} (ID: {product.id})")

# Get a specific product by ID
product = galtea.products.get(id="YOUR_PRODUCT_ID")
print(f"Product: {product.name}")
print(f"Description: {product.description}")

# Get product by name
product = galtea.products.get_by_name(name="My Product")
print(f"Found: {product.name}")
```

--------------------------------

### GET /inference_results/{inference_result_id}

Source: https://docs.galtea.ai/sdk/api/inference-result/get

Retrieves a specific inference result log using its unique identifier. This endpoint is useful for accessing detailed logs of past inferences.

```APIDOC
## GET /inference_results/{inference_result_id}

### Description
Retrieve an inference result log by its ID.

### Method
GET

### Endpoint
/inference_results/{inference_result_id}

### Parameters
#### Path Parameters
- **inference_result_id** (string) - Required - The ID of the inference result log to retrieve.

### Request Example
```python
# No request body for GET requests
```

### Response
#### Success Response (200)
- **InferenceResult** (object) - An object containing the inference result details.

#### Response Example
```json
{
  "inference_result_id": "YOUR_INFERENCE_RESULT_ID",
  "status": "completed",
  "output": {
    "prediction": "example_prediction"
  },
  "timestamp": "2023-10-27T10:00:00Z"
}
```
```

--------------------------------

### Create a Test using Galtea SDK

Source: https://docs.galtea.ai/sdk/usage

This snippet illustrates how to create a test for evaluating a product using the Galtea SDK. You can specify the test type (e.g., QUALITY, RED_TEAM), the product ID, a ground truth file path, and the language for the test.

```python
# Create a test
test = galtea.tests.create(
    name="example-test-tutorial",
    type="QUALITY",
    product_id=product.id,
    ground_truth_file_path="path/to/knowledge_file.pdf", # The ground truth is also known as as the knowledge base.
    language='spanish'
)
```

--------------------------------

### Get or Create Session

Source: https://docs.galtea.ai/sdk/api/session/get-or-create

Retrieves an existing session using a custom ID or creates a new one if it does not exist. This is useful for associating Galtea sessions with your application's sessions.

```APIDOC
## POST /sessions/get_or_create

### Description
Retrieves an existing session using a custom ID or creates a new one if it does not exist. This is useful for associating Galtea sessions with your application's sessions.

### Method
POST

### Endpoint
/sessions/get_or_create

### Parameters
#### Request Body
- **custom_id** (string) - Required - The Custom ID of the session to fetch or create. This is useful to associate sessions in the Galtea platform with those in your application.
- **version_id** (string) - Required - The ID of the version to associate with this session if it needs to be created.
- **test_case_id** (string) - Optional - The test case ID. Required for non-production sessions.
- **context** (dict) - Optional - A flexible key-value context.
- **is_production** (boolean) - Optional - A flag to indicate if the session is from a production environment. Defaults to `False`.

### Request Example
```json
{
  "custom_id": "existing-session-id",
  "version_id": "YOUR_VERSION_ID",
  "test_case_id": "YOUR_TEST_CASE_ID",
  "context": {},
  "is_production": false
}
```

### Response
#### Success Response (200)
- **session** (object) - A Session object containing details of the retrieved or created session.

#### Response Example
```json
{
  "session": {
    "id": "session-id-123",
    "custom_id": "existing-session-id",
    "version_id": "YOUR_VERSION_ID",
    "test_case_id": "YOUR_TEST_CASE_ID",
    "context": {},
    "is_production": false,
    "created_at": "2023-10-27T10:00:00Z",
    "updated_at": "2023-10-27T10:00:00Z"
  }
}
```
```

--------------------------------

### Understanding Pagination Parameters

Source: https://docs.galtea.ai/sdk/usage

This snippet illustrates how to use the `offset` and `limit` parameters for pagination in the Galtea SDK. It shows fetching the first 10 items, the next 10, and the default maximum of 1000 items.

```python
# Get first 10 products
first_page = galtea.products.list(offset=0, limit=10)

# Get next 10 products
second_page = galtea.products.list(offset=10, limit=10)

# Get default number of products (up to 1000)
products = galtea.products.list()  # Returns up to 1000 items
```

--------------------------------

### Get Inference Result by ID (Python)

Source: https://docs.galtea.ai/sdk/api/inference-result/get

Retrieves an inference result log using its unique identifier. This function requires the inference result ID as a string parameter and returns an InferenceResult object.

```python
inference_result = galtea.inference_results.get(inference_result_id="YOUR_INFERENCE_RESULT_ID")
```

--------------------------------

### Manually Create Test Cases with Galtea AI SDK

Source: https://docs.galtea.ai/sdk/api/test/create

Demonstrates how to manually create test cases using the `test_cases.create()` function in the Galtea AI SDK. This is useful when needing more control or when splitting knowledge bases.

```python
from galtea import test_cases

# Example of manually creating a test case
test_cases.create(input="User query", expected_output="Expected response", tag="example_tag")
```

--------------------------------

### Start Trace Collection Context (Python)

Source: https://docs.galtea.ai/sdk/api/trace/start-collection-context

Initializes trace collection for the current context. This method is automatically called when using `galtea.inference_results.generate()`, but should be manually invoked for manual trace collection. It takes no parameters and returns `None`.

```python
galtea.traces.start_collection_context()
```

--------------------------------

### Python: Production Data Logging with Single-Turn Evaluations (SDK v2.0)

Source: https://docs.galtea.ai/migration-to-v2

Shows how to log production data using the `create_single_turn` method in SDK v2.0 without requiring a predefined test case. This is useful for monitoring real user interactions.

```python
# Production data logging (no test case required)
galtea.evaluations.create_single_turn(
    version_id=version.id,
    input="User's actual input",
    actual_output="Model's actual output",
    metrics=["your_metric"],
    is_production=True
)
```

--------------------------------

### Python: Create Version and Single-Turn Evaluation (SDK v2.0)

Source: https://docs.galtea.ai/migration-to-v2

Demonstrates creating a new version using direct keyword arguments and performing a single-turn evaluation with the `create_single_turn` method in SDK v2.0. This replaces the older explicit evaluation creation and `evaluations.create()` method.

```python
from galtea import Galtea
galtea = Galtea(api_key="...")

# 1. Create version with direct keyword arguments
version = galtea.versions.create(
    name="v2.0-summarizer",
    product_id="YOUR_PRODUCT_ID",
    description="New version using gpt-4",
    model_id="gpt-4"
)

# 2. Skip explicit evaluation creation (it's automatic!)

# 3. Create evaluations using version_id
galtea.evaluations.create_single_turn(
    version_id=version.id,
    test_case_id="YOUR_TEST_CASE_ID",
    metrics=["your_metric"],
    actual_output="This is the model's output."
)
```

--------------------------------

### Get Current Trace ID from Context (Python)

Source: https://docs.galtea.ai/sdk/api/trace/get-current-id-from-context

Retrieves the current active trace ID from the context. Returns the trace ID as a string or None if no trace is active. This is an advanced method primarily for debugging purposes.

```python
current_trace_id = galtea.traces.get_current_id_from_context()
```

--------------------------------

### List Products (Python)

Source: https://docs.galtea.ai/sdk/api/product/list

This Python code snippet demonstrates how to list products using the Galtea AI API. It utilizes the `galtea.products.list()` method to fetch product data. No external dependencies are required beyond the Galtea library.

```python
products = galtea.products.list()
```
Tokens:
Raw