### Create Example Instance

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Factory method to create an `Example` instance with arbitrary key-value pairs. Useful for setting up examples for Q&A, RAG, or including custom fields.

```python
from judgeval.data import Example

# For Q&A evaluation
example = Example.create(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris",
)
```

```python
from judgeval.data import Example

# For RAG evaluation
example = Example.create(
    input="What is Python?",
    actual_output="A programming language.",
    retrieval_context=["Python is a high-level programming language..."],
)
```

```python
from judgeval.data import Example

# With custom fields
example = Example.create(
    input="user question",
    actual_output="assistant response",
    expected_output="correct response",
    context="some context",
    custom_field="custom value",
)
```

--------------------------------

### Add Examples from YAML File

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Use this method to upload examples from a YAML file. The YAML file should contain a list of dictionaries, each representing an example with 'input', 'expected_output', and optional 'context' keys. The `batch_size` parameter determines how many examples are sent in each API request.

```python
def add_from_yaml(
    self,
    file_path: str,
    batch_size: int = 100,
) -> None:
    """Upload examples from a YAML file."""
    pass
```

```yaml
- input: What is AI?
  expected_output: Artificial Intelligence
  context: AI basics
- input: What is ML?
  expected_output: Machine Learning
```

```python
dataset.add_from_yaml("./data/golden-set.yaml")
```

--------------------------------

### Example.__getitem__()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Access example properties using bracket notation. This method allows retrieving specific properties of an example by their key.

```APIDOC
## `Example.__getitem__()`

Access example properties using bracket notation.

### Method
`__getitem__(self, key: str) -> Any`

### Parameters
#### Path Parameters
- **key** (str) - Required - Property name.

### Returns
The property value.

### Raises
- `KeyError` if the key does not exist.

### Example
```python
example = Example.create(input="test", output="result")
print(example["input"])
print(example["output"])
```
```

--------------------------------

### Example.create()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Creates a new `Example` instance with arbitrary key-value properties. This is useful for defining evaluation examples with various inputs, outputs, and contextual information.

```APIDOC
## Example.create()

### Description
Creates a new `Example` instance with arbitrary key-value properties. This is useful for defining evaluation examples with various inputs, outputs, and contextual information.

### Method
```python
@classmethod
def create(cls, **kwargs: Any) -> Example
```

### Parameters
#### Keyword Arguments
- **kwargs** (Any) - Any key-value pairs. Common keys: `input`, `actual_output`, `expected_output`, `retrieval_context`, `context`.

### Returns
A new `Example` instance.

### Example
```python
from judgeval.data import Example

# For Q&A evaluation
example = Example.create(
    input="What is the capital of France?",
    actual_output="Paris is the capital of France.",
    expected_output="Paris",
)

# For RAG evaluation
example = Example.create(
    input="What is Python?",
    actual_output="A programming language.",
    retrieval_context=["Python is a high-level programming language..."]
)

# With custom fields
example = Example.create(
    input="user question",
    actual_output="assistant response",
    expected_output="correct response",
    context="some context",
    custom_field="custom value"
)
```
```

--------------------------------

### Add Examples from JSON File

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Use this method to upload examples from a JSON file. The JSON file must contain an array of objects, where each object represents an example with 'input', 'expected_output', and optional 'context' fields. The `batch_size` parameter controls the number of examples uploaded per API call.

```python
def add_from_json(
    self,
    file_path: str,
    batch_size: int = 100,
) -> None:
    """Upload examples from a JSON file."""
    pass
```

```json
[
    {
        "input": "What is AI?",
        "expected_output": "Artificial Intelligence",
        "context": "AI basics"
    },
    {
        "input": "What is ML?",
        "expected_output": "Machine Learning"
    }
]
```

```python
dataset.add_from_json("./data/golden-set.json")
```

--------------------------------

### Add Examples to Dataset

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Shows how to add a list of examples to an existing dataset, with an option to specify the batch size for API calls.

```python
examples = [
    Example.create(input="Q1", expected_output="A1"),
    Example.create(input="Q2", expected_output="A2"),
]
dataset.add_examples(examples, batch_size=50)
```

--------------------------------

### Create an Example Object

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Instantiate an Example object using the create class method. This is used to define input, actual, and expected outputs for an evaluation.

```python
from judgeval.data import Example

example = Example.create(
    input="What is AI?",
    actual_output="Artificial Intelligence",
    expected_output="AI is...",
)
```

--------------------------------

### Example.properties

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Get all custom properties as a dictionary. This property provides access to all custom properties stored within an example object.

```APIDOC
## `Example.properties`

Get all custom properties as a dictionary.

### Method
`@property
def properties(self) -> Dict[str, Any]`

### Returns
A copy of the example's custom properties.

### Example
```python
example = Example.create(input="q", output="a", score=0.9)
props = example.properties
# {"input": "q", "output": "a", "score": 0.9}
```
```

--------------------------------

### Install Judgeval SDK

Source: https://github.com/judgmentlabs/judgeval/blob/main/README.md

Install the Judgeval SDK using pip. This is the first step to integrating Judgeval into your project.

```bash
pip install judgeval
```

--------------------------------

### Create Datasets with Examples

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/INDEX.md

Instantiate Judgeval with your project name and create datasets. Add examples to the dataset using Example.create, specifying input and expected output.

```python
from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="my-project")

dataset = client.datasets.create(name="golden-set")
dataset.add_examples([
    Example.create(input="What is AI?", expected_output="Artificial Intelligence"),
    Example.create(input="What is ML?", expected_output="Machine Learning"),
])
```

--------------------------------

### Dataset.add_examples

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Adds a collection of examples to an existing dataset. The examples can be provided as a list or a generator, and they are uploaded in batches to manage performance.

```APIDOC
## Dataset.add_examples

### Description
Adds examples to the dataset.

### Method
`add_examples`

### Parameters
#### Parameters
- **examples** (Iterable[Example]) - Required - Examples to add. Can be a list or generator.
- **batch_size** (int) - Optional - Number of examples uploaded per API call. Default: 100
```

--------------------------------

### Get a Dataset

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Fetch an existing dataset by its name. Use this when you need to access a dataset and its examples.

```python
dataset = client.datasets.get(name="golden-set")
print(len(dataset))  # number of examples
```

--------------------------------

### Iterate Over Dataset Examples

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Loop through each example within a dataset. This is useful for processing individual data points.

```python
dataset = client.datasets.get(name="golden-set")
for example in dataset:
    print(example["input"], example["expected_output"])
```

--------------------------------

### Create a New Dataset

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Create a new dataset with an optional initial set of examples. Use this to initialize a new dataset.

```python
dataset = client.datasets.create(
    name="golden-set",
    examples=[
        Example.create(input="What is AI?", expected_output="Artificial Intelligence"),
    ],
)
```

--------------------------------

### Access Example Properties with __getitem__

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Use bracket notation to access properties of an Example object. This method is useful for retrieving specific data points like input or output.

```python
example = Example.create(input="test", output="result")
print(example["input"])   # "test"
print(example["output"])  # "result"
```

--------------------------------

### Dataset.add_from_yaml()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Upload examples from a YAML file. This method takes a file path to a YAML file containing a list of objects, where each object represents a dataset example.

```APIDOC
## `Dataset.add_from_yaml()`

### Description
Upload examples from a YAML file. This method takes a file path to a YAML file containing a list of objects, where each object represents a dataset example.

### Method Signature
```python
def add_from_yaml(
    self,
    file_path: str,
    batch_size: int = 100,
) -> None
```

### Parameters
#### Path Parameters
- **file_path** (str) - Required - Path to the YAML file.
- **batch_size** (int) - Optional - Default: 100 - Number of examples uploaded per API call.

### YAML Format
```yaml
- input: What is AI?
  expected_output: Artificial Intelligence
  context: AI basics
- input: What is ML?
  expected_output: Machine Learning
```

### Example
```python
dataset.add_from_yaml("./data/golden-set.yaml")
```
```

--------------------------------

### Framework Integrations (LangGraph Example)

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Shows how to integrate Judgeval with LangGraph for agentic workflows.

```python
from judgeval.integrations.langgraph import JudgevalLangGraph

# Assuming you have a LangGraph graph instance
graph = ...

judgeval_integration = JudgevalLangGraph(graph)
# Use judgeval_integration to add Judgeval capabilities to your graph
```

--------------------------------

### Get Dataset Size

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Retrieve the total number of examples contained within a dataset. This provides a quick count of the dataset's entries.

```python
dataset = client.datasets.get(name="golden-set")
print(len(dataset))  # 42
```

--------------------------------

### DatasetFactory.create()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Creates a new dataset with an optional initial set of examples. Returns the created Dataset object or None if the project is not resolved.

```APIDOC
## DatasetFactory.create()

### Description
Create a new dataset.

### Method
POST (conceptual, SDK method)

### Endpoint
(Not applicable for SDK method)

### Parameters
#### Path Parameters
(Not applicable for SDK method)

#### Query Parameters
(Not applicable for SDK method)

#### Request Body
(Not applicable for SDK method)

### Parameters
- **name** (str) - Required - Name for the dataset.
- **examples** (Iterable[Example]) - Optional - Initial examples to add.

### Request Example
```python
dataset = client.datasets.create(
    name="golden-set",
    examples=[
        Example.create(input="What is AI?", expected_output="Artificial Intelligence"),
    ],
)
```

### Response
#### Success Response
- **Dataset** - The created `Dataset` object.

#### Response Example
(Response structure depends on the Dataset object, not explicitly defined in source)

### Error Handling
- Returns `None` if the project is not resolved.
```

--------------------------------

### Handling KeyError for Non-existent Example Properties

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/errors.md

Demonstrates how to catch a KeyError when accessing a property that does not exist on a Judgeval Example object. It also shows how to inspect available properties.

```python
from judgeval.data import Example

example = Example.create(input="test")

try:
    output = example["output"]  # Property doesn't exist
except KeyError as e:
    print(f"Property not found: {e}")
    # Check example.properties for available keys
    print(f"Available: {example.properties.keys()}")
```

--------------------------------

### Retrieve All Properties with Example.properties

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Access all custom properties of an Example object as a dictionary. This is useful for inspecting all associated data, including custom fields.

```python
example = Example.create(input="q", output="a", score=0.9)
props = example.properties
# {"input": "q", "output": "a", "score": 0.9}
```

--------------------------------

### Running Evaluations with Hosted Scorers

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Example of setting up and running an evaluation using a hosted scorer.

```python
from judgeval.evaluation import evaluate
from judgeval.data import Dataset

dataset = Dataset()
dataset.add_record(input="Test input", output="Expected output")

# Assuming 'hosted-scorer-name' is a valid scorer
results = await evaluate(dataset, scorer="hosted-scorer-name")
```

--------------------------------

### LLM Client Wrapping (OpenAI Example)

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Shows how to wrap an OpenAI client for use with Judgeval's tracing capabilities.

```python
from judgeval.instrumentation.llm import wrap_openai
import openai

# Assuming openai client is configured
client = openai.OpenAI()
wrapped_client = wrap_openai(client)

# Use wrapped_client for API calls to enable tracing
```

--------------------------------

### Dataset.add_from_json()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Upload examples from a JSON file. This method takes a file path to a JSON file containing an array of objects, where each object represents a dataset example.

```APIDOC
## `Dataset.add_from_json()`

### Description
Upload examples from a JSON file. This method takes a file path to a JSON file containing an array of objects, where each object represents a dataset example.

### Method Signature
```python
def add_from_json(
    self,
    file_path: str,
    batch_size: int = 100,
) -> None
```

### Parameters
#### Path Parameters
- **file_path** (str) - Required - Path to the JSON file. Must contain an array of objects.
- **batch_size** (int) - Optional - Default: 100 - Number of examples uploaded per API call.

### JSON Format
```json
[
    {
        "input": "What is AI?",
        "expected_output": "Artificial Intelligence",
        "context": "AI basics"
    },
    {
        "input": "What is ML?",
        "expected_output": "Machine Learning"
    }
]
```

### Example
```python
dataset.add_from_json("./data/golden-set.json")
```
```

--------------------------------

### Initialize Tracer with Sampler

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Configures trace sampling using an OpenTelemetry Sampler. This example uses TraceIdRatioBased to sample 10% of traces.

```python
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
from judgeval import Tracer

sampler = TraceIdRatioBased(0.1)  # Sample 10% of traces
tracer = Tracer.init(
    project_name="my-project",
    sampler=sampler,
)
```

--------------------------------

### Dataset.__iter__()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Allows iteration over the examples within a dataset. Each yielded item is an example object.

```APIDOC
## `Dataset.__iter__()`

### Description
Iterate over examples in the dataset.

### Method
```python
def __iter__(self) -> Iterator[Example]
```

### Returns
- Iterator[Example] - An iterator over the loaded examples.

### Example
```python
dataset = client.datasets.get(name="golden-set")
for example in dataset:
    print(example["input"], example["expected_output"])
```
```

--------------------------------

### Example.__contains__()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Check if a property exists within an example. This method returns a boolean indicating whether a specified property key is present in the example.

```APIDOC
## `Example.__contains__()`

Check if a property exists.

### Method
`__contains__(self, key: object) -> bool`

### Parameters
#### Path Parameters
- **key** (str) - Required - Property name.

### Returns
`True` if the property exists.

### Example
```python
if "input" in example:
    print(example["input"])
```
```

--------------------------------

### List All Datasets

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

List all available datasets in the project. This is useful for getting an overview of existing datasets.

```python
for info in client.datasets.list():
    print(f"{info.name}: {info.entries} examples")
```

--------------------------------

### Example BinaryResponse Usage

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Demonstrates how to instantiate and return a BinaryResponse object. This is typically used as the return value for a scoring function.

```python
return BinaryResponse(
    value=True,
    reason="Output correctly answers the question.",
)
```

--------------------------------

### Example NumericResponse Usage

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Shows how to create a NumericResponse instance with a score and explanation. This is useful for quantitative assessments.

```python
return NumericResponse(
    value=0.85,
    reason="Output covers most of the expected content.",
)
```

--------------------------------

### Example Usage of Judgeval offline_tracer

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judgeval.md

Demonstrates how to initialize an `OfflineTracer` with a dataset and custom example fields, then use the `@tracer.observe` decorator to track function calls.

```python
dataset = []
tracer = client.offline_tracer(
    dataset=dataset,
    example_fields={"input": "test", "golden_output": "expected"},
)

@tracer.observe(span_type="tool")
def search(query: str) -> str:
    return "mock result"

search("test query")
# Each root span adds an Example to dataset
```

--------------------------------

### Dataset.__len__()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Returns the total number of examples contained within the dataset.

```APIDOC
## `Dataset.__len__()`

### Description
Get the number of examples in the dataset.

### Method
```python
def __len__(self) -> int
```

### Returns
- int - Number of examples.

### Example
```python
dataset = client.datasets.get(name="golden-set")
print(len(dataset))  # 42
```
```

--------------------------------

### Example ScorerResponse Usage

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Demonstrates how to return a ScorerResponse, specifically a BinaryResponse. Ensure the correct response type is imported.

```python
from judgeval.hosted.responses import ScorerResponse, BinaryResponse

def get_response() -> ScorerResponse:
    return BinaryResponse(value=True, reason="Looks good")
```

--------------------------------

### Example Class Definition

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Defines the structure of a single evaluation example, including its ID, creation timestamp, optional name, custom properties, and an associated trace.

```python
from dataclasses import dataclass
from typing import Optional, Dict, Any

# Assuming Trace is defined elsewhere
class Trace:
    pass

@dataclass(slots=True)
class Example:
    example_id: str
    created_at: str
    name: Optional[str] = None
    _properties: Dict[str, Any]
    trace: Optional[Trace] = None
```

--------------------------------

### Run Evaluations with Custom Judges

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/INDEX.md

Create an evaluation run and execute it with a list of examples and custom scorers (Judges). The results indicate the success of each example against the provided judges.

```python
from judgeval.judges import Judge
from judgeval.hosted.responses import BinaryResponse

class ToxicityJudge(Judge[BinaryResponse]):
    async def score(self, data: Example) -> BinaryResponse:
        output = data["actual_output"]
        is_clean = not any(word in output for word in BLOCKED_WORDS)
        return BinaryResponse(
            value=is_clean,
            reason="No blocked words found" if is_clean else "Contains blocked content",
        )

evaluation = client.evaluation.create()
results = evaluation.run(
    examples=examples,
    scorers=[ToxicityJudge()],
    eval_run_name="toxicity-check",
)

for result in results:
    print(f"Success: {result.success}")
```

--------------------------------

### Initialize Tracer with SpanLimits

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Configures OpenTelemetry span limits for trace generation. This example sets maximum attributes and events per span.

```python
from opentelemetry.sdk.trace import SpanLimits
from judgeval import Tracer

limits = SpanLimits(
    max_num_attributes=256,
    max_num_events=128,
)
tracer = Tracer.init(
    project_name="my-project",
    span_limits=limits,
)
```

--------------------------------

### Initialize Tracer with Environment Variables

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/tracer.md

Initialize a tracer instance using project name and relying on environment variables for API key, organization ID, and API URL. This is a common setup for basic tracing.

```python
from judgeval import Tracer

# Basic setup with environment variables
tracer = Tracer.init(project_name="search-assistant")
```

--------------------------------

### Create and Run an Evaluation with Judgeval

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/README.md

Instantiate the Judgeval client to create evaluations, run them with examples and scorers, and manage datasets. This also shows how to create agent judges.

```python
from judgeval import Judgeval

client = Judgeval(project_name="my-project")

# Create an evaluation
evaluation = client.evaluation.create()
results = evaluation.run(
    examples=examples,
    scorers=["faithfulness", "answer_relevancy"],
    eval_run_name="nightly",
)

# Manage datasets
dataset = client.datasets.create(name="golden-set")
dataset.add_examples([...])

# Create agent judges
judge = client.agent_judges.create(
    name="helpfulness",
    prompt="Rate helpfulness from 0 to 1.",
    model="gpt-5.2",
    score_type="numeric",
)
```

--------------------------------

### Dataset Methods

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/API-SUMMARY.md

Methods for managing and interacting with datasets, including adding, loading, and exporting examples.

```APIDOC
## Dataset.add_examples()

### Description
Adds examples to an existing dataset.

### Method
`Dataset.add_examples()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
dataset.add_examples(examples)
```

### Response
This method's response is not explicitly detailed in the source.
```

```APIDOC
## Dataset.add_from_json()

### Description
Loads examples into the dataset from a JSON file.

### Method
`Dataset.add_from_json()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
dataset.add_from_json('path/to/file.json')
```

### Response
This method's response is not explicitly detailed in the source.
```

```APIDOC
## Dataset.add_from_yaml()

### Description
Loads examples into the dataset from a YAML file.

### Method
`Dataset.add_from_yaml()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
dataset.add_from_yaml('path/to/file.yaml')
```

### Response
This method's response is not explicitly detailed in the source.
```

```APIDOC
## Dataset.save_as()

### Description
Exports the dataset to a specified file format (JSON or YAML).

### Method
`Dataset.save_as()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
dataset.save_as('path/to/output.json')
```

### Response
This method's response is not explicitly detailed in the source.
```

```APIDOC
## Dataset.__iter__()

### Description
Provides an iterator to loop through the examples in the dataset.

### Method
`Dataset.__iter__()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
for example in dataset:
    print(example)
```

### Response
This method's response is not explicitly detailed in the source.
```

```APIDOC
## Dataset.__len__()

### Description
Returns the total number of examples in the dataset.

### Method
`Dataset.__len__()`

### Parameters
This method does not explicitly list parameters in the source.

### Request Example
```
# Example usage (conceptual)
count = len(dataset)
```

### Response
This method's response is not explicitly detailed in the source.
```

--------------------------------

### Create and Run an Evaluation

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judgeval.md

Use the `evaluation` factory to create an evaluation runner and execute evaluations with specified examples and scorers.

```python
eval_runner = client.evaluation.create()
results = eval_runner.run(
    examples=examples,
    scorers=["faithfulness", "answer_relevancy"],
    eval_run_name="nightly-eval",
)
```

--------------------------------

### Run Evaluation with Hosted Scorers

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/evaluation.md

Example of how to use the Evaluation.run method with a list of hosted scorer names. This is useful for quick evaluations using pre-defined scoring metrics.

```python
from judgeval import Judgeval
from judgeval.data import Example

client = Judgeval(project_name="my-project")
evaluation = client.evaluation.create()

examples = [
    Example.create(
        input="What is Python?",
        actual_output="A programming language.",
        expected_output="A high-level programming language.",
    ),
    Example.create(
        input="What is Rust?",
        actual_output="A systems programming language.",
        expected_output="A systems language focused on memory safety.",
    ),
]

results = evaluation.run(
    examples=examples,
    scorers=["faithfulness", "answer_relevancy"],
    eval_run_name="quick-test",
)

for result in results:
    print(f"Success: {result.success}")
    for scorer in result.scorers_data:
        print(f"  {scorer.name}: {scorer.score} - {scorer.reason}")
```

--------------------------------

### Run Evaluation and Print Results

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/data.md

Demonstrates how to run an evaluation with specified examples and scorers, then iterate through the results to print success status, duration, cost, and detailed scorer information if the evaluation failed.

```python
results = evaluation.run(
    examples=examples,
    scorers=["faithfulness", "answer_relevancy"],
    eval_run_name="nightly",
)

for result in results:
    print(f"Success: {result.success}")
    print(f"Duration: {result.run_duration}s")
    print(f"Cost: ${result.evaluation_cost}")
    
    if not result.success:
        for scorer in result.scorers_data:
            print(f"  {scorer.name}: {scorer.score} - {scorer.reason}")
```

--------------------------------

### Manage Datasets with Judgeval

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judgeval.md

Create, retrieve, and list datasets using the `client.datasets` factory. Datasets store examples for evaluation.

```python
dataset = client.datasets.create(
    name="golden-set",
    examples=[
        Example.create(input="What is AI?", expected_output="Artificial Intelligence"),
    ],
)

dataset = client.datasets.get(name="golden-set")

for info in client.datasets.list():
    print(info.name, info.entries)
```

--------------------------------

### BackgroundQueue.get_instance()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/background-queue.md

Gets the singleton background queue instance, creating it if it does not yet exist.

```APIDOC
## BackgroundQueue.get_instance()

### Description
Get the singleton background queue instance.

### Method Signature
`@classmethod
def get_instance(cls) -> BackgroundQueue`

### Returns
* `BackgroundQueue` - The singleton `BackgroundQueue` instance, creating it if needed.

### Example
```python
from judgeval.background_queue import BackgroundQueue

queue = BackgroundQueue.get_instance()
```
```

--------------------------------

### Record LLMMetadata

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Example of recording LLM metadata using the Tracer. Ensure all provided fields conform to the LLMMetadata TypedDict structure.

```python
from judgeval import Tracer

Tracer.recordLLMMetadata({
    "model": "gpt-4o",
    "provider": "openai",
    "non_cached_input_tokens": 150,
    "output_tokens": 80,
    "total_cost_usd": 0.003,
})
```

--------------------------------

### DatasetFactory.get()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Fetches an existing dataset by its name. If the dataset exists, it is returned with all its examples loaded. Returns None if the dataset cannot be found or the project is not resolved.

```APIDOC
## DatasetFactory.get()

### Description
Fetch an existing dataset with all examples loaded.

### Method
GET (conceptual, SDK method)

### Endpoint
(Not applicable for SDK method)

### Parameters
#### Path Parameters
(Not applicable for SDK method)

#### Query Parameters
(Not applicable for SDK method)

#### Request Body
(Not applicable for SDK method)

### Parameters
- **name** (str) - Required - The dataset name.

### Request Example
```python
dataset = client.datasets.get(name="golden-set")
print(len(dataset))  # number of examples
```

### Response
#### Success Response
- **Dataset** - A `Dataset` object with examples populated.

#### Response Example
(Response structure depends on the Dataset object, not explicitly defined in source)

### Error Handling
- Returns `None` if the project is not resolved or the dataset is not found.
```

--------------------------------

### Get and Compile a Versioned Prompt

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/types.md

Retrieve a specific version of a prompt template from the platform and compile it with specified parameters. Ensure the 'client' object is initialized and the prompt name and tag are correct.

```python
from judgeval.prompts import Prompt

prompt = client.prompts.get(name="system-prompt", tag="production")
text = prompt.compile(tone="helpful")
```

--------------------------------

### Retrieve a Prompt by Tag or Commit ID

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/prompts.md

Use `get` to retrieve a specific prompt version by its name and either a tag or a commit ID. The retrieved prompt can then be compiled with variables.

```python
@overload
def get(
    self,
    name: str,
    tag: str,
) -> Optional[Prompt]: ...

@overload
def get(
    self,
    name: str,
    commit_id: str,
) -> Optional[Prompt]: ...
```

```python
# Retrieve by tag
prompt = client.prompts.get(name="system-prompt", tag="production")

# Retrieve by version
prompt = client.prompts.get(name="system-prompt", commit_id="abc123")

# Compile with variables
text = prompt.compile(tone="helpful", product="Acme Search")
```

--------------------------------

### Example NumericResponse with Citations

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Demonstrates how to include citations when returning a NumericResponse. Citations link the score to specific trace spans for detailed analysis.

```python
return NumericResponse(
    value=0.8,
    reason="Based on span output quality",
    citations=[
        Citation(span_id="span-123", span_attribute="output"),
    ],
)
```

--------------------------------

### Setup Claude Agent SDK Integration

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/integrations.md

Patches Claude Agent SDK components for automatic tracing of agent turns, LLM calls, and tool invocations. Returns True on success, False on error.

```python
from judgeval.integrations import setup_claude_agent_sdk

setup_claude_agent_sdk() -> bool
```

```python
from judgeval import Tracer
from judgeval.integrations import setup_claude_agent_sdk
from claude_agent_sdk import ClaudeSDKClient

# Initialize Judgment tracing
Tracer.init(project_name="my-claude-agent")

# Patch Claude Agent SDK for automatic tracing
success = setup_claude_agent_sdk()
if not success:
    print("Failed to setup integration")

# Use Claude Agent SDK normally
client = ClaudeSDKClient()
response = client.messages.create(
    model="claude-opus-4-1-20250805",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello"}],
)
# All traces automatically recorded
```

--------------------------------

### Creating and Managing Datasets

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Illustrates the creation of a dataset and adding records to it.

```python
from judgeval.data import Dataset

dataset = Dataset()
dataset.add_record(input="What is the capital of France?", output="Paris")
dataset.add_record(input="What is 2+2?", output="4")
```

--------------------------------

### Add Observability to Agent with Judgeval

Source: https://github.com/judgmentlabs/judgeval/blob/main/README.md

Instrument your agent functions with Judgeval's Tracer to automatically capture inputs, outputs, and LLM token usage. This example demonstrates observing a search function and an agent run function.

```python
from judgeval import Tracer, wrap
from openai import OpenAI

Tracer.init(project_name="my-project")
client = wrap(OpenAI())

@Tracer.observe(span_type="tool")
def search(query: str) -> str:
    results = vector_db.search(query)
    return results

@Tracer.observe(span_type="agent")
def run_agent(question: str) -> str:
    context = search(question)
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"{context}\n\n{question}"}],
    )
    return response.choices[0].message.content

run_agent("What is the capital of the United States?")
```

--------------------------------

### Tracer Initialization and Decoration

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Demonstrates how to initialize the Tracer and use it to decorate functions for tracing.

```python
from judgeval.trace import Tracer

tracer = Tracer()

@tracer.trace
def my_function(arg1: str, arg2: int) -> str:
    return f"Processed {arg1} with {arg2}"
```

--------------------------------

### Handle ImportError for Claude Agent SDK

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/integrations.md

Catch ImportError if the Claude Agent SDK is not installed. Provides guidance on how to install it.

```python
try:
    from judgeval.integrations import setup_claude_agent_sdk
except ImportError as e:
    print("Claude Agent SDK not installed")
    # pip install claude-agent-sdk
```

--------------------------------

### Initialize Tracer with Options

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/configuration.md

Configure Tracer with various optional parameters like API key, organization ID, environment, and sampling.

```python
tracer = Tracer.init(
    project_name="my-project",
    api_key="jdg_...",                          # Optional: override env var
    organization_id="org_...",                  # Optional: override env var
    api_url="https://api.judgmentlabs.ai",      # Optional: override env var
    environment="production",                    # Optional: label for traces
    set_active=True,                            # Optional: make global tracer
    serializer=custom_serializer,               # Optional: custom serializer
    resource_attributes={"service": "api"},     # Optional: OTel attributes
    sampler=TraceIdRatioBased(0.1),             # Optional: sample 10%
    span_limits=SpanLimits(...),                # Optional: span limits
    span_processors=[CustomProcessor()],        # Optional: extra processors
)
```

--------------------------------

### Prompt Versioning and Compilation

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/MANIFEST.txt

Demonstrates creating a versioned prompt and compiling it.

```python
from judgeval.prompts import Prompt

my_prompt = Prompt.create(
    "My Prompt",
    "This is version {version} of my prompt."
)

compiled_prompt = my_prompt.compile(version=1)
print(compiled_prompt.text)
```

--------------------------------

### Implement a Numeric Judge for Output Length

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Example of a custom judge that scores responses numerically based on the length of the output. Requires NumericResponse and Example types.

```python
from judgeval.hosted.responses import NumericResponse

class LengthScorer(Judge[NumericResponse]):
    """Score based on output length."""
    
    async def score(self, data: Example) -> NumericResponse:
        output = data["actual_output"]
        length = len(output)
        
        # Score 1.0 if >= 500 chars, 0.0 if < 100
        if length >= 500:
            score = 1.0
        elif length < 100:
            score = 0.0
        else:
            score = (length - 100) / 400
        
        return NumericResponse(
            value=score,
            reason=f"Output length: {length} characters",
        )
```

--------------------------------

### Implement a Binary Judge for Toxicity

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Example of a custom judge that scores responses as binary (pass/fail) based on the presence of blocked words. Requires BinaryResponse and Example types.

```python
from judgeval.judges import Judge
from judgeval.hosted.responses import BinaryResponse
from judgeval.data import Example

class ToxicityJudge(Judge[BinaryResponse]):
    """Check if output contains blocked content."""
    
    async def score(self, data: Example) -> BinaryResponse:
        output = data["actual_output"]
        blocked_words = ["badword1", "badword2"]
        is_clean = not any(word in output for word in blocked_words)
        
        return BinaryResponse(
            value=is_clean,
            reason="No blocked words found" if is_clean else "Contains blocked content",
        )

# Use in evaluation
evaluation = client.evaluation.create()
results = evaluation.run(
    examples=examples,
    scorers=[ToxicityJudge()],
    eval_run_name="toxicity-check",
)
```

--------------------------------

### Initialize and Use Tracer for Observability

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/README.md

Initialize the Tracer for your project and use the @tracer.observe decorator to instrument functions. Wrap LLM clients for automatic token tracking.

```python
from judgeval import Tracer

tracer = Tracer.init(project_name="my-project")

@tracer.observe(span_type="tool")
def my_tool(input: str) -> str:
    return "output"

# Wrap LLM clients for automatic token tracking
client = Tracer.wrap(OpenAI())
```

--------------------------------

### Tracer.init()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/tracer.md

Initializes and activates a global tracer instance for your application. It allows configuration of project details, API credentials, environment, serialization, resource attributes, sampling, and span processing.

```APIDOC
## Tracer.init()

### Description
Initializes and activates a global tracer instance for your application. It allows configuration of project details, API credentials, environment, serialization, resource attributes, sampling, and span processing.

### Method
`classmethod`

### Parameters
#### Parameters
- **project_name** (Optional[str]) - No - None - Your Judgment project name. Required for span export.
- **api_key** (Optional[str]) - No - None - Judgment API key. Defaults to `JUDGMENT_API_KEY` environment variable.
- **organization_id** (Optional[str]) - No - None - Organization ID. Defaults to `JUDGMENT_ORG_ID` environment variable.
- **api_url** (Optional[str]) - No - None - API endpoint URL. Defaults to `JUDGMENT_API_URL` environment variable.
- **environment** (Optional[str]) - No - None - Deployment environment label (e.g. "production", "staging").
- **set_active** (bool) - No - True - If True, sets this as the global tracer for `@Tracer.observe()` and other static methods.
- **serializer** (Callable[[Any], str]) - No - safe_serialize - Custom function to serialize span inputs/outputs.
- **resource_attributes** (Optional[Dict[str, Any]]) - No - None - Extra OpenTelemetry resource attributes.
- **sampler** (Optional[Sampler]) - No - None - Custom OpenTelemetry sampler for trace sampling.
- **span_limits** (Optional[SpanLimits]) - No - None - OpenTelemetry span limits configuration.
- **span_processors** (Optional[Sequence[SpanProcessor]]) - No - None - Additional span processors appended after the default.

### Returns
A configured and active `Tracer` instance.

### Example
```python
from judgeval import Tracer

# Basic setup with environment variables
tracer = Tracer.init(project_name="search-assistant")

# Explicit credentials
tracer = Tracer.init(
    project_name="search-assistant",
    api_key="jdg_...",
    organization_id="org_...",
    environment="production",
)
```
```

--------------------------------

### Judge.score()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judges.md

Evaluates a single example and returns a score. This method is abstract and must be implemented by subclasses.

```APIDOC
## Judge.score()

### Description
Evaluate a single example and return a score.

### Method
`async def score(self, data: Example) -> R`

### Parameters
#### Path Parameters
- **data** (Example) - Required - The example to evaluate. Access properties via bracket notation (e.g. `data["input"]`, `data["actual_output"]`).

### Returns
A response object (`BinaryResponse`, `NumericResponse`, or `CategoricalResponse`) with the score and reason.
```

--------------------------------

### Get singleton BackgroundQueue instance

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/background-queue.md

Retrieve the singleton BackgroundQueue instance. It will be created if it does not already exist.

```python
from judgeval.background_queue import BackgroundQueue

queue = BackgroundQueue.get_instance()
success = queue.enqueue(my_task)
```

--------------------------------

### Import Judgeval Client

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/README.md

Import the main Judgeval client class for project initialization and interaction.

```python
from judgeval import Judgeval
```

--------------------------------

### Dataset Class Definition

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/datasets.md

Defines the structure of a Dataset object, including its name, project association, kind, and examples.

```python
from dataclasses import dataclass
from typing import Optional, List

from judgeval.data import Example
from judgeval.judgment_sync_client import JudgmentSyncClient

@dataclass
class Dataset:
    name: str
    project_id: str
    project_name: str
    dataset_kind: str = "example"
    examples: Optional[List[Example]] = None
    client: Optional[JudgmentSyncClient] = None
```

--------------------------------

### Create a New Agent Judge

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/agent-judges.md

Demonstrates how to instantiate the Judgeval client and create a new agent judge with specified parameters. The judge ID and prompt are printed upon successful creation.

```python
from judgeval import Judgeval

client = Judgeval(project_name="my-project")

judge = client.agent_judges.create(
    name="helpfulness",
    prompt="Score the assistant's helpfulness from 0 to 1.",
    model="gpt-5.2",
    score_type="numeric",
)

print(judge.judge_id)
print(judge.prompt)
```

--------------------------------

### Create a New Prompt Version

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/prompts.md

Use `create` to add a new prompt or a new version of an existing prompt. If a prompt with the same name exists, a new version is automatically created and linked.

```python
class PromptFactory:
    def create(
        self,
        name: str,
        prompt: str,
        tags: Optional[List[str]] = None,
    ) -> Optional[Prompt]:
```

```python
prompt = client.prompts.create(
    name="system-prompt",
    prompt="You are a {{tone}} assistant for {{product}}.",
    tags=["v2"],
)

print(prompt.commit_id)
print(prompt.parent_commit_id)  # Previous version's ID
```

--------------------------------

### Initialize Tracer with Explicit Credentials

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/tracer.md

Initialize a tracer instance by explicitly providing all necessary credentials including project name, API key, organization ID, and environment. Use this when environment variables are not set or for specific configurations.

```python
tracer = Tracer.init(
    project_name="search-assistant",
    api_key="jdg_...",
    organization_id="org_...",
    environment="production",
)
```

--------------------------------

### Initialize Judgeval Client

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/judgeval.md

Instantiate the Judgeval client to connect to your Judgment project. You can provide credentials explicitly or rely on environment variables.

```python
from judgeval import Judgeval

# Basic setup with environment variables
client = Judgeval(project_name="search-assistant")

# Explicit credentials
client = Judgeval(
    project_name="search-assistant",
    api_key="jdg_...",
    organization_id="org_...",
)
```

--------------------------------

### Wrap OpenAI Client

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/instrumentation.md

Automatically instrument an OpenAI client for token tracking and cost reporting. Ensure Tracer is initialized before wrapping.

```python
from openai import OpenAI
from judgeval import Tracer

Tracer.init(project_name="my-project")

# Wrap the client
client = Tracer.wrap(OpenAI())

# Use normally — token usage is auto-recorded
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "What is AI?"}],
)
```

--------------------------------

### PromptFactory.list()

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/prompts.md

Lists all available versions of a given prompt by its name. It returns a list of Prompt objects representing different versions, or None if the prompt is not found or the project is not resolved.

```APIDOC
## PromptFactory.list()

### Description
List all versions of a prompt.

### Method
```python
def list(self, name: str) -> Optional[List[Prompt]]
```

### Parameters
#### Path Parameters
- **name** (str) - Required - Prompt name.

### Returns
A list of all versions of this prompt, or `None` if not found or project is unresolved.

### Example
```python
versions = client.prompts.list(name="system-prompt")
for version in versions:
    print(f"Version {version.commit_id}: tags={version.tags}")
```
```

--------------------------------

### Evaluation.run

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/evaluation.md

Runs a batch evaluation on a list of examples using specified scorers. This method can be used with hosted scorers or custom judges.

```APIDOC
## Evaluation.run

### Description
Runs a batch evaluation on a list of examples using specified scorers. This method can be used with hosted scorers or custom judges.

### Method
`run`

### Parameters
#### Path Parameters
- None

#### Query Parameters
- None

#### Request Body
- **examples** (List[Example]) - Required - The `Example` objects to evaluate.
- **scorers** (Union[List[str], List[Judge]]) - Required - Hosted scorer names (e.g. `["faithfulness"]`) or `Judge` instances. Do not mix both types.
- **eval_run_name** (str) - Required - Name for this evaluation run, visible in the dashboard.
- **assert_test** (bool) - Optional - If True, raises an exception when any scorer fails its threshold. Useful for CI/CD pipelines. Defaults to `False`.
- **timeout_seconds** (int) - Optional - Maximum seconds to wait for hosted scorer results before timing out. Defaults to `300`.

### Request Example
```python
# Example using hosted scorers
results = evaluation.run(
    examples=examples,
    scorers=["faithfulness", "answer_relevancy"],
    eval_run_name="quick-test"
)

# Example using custom judges
results = evaluation.run(
    examples=examples,
    scorers=[ToxicityJudge()],
    eval_run_name="toxicity-check",
    assert_test=True
)
```

### Response
#### Success Response (200)
- Returns a list of `ScoringResult` objects, one per example.

#### Response Example
```json
[
  {
    "success": true,
    "scorers_data": [
      {
        "name": "faithfulness",
        "score": 0.9,
        "reason": "The output is faithful to the input."
      }
    ]
  }
]
```

### Raises
- `TimeoutError`: If hosted scorers don't return within `timeout_seconds`.
- Exception if `assert_test=True` and any scorer fails its threshold.
```

--------------------------------

### Compile Prompt with Variables

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/api-reference/prompts.md

Shows how to use the `compile` method to substitute variables within a prompt template. Ensure all template variables are provided as keyword arguments.

```python
prompt = client.prompts.get(name="system-prompt", tag="production")

# Template: "You are a {{tone}} assistant for {{product}}."
text = prompt.compile(tone="helpful", product="Acme Search")
# Result: "You are a helpful assistant for Acme Search."

print(text)
```

--------------------------------

### Data Structures

Source: https://github.com/judgmentlabs/judgeval/blob/main/_autodocs/API-SUMMARY.md

Defines the core data structures used for representing evaluation examples, traces, and scoring results within the Judgeval SDK.

```APIDOC
## Data Structures

### Classes

- **`Example`**
  - Location: `judgeval.data`
  - Purpose: Single evaluation example.

- **`Trace`**
  - Location: `judgeval.data`
  - Purpose: Recorded execution trace.

- **`ScoringResult`**
  - Location: `judgeval.data`
  - Purpose: Result of scoring one example.

- **`ScorerData`**
  - Location: `judgeval.data`
  - Purpose: Result from one scorer.

### Example Methods

- **`Example.create(**kwargs)`**
  - Purpose: Create example with properties.

- **`example["key"]`**
  - Purpose: Access property.

- **`example.properties`**
  - Purpose: Get all properties.
```