Judgeval

https://docs.judgmentlabs.ai/documentation
Admin
Judgeval is an Agent Behavior Monitoring (ABM) library that helps track and judge any agent behavior...
Tokens:39,102
Snippets:414
Trust Score:5.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Multi-Agent Delegation Example Setup

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Set up specialized agents for time and weather information, including their models, descriptions, instructions, and tools. This code is part of a larger multi-agent delegation example.

```python
import asyncio
from dotenv import load_dotenv

load_dotenv()

from judgeval.trace import JudgmentTracerProvider, Tracer
from google.adk.agents import Agent
from google.adk.runners import InMemoryRunner
from google.genai import types

Tracer.init(project_name="adk-demo")
JudgmentTracerProvider.install_as_global_tracer_provider()


def get_current_time(city: str) -> dict:
    """Returns the current time in a specified city."""
    return {"status": "success", "city": city, "time": "10:30 AM"}


def get_weather(city: str) -> dict:
    """Returns the current weather in a specified city."""
    return {"status": "success", "city": city, "weather": "sunny", "temp": "72°F"}


# Define specialized agents
time_agent = Agent(
    model="gemini-2.0-flash",
    name="time_agent",
    description="Provides the current time for a given city.",
    instruction="Use the get_current_time tool to answer time-related questions.",
    tools=[get_current_time],
)

weather_agent = Agent(
    model="gemini-2.0-flash",
    name="weather_agent",
    description="Provides current weather information for a given city.",
    instruction="Use the get_weather tool to answer weather-related questions.",
    tools=[get_weather],
)

```

--------------------------------

### Initialize Tracer and Install Global Tracer Provider

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Initialize the Tracer with a project name and install JudgmentTracerProvider as the global tracer provider. This setup is necessary for tracing operations.

```python
Tracer.init(project_name="adk-demo")
JudgmentTracerProvider.install_as_global_tracer_provider()
```

--------------------------------

### Initialize OpenLit with Separate openlit.init()

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openlit

This example shows initializing OpenLit with a separate openlit.init() call before Judgment's integration initialization. This is an alternative setup.

```python
import openlit
openlit.init()

from judgeval import Tracer
from judgeval.integrations.openlit import Openlit
Tracer.init(project_name="openlit_project")
Openlit.initialize()

from openai import OpenAI
client = OpenAI()
```

--------------------------------

### Install OpenLit Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openlit

Install the necessary packages for OpenLit integration using pip or uv.

```bash
uv add openlit judgeval openai
```

```bash
pip install openlit judgeval openai
```

--------------------------------

### Install Dependencies with bun

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/vercel-ai-sdk

Install the necessary Vercel AI SDK and Judgeval packages using bun.

```bash
bun add ai @ai-sdk/openai judgeval zod dotenv mathjs
```

--------------------------------

### Install LiveKit Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/livekit

Install the judgeval and livekit packages using uv or pip.

```bash
uv add judgeval livekit
```

```bash
pip install judgeval livekit
```

--------------------------------

### Install Vertex AI Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/vertexai

Install the necessary OpenTelemetry package for Vertex AI integration.

```bash
pip install opentelemetry-instrumentation-vertexai
```

--------------------------------

### Install Ollama OpenTelemetry Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/ollama

Install the necessary package for Ollama instrumentation.

```bash
pip install opentelemetry-instrumentation-ollama
```

--------------------------------

### Install OpenLLMetry Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openllmetry

Install the necessary packages for OpenLLMetry integration using pip or uv.

```bash
uv add opentelemetry-instrumentation-openai judgeval openai
```

```bash
pip install opentelemetry-instrumentation-openai judgeval openai
```

--------------------------------

### Install Dependencies with npm

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/vercel-ai-sdk

Install the necessary Vercel AI SDK and Judgeval packages using npm.

```bash
npm install ai @ai-sdk/openai judgeval zod dotenv mathjs
```

--------------------------------

### Install Dependencies with pnpm

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/vercel-ai-sdk

Install the necessary Vercel AI SDK and Judgeval packages using pnpm.

```bash
pnpm add ai @ai-sdk/openai judgeval zod dotenv mathjs
```

--------------------------------

### Install Dependencies with yarn

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/vercel-ai-sdk

Install the necessary Vercel AI SDK and Judgeval packages using yarn.

```bash
yarn add ai @ai-sdk/openai judgeval zod dotenv mathjs
```

--------------------------------

### Install Bedrock OpenTelemetry Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/aws_bedrock

Install the necessary package for AWS Bedrock OpenTelemetry instrumentation.

```bash
pip install opentelemetry-instrumentation-bedrock
```

--------------------------------

### Initialize Groq Tracing with OpenTelemetry

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/groq

Initialize the Judgment Tracer and register the GroqInstrumentor for OpenTelemetry. Ensure the 'opentelemetry-instrumentation-groq' dependency is installed.

```python
from judgeval import Tracer
# Make sure you installed the dependency: opentelemetry-instrumentation-groq
from opentelemetry.instrumentation.groq import GroqInstrumentor

Tracer.init(project_name="groq_project")
Tracer.registerOTELInstrumentation(GroqInstrumentor())
```

--------------------------------

### Install Google ADK Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Install the necessary packages for Google ADK integration using uv or pip.

```bash
uv add google-adk judgeval python-dotenv
```

```bash
pip install google-adk judgeval python-dotenv
```

--------------------------------

### Example: Agent with Tool Execution

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/claude-agent-sdk

Demonstrates an agent using a custom calculator tool. It includes tool definition, MCP server creation, and agent options configuration for tool usage.

```python
import dotenv
import asyncio
from typing import Any, Dict

dotenv.load_dotenv()

from judgeval import Tracer
from judgeval.integrations.claude_agent_sdk import setup_claude_agent_sdk

Tracer.init(project_name="claude_agents_project")
setup_claude_agent_sdk()

from claude_agent_sdk import (
    ClaudeSDKClient,
    ClaudeAgentOptions,
    tool,
    create_sdk_mcp_server,
)

# Define a calculator tool
@tool(
    "calculator",
    "Calculates simple math expressions. Input should be a string like '2+2' or '10*5'.",
    {"expression": str},
)
async def calculator(args: Dict[str, Any]) -> Dict[str, Any]:
    """Simple calculator tool."""
    try:
        expression = args.get("expression", "")
        # Security: only allow basic math
        allowed_chars = set("0123456789+-*/(). ")
        if not all(c in allowed_chars for c in expression):
            return {
                "content": [{"type": "text", "text": "Error: Invalid characters in expression"}],
                "is_error": True,
            }

        result = eval(expression, {"__builtins__": {}}, {}) # This is just an example, we don't recommend using this code in production as it can lead to security vulnerabilities
        return {"content": [{"type": "text", "text": f"{expression} = {result}"}]}
    except Exception as e:
        return {
            "content": [{"type": "text", "text": f"Error: {str(e)}"}],
            "is_error": True,
        }

# Create MCP server with the tool
calc_server = create_sdk_mcp_server(
    name="math_tools",
    version="1.0.0",
    tools=[calculator]
)

@Tracer.observe(span_type="function")  
async def main():
    options = ClaudeAgentOptions(
        model="claude-sonnet-4-20250514",
        system_prompt="You are a helpful assistant. Use the calculator tool for any math.",
        mcp_servers={"math": calc_server},
        allowed_tools=["mcp__math__calculator"],
        permission_mode="acceptEdits",
    )

    # Make API call with tool
    async with ClaudeSDKClient(options=options) as client:
        await client.query("What is 15 multiplied by 23? Use the calculator tool.")

        response_text = []
        async for message in client.receive_response():
            if hasattr(message, "content"):
                for block in message.content:
                    if hasattr(block, "text"):
                        response_text.append(block.text)

        print("".join(response_text))

if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Install Langgraph Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/langgraph

Install the necessary packages for Langgraph integration using uv or pip.

```bash
uv add langgraph judgeval langchain-openai
```

```bash
pip install langgraph judgeval langchain-openai
```

--------------------------------

### Complete Example with Tracer Observation

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/claude-agent-sdk

Demonstrates how to use `@Tracer.observe()` to track functions outside the SDK and integrate with the Claude Agent SDK client.

```APIDOC
```python
from judgeval import Tracer
from judgeval.integrations.claude_agent_sdk import setup_claude_agent_sdk
from claude_agent_sdk import ClaudeSDKClient, ClaudeAgentOptions
import asyncio

Tracer.init(project_name="claude_agents_project")
setup_claude_agent_sdk()

@Tracer.observe(span_type="function")
def preprocess_input(data: str) -> str:
    return f"Preprocessed: {data}"

async def run_agent(user_input: str):
    # Preprocess input (traced with @Tracer.observe)
    processed_input = preprocess_input(user_input)

    # Agent execution (automatically traced)
    options = ClaudeAgentOptions(model="claude-sonnet-4-20250514")

    async with ClaudeSDKClient(options=options) as client:
        await client.query(processed_input)

        response_text = []
        async for message in client.receive_response():
            if hasattr(message, "content"):
                for block in message.content:
                    if hasattr(block, "text"):
                        response_text.append(block.text)

        return "".join(response_text)

# Execute - both helper functions and agents are traced
result = asyncio.run(run_agent("Hello World"))
print(result)
```
```

--------------------------------

### Multi-Agent Langgraph Workflow Example

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/langgraph

Demonstrates a multi-agent workflow using Langgraph, including research, planning, and execution agents. Initialization of Tracer and Langgraph integration is required.

```python
from judgeval import Tracer
from judgeval.integrations.langgraph import Langgraph
from langgraph.graph import StateGraph, START, END
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage
from typing import TypedDict, List

Tracer.init(project_name="langgraph_project")
Langgraph.initialize()

class AgentState(TypedDict):
    messages: List[dict]
    task: str
    result: str

def research_agent(state: AgentState) -> AgentState:
    llm = ChatOpenAI(model="gpt-5.2")
    response = llm.invoke([HumanMessage(content=f"Research: {state['task']}")])

    return {
        **state,
        "messages": state["messages"] + [{"role": "assistant", "content": response.content}],
        "result": f"Research completed for: {state['task']}"
    }

def planning_agent(state: AgentState) -> AgentState:
    llm = ChatOpenAI(model="gpt-5.2")
    response = llm.invoke([HumanMessage(content=f"Create plan for: {state['task']}")])

    return {
        **state,
        "messages": state["messages"] + [{"role": "assistant", "content": response.content}],
        "result": f"Plan created for: {state['task']}"
    }

def execution_agent(state: AgentState) -> AgentState:
    llm = ChatOpenAI(model="gpt-5.2")
    response = llm.invoke([HumanMessage(content=f"Execute: {state['task']}")])

    return {
        **state,
        "messages": state["messages"] + [{"role": "assistant", "content": response.content}],
        "result": f"Task completed: {state['task']}"
    }

@Tracer.observe(span_type="function")  
def main():
    graph = StateGraph(AgentState)
    graph.add_node("research", research_agent)
    graph.add_node("planning", planning_agent)
    graph.add_node("execution", execution_agent)

    graph.set_entry_point("research")
    graph.add_edge("research", "planning")
    graph.add_edge("planning", "execution")
    graph.add_edge("execution", END)

    workflow = graph.compile()

    result = workflow.invoke({
        "messages": [],
        "task": "Build a customer service bot",
        "result": ""
    })
    print(result)

if __name__ == "__main__":
    main()
```

--------------------------------

### Install OpenAI Agents SDK Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/openai-agents

Install the necessary libraries for OpenAI Agents SDK integration using either uv or pip.

```bash
uv add openai-agents-sdk judgeval openinference-instrumentation-openai-agents python-dotenv
```

```bash
pip install openai-agents-sdk judgeval openinference-instrumentation-openai-agents python-dotenv
```

--------------------------------

### Install OpenTelemetry Transformers Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/huggingface

Install the required package for OpenTelemetry instrumentation of HuggingFace Transformers. This is a prerequisite for tracing.

```bash
pip install opentelemetry-instrumentation-transformers
```

--------------------------------

### Install OpenInference Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openinference

Install the necessary OpenInference packages for OpenAI instrumentation and Judgment evaluation using either uv or pip.

```bash
uv add openinference-instrumentation-openai judgeval openai
```

```bash
pip install openinference-instrumentation-openai judgeval openai
```

--------------------------------

### Install Judgeval and Requests for Python

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Install the necessary Python packages for using Judgment's tracing and making HTTP requests.

```bash
uv add judgeval requests
```

```bash
pip install judgeval requests
```

--------------------------------

### Install judgeval with npm

Source: https://docs.judgmentlabs.ai/documentation

Use 'npm install' for managing judgeval dependencies in Node.js projects.

```bash
npm install judgeval
```

--------------------------------

### Install Judgment CLI with Homebrew

Source: https://docs.judgmentlabs.ai/documentation/cli

Installs the Judgment CLI using the Homebrew package manager. This command also handles upgrades.

```bash
brew install JudgmentLabs/tap/judgment-cli
```

--------------------------------

### Install judgeval with pip

Source: https://docs.judgmentlabs.ai/documentation

Use 'pip install' for managing judgeval dependencies in Python projects.

```bash
pip install judgeval
```

--------------------------------

### Initialize Judgment Tracer for ADK

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Initialize the Judgment tracer and install it as the global tracer provider to capture ADK spans automatically.

```python
from judgeval.trace import JudgmentTracerProvider, Tracer

Tracer.init(project_name="adk-demo")
JudgmentTracerProvider.install_as_global_tracer_provider()
```

--------------------------------

### Install Judgeval and OpenTelemetry API for Node.js/TypeScript

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Install the necessary npm packages for using Judgment's tracing in a Node.js or TypeScript environment.

```bash
npm install judgeval @opentelemetry/api
```

```bash
yarn add judgeval @opentelemetry/api
```

```bash
pnpm add judgeval @opentelemetry/api
```

```bash
bun add judgeval @opentelemetry/api
```

--------------------------------

### Install Claude Agent SDK Dependencies

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/claude-agent-sdk

Install the necessary Python packages for Claude Agent SDK integration using pip or uv.

```bash
uv add claude-agent-sdk judgeval python-dotenv
```

```bash
pip install claude-agent-sdk judgeval python-dotenv
```

--------------------------------

### Install judgeval with uv

Source: https://docs.judgmentlabs.ai/documentation

Use 'uv add' for managing judgeval dependencies in Python projects.

```bash
uv add judgeval
```

--------------------------------

### Add OpenLit Integration to Existing Code

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openlit

Integrate OpenLit tracing into an existing application using OpenAI. The Openlit.initialize() call handles OpenLit setup automatically.

```python
from openai import OpenAI
from judgeval import Tracer
from judgeval.integrations.openlit import Openlit

Tracer.init(project_name="openlit_project")
Openlit.initialize()

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.2",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

print(response.choices[0].message.content)
```

--------------------------------

### Example Workflow: Movie Recommendation Agent

Source: https://docs.judgmentlabs.ai/documentation/optimization/prompt-versioning

Shows a complete Python class that uses Judgeval to retrieve a prompt and OpenAI to generate a response based on user inputs for genre and decade. Requires OpenAI and Judgeval libraries.

```python
from openai import OpenAI
from judgeval import Judgeval

# Retrieve and use a tagged version
class MovieRecommendationAgent:
    def __init__(self):
        self.client = OpenAI()
        self.judgeval = Judgeval(project_name="default_project")

    def get_recommendation(self, user_genre: str, user_decade: str):
        prompt = self.judgeval.prompts.get(
          name="Movie Recommendation Prompt",
          tag="production"
        )
        response = self.client.chat.completions.create(
            model="gpt-5.2",
            messages=[
              {
                "role": "user",
                "content": prompt.compile(genre=user_genre, decade=user_decade)
              }
            ]
        )
        return f"Response: {response.choices[0].message.content}"

if __name__ == "__main__":
    agent = MovieRecommendationAgent()
    agent.get_recommendation("horror", "90s")
```

--------------------------------

### Install Claude Code CLI

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/claude-agent-sdk

Install the Claude Code CLI globally using npm, which is a requirement for the Claude Agent SDK.

```bash
npm install -g @anthropic-ai/claude-code
```

--------------------------------

### Initialize OpenInference Tracer

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openinference

Initialize the Judgment Tracer and register the OpenAI instrumentor for OpenInference. This setup is required before instrumenting OpenAI calls.

```python
from judgeval import Tracer
from openinference.instrumentation.openai import OpenAIInstrumentor

Tracer.init(project_name="openinference_project")
Tracer.registerOTELInstrumentation(OpenAIInstrumentor())
```

--------------------------------

### Get a Specific Documentation Page

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves the content of a specific documentation page using its path.

```bash
judgment docs get-page /docs/getting-started
```

--------------------------------

### Initialize Tracer and Register Vertex AI Instrumentor

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/vertexai

Initialize the Judgment Tracer and register the VertexAIInstrumentor to enable tracing for Vertex AI calls. Ensure the dependency is installed.

```python
from judgeval import Tracer
# Make sure you installed the dependency: opentelemetry-instrumentation-vertexai
from opentelemetry.instrumentation.vertexai import VertexAIInstrumentor

Tracer.init(project_name="vertexai_project")
Tracer.registerOTELInstrumentation(VertexAIInstrumentor())
```

--------------------------------

### Install judgeval with bun

Source: https://docs.judgmentlabs.ai/documentation

Use 'bun add' for managing judgeval dependencies in Node.js projects.

```bash
bun add judgeval
```

--------------------------------

### Run a Multi-Agent Delegation Workflow

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Execute a workflow where a coordinator agent delegates tasks to sub-agents. This example demonstrates running a session with a coordinator agent and processing user questions.

```python
@Tracer.observe(span_type="function")  
async def main():
    runner = InMemoryRunner(agent=coordinator, app_name="demo")
    session = await runner.session_service.create_session(
        app_name="demo", user_id="user1")

    questions = [
        "What time is it in Tokyo?",
        "What's the weather like in Paris?",
    ]

    for question in questions:
        msg = types.Content(
            role="user", parts=[types.Part(text=question)]
        )
        print(f"Question: {question}")
        async for event in runner.run_async(
            user_id="user1", session_id=session.id, new_message=msg
        ):
            if event.content and event.content.parts:
                for p in event.content.parts:
                    if p.text:
                        print(f"Response: {p.text}")
        print()


if __name__ == "__main__":
    asyncio.run(main())
```

--------------------------------

### Integrate OpenInference with OpenAI Calls

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openinference

Add initialization code and use the OpenAI client to send traces from your application to Judgment. This example demonstrates a basic chat completion call.

```python
from openai import OpenAI
from judgeval import Tracer
from openinference.instrumentation.openai import OpenAIInstrumentor

Tracer.init(project_name="openinference_project")
Tracer.registerOTELInstrumentation(OpenAIInstrumentor())

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-5.2",
    messages=[{"role": "user", "content": "Hello, world!"}]
)

print(response.choices[0].message.content)
```

--------------------------------

### Initialize OpenAI Tracing with OpenTelemetry

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/openai

Use the `OpenTelemetry` library to instrument OpenAI SDK calls and send spans to Judgment. Ensure the `opentelemetry-instrumentation-openai` dependency is installed and Tracer is initialized.

```python
from judgeval import Tracer
# Make sure you installed the dependency: opentelemetry-instrumentation-openai
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

Tracer.init(project_name="openai_project")
Tracer.registerOTELInstrumentation(OpenAIInstrumentor())
```

--------------------------------

### Initialize Judgment Tracer and Register Ollama Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/ollama

Initialize the Judgment Tracer and register the Ollama instrumentor to automatically trace Ollama API calls. Ensure the dependency is installed.

```python
from judgeval import Tracer
# Make sure you installed the dependency: opentelemetry-instrumentation-ollama
from opentelemetry.instrumentation.ollama import OllamaInstrumentor

Tracer.init(project_name="ollama_project")
Tracer.registerOTELInstrumentation(OllamaInstrumentor())
```

--------------------------------

### Implement Auto-Instrumentation with Judgeval and OpenAI

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

This TypeScript example demonstrates setting up auto-instrumentation for OpenAI calls using Judgeval. It requires initializing instrumentation, setting up the OpenAI client, and using `Tracer.observe` for specific functions.

```typescript
import "./instrumentation";
import { Tracer } from "judgeval";
import OpenAI from "openai";

function requireEnv(name: string): string {
    const value = process.env[name];
    if (!value) {
        throw new Error(`Environment variable ${name} is not set`);
    }
    return value;
}

const OPENAI_API_KEY = requireEnv("OPENAI_API_KEY");

const openai = new OpenAI({
    apiKey: OPENAI_API_KEY,
});

async function _chatWithUser(userMessage: string): Promise<string> {
    const messages: OpenAI.Chat.ChatCompletionMessageParam[] = [
        { role: "system", content: "You are a helpful assistant." },
        { role: "user", content: userMessage },
    ];

    const completion = await openai.chat.completions.create({
        model: "gpt-5.2",
        messages,
    });

    const result = completion.choices[0].message.content || "";

    console.log(`User: ${userMessage}`);
    console.log(`Assistant: ${result}`);

    Tracer.asyncEvaluate("answer_relevancy", { 
        input: "What is the capital of France?",
        actual_output: result,
    });

    return result;
}

(async () => {
    const chatWithUser = Tracer.observe(_chatWithUser); 

    const result = await chatWithUser("What is the capital of France?");
    console.log(result);

    await new Promise((resolve) => setTimeout(resolve, 10000));
    await Tracer.shutdown();
})();
```

--------------------------------

### Define a Coordinator Agent for Multi-Agent Delegation

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Define a coordinator agent that delegates tasks to specialized sub-agents based on the user's query. This example shows how to set up an agent with sub-agents for time and weather.

```python
coordinator = Agent(
    model="gemini-2.0-flash",
    name="coordinator",
    description="Routes user questions to the appropriate specialized agent.",
    instruction=(
        "You are a coordinator. Delegate time questions to the time_agent "
        "and weather questions to the weather_agent."
    ),
    sub_agents=[time_agent, weather_agent],
)
```

--------------------------------

### Install judgeval for Python

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Install the necessary judgeval package and its dependencies for Python using pip.

```bash
uv add judgeval fastapi uvicorn
```

```bash
pip install judgeval fastapi uvicorn
```

--------------------------------

### Create a New Project

Source: https://docs.judgmentlabs.ai/documentation/cli

Creates a new project with the specified name.

```bash
judgment projects create "My Project"
```

--------------------------------

### Add Best Practices Skill for Windsurf

Source: https://docs.judgmentlabs.ai/documentation/mcp-server

Create the directory and download the MCP server best practices skill for Windsurf. This helps optimize query patterns.

```bash
mkdir -p .windsurf/skills/judgment-mcp 
curl -fLo .windsurf/skills/judgment-mcp/SKILL.md 
  https://docs.judgmentlabs.ai/skills/mcp-server-best-practices.md 
```

--------------------------------

### Initialize OpenLit Integration

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openlit

Initialize the Tracer and the OpenLit integration. Ensure Tracer is initialized before Openlit.

```python
from judgeval import Tracer
from judgeval.integrations.openlit import Openlit

Tracer.init(project_name="openlit_project")
Openlit.initialize()
```

--------------------------------

### Add Best Practices Skill for Cursor

Source: https://docs.judgmentlabs.ai/documentation/mcp-server

Create the directory and download the MCP server best practices skill for Cursor. This helps optimize query patterns.

```bash
mkdir -p .cursor/skills/judgment-mcp
curl -fLo .cursor/skills/judgment-mcp/SKILL.md \
  https://docs.judgmentlabs.ai/skills/mcp-server-best-practices.md
```

--------------------------------

### Initialize Claude Agent SDK Integration

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/claude-agent-sdk

Initialize the Tracer and set up the Claude Agent SDK integration. Ensure Tracer.init is called before setup_claude_agent_sdk().

```python
from judgeval import Tracer
from judgeval.integrations.claude_agent_sdk import setup_claude_agent_sdk

Tracer.init(project_name="claude_agents_project")
setup_claude_agent_sdk()
```

--------------------------------

### Install judgeval for TypeScript

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Install the judgeval package and OpenTelemetry API for TypeScript using npm, yarn, pnpm, or bun.

```bash
npm install judgeval @opentelemetry/api express
```

```bash
yarn add judgeval @opentelemetry/api express
```

```bash
pnpm add judgeval @opentelemetry/api express
```

```bash
bun add judgeval @opentelemetry/api express
```

--------------------------------

### Initialize Tracer in Python

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Call Tracer.init() to set up tracing for your project. Ensure you provide a project_name.

```python
from judgeval import Tracer

Tracer.init(project_name="default_project")
```

--------------------------------

### Compile and Print a Prompt

Source: https://docs.judgmentlabs.ai/documentation/optimization/prompt-versioning

Demonstrates how to retrieve a specific prompt version and compile it with given variables, then print the resulting string. Ensure Judgeval is initialized.

```python
client = Judgeval(project_name="default_project")

prompt = client.prompts.get(
  name="Movie Recommendation Prompt",
  tag="production"
)

response = prompt.compile(genre="horror", decade="90s")
print(response) # "Given this genre: horror and this decade: 90s, can you recommend a movie of that genre that was released in that decade?"
```

--------------------------------

### Add Best Practices Skill for Codex

Source: https://docs.judgmentlabs.ai/documentation/mcp-server

Create the directory and download the MCP server best practices skill for Codex. This helps optimize query patterns.

```bash
mkdir -p .codex/skills/judgment-mcp
curl -fLo .codex/skills/judgment-mcp/SKILL.md 
  https://docs.judgmentlabs.ai/skills/mcp-server-best-practices.md
```

--------------------------------

### Install a Specific Version of Judgment CLI

Source: https://docs.judgmentlabs.ai/documentation/cli

Installs a specific version of the Judgment CLI by downloading the release script from GitHub. This is useful for pinning to a known stable version.

```bash
curl -fsSL https://github.com/JudgmentLabs/cli/releases/download/v0.1.0/install.sh | bash
```

--------------------------------

### Install Judgment CLI with curl

Source: https://docs.judgmentlabs.ai/documentation/cli

Installs the latest version of the Judgment CLI using a curl script. Ensure your system meets the Python version requirement (≥ 3.9).

```bash
curl -fsSL https://judgmentlabs.ai/install.sh | bash
```

--------------------------------

### Initialize OpenLLMetry Integration

Source: https://docs.judgmentlabs.ai/documentation/integrations/tracing-providers/openllmetry

Initialize the Judgment Tracer and register the OpenAI instrumentor for tracing.

```python
from judgeval import Tracer
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

Tracer.init(project_name="openllmetry_project")
Tracer.registerOTELInstrumentation(OpenAIInstrumentor())
```

--------------------------------

### Get Spans for a Trace

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves all spans associated with a specific trace.

```bash
judgment traces spans     <PROJECT_ID> <TRACE_ID>
```

--------------------------------

### Get Trace Behaviors for a Session

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves the trace behaviors associated with a specific session.

```bash
judgment sessions trace-behaviors <PROJECT_ID> <SESSION_ID>
```

--------------------------------

### Initialize and Wrap OpenAI Client for OpenRouter

Source: https://docs.judgmentlabs.ai/documentation/integrations/model-providers/openrouter

Initialize the OpenAI client with OpenRouter's base URL and API key, then wrap it with Judgeval's Tracer for monitoring. Ensure to set appropriate headers for identification.

```python
from openai import OpenAI
from judgeval import Tracer, wrap

Tracer.init(project_name="openrouter_project")

openai_client = OpenAI(
    api_key="YOUR_OPENROUTER_API_KEY",
    base_url="https://openrouter.ai/api/v1",
    default_headers={
        "HTTP-Referer": "YOUR_OPENROUTER_URL",
        "X-Title": "YOUR_OPENROUTER_NAME",
    }
)
wrapped_client = wrap(openai_client)  # Use wrapped_client from here on
```

--------------------------------

### Initialize LiveKit Tracer

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/livekit

Initialize the Tracer with your project name and set up the OpenTelemetry TracerProvider to capture LiveKit agent traces.

```python
from judgeval import Tracer
from livekit.agents.telemetry import set_tracer_provider
from opentelemetry.sdk.trace import TracerProvider

tracer = Tracer.init(project_name="livekit_project")

trace_provider = TracerProvider()
span_processor = tracer.get_span_processor()
if span_processor:
    trace_provider.add_span_processor(span_processor)

set_tracer_provider(trace_provider)
```

--------------------------------

### Install judgeval with pnpm

Source: https://docs.judgmentlabs.ai/documentation

Use 'pnpm add' for managing judgeval dependencies in Node.js projects.

```bash
pnpm add judgeval
```

--------------------------------

### Install judgeval with yarn

Source: https://docs.judgmentlabs.ai/documentation

Use 'yarn add' for managing judgeval dependencies in Node.js projects.

```bash
yarn add judgeval
```

--------------------------------

### Run an Agent with Preprocessing and Tracing

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/google-adk

Execute an agent workflow that includes preprocessing user input and running the agent. Both the helper function and the agent execution are traced using @Tracer.observe.

```python
async def run_agent(user_input: str):
    # Preprocess input (traced with @Tracer.observe)
    processed_input = preprocess_input(user_input)

    agent = Agent(
        model="gemini-2.0-flash",
        name="assistant",
        description="A helpful assistant.",
        instruction="You are a helpful assistant.",
    )

    runner = InMemoryRunner(agent=agent, app_name="demo")
    session = await runner.session_service.create_session(
        app_name="demo", user_id="user1")
    msg = types.Content(
        role="user", parts=[types.Part(text=processed_input)]
    )
    async for event in runner.run_async(
        user_id="user1", session_id=session.id, new_message=msg
    ):
        if event.content and event.content.parts:
            for p in event.content.parts:
                if p.text:
                    print(p.text)

# Execute - both helper functions and agents are traced
asyncio.run(run_agent("Hello World"))
```

--------------------------------

### List Behaviors

Source: https://docs.judgmentlabs.ai/documentation/cli

Lists all defined behaviors within a specified project.

```bash
judgment behaviors list   <PROJECT_ID>
```

--------------------------------

### List Projects

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves a list of all projects associated with your account.

```bash
judgment projects list
```

--------------------------------

### Set up and Invoke LangGraph Workflow

Source: https://docs.judgmentlabs.ai/documentation/integrations/agent-frameworks/langgraph

Define a simple LangGraph state, add a node, compile the graph, and invoke it. Ensure Judgment is configured to trace the workflow execution.

```python
from langgraph.graph import StateGraph, START, END

# Assume 'langgraph_node' is a defined function or callable
def langgraph_node(state):
    # Example node logic
    return {"result": f"Processed: {state['input']}"}

graph = StateGraph(dict)
graph.add_node("process", langgraph_node)
graph.add_edge(START, "process")
graph.add_edge("process", END)

workflow = graph.compile()

# Execute the workflow - both Langgraph and helper functions are traced
result = workflow.invoke({"input": "Hello World"})
print(result["result"])  # Output: "Processed: Hello World"
```

--------------------------------

### Get a Specific Trace

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves details for a specific trace identified by its project ID and trace ID.

```bash
judgment traces get       <PROJECT_ID> <TRACE_ID>
```

--------------------------------

### Add Best Practices Skill for Claude Code

Source: https://docs.judgmentlabs.ai/documentation/mcp-server

Create the directory and download the MCP server best practices skill for Claude Code. This helps optimize query patterns.

```bash
mkdir -p .agents/skills/mcp-server-best-practices 
curl -fLo .agents/skills/mcp-server-best-practices/SKILL.md 
  https://docs.judgmentlabs.ai/skills/mcp-server-best-practices.md 
```

--------------------------------

### Get a Specific Session

Source: https://docs.judgmentlabs.ai/documentation/cli

Retrieves detailed information about a specific session using its project ID and session ID.

```bash
judgment sessions get             <PROJECT_ID> <SESSION_ID>
```

--------------------------------

### List Automations

Source: https://docs.judgmentlabs.ai/documentation/cli

Lists all configured automations within a project.

```bash
judgment automations list <PROJECT_ID>
```

--------------------------------

### Initialize OpenTelemetry Instrumentation for Auto-Instrumentation

Source: https://docs.judgmentlabs.ai/documentation/performance/tracing

Register OpenTelemetry instrumentations, such as `OpenAIInstrumentation`, with `Tracer` before initializing it. This setup is crucial for the auto-instrumentation of LLM calls.

```typescript
import { OpenAIInstrumentation } from "@opentelemetry/instrumentation-openai";
import { Tracer } from "judgeval";

Tracer.registerOTELInstrumentation(new OpenAIInstrumentation()); 

await Tracer.init({
    projectName: "auto_instrumentation_example",
});
```

--------------------------------

### Create and Run Agent Evaluation

Source: https://docs.judgmentlabs.ai/documentation/evaluation/introduction

Initializes the Judgeval client and defines examples for an accuracy test. The evaluation is then run with specified scorers and a name.

```python
from judgeval import Judgeval
from judgeval.data.example import Example

client = Judgeval(project_name="default_project")

test_examples = [
    Example.create(
        input="What is the capital of the United States?",
        actual_output="The capital of the U.S. is Washington, D.C."
    ),
    Example.create(
        input="What is the capital of the United States?",
        actual_output="I think it's New York City."
    )
]

results = client.evaluation.create().run(
    examples=test_examples,
    scorers=["AccuracyScorer"],
    eval_run_name="accuracy_test"
)
```

--------------------------------

### Add TypeScript Agent Rules (Project-Specific)

Source: https://docs.judgmentlabs.ai/documentation/agent-rules

Installs Judgment agent rules for TypeScript to your project's local Claude configuration. Use this for project-specific settings.

```bash
mkdir -p .claude/rules
curl https://docs.judgmentlabs.ai/agent-rules/typescript -o .claude/rules/judgment.md
```

--------------------------------

### Add Python Agent Rules (Project-Specific)

Source: https://docs.judgmentlabs.ai/documentation/agent-rules

Installs Judgment agent rules for Python to your project's local Claude configuration. Use this for project-specific settings.

```bash
mkdir -p .claude/rules
curl https://docs.judgmentlabs.ai/agent-rules/python -o .claude/rules/judgment.md
```

--------------------------------

### Compile Prompt with Variables

Source: https://docs.judgmentlabs.ai/documentation/optimization/prompt-versioning

After retrieving a prompt, use the `compile` method to integrate it with specific variables for dynamic content generation.

```python
from judgeval import Judgeval

# Assume "Movie Recommendation Prompt" is a prompt whose latest commit has the following content:

```
Tokens:
Raw