### Install agentevals and @langchain/core

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Install the necessary packages for using AgentEvals and LangChain core functionalities.

```bash
npm install agentevals @langchain/core
```

--------------------------------

### Install Agentevals

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Install the agentevals library for Python or TypeScript/JavaScript.

```bash
# Python
pip install agentevals
```

```bash
# TypeScript/JavaScript
npm install agentevals @langchain/core
```

--------------------------------

### Install OpenAI Client

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Optionally, install the OpenAI client directly if you prefer not to use LangChain's integrations for LLM-as-judge evaluators.

```bash
pip install openai
```

--------------------------------

### Install OpenAI Client (Optional)

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Install the OpenAI client directly if you prefer to use it instead of LangChain's integrations for LLM-as-judge evaluators.

```bash
npm install openai
```

--------------------------------

### Install AgentEvals

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/00-START-HERE.md

Install the AgentEvals library for Python or TypeScript/JavaScript. For TypeScript/JavaScript, also install the core LangChain package.

```bash
# Python
pip install agentevals

# TypeScript/JavaScript
npm install agentevals @langchain/core
```

--------------------------------

### Unordered Trajectory Match Example

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Demonstrates how to use the 'unordered' trajectory match mode to evaluate if two trajectories contain the same tool calls in any order. Ensure that the 'agentevals' library is installed and imported.

```typescript
import {
  createTrajectoryMatchEvaluator,
  type FlexibleChatCompletionMessage,
} from "agentevals";

const outputs = [
  { role: "user", content: "What is the weather in SF and is there anything fun happening?" },
  {
    role: "assistant",
    content: "",
    tool_calls: [{
      function: {
        name: "get_weather",
        arguments: JSON.stringify({ city: "SF" }),
      }
    }],
  },
  { role: "tool", content: "It's 80 degrees and sunny in SF." },
  {
    role: "assistant",
    content: "",
    tool_calls: [{
      function: {
        name: "get_fun_activities",
        arguments: JSON.stringify({ city: "SF" }),
      }
    }],
  },
  { role: "tool", content: "Nothing fun is happening, you should stay indoors and read!" },
  { role: "assistant", content: "The weather in SF is 80 degrees and sunny, but there is nothing fun happening." },
] satisifes FlexibleChatCompletionMessage[];

const referenceOutputs = [
  { role: "user", content: "What is the weather in SF and is there anything fun happening?" },
  {
    role: "assistant",
    content: "",
    tool_calls: [
      {
        function: {
          name: "get_fun_activities",
          arguments: JSON.stringify({ city: "San Francisco" }),
        }
      },
      {
        function: {
          name: "get_weather",
          arguments: JSON.stringify({ city: "San Francisco" }),
        }
      },
    ],
  },
  { role: "tool", content: "Nothing fun is happening, you should stay indoors and read!" },
  { role: "tool", content: "It's 80 degrees and sunny in SF." },
  { role: "assistant", content: "In SF, it's 80˚ and sunny, but there is nothing fun happening." },
] satisfies FlexibleChatCompletionMessage[];

const evaluator = createTrajectoryMatchEvaluator({
  trajectoryMatchMode: "unordered",
});

const result = await evaluator({
  outputs,
  referenceOutputs,
});

console.log(result)
```

```json
{
    'key': 'trajectory_unordered_match',
    'score': true,
}
```

--------------------------------

### GraphTrajectory Example

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/types-and-utilities.md

An example demonstrating the structure of a GraphTrajectory, showing inputs, results, and a sequence of execution steps.

```typescript
const trajectory: GraphTrajectory = {
  inputs: [{ query: "What is 2+2?" }],
  results: [{ answer: "4" }],
  steps: [["__start__", "agent", "calculator", "agent"]],
};
```

--------------------------------

### Environment Setup for AgentEval

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Set the OPENAI_API_KEY for required functionality. Optionally set LANGSMITH_API_KEY and LANGSMITH_TRACING for logging.

```bash
# Required
export OPENAI_API_KEY="sk-..."

# Optional (for LangSmith logging)
export LANGSMITH_API_KEY="ls-..."
export LANGSMITH_TRACING="true"
```

--------------------------------

### ToolArgsMatchOverrides Example

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/types-and-utilities.md

An example illustrating how to configure ToolArgsMatchOverrides for different tools, including ignoring arguments, partial matching, and custom comparison logic.

```typescript
const overrides: ToolArgsMatchOverrides = {
  get_weather: "ignore",  // Any weather args match
  search: ["query"],  // Only compare 'query' field
  calculate: (a, b) => {  // Custom logic
    return Math.abs(a.value - b.value) < 0.01;
  },
};
```

--------------------------------

### Python Async Support Example

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Demonstrates how to use async evaluators with AgentEvals. Ensure your OpenAI client is configured for async operations if needed.

```python
from agentevals.trajectory.llm import create_async_trajectory_llm_as_judge

evaluator = create_async_trajectory_llm_as_judge(
    prompt="What is the weather in {inputs}?",
)

result = await evaluator(inputs="San Francisco")
```

```python
from openai import AsyncOpenAI

evaluator = create_async_trajectory_llm_as_judge(
    prompt="What is the weather in {inputs}?",
    judge=AsyncOpenAI(),
    model="o3-mini",
)

result = await evaluator(inputs="San Francisco")
```

--------------------------------

### Few-Shot Examples for LLM Judge (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Provide few-shot examples to an LLM judge evaluator to improve its understanding and scoring consistency. Examples should include inputs, outputs (trajectory), reasoning, and score.

```python
evaluator = create_trajectory_llm_as_judge(
    model="openai:o3-mini",
    few_shot_examples=[
        {
            "inputs": "What is 2+2?",
            "outputs": [...],  # trajectory
            "reasoning": "Good efficiency",
            "score": 1
        }
    ]
)
```

--------------------------------

### Install agentevals Package

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Install the agentevals package using pip. For LLM-as-judge evaluators, ensure an LLM client like OpenAI is configured.

```bash
pip install agentevals
```

--------------------------------

### LLM-as-judge Few-Shot Examples Structure

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Defines the structure for few-shot examples used to guide the LLM judge. Each example includes inputs, outputs, reasoning, and a score.

```typescript
const fewShotExamples = [
  {
    inputs: "What color is the sky?",
    outputs: "The sky is red.",
    reasoning: "The sky is red because it is early evening.",
    score: 1,
  }
];
```

--------------------------------

### LLM-as-judge Few-Shot Examples Structure

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Defines the structure for providing few-shot examples to the LLM-as-judge evaluator, which helps in guiding the model's evaluation process.

```python
few_shot_examples = [
    {
        "inputs": "What color is the sky?",
        "outputs": "The sky is red.",
        "reasoning": "The sky is red because it is early evening.",
        "score": 1,
    }
]
```

--------------------------------

### Trajectory LLM-as-judge Evaluator Output Example (With Reference)

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

An example of the expected output format from the Trajectory LLM-as-judge evaluator when a reference trajectory is provided.

```json
{
    'key': 'trajectory_accuracy',
    'score': True,
    'comment': 'The provided agent trajectory is consistent with the reference. Both trajectories start with the same user query and then correctly invoke a weather lookup through a tool call. Although the reference uses "San Francisco" while the provided trajectory uses "SF" and there is a minor formatting difference (degrees vs. ˚), these differences do not affect the correctness or essential steps of the process. Thus, the score should be: true.'
}
```

--------------------------------

### Trajectory LLM-as-judge Evaluator Output Example

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

An example of the expected output format from the Trajectory LLM-as-judge evaluator when no reference trajectory is provided.

```json
{
    'key': 'trajectory_accuracy',
    'score': True,
    'comment': 'The provided agent trajectory is reasonable...'
}
```

--------------------------------

### Graph-based Trajectory Example (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/README.md

Represents the execution path in a LangGraph system, including inputs, results, and the sequence of nodes visited.

```python
{
    "inputs": [{"query": "..."}],
    "results": [{"answer": "..."}],
    "steps": [["__start__", "agent", "tool", "agent"]]
}
```

--------------------------------

### Setup LangGraph Agent and Extract Trajectory (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/README.md

Demonstrates setting up a LangGraph agent with tools, invoking it, and simulating a human-in-the-loop interaction to capture the graph's execution trajectory.

```python
from agentevals.graph_trajectory.utils import (
    extract_langgraph_trajectory_from_thread,
)
from agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge

from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import MemorySaver
from langgraph.types import Command, interrupt

from langchain_core.tools import tool

@tool
def search(query: str):
    """Call to surf the web."""
    user_answer = interrupt("Tell me the answer to the question.")
    return user_answer

tools = [search]

checkpointer = MemorySaver()
graph = create_react_agent(
    model="gpt-4o-mini",
    checkpointer=checkpointer,
    tools=[search],
)

graph.invoke(
    {"messages": [{"role": "user", "content": "what's the weather in sf?"}]},
    config={"configurable": {"thread_id": "1"}},
)
# Resume the agent with a new command, simulating a human-in-the-loop workflow
graph.invoke(
    Command(resume="It is rainy and 70 degreesנק"),
    config={"configurable": {"thread_id": "1"}},
)

```

--------------------------------

### LangGraph Agent Setup and Trajectory Extraction

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Demonstrates setting up a LangGraph agent with tools and a checkpointer, invoking it, and resuming it with a command to simulate a conversation thread. This is a prerequisite for extracting graph trajectories.

```python
from agentevals.graph_trajectory.utils import (
    extract_langgraph_trajectory_from_thread,
)
from agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge

from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import MemorySaver
from langgraph.types import Command, interrupt

from langchain_core.tools import tool

@tool
def search(query: str):
    """Call to surf the web."""
    user_answer = interrupt("Tell me the answer to the question.")
    return user_answer

tools = [search]

checkpointer = MemorySaver()
graph = create_react_agent(
    model="gpt-4o-mini",
    checkpointer=checkpointer,
    tools=[search],
)

graph.invoke(
    {"messages": [{"role": "user", "content": "what's the weather in sf?"}]},
    config={"configurable": {"thread_id": "1"}},
)
# Resume the agent with a new command, simulating a human-in-the-loop workflow
graph.invoke(
    Command(resume="It is rainy and 70 degreesנק!"),
    config={"configurable": {"thread_id": "1"}},
)

```

--------------------------------

### FewShotExample Structure

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Defines the structure for individual few-shot learning examples. Each example includes optional inputs, required outputs, an optional reasoning, and the expected score (boolean or number).

```typescript
type FewShotExample = {
  inputs?: string | Record<string, unknown>;
  outputs: any;
  reasoning?: string;
  score: boolean | number;
};

type FewShotExamples = FewShotExample[];

```

--------------------------------

### Customizing the Prompt for Graph Trajectory Evaluation

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

This example demonstrates how to customize the prompt for the `GraphTrajectoryLLMAsJudge` evaluator. This allows for more specific grading criteria and instructions for the LLM judge.

```typescript
const CUSTOM_PROMPT = `You are an expert data labeler.
Your task is to grade the accuracy of an AI agent's internal steps in resolving a user queries.

<Rubric>
  An accurate trajectory:
  - Makes logical sense between steps
  - Shows clear progression
  - Is perfectly efficient, with no more than one tool call
  - Is semantically equivalent to the provided reference trajectory, if present
</Rubric>

<Instructions>
  Grade the following thread, evaluating whether the agent's overall steps are logical and relatively efficient.
  For the trajectory, "__start__" denotes an initial entrypoint to the agent, and "__interrupt__" corresponds to the agent
  interrupting to await additional data from another source ("human-in-the-loop"):
</Instructions>

<thread>
{thread}
</thread>

{reference_outputs}
`

const graphTrajectoryEvaluator = createGraphTrajectoryLLMAsJudge({
  prompt: CUSTOM_PROMPT,
  model: "openai:o3-mini",
})
const res = await graphTrajectoryEvaluator({
  inputs: extractedTrajectory.inputs,
  outputs: extractedTrajectory.outputs,
});
```

--------------------------------

### Case-Insensitive Tool Argument Matching

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Example demonstrating how to create a trajectory match evaluator with custom logic for case-insensitive matching of the 'city' argument for the 'get_weather' tool.

```python
import json
from agentevals.trajectory.match import create_trajectory_match_evaluator

outputs = [
    {"role": "user", "content": "What is the weather in SF?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "san francisco"}),
                }
            }
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny."},
]
reference_outputs = [
    {"role": "user", "content": "What is the weather in San Francisco?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "San Francisco"}),
                }
            }
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in San Francisco."},
    {"role": "assistant", "content": "The weather in SF is 80˚ and sunny."},
]

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",
    tool_args_match_mode="exact",  # Default value
    tool_args_match_overrides={
        "get_weather": lambda x, y: x["city"].lower() == y["city"].lower()
    }
)

result = evaluator(
    outputs=outputs, reference_outputs=reference_outputs
)

print(result)
```

```text
{
    'key': 'trajectory_strict_match',
    'score': True,
    'comment': None,
}

```

--------------------------------

### Message-based Trajectory Example (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/README.md

Represents a sequence of messages exchanged in an agent system, including user inputs, assistant responses, and tool calls.

```python
[
    {"role": "user", "content": "What is the weather?"},
    {"role": "assistant", "tool_calls": [...]},
    {"role": "tool", "content": "Sunny"},
    {"role": "assistant", "content": "It is sunny"}
]
```

--------------------------------

### Custom Prompt for LLM as Judge (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Create a custom prompt string to guide the LLM judge for trajectory evaluation. This allows for more specific scoring criteria.

```python
custom_prompt = """Score this trajectory:
{outputs}

Reference:
{reference_outputs}

Rate on scale 1-5."""

evaluator = create_trajectory_llm_as_judge(
    prompt=custom_prompt,
    model="openai:gpt-4"
)
```

--------------------------------

### Create Trajectory LLM Judge with Few-Shot Examples (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/trajectory-llm-evaluators.md

Instantiates a trajectory LLM evaluator with predefined few-shot examples for more nuanced evaluation. Ensure the model identifier and prompt are correctly configured.

```typescript
const evaluator = createTrajectoryLLMAsJudge({
  prompt: TRAJECTORY_ACCURACY_PROMPT,
  model: "openai:o3-mini",
  fewShotExamples: [
    {
      inputs: "What is the weather?",
      outputs: [
        { role: "user", content: "What is the weather?" },
        {
          role: "assistant",
          content: "",
          tool_calls: [{
            function: {
              name: "get_weather",
              arguments: JSON.stringify({ city: "SF" })
            }
          }]
        },
        { role: "tool", content: "Sunny" },
        { role: "assistant", content: "It is sunny" }
      ],
      reasoning: "Efficient trajectory",
      score: 1
    }
  ]
});
```

--------------------------------

### FewShotExample Type

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/types-and-utilities.md

Defines the structure for a single few-shot example used in LLM-as-judge evaluators, including inputs, outputs, reasoning, and score.

```typescript
type FewShotExample = {
  inputs?: string | Record<string, unknown>;
  outputs: any;
  reasoning?: string;
  score: boolean | number;
};
```

--------------------------------

### Unordered Trajectory Match Example

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Use the 'unordered' trajectory match mode to compare tool calls in any order. This is useful when the sequence of tool calls is not critical, but ensuring all required calls are made is important. Ensure the `create_trajectory_match_evaluator` function is imported.

```python
import json
from agentevals.trajectory.match import create_trajectory_match_evaluator

inputs = {}
outputs = [
    {"role": "user", "content": "What is the weather in SF and is there anything fun happening?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [{
            "function": {
                "name": "get_weather",
                "arguments": json.dumps({"city": "San Francisco"}),
            }
        }],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [{
            "function": {
                "name": "get_fun_activities",
                "arguments": json.dumps({"city": "San Francisco"}),
            }
        }],
    },
    {"role": "tool", "content": "Nothing fun is happening, you should stay indoors and read!"},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny, but there is nothing fun happening."},
]
reference_outputs = [
    {"role": "user", "content": "What is the weather in SF and is there anything fun happening?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_fun_activities",
                    "arguments": json.dumps({"city": "San Francisco"}),
                }
            },
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "San Francisco"}),
                }
            },
        ],
    },
    {"role": "tool", "content": "Nothing fun is happening, you should stay indoors and read!"},
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "In SF, it's 80˚ and sunny, but there is nothing fun happening."},
]

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="unordered"
)

result = evaluator(
    outputs=outputs, reference_outputs=reference_outputs
)

print(result)
```

```text
{
    'key': 'trajectory_unordered_match',
    'score': True,
    'comment': None,
}
```

--------------------------------

### Custom Prompt Example for Trajectory Evaluation (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/trajectory-llm-evaluators.md

Demonstrates how to create a custom prompt for trajectory evaluation by defining a template string with variables like {task}, {outputs}, and {reference_outputs}. This custom prompt is then used with `createTrajectoryLLMAsJudge`.

```typescript
const customPrompt = `You are a trajectory evaluator.
Task: {task}
Trajectory:
{outputs}
Reference:
{reference_outputs}

Grade on scale 0-10.`;

const evaluator = createTrajectoryLLMAsJudge({
  prompt: customPrompt,
  model: "openai:gpt-4"
});

const result = await evaluator({
  outputs,
  referenceOutputs,
  task: "Find the shortest path"
});
```

--------------------------------

### Custom Tool Argument Matching for Trajectory Evaluation

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/README.md

Customize how tool arguments are matched during trajectory evaluation by providing a lambda function for specific tools. This example overrides matching for the 'get_weather' tool.

```python
evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",
    tool_args_match_overrides={
        "get_weather": lambda out, ref: out["city"].lower() == ref["city"].lower()
    }
)
```

--------------------------------

### Configure LLM Evaluator with Python

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Use `create_trajectory_llm_as_judge` to set up an evaluator with a custom prompt, model, scoring choices, reasoning, and few-shot examples. This is useful for defining specific evaluation criteria for agent trajectories.

```python
evaluator = create_trajectory_llm_as_judge(
    prompt="""Evaluate efficiency:
{outputs}

Reference:
{reference_outputs}

{custom}   """,
    
    model="openai:gpt-4",
    
    continuous=True,
    choices=[0, 2, 4, 6, 8, 10],
    
    use_reasoning=True,
    feedback_key="efficiency_score",
    
    few_shot_examples=[
        {
            "inputs": "Navigate to A then B",
            "outputs": ["Start", "A", "B"],
            "reasoning": "Direct path",
            "score": 8
        }
    ]
)

result = evaluator(
    outputs=outputs,
    reference_outputs=reference,
    custom="Consider path length."
)

print(result)
# {'key': 'efficiency_score', 'score': 8.5, 'comment': '...'}
```

--------------------------------

### Evaluate LangGraph Trajectories

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/00-START-HERE.md

This example shows how to extract a trajectory from a LangGraph thread and then evaluate it using an LLM judge. It requires importing utility functions for extraction and evaluation.

```python
from agentevals.graph_trajectory.utils import extract_langgraph_trajectory_from_thread
from agentevals.graph_trajectory.llm import create_graph_trajectory_llm_as_judge

extracted = extract_langgraph_trajectory_from_thread(graph, config)
evaluator = create_graph_trajectory_llm_as_judge(model="openai:gpt-4")
result = evaluator(inputs=extracted["inputs"], outputs=extracted["outputs"])
```

--------------------------------

### Evaluate Agent Trajectory with LLM-as-judge

Source: https://github.com/langchain-ai/agentevals/blob/main/README.md

This example demonstrates how to use the `evaluator` to compare an agent's output trajectory against a reference trajectory. It requires defining both `outputs` and `referenceOutputs` as arrays of messages.

```typescript
const outputs = [
  {role: "user", content: "What is the weather in SF?"},
  {
    role: "assistant",
    content: "",
    tool_calls: [
      {
        function: {
          name: "get_weather",
          arguments: JSON.stringify({ city: "San Francisco" }),
        }
      }
    ],
  },
  {role: "tool", content: "It's 80 degrees and sunny in SF."},
  {role: "assistant", content: "The weather in SF is 80 degrees and sunny."},
] satisfies FlexibleChatCompletionMessage[];

const referenceOutputs = [
  {role: "user", content: "What is the weather in SF?"},
  {
    role: "assistant",
    content: "",
    tool_calls: [
      {
        function: {
          name: "get_weather",
          arguments: JSON.stringify({ city: "San Francisco" }),
        }
      }
    ],
  },
  {role: "tool", content: "It's 80 degrees and sunny in San Francisco."},
  {role: "assistant", content: "The weather in SF is 80˚ and sunny."},
] satisfies FlexibleChatCompletionMessage[];

const result = await evaluator({
  outputs,
  referenceOutputs,
});

console.log(result)
```

--------------------------------

### ToolArgsMatchOverrides: String Mode

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Example of customizing tool argument matching using a string to specify the match mode for a specific tool. This allows for per-tool argument comparison strategies.

```typescript
{ "tool_name": "ignore" }
```

--------------------------------

### Detailed Continuous Scoring with LLM Judge

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Configure for detailed, continuous scoring that includes reasoning. This setup is useful for nuanced evaluations requiring explanations and a range of scores.

```python
create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,
    model="openai:gpt-4",
    continuous=True,
    choices=[0, 0.25, 0.5, 0.75, 1.0],
    use_reasoning=True,
    few_shot_examples=[...]  # Improve consistency
)
```

--------------------------------

### Get Tool Argument Matcher (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/types-and-utilities.md

Python equivalent for obtaining a tool argument matching function. It considers the tool call name, argument matching mode, and optional overrides.

```python
def _get_matcher_for_tool_name(
    tool_call_name: str,
    tool_args_match_mode: ToolArgsMatchMode,
    tool_args_match_overrides: Optional[ToolArgsMatchOverrides] = None
) -> Callable[[dict, dict], bool]
```

--------------------------------

### Configure LLM Evaluator with TypeScript

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Use `createTrajectoryLLMAsJudge` to set up an evaluator with a custom prompt, model, scoring choices, reasoning, and few-shot examples. This is useful for defining specific evaluation criteria for agent trajectories.

```typescript
const evaluator = createTrajectoryLLMAsJudge({
  prompt: `You are evaluating agent performance.
Trajectory:
{outputs}

Reference trajectory:
{reference_outputs}

{custom_instruction}

Rate on scale 0-10 for efficiency.`,
  
  model: "openai:gpt-4",
  
  continuous: true,
  choices: [0, 2, 4, 6, 8, 10],
  
  useReasoning: true,
  feedbackKey: "efficiency_score",
  
  fewShotExamples: [
    {
      inputs: "Navigate to A then B",
      outputs: ["Start", "Go to A", "Go to B"],
      reasoning: "Direct path",
      score: 8
    },
    {
      inputs: "Navigate to A then B",
      outputs: ["Start", "Go to C", "Go to A", "Go to B"],
      reasoning: "Unnecessary detour",
      score: 3
    }
  ]
});

const result = await evaluator({
  outputs,
  referenceOutputs,
  custom_instruction: "Consider path length."
});

console.log(result);
// { key: 'efficiency_score', score: 8.5, comment: '...' }
```

--------------------------------

### Import Built-in Trajectory Prompts (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/trajectory-llm-evaluators.md

Import the predefined trajectory accuracy prompts for use in evaluators.

```python
from agentevals.trajectory.llm import (
    TRAJECTORY_ACCURACY_PROMPT,
    TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE
)
```

--------------------------------

### Fast Binary Evaluation with LLM Judge

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Use this configuration for quick, binary evaluations where reasoning is not required. It prioritizes speed by disabling explanation generation.

```python
create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="openai:gpt-4o-mini",
    continuous=False,
    use_reasoning=False  # Faster, no explanation
)
```

--------------------------------

### Async Evaluator Creation in Python

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/README.md

Shows how to create and use an asynchronous evaluator in Python using the `create_async_*` prefix. Specify the desired model when creating the evaluator.

```python
evaluator = create_async_trajectory_llm_as_judge(model="openai:o3-mini")
result = await evaluator(outputs=outputs)
```

--------------------------------

### OpenAI Model Identifier

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

Use this format to specify OpenAI models. Examples include gpt-4, o3-mini, and gpt-4o.

```text
"openai:{model_name}"
```

--------------------------------

### Trajectory Utility Functions

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

Internal utility functions for trajectory matching, such as normalizing messages and getting tool name matchers.

```python
_normalize_to_openai_messages_list(...)
_get_matcher_for_tool_name(...)
```

--------------------------------

### Create and Run Trajectory Evaluator (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Demonstrates how to create a trajectory evaluator using OpenAI's o3-mini model and run it with a sample agent output. The evaluator returns a score and a comment.

```typescript
import {
  createTrajectoryLLMAsJudge,
  type FlexibleChatCompletionMessage,
  TRAJECTORY_ACCURACY_PROMPT,
} from "agentevals";

const trajectoryEvaluator = createTrajectoryLLMAsJudge({
  prompt: TRAJECTORY_ACCURACY_PROMPT,
  model: "openai:o3-mini",
});

const outputs = [
  { role: "user", content: "What is the weather in SF?" },
  {
    role: "assistant",
    content: "",
    tool_calls: [
      {
        function: {
          name: "get_weather",
          arguments: JSON.stringify({ city: "SF" }),
        },
      },
    ],
  },
  { role: "tool", content: "It's 80 degrees and sunny in SF." },
  {
    role: "assistant",
    content: "The weather in SF is 80 degrees and sunny.",
  },
] satisfies FlexibleChatCompletionMessage[];

const evalResult = await trajectoryEvaluator({
  outputs,
});

console.log(evalResult);
```

```json
{
    key: 'trajectory_accuracy',
    score: true,
    comment: '...'
}
```

--------------------------------

### Python Unordered Trajectory Match

Source: https://github.com/langchain-ai/agentevals/blob/main/README.md

Use this snippet to evaluate if two trajectories contain the same tool calls in any order. Ensure the 'agentevals' library is installed.

```python
import json
from agentevals.trajectory.match import create_trajectory_match_evaluator

inputs = {}
outputs = [
    {"role": "user", "content": "What is the weather in SF and is there anything fun happening?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [{"function": {"name": "get_weather", "arguments": json.dumps({"city": "San Francisco"})}}],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [{"function": {"name": "get_fun_activities", "arguments": json.dumps({"city": "San Francisco"})}}],
    },
    {"role": "tool", "content": "Nothing fun is happening, you should stay indoors and read!"},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny, but there is nothing fun happening."},
]
reference_outputs = [
    {"role": "user", "content": "What is the weather in SF and is there anything fun happening?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {"function": {"name": "get_fun_activities", "arguments": json.dumps({"city": "San Francisco"})}},
            {"function": {"name": "get_weather", "arguments": json.dumps({"city": "San Francisco"})}},
        ],
    },
    {"role": "tool", "content": "Nothing fun is happening, you should stay indoors and read!"},
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "In SF, it's 80˚ and sunny, but there is nothing fun happening."},
]

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="unordered"
)

result = evaluator(
    outputs=outputs, reference_outputs=reference_outputs
)

print(result)
```

```text
{
    'key': 'trajectory_unordered_match',
    'score': True,
    'comment': None,
}
```

--------------------------------

### ToolArgsMatchOverrides: Array Mode

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Example of customizing tool argument matching by providing an array of field names. Only the specified fields will be compared for the given tool.

```typescript
{ "tool_name": ["field1", "field2"] }
```

--------------------------------

### Setting up and Extracting LangGraph Trajectory

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

This snippet demonstrates how to set up a LangGraph agent, invoke it, and then extract the execution trajectory using `extractLangGraphTrajectoryFromThread`. This is a prerequisite for using the evaluator.

```typescript
import { tool } from "@langchain/core/tools";
import { ChatOpenAI } from "@langchain/openai";
import { createReactAgent } from "@langchain/langgraph/prebuilt";
import { MemorySaver, interrupt } from "@langchain/langgraph";
import { z } from "zod";
import { extractLangGraphTrajectoryFromThread } from "agentevals";

const search = tool((_): string => {
  const userAnswer = interrupt("Tell me the answer to the question.")
  return userAnswer;
}, {
  name: "search",
  description: "Call to surf the web.",
  schema: z.object({
      query: z.string()
  })
})

const tools = [search];

// Create a checkpointer
const checkpointer = new MemorySaver();

// Create the React agent
const graph = createReactAgent({
  llm: new ChatOpenAI({ model: "gpt-4o-mini" }),
  tools,
  checkpointer,
});

// Invoke the graph with initial message
await graph.invoke(
  { messages: [{ role: "user", content: "what's the weather in sf?" }] },
  { configurable: { thread_id: "1" } }
);

// Resume the agent with a new command (simulating human-in-the-loop)
await graph.invoke(
  { messages: [{ role: "user", content: "It is rainy and 70 degrees!" }] },
  { configurable: { thread_id: "1" } }
);

const extractedTrajectory = await extractLangGraphTrajectoryFromThread(
  graph,
  { configurable: { thread_id: "1" } },
);

console.log(extractedTrajectory);
```

--------------------------------

### Create Trajectory LLM-as-judge Evaluator (With Reference)

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Initializes the LLM-as-judge evaluator for trajectories, incorporating a reference trajectory for comparison. Uses the TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE for evaluation.

```python
import json
from agentevals.trajectory.llm import create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE

evaluator = create_trajectory_llm_as_judge(
  prompt=TRAJECTORY_ACCURACY_PROMPT_WITH_REFERENCE,
  model="openai:o3-mini"
)
outputs = [
    {"role": "user", "content": "What is the weather in SF?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "SF"}),
                }
            }
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF."},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny."},
]
reference_outputs = [
    {"role": "user", "content": "What is the weather in SF?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "San Francisco"}),
                }
            }
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in San Francisco."},
    {"role": "assistant", "content": "The weather in SF is 80˚ and sunny."},
]
eval_result = evaluator(
    outputs=outputs,
    reference_outputs=reference_outputs,
)

print(eval_result)
```

--------------------------------

### ToolArgsMatchOverrides: Function Mode

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Example of customizing tool argument matching using a custom comparator function. This provides maximum flexibility for complex argument comparisons.

```typescript
{
  "tool_name": (output, reference) => {
    return output.value === reference.value;
  }
}
```

--------------------------------

### Use Async Evaluators in Python

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

Utilize the `create_async_*` factory prefix for async versions of evaluators in Python. These follow asyncio conventions with `async def` and `await` support.

```python
from agentevals.trajectory.llm import create_async_trajectory_llm_as_judge

evaluator = create_async_trajectory_llm_as_judge(model="openai:o3-mini")
result = await evaluator(outputs=outputs)
```

--------------------------------

### Migrate Legacy Tool Matching

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Replace the legacy `tool_call_args_exact_match` boolean flag with the more flexible `tool_args_match_mode` and `tool_args_match_overrides` for tool argument matching.

```python
# Old
tool_call_args_exact_match=True/False

# New
tool_args_match_mode="exact"  # or "ignore"
tool_args_match_overrides={...}  # for custom per-tool logic
```

--------------------------------

### Trajectory LLM-as-judge Evaluation Result with Reference

Source: https://github.com/langchain-ai/agentevals/blob/main/js/README.md

Example output format for the Trajectory LLM-as-judge evaluator when a reference trajectory is provided. The comment field details the comparison between the output and reference trajectories.

```python
{
    'key': 'trajectory_accuracy',
    'score': true,
    'comment': 'The provided agent trajectory is consistent with the reference. Both trajectories start with the same user query and then correctly invoke a weather lookup through a tool call. Although the reference uses "San Francisco" while the provided trajectory uses "SF" and there is a minor formatting difference (degrees vs. ˚), these differences do not affect the correctness or essential steps of the process. Thus, the score should be: true.'
}
```

--------------------------------

### Subset and Superset Trajectory Matching

Source: https://github.com/langchain-ai/agentevals/blob/main/python/README.md

Use 'superset' mode to ensure key tools were called, accepting extra tools. Use 'subset' mode to ensure no unexpected tools were called. Configure tool call equality checks in a separate section.

```python
import json
from agentevals.trajectory.match import create_trajectory_match_evaluator

outputs = [
    {"role": "user", "content": "What is the weather in SF and London?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [{"function": {"name": "get_weather", "arguments": json.dumps({"city": "SF and London"})}},
                       {"function": {"name": "accuweather_forecast", "arguments": json.dumps({"city": "SF and London"})}}],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in SF, and 90 degrees and rainy in London."},
    {"role": "tool", "content": "Unknown."},
    {"role": "assistant", "content": "The weather in SF is 80 degrees and sunny. In London, it's 90 degrees and rainy."},
]
reference_outputs = [
    {"role": "user", "content": "What is the weather in SF and London?"},
    {
        "role": "assistant",
        "content": "",
        "tool_calls": [
            {
                "function": {
                    "name": "get_weather",
                    "arguments": json.dumps({"city": "SF and London"})
                }
            },
        ],
    },
    {"role": "tool", "content": "It's 80 degrees and sunny in San Francisco, and 90 degrees and rainy in London."},
    {"role": "assistant", "content": "The weather in SF is 80˚ and sunny. In London, it's 90˚ and rainy."},
]

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="superset", # or "subset"
)

result = evaluator(
    outputs=outputs, reference_outputs=reference_outputs
)

print(result)
```

```json
{
    'key': 'trajectory_superset_match',
    'score': True,
    'comment': None,
}
```

--------------------------------

### Create Trajectory Match Evaluator (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/INDEX.md

Use this to create a trajectory match evaluator in Python. It compares actual trajectories against known good execution paths.

```python
from agentevals.trajectory_match import create_trajectory_match_evaluator

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",
    tool_args_match_mode="exact",
)
```

--------------------------------

### Create Trajectory LLM-as-judge Evaluator (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/README.md

Initializes the LLM-as-judge evaluator for trajectories in TypeScript, using a specified prompt and model. This setup is suitable for evaluating agent performance without a reference.

```typescript
import {
  createTrajectoryLLMAsJudge,
  TRAJECTORY_ACCURACY_PROMPT,
  type FlexibleChatCompletionMessage,
} from "agentevals";

const evaluator = createTrajectoryLLMAsJudge({
  prompt: TRAJECTORY_ACCURACY_PROMPT,
  model: "openai:o3-mini",
});

const outputs = [
  {role: "user", content: "What is the weather in SF?"},
  {
    role: "assistant",
    content: "",
    tool_calls: [
      {
        function: {
          name: "get_weather",
          arguments: JSON.stringify({ city: "SF" }),
        }
      }
    ],
  },
  {role: "tool", content: "It's 80 degrees and sunny in SF."},
  {role: "assistant", content: "The weather in SF is 80 degrees and sunny."},
] satisfies FlexibleChatCompletionMessage[];

const result = await evaluator({ outputs });

console.log(result)
```

--------------------------------

### Create Async Trajectory Match Evaluator

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

Instantiate an asynchronous trajectory match evaluator. This is the async counterpart to `create_trajectory_match_evaluator`.

```python
create_async_trajectory_match_evaluator(...) -> SimpleAsyncEvaluator
```

--------------------------------

### Model Identifiers for OpenAI and LangChain Integrations

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/README.md

Use these string identifiers to specify models for evaluators. Examples include various OpenAI models and integrations with other providers like Anthropic and Google GenAI.

```python
# OpenAI models
"openai:gpt-4"
"openai:gpt-4-turbo"
"openai:gpt-4o"
"openai:o3-mini"

# LangChain integrations (see documentation)
"anthropic:claude-3-5-sonnet-20241022"
"google_genai:gemini-2.0-flash"
```

--------------------------------

### Import Agentevals Utility Functions

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

Import utility functions for running evaluators, including synchronous and asynchronous versions.

```python
from agentevals.utils import (
    _run_evaluator,
    _arun_evaluator,
)
```

--------------------------------

### Default Reference Comparison Prompt

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/module-exports.md

The default prompt used for comparing reference graph trajectories.

```python
DEFAULT_REF_COMPARE_PROMPT: str
```

--------------------------------

### Custom Tool Argument Matching (Python)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/quick-reference.md

Define a custom Python function to match specific tool arguments, like ignoring case for city names. This is useful when default argument matching is too strict.

```python
# Python
def custom_matcher(output, reference):
    return output["city"].lower() == reference["city"].lower()

evaluator = create_trajectory_match_evaluator(
    trajectory_match_mode="strict",
    tool_args_match_overrides={
        "get_weather": custom_matcher
    }
)
```

--------------------------------

### Custom Evaluation Logic with LLM Judge

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/evaluator-factory-options.md

Implement custom evaluation logic by providing your own OpenAI client and specifying feedback keys. This allows for tailored evaluation workflows.

```python
create_trajectory_llm_as_judge(
    prompt=custom_prompt_template,
    judge=openai.AsyncOpenAI(),  # Use your own client
    model="gpt-4",  # Model name for the client
    continuous=True,
    feedback_key="my_custom_eval"
)
```

--------------------------------

### Get Tool Argument Matcher (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/types-and-utilities.md

Retrieves a matching function for a tool's arguments based on the tool name, matching mode, and optional overrides. It prioritizes per-tool overrides over the global matching mode.

```typescript
function _getMatcherForToolName(
  toolCallName: string,
  toolArgsMatchMode: ToolArgsMatchMode,
  toolArgsMatchOverrides?: ToolArgsMatchOverrides
): ToolArgsMatcher
```

--------------------------------

### Create and Use Trajectory LLM Judge in TypeScript

Source: https://github.com/langchain-ai/agentevals/blob/main/README.md

Shows how to set up a trajectory LLM judge in TypeScript using `createTrajectoryLLMAsJudge` and integrate it with LangSmith's `evaluate` function. This example includes a sample conversation structure for evaluation.

```typescript
import { evaluate } from "langsmith/evaluation";
import { createTrajectoryLLMAsJudge, TRAJECTORY_ACCURACY_PROMPT } from "agentevals";

const trajectoryEvaluator = createTrajectoryLLMAsJudge({
  model: "openai:o3-mini",
  prompt: TRAJECTORY_ACCURACY_PROMPT
});

await evaluate(
  (inputs) => [
      {role: "user", content: "What is the weather in SF?"},
      {
          role: "assistant",
          content: "",
          tool_calls: [
              {
                  function: {
                      name: "get_weather",
                      arguments: json.dumps({"city": "SF"}),
                  }
              }
          ],
      },
      {role: "tool", content: "It's 80 degrees and sunny in SF."},
      {role: "assistant", content: "The weather in SF is 80 degrees and sunny."},
    ],
  {
    data: datasetName,
    evaluators: [trajectoryEvaluator],
  }
);
```

--------------------------------

### Create Trajectory LLM-as-Judge with Custom Prompt (TypeScript)

Source: https://github.com/langchain-ai/agentevals/blob/main/_autodocs/trajectory-llm-evaluators.md

Demonstrates creating a trajectory LLM-as-judge evaluator with a custom prompt focused on efficiency and specific scoring choices. This allows for tailored evaluation criteria.

```typescript
const customPrompt = `You are evaluating an agent trajectory.
Score the trajectory from 0-10 based on efficiency.

<trajectory>
{outputs}
</trajectory>`;

const evaluator = createTrajectoryLLMAsJudge({
  prompt: customPrompt,
  model: "openai:gpt-4",
  continuous: true,
  choices: [0, 2, 4, 6, 8, 10]
});
```