Open Evals

https://open-evals.com/docs
Admin
Open Evals is an open-source evaluation framework for testing LLM applications with built-in...
Tokens:22,165
Snippets:259
Trust Score:1.7
Update:5 months ago
Show doc for...
Tokens:
Raw
### Basic Evaluation Setup

Source: https://open-evals.com/docs/core/evaluate

Import necessary components and set up a basic evaluation with a dataset and metrics. This is the starting point for most evaluations.

```typescript
import { evaluate, EvaluationDataset } from '@open-evals/core'
import { Faithfulness } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

const dataset = new EvaluationDataset([
  /* samples */
])
const metrics = [new Faithfulness({ model: openai('gpt-4o-mini') })]

const results = await evaluate(dataset, metrics)
console.log(results.statistics.averages) // { faithfulness: 0.92 }
```

--------------------------------

### Install @open-evals/metrics and Evalite

Source: https://open-evals.com/docs/metrics/evalite

Install the necessary packages using npm, pnpm, yarn, or bun.

```bash
npm i @open-evals/metrics evalite
```

--------------------------------

### Install @open-evals/metrics

Source: https://open-evals.com/docs/metrics

Install the package using npm.

```bash
npm install @open-evals/metrics
```

--------------------------------

### Sampling and Splitting Examples

Source: https://open-evals.com/docs/core/dataset

Examples demonstrating how to create subsets, split, shuffle, and slice datasets.

```APIDOC
## Sampling and Splitting

Datasets provide convenient methods for creating subsets, essential for ML workflows:

```javascript
// Get a random subset - perfect for quick testing
const testSubset = dataset.sample(10)

// Split into train/test sets - ratio determines the first set size
const [train, test] = dataset.split(0.8) // 80% train, 20% test

// For train/val/test splits, split twice
const [trainVal, test] = dataset.split(0.9)
const [train, val] = trainVal.split(0.89) // Results in ~80/10/10

// Shuffle for randomization
const shuffled = dataset.shuffle()

// Slice for specific ranges
const first100 = dataset.slice(0, 100)
const samples20to30 = dataset.slice(20, 30)
```
```

--------------------------------

### Persistence Examples

Source: https://open-evals.com/docs/core/dataset

Examples showing how to save and load datasets using JSON and JSONL formats.

```APIDOC
## Persistence

Save and load datasets using JSON or JSONL format. JSONL (one JSON object per line) is recommended for large datasets as it's more memory-efficient and easier to stream:

```javascript
import { writeFile, readFile } from 'fs/promises'

// Save as JSONL (recommended)
const jsonl = dataset.toJSONL()
await writeFile('dataset.jsonl', jsonl)

// Load from JSONL
const content = await readFile('dataset.jsonl', 'utf-8')
const loaded = EvaluationDataset.fromJSONL(content)

// Or use JSON for smaller datasets
await writeFile('dataset.json', dataset.toJSON())
const loaded2 = EvaluationDataset.fromJSON(
  await readFile('dataset.json', 'utf-8')
)
```
```

--------------------------------

### Install Faithfulness Metric

Source: https://open-evals.com/docs/metrics/faithfulness

Install the @open-evals/metrics package using npm.

```bash
npm i @open-evals/metrics
```

--------------------------------

### Install Open-Evals Core Packages

Source: https://open-evals.com/docs/core/quickstart

Install the necessary packages for @open-evals/core, including metrics and the OpenAI SDK.

```bash
npm install @open-evals/core @open-evals/metrics @ai-sdk/openai
```

--------------------------------

### Install Open-Evals Generator

Source: https://open-evals.com/docs/generator/quickstart

Install the necessary packages for @open-evals/generator, @open-evals/rag, and @ai-sdk/openai using npm.

```bash
npm install @open-evals/generator @open-evals/rag @ai-sdk/openai
```

--------------------------------

### Example Multi-Hop Abstract Questions

Source: https://open-evals.com/docs/generator/synthesis/multi-hop-abstract

These examples illustrate the conceptual and multi-source nature of questions generated by the multi-hop abstract synthesizer, applicable to different documentation domains.

```text
// TypeScript documentation
"How does TypeScript's type system improve code quality?"
"What are the tradeoffs between different typing approaches?"
"How do generics relate to type safety and reusability?"

// Architecture documentation
"How do microservices compare to monolithic architectures?"
"What factors influence database selection for a project?"
"How does caching improve system performance?"
```

--------------------------------

### Example Multi-Hop Specific Questions

Source: https://open-evals.com/docs/generator/synthesis/multi-hop-specific

These examples illustrate the detailed and multi-source nature of questions generated by the multi-hop specific synthesizer, covering technical scenarios from documentation.

```text
// TypeScript documentation
'How would you implement a type-safe event emitter using TypeScript generics and strict null checks?'
"What's the correct way to type a React component that accepts either a string or number prop and renders differently based on the type?"

// API documentation
'How do you implement pagination with cursor-based pagination while maintaining sort order and filtering by multiple fields?'
"What's the proper error handling strategy when making parallel API calls with different retry policies?"
```

--------------------------------

### Install Generator and RAG Packages

Source: https://open-evals.com/docs/generator

Install the necessary packages for the generator and RAG functionalities using npm.

```bash
npm install @open-evals/generator @open-evals/rag
```

--------------------------------

### Basic Example: Generate Test Data

Source: https://open-evals.com/docs/generator/quickstart

A complete example demonstrating the generation of a synthetic test dataset. It includes creating document nodes, building a knowledge graph, generating user personas, and synthesizing test samples with ground truth.

```typescript
import {
  graph,
  DocumentNode,
  chunk,
  embed,
  relationship,
  generatePersonas,
  synthesize,
  createSynthesizer,
  transform,
} from '@open-evals/generator'
import { RecursiveCharacterSplitter } from '@open-evals/rag'
import { openai } from '@ai-sdk/openai'

// 1. Create documents from your content
const documents = [new DocumentNode('typescript-guide.md', fileContent, {})]

// 2. Build knowledge graph with transforms
const knowledgeGraph = await transform(graph(documents))
  .pipe(chunk(new RecursiveCharacterSplitter({ chunkSize: 512 })))
  .pipe(embed(openai.embedding('text-embedding-3-small')))
  .pipe(relationship())
  .apply()

// 3. Generate diverse user personas
const personas = await generatePersonas(knowledgeGraph, openai.chat('gpt-4o'), {
  count: 3,
})

// 4. Synthesize test samples
const testSamples = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [
    [createSynthesizer(openai.chat('gpt-4o'), 'single-hop-specific'), 20],
  ],
  personas,
  count: 5,
  config: { generateGroundTruth: true },
})

console.log(`Generated ${testSamples.length} test samples`)
```

--------------------------------

### Evaluate Test Samples with Metrics

Source: https://open-evals.com/docs/generator/quickstart

Use the core evaluation package to assess generated test samples against specified metrics. Ensure '@open-evals/core' and '@open-evals/metrics' are installed.

```javascript
import { evaluate } from '@open-evals/core'
import { Faithfulness } from '@open-evals/metrics'

const metric = new Faithfulness({ model: openai('gpt-4o-mini') })
const results = await evaluate(testSamples, [metric])

console.log('Average faithfulness:', results.statistics.averages.faithfulness)
```

--------------------------------

### Example Generated Personas

Source: https://open-evals.com/docs/generator/synthesis/personas

This is an example of the structure of personas generated by the `generatePersonas` function. Each persona includes a name and a description.

```json
;[
  {
    name: 'Junior Developer',
    description:
      'New to TypeScript, learning basic syntax and type system features...',
  },
  {
    name: 'Senior Architect',
    description:
      'Experienced developer evaluating TypeScript for large-scale applications...',
  },
  {
    name: 'Technical Writer',
    description:
      'Creating documentation and tutorials, needs clear explanations...',
  },
]
```

--------------------------------

### Sampling and Splitting Datasets

Source: https://open-evals.com/docs/core/dataset

Use these methods to create subsets for ML workflows. `sample` gets a random subset, `split` divides into train/test sets, `shuffle` randomizes order, and `slice` extracts specific ranges.

```javascript
// Get a random subset - perfect for quick testing
const testSubset = dataset.sample(10)
```

```javascript
// Split into train/test sets - ratio determines the first set size
const [train, test] = dataset.split(0.8) // 80% train, 20% test
```

```javascript
// For train/val/test splits, split twice
const [trainVal, test] = dataset.split(0.9)
const [train, val] = trainVal.split(0.89) // Results in ~80/10/10
```

```javascript
// Shuffle for randomization
const shuffled = dataset.shuffle()
```

```javascript
// Slice for specific ranges
const first100 = dataset.slice(0, 100)
const samples20to30 = dataset.slice(20, 30)
```

--------------------------------

### Example Single-Hop Specific Questions

Source: https://open-evals.com/docs/generator/synthesis/single-hop-specific

Illustrates the type of factual and direct questions generated by the single-hop specific synthesizer, suitable for testing basic retrieval and factual accuracy.

```plaintext
// TypeScript documentation
'What is the syntax for optional properties in TypeScript?'
'How do you declare a string array in TypeScript?'
'What does the readonly modifier do in TypeScript?'

// API documentation
'What HTTP method does the /users endpoint accept?'
'What is the rate limit for the search API?'
'What format does the API return error messages in?'
```

--------------------------------

### Basic Evalite Integration with a Single Metric

Source: https://open-evals.com/docs/metrics/evalite

Convert a single metric to an Evalite scorer using `toEvaliteScorer`. This example demonstrates integrating the Faithfulness metric.

```typescript
import { evalite } from 'evalite'
import { Faithfulness, toEvaliteScorer } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

evalite('RAG Evaluation', {
  data: async () => [
    {
      input: {
        query: 'What is TypeScript?',
        retrievedContexts: ['TypeScript adds static typing to JavaScript.'],
      },
    },
  ],
  task: async (sample) => {
    // Your LLM task implementation
    return 'TypeScript is a typed superset of JavaScript.'
  },
  scorers: [
    toEvaliteScorer(
      new Faithfulness({
        model: openai('gpt-4.1-mini'),
      })
    ),
  ],
})
```

--------------------------------

### Evalite Integration with Multiple Metrics

Source: https://open-evals.com/docs/metrics/evalite

Integrate multiple metrics into Evalite by mapping each metric to a scorer using `toEvaliteScorer`. This example includes Faithfulness, AnswerSimilarity, and ContextRecall.

```typescript
import { evalite } from 'evalite'
import { generateText } from 'ai'
import { openai } from '@ai-sdk/openai'
import {
  Faithfulness,
  AnswerSimilarity,
  ContextRecall,
  toEvaliteScorer,
} from '@open-evals/metrics'

evalite('RAG Evaluation', {
  data: async () => [
    {
      input: {
        query: 'What is TypeScript?',
        retrievedContexts: ['TypeScript adds static typing to JavaScript.'],
        reference: 'TypeScript is a typed superset of JavaScript.',
      },
    },
  ],

  task: async (sample) => {
    const result = await generateText({
      model: openai('gpt-4.1'),
      prompt: sample.query,
    })
    return result.text
  },

  scorers: [
    toEvaliteScorer(new Faithfulness({ model: openai('gpt-4.1-mini') })),
    toEvaliteScorer(
      new AnswerSimilarity({ model: openai('text-embedding-3-small') })
    ),
    toEvaliteScorer(new ContextRecall({ model: openai('gpt-4.1-mini') })),
  ],
})
```

--------------------------------

### Control Concurrency for Test Synthesis

Source: https://open-evals.com/docs/generator/synthesis/configuration

Balance generation speed against API rate limits and costs by setting the concurrency level. A concurrency of 3 is used in this example.

```javascript
await synthesize({
  graph,
  synthesizers,
  personas,
  count: 100,
  config: { concurrency: 3 },
})
```

--------------------------------

### Get Random Dataset Subset

Source: https://open-evals.com/docs/core/quickstart

Obtain a random subset of samples from an `EvaluationDataset` using the `sample` method. Useful for quick testing.

```typescript
// Get a random subset for testing
const sample = dataset.sample(10)
```

--------------------------------

### Sample Output Structure for Single-Hop Questions

Source: https://open-evals.com/docs/generator/synthesis/single-hop-specific

Demonstrates the structure of the output generated by the synthesizer, including the query, reference answer, retrieved contexts, and metadata.

```json
{
  query: "What is the syntax for optional properties in TypeScript?",
  reference: "Optional properties in TypeScript use the ? modifier after the property name. For example: `interface User { name: string; age?: number; }` makes age optional.",
  retrievedContexts: [
    "TypeScript provides several ways to mark properties as optional. The ? modifier after a property name indicates it may be undefined..."
  ],
  metadata: {
    persona: "Junior Developer",
    queryType: "specific",
    queryLength: "medium",
    queryStyle: "conversational"
  }
}
```

--------------------------------

### Step 1: Create Document Nodes

Source: https://open-evals.com/docs/generator/quickstart

Initialize DocumentNodes to wrap your raw content with essential metadata like ID, content, and custom attributes.

```typescript
const doc = new DocumentNode(
  'my-doc.md', // ID/filename
  'Content here...', // The actual text
  {
    // Custom metadata
    category: 'tutorial',
    author: 'team',
  }
)
```

--------------------------------

### Configure Sample Distribution by Synthesizer

Source: https://open-evals.com/docs/generator/synthesis/configuration

Control the mix of query types by specifying weights for each synthesizer. Percentages do not need to sum to 100; numbers represent weights.

```javascript
const samples = await synthesize({
  graph,
  synthesizers: [
    [createSynthesizer(llm, 'single-hop-specific'), 60], // 60% simple
    [createSynthesizer(llm, 'multi-hop-abstract'), 30], // 30% abstract
    [createSynthesizer(llm, 'multi-hop-specific'), 10], // 10% complex
  ],
  personas,
  count: 100,
})
```

--------------------------------

### Initialize Faithfulness and FactualCorrectness Metrics

Source: https://open-evals.com/docs/metrics

Import and initialize metrics for evaluating RAG systems and Q&A applications. Specify the LLM model to be used for evaluation.

```typescript
import { Faithfulness, FactualCorrectness } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

// For RAG applications - check if responses are grounded in context
const faithfulness = new Faithfulness({
  model: openai('gpt-4o-mini'),
})

// For Q&A systems - check accuracy against reference answers
const factualCorrectness = new FactualCorrectness({
  model: openai('gpt-4o-mini'),
  mode: 'f1',
})
```

--------------------------------

### Create and Use Multi-Hop Abstract Synthesizer

Source: https://open-evals.com/docs/generator/synthesis/multi-hop-abstract

Instantiate the multi-hop-abstract synthesizer with a specified model and use it within the synthesize function to generate samples. Ensure the 'openai' SDK is imported and configured.

```typescript
import { createSynthesizer } from '@open-evals/generator'
import { openai } from '@ai-sdk/openai'

const synthesizer = createSynthesizer(openai.chat('gpt-4o'), 'multi-hop-abstract')

// Use in synthesis
const samples = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [[synthesizer, 100]], // 100% of samples
  personas,
  count: 100,
})
```

--------------------------------

### Step 4: Configure Synthesizers

Source: https://open-evals.com/docs/generator/quickstart

Define the types of questions to generate by configuring synthesizers and their weights. Options include single-hop, multi-hop abstract, and multi-hop specific questions.

```typescript
const synthesizers = [
  // [synthesizer, weight]
  [createSynthesizer(llm, 'single-hop-specific'), 50],
  [createSynthesizer(llm, 'multi-hop-abstract'), 25],
  [createSynthesizer(llm, 'multi-hop-specific'), 25],
]
```

--------------------------------

### Controlling Evaluation Concurrency

Source: https://open-evals.com/docs/core/evaluate

Adjust the `concurrency` setting to manage how many samples are evaluated simultaneously. Start conservatively and increase as needed, especially for rate-limited APIs.

```typescript
// Conservative - good for rate-limited APIs
await evaluate(dataset, metrics, { concurrency: 3 })
```

```typescript
// Aggressive - good for high rate limits
await evaluate(dataset, metrics, { concurrency: 50 })
```

--------------------------------

### Initialize and Evaluate Faithfulness

Source: https://open-evals.com/docs/metrics/faithfulness

Create a Faithfulness metric instance with a specified language model and evaluate a single turn of conversation. Ensure 'query', 'response', and 'retrievedContexts' are provided.

```javascript
import { Faithfulness } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

// Create the metric
const faithfulness = new Faithfulness({
  model: openai('gpt-4.1'),
})

// Evaluate a sample
const result = await faithfulness.evaluateSingleTurn({
  query: 'What is the capital of France?',
  response: 'The capital of France is Paris. It is located on the Seine River.',
  retrievedContexts: [
    'Paris is the capital and most populous city of France. It is situated on the Seine River.',
  ],
})

console.log(result)
// {
//   name: 'faithfulness',
//   score: 1.0,
//   reason: '2 out of 2 statements were faithful to the context',
//   metadata: {
//     statements: [...] 
//   }
// }
```

--------------------------------

### Configure Question Synthesizers

Source: https://open-evals.com/docs/generator

Sets up different types of question synthesizers with specified weights. This configuration determines the mix of simple and complex questions to be generated.

```typescript
import { createSynthesizer } from '@open-evals/generator'

const synthesizers = [
  [createSynthesizer(openai.chat('gpt-4o'), 'single-hop-specific'), 50], // 50 simple questions
  [createSynthesizer(openai.chat('gpt-4o'), 'multi-hop-abstract'), 25], // 25 complex questions
  [createSynthesizer(openai.chat('gpt-4o'), 'multi-hop-specific'), 25], // 25 detailed complex
]
```

--------------------------------

### Single-Turn Sample Format

Source: https://open-evals.com/docs/core/quickstart

Use this JSON format for simple query-response pairs, suitable for QA systems or RAG applications. Reference answers and retrieved contexts are optional.

```json
{
  "query": "What is the capital of France?",
  "response": "Paris is the capital of France.",
  "reference": "The capital of France is Paris.",           // Optional: ground truth answer
  "retrievedContexts": ["Paris is the capital city of France."]  // Optional: for RAG evaluation
}
```

--------------------------------

### Multi-Turn Sample Format

Source: https://open-evals.com/docs/core/quickstart

Use this JSON format for conversational scenarios where the context of previous messages is important. It evaluates the full conversation flow.

```json
{
  "messages": [
    { "role": "user", "content": "Hello!" },
    { "role": "assistant", "content": "Hi! How can I help?" },
    { "role": "user", "content": "Tell me about Paris" },
    { "role": "assistant", "content": "Paris is the capital of France..." }
  ]
}
```

--------------------------------

### Load and Process Multiple Documents

Source: https://open-evals.com/docs/generator/quickstart

Load all markdown files from a directory and process them into a knowledge graph. Ensure the 'docs' directory exists and contains markdown files.

```javascript
import { readdir, readFile } from 'fs/promises'

// Load all markdown files from a directory
const files = await readdir('./docs')
const documents = await Promise.all(
  files
    .filter((f) => f.endsWith('.md'))
    .map(async (file) => {
      const content = await readFile(`./docs/${file}`, 'utf-8')
      return new DocumentNode(file, content, { category: 'docs' })
    })
)

// Process them all together
const graph = await transform(graph(documents))
  .pipe(chunk(new RecursiveCharacterSplitter()))
  .pipe(embed(openai.embedding('text-embedding-3-small')))
  .pipe(relationship())
  .apply()
```

--------------------------------

### Initialize EvaluationDataset

Source: https://open-evals.com/docs/core/dataset

Create a new dataset instance by passing an array of samples to the constructor. Supports both single-turn and multi-turn sample formats.

```typescript
import { EvaluationDataset } from '@open-evals/core'

const dataset = new EvaluationDataset([
  {
    query: 'What is TypeScript?',
    response: 'TypeScript is a typed superset of JavaScript.',
    reference: 'TypeScript is a strongly typed programming language.',
  },
])
```

--------------------------------

### Create and Use DocumentNode

Source: https://open-evals.com/docs/generator/knowledge-graph/usage

Represents a source document. Instantiate with an ID, content, and optional metadata. Properties like ID, content, and type are accessible.

```javascript
import { DocumentNode } from '@open-evals/generator'

const doc = new DocumentNode(
  'my-doc', // ID
  'Content here...', // Text content
  {
    // Custom metadata
    author: 'team',
    version: '2.0',
  }
)

// Access properties
console.log(doc.id) // 'my-doc'
console.log(doc.content) // 'Content here...'
console.log(doc.type) // 'document'
console.log(doc.summary) // Set by summarize transform
```

--------------------------------

### Dataset Persistence with JSON/JSONL

Source: https://open-evals.com/docs/core/dataset

Save and load datasets using JSON or JSONL. JSONL is recommended for large datasets due to memory efficiency and streaming capabilities. Ensure `fs/promises` is imported for file operations.

```javascript
import { writeFile, readFile } from 'fs/promises'

// Save as JSONL (recommended)
const jsonl = dataset.toJSONL()
await writeFile('dataset.jsonl', jsonl)
```

```javascript
// Load from JSONL
const content = await readFile('dataset.jsonl', 'utf-8')
const loaded = EvaluationDataset.fromJSONL(content)
```

```javascript
// Or use JSON for smaller datasets
await writeFile('dataset.json', dataset.toJSON())
const loaded2 = EvaluationDataset.fromJSON(
  await readFile('dataset.json', 'utf-8')
)
```

--------------------------------

### Structure of a Generated Test Sample

Source: https://open-evals.com/docs/generator/synthesis

Each generated sample is a complete test case including a query, reference answer, retrieved contexts, and metadata about its creation. This format is ready for evaluation.

```json
{
  "query": "What are TypeScript's primitive types?",
  "reference": "TypeScript has several primitive types: string, number, boolean...",
  "retrievedContexts": [
    "TypeScript provides several primitive types including...",
    "The basic types in TypeScript are..."
  ],
  "metadata": {
    "persona": "Junior Developer",
    "queryType": "specific",
    "queryLength": "medium",
    "queryStyle": "conversational"
  }
}
```

--------------------------------

### Create and Use Multi-Hop Specific Synthesizer

Source: https://open-evals.com/docs/generator/synthesis/multi-hop-specific

Instantiate the multi-hop specific synthesizer using a specified model and then use it within the synthesize function to generate question samples. Ensure the synthesizer is included in the list of synthesizers with its desired weight.

```typescript
import {
  createSynthesizer,
} from '@open-evals/generator'
import { openai } from '@ai-sdk/openai'

const synthesizer = createSynthesizer(
  openai.chat('gpt-4o'),
  'multi-hop-specific'
)

// Use in synthesis
const samples = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [[synthesizer, 100]], // 100% of samples
  personas,
  count: 100,
})
```

--------------------------------

### Create Document Nodes for Knowledge Graph

Source: https://open-evals.com/docs/generator/knowledge-graph

Initialize document nodes with unique identifiers, content, and custom metadata. These form the basis of your knowledge graph.

```typescript
import { graph, DocumentNode } from '@open-evals/generator'

const documents = [
  new DocumentNode(
    'typescript-basics.md', // Unique identifier
    'TypeScript is a typed...', // Content
    { category: 'tutorial' } // Custom metadata
  ),
  new DocumentNode('advanced-types.md', 'Advanced TypeScript features...', {
    category: 'reference',
  }),
]

const knowledgeGraph = graph(documents)
```

--------------------------------

### Step 5: Generate Samples

Source: https://open-evals.com/docs/generator/quickstart

Orchestrate the test sample generation process using the synthesize function, providing the knowledge graph, configured synthesizers, personas, and desired sample count.

```typescript
const samples = await synthesize({
  graph: knowledgeGraph, // Your knowledge
  synthesizers, // What to generate
  personas, // Who asks
  count: 10, // Number of samples
})
```

--------------------------------

### Evaluate Dataset with Metrics

Source: https://open-evals.com/docs/metrics

Integrate metrics with the evaluation framework to evaluate a dataset. Ensure the EvaluationDataset is properly initialized with your samples.

```javascript
import { evaluate, EvaluationDataset } from '@ai-sdk-eval/core'

const dataset = new EvaluationDataset([
  /* your samples */
])

const results = await evaluate(dataset, [faithfulness, factualCorrectness])
```

--------------------------------

### Basic LLM Evaluation Workflow

Source: https://open-evals.com/docs/core/quickstart

Evaluate LLM responses by creating a dataset, configuring a metric with a model, and running the evaluation. The evaluate function handles parallelization.

```typescript
import { EvaluationDataset, evaluate } from '@open-evals/core'
import { Faithfulness } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

// 1. Create a dataset with your test samples
const dataset = new EvaluationDataset([
  {
    query: 'What is TypeScript?',
    response: 'TypeScript is a typed superset of JavaScript.',
    retrievedContexts: ['TypeScript adds static typing to JavaScript.'],
  },
])

// 2. Configure the metric with your chosen model
const metric = new Faithfulness({ model: openai('gpt-4o-mini') })

// 3. Run evaluation - the function handles parallelization automatically
const results = await evaluate(dataset, [metric])

console.log(results.statistics.averages) // { faithfulness: 0.92 }
```

--------------------------------

### Evalite Input to Metric Format Transformation

Source: https://open-evals.com/docs/metrics/evalite

Illustrates how Evalite's input and output are transformed to match the expected format for a metric. The adapter combines the input properties with the output as the 'response'.

```javascript
// Evalite provides: { input, output }
// Adapter transforms to: { ...input, response: output }
const result = await metric.evaluate({ ...input, response: output })
```

--------------------------------

### Step 3: Generate Personas

Source: https://open-evals.com/docs/generator/quickstart

Create diverse user personas by analyzing the knowledge graph with an LLM. Specify the number of personas and concurrency for parallel LLM calls.

```typescript
const personas = await generatePersonas(knowledgeGraph, openai.chat('gpt-4o'), {
  count: 5, // Number of personas to generate
  concurrency: 3, // Parallel LLM calls
})
```

--------------------------------

### Build Datasets Incrementally

Source: https://open-evals.com/docs/core/dataset

Add samples to an empty dataset one by one using `add()` or in batches using `addMany()`. This is useful for loading data from multiple sources or programmatic generation.

```typescript
const dataset = new EvaluationDataset([])

// Add individual samples as they become available
dataset.add({ query: '...', response: '...' })

// Or add multiple samples at once
dataset.addMany([sample1, sample2, sample3])
```

--------------------------------

### Sample Metadata Structure

Source: https://open-evals.com/docs/generator/synthesis/configuration

Each generated sample includes metadata for filtering and analysis, such as persona, query type, length, and style.

```json
{
  "query": "...",
  "reference": "...",
  "retrievedContexts": [...],
  "metadata": {
    "persona": "Junior Developer",      // Which persona asked
    "queryType": "specific",            // Type of question (specific/abstract)
    "queryLength": "medium",            // Length (short/medium/long)
    "queryStyle": "conversational"      // Style (web-search/conversational/technical)
  }
}
```

--------------------------------

### Multiple Metrics Simultaneously

Source: https://open-evals.com/docs/core/evaluate

Demonstrates how to run multiple metrics concurrently on the same dataset within a single evaluation pass.

```APIDOC
## Multiple Metrics Simultaneously

```typescript
const metrics = [new Faithfulness({ model }), new FactualCorrectness({ model })]

const results = await evaluate(dataset, metrics)

// Get individual metric averages
console.log('Faithfulness:', results.statistics.averages.faithfulness)
console.log('F1:', results.statistics.averages.factual_correctness)
```

The `evaluate` function processes all specified metrics for each sample in parallel, providing aggregated results for each.
```

--------------------------------

### A/B Test Different Models

Source: https://open-evals.com/docs/core/evaluate

Compare performance of different models or configurations by evaluating them on separate datasets and analyzing the results. Ensure datasets and metrics are correctly prepared for each model.

```javascript
const datasetA = createDatasetForModelA()
const datasetB = createDatasetForModelB()

const [resultsA, resultsB] = await Promise.all([
  evaluate(datasetA, metrics, { metadata: { model: 'gpt-4o-mini' } }),
  evaluate(datasetB, metrics, { metadata: { model: 'gpt-4o' } }),
])

console.log('Model A average:', resultsA.statistics.averages)
console.log('Model B average:', resultsB.statistics.averages)

const improved = Object.keys(resultsA.statistics.averages).filter(
  (metric) =>
    resultsB.statistics.averages[metric] > resultsA.statistics.averages[metric]
)
console.log('Model B improved on:', improved)
```

--------------------------------

### Apply Transforms in Correct Sequence

Source: https://open-evals.com/docs/generator/knowledge-graph/advanced

Demonstrates the correct order for applying transforms, emphasizing summarizing before chunking to maintain document context.

```typescript
// Good: summarize before chunking
await transform(graph)
  .pipe(summarize(llm)) // Document-level
  .pipe(chunk(splitter)) // Then split
  .pipe(embed(embedModel)) // Then embed chunks
  .apply()

// Bad: chunking before summarizing loses document context
await transform(graph)
  .pipe(chunk(splitter)) // Splits document
  .pipe(summarize(llm)) // Can't summarize chunks effectively!
  .apply()
```

--------------------------------

### Evaluate Function Signature

Source: https://open-evals.com/docs/core/evaluate

The main entry point for running evaluations. It takes a dataset and metrics, runs them in parallel, and returns aggregated statistics. Configuration options for concurrency, error handling, and metadata are available.

```APIDOC
## Function Signature
```typescript
function evaluate<T extends readonly Metric[]>(
  dataset: EvaluationDataset,
  metrics: T,
  config?: {
    concurrency?: number // Default: 10
    throwOnError?: boolean // Default: false
    metadata?: Record<string, unknown>
  }
): Promise<EvaluationResult<T>>
```

### Parameters
- **dataset** (`EvaluationDataset`): The dataset to evaluate.
- **metrics** (`Metric[]`): An array of metrics to apply to the dataset.
- **config?** (`object`): Optional configuration object.
  - **config.concurrency?** (`number`): The maximum number of samples to evaluate concurrently. Defaults to 10.
  - **config.throwOnError?** (`boolean`): If true, the evaluation will stop on the first error. Defaults to false.
  - **config.metadata?** (`Record<string, unknown>`): Arbitrary metadata to attach to the evaluation run.

**Returns** an `EvaluationResult` with the dataset and computed statistics.
```

--------------------------------

### Step 2: Transform Pipeline

Source: https://open-evals.com/docs/generator/quickstart

Build a knowledge graph by applying a series of transforms to your documents, including chunking, embedding, and relationship detection.

```typescript
const graph = await transform(documents)
  .pipe(chunk(splitter)) // Break into smaller pieces
  .pipe(embed(embedModel)) // Add vector embeddings
  .pipe(relationship()) // Find connections
  .apply()
```

--------------------------------

### Create Single-Hop Specific Synthesizer

Source: https://open-evals.com/docs/generator/synthesis/single-hop-specific

Instantiates a single-hop specific synthesizer using a specified language model. This synthesizer is used to generate focused questions from single context chunks.

```typescript
import {
  createSynthesizer
} from '@open-evals/generator'
import { openai } from '@ai-sdk/openai'

const synthesizer = createSynthesizer(
  openai.chat('gpt-4o'),
  'single-hop-specific'
)

// Use in synthesis
const samples = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [[synthesizer, 100]], // 100% of samples
  personas,
  count: 100,
})
```

--------------------------------

### End-to-End LLM Evaluation with Open Evals

Source: https://open-evals.com

This snippet demonstrates a full evaluation workflow using Open Evals, including document processing, knowledge graph creation, persona generation, synthetic data synthesis, and evaluation with the Faithfulness metric. Ensure necessary imports and models are configured.

```typescript
import { EvaluationDataset, evaluate } from '@open-evals/core'
import { Faithfulness } from '@open-evals/metrics'
import { synthesize, graph, DocumentNode, chunk, embed, summarize, relationship } from '@open-evals/generator'
import { openai } from '@ai-sdk/openai'

const documents = [new DocumentNode('typescript-guide.md', content, {})]

const knowledgeGraph = await transform(graph(documents))
  .pipe(summarize(openai.chat('gpt-4.1')))
  .pipe(embedProperty(openai.embedding('text-embedding-3-small'), {
    embedProperty: 'summary',
    propertyName: 'summaryEmbedding',
  }))
  .pipe(chunk(new RecursiveCharacterSplitter()))
  .pipe(embed(openai.embedding('text-embedding-3-small')))
  .pipe(relationship())
  .apply()

const personas = await generatePersonas(knowledgeGraph, openai.chat('gpt-4.1'), {
  count: 5,
})

const dataset = await synthesize({
  graph: knowledgeGraph,
  synthesizers: [
    [createSynthesizer(openai.chat('gpt-4.1'), 'single-hop-specific'), 1],
  ],
  personas,
  count: 10,
})

const results = await evaluate(dataset, 
  [new Faithfulness({ model: openai.chat('gpt-4.1') })],
  openai.chat('gpt-4.1'))

console.log(results)

```

--------------------------------

### Generated Sample Structure

Source: https://open-evals.com/docs/generator/quickstart

Examine the structure of a generated test sample, which includes the query, reference answer, retrieved contexts, and metadata about the persona, query type, and difficulty.

```json
{
  "query": "How do I define optional properties in TypeScript interfaces?",
  "reference": "In TypeScript, optional properties are marked with a question mark...",
  "retrievedContexts": [
    "TypeScript interfaces allow optional properties using the ? syntax...",
    "Optional properties can be undefined or the specified type..."
  ],
  "metadata": {
    "persona": "Junior Developer",
    "queryType": "single-hop-specific",
    "difficulty": "beginner",
    "nodeIds": ["chunk-1", "chunk-2"]
  }
}
```

--------------------------------

### Map Dataset Samples

Source: https://open-evals.com/docs/core/dataset

Transform each sample in the dataset into a new form, returning a new dataset. Useful for preprocessing or adding metadata.

```typescript
// Map transforms each sample - useful for adding metadata or preprocessing
const withTimestamps = dataset.map((s) => ({
  ...s,
  metadata: { ...s.metadata, processedAt: Date.now() },
}))
```

--------------------------------

### Generate Responses for Dataset Queries

Source: https://open-evals.com/docs/core/quickstart

Populate empty responses in a dataset using the `generate()` method, which handles parallel execution. This is useful for testing LLM applications.

```typescript
import { EvaluationDataset, isSingleTurnSample } from '@open-evals/core'
import { generateText } from 'ai'
import { openai } from '@ai-sdk/openai'

// 1. Start with queries and reference answers
const dataset = new EvaluationDataset([
  {
    query: 'What is the capital of France?',
    response: '', // Empty - will be filled by generate()
    reference: 'The capital of France is Paris.',
  },
  {
    query: 'Explain TypeScript',
    response: '',
    reference: 'TypeScript is a typed superset of JavaScript.',
  },
])

// 2. Generate responses using your model
const populatedDataset = await dataset.generate(
  async (sample) => {
    // Use type guard for type safety
    if (isSingleTurnSample(sample)) {
      const result = await generateText({
        model: openai('gpt-4o-mini'),
        prompt: sample.query,
      })
      return result.text
    }
    return ''
  },
  { concurrency: 10 } // Process 10 samples at a time
)

// 3. Evaluate the generated responses
const results = await evaluate(populatedDataset,
  [
    new Faithfulness({ model: openai('gpt-4o-mini') }),
  ]
)
```

--------------------------------

### Access Methods

Source: https://open-evals.com/docs/core/dataset

Methods for accessing individual samples or the entire dataset as an array.

```APIDOC
## Access Methods

* `at(index: number)` - Get sample at index (supports negative indices)
* `toArray()` - Convert to plain array
* `forEach(callback)` - Iterate with index
* `[Symbol.iterator]()` - Makes dataset iterable
```

--------------------------------

### Synthesize Complete Test Samples

Source: https://open-evals.com/docs/generator

Generates a complete test dataset including queries, reference answers, retrieved contexts, and metadata. This function requires the knowledge graph, configured synthesizers, personas, and desired sample count.

```typescript
import { synthesize } from '@open-evals/generator'

const testSamples = await synthesize({
  graph: knowledgeGraph,
  synthesizers,
  personas,
  count: 10,
  config: { generateGroundTruth: true },
})
```

--------------------------------

### Load Knowledge Graph

Source: https://open-evals.com/docs/generator/quickstart

Load a previously saved knowledge graph from a JSON file. Use 'JSON.parse' to deserialize the JSON string and 'KnowledgeGraph.fromJSON' to reconstruct the graph object.

```javascript
import { writeFile, readFile } from 'fs/promises'

// Save knowledge graph
await writeFile('knowledge-graph.json', JSON.stringify(graph.toJSON()))

// Load later
import { KnowledgeGraph } from '@open-evals/generator'
const saved = JSON.parse(await readFile('knowledge-graph.json', 'utf-8'))
const loadedGraph = KnowledgeGraph.fromJSON(saved)
```

--------------------------------

### Serialization Methods

Source: https://open-evals.com/docs/core/dataset

Methods for saving and loading datasets in JSON or JSONL format.

```APIDOC
## Serialization Methods

* `toJSON()` - Serialize to JSON string
* `toJSONL()` - Serialize to JSONL string
* `static fromJSON(json)` - Deserialize from JSON
* `static fromJSONL(jsonl)` - Deserialize from JSONL
```

--------------------------------

### Save Generated Samples

Source: https://open-evals.com/docs/generator/quickstart

Save generated test samples in JSONL format for easy loading. Use the 'toJSONL()' method on the samples object.

```javascript
// Save as JSONL for easy loading
await writeFile('test-samples.jsonl', testSamples.toJSONL())
```

--------------------------------

### Generate Ground Truth for Test Samples

Source: https://open-evals.com/docs/generator/synthesis/configuration

Configure whether to generate ground truth (reference answers) for each sample. Generating ground truth is essential for metrics like FactualCorrectness but doubles LLM calls.

```javascript
// With ground truth (default)
await synthesize({
  graph,
  synthesizers,
  personas,
  count: 100,
  config: { generateGroundTruth: true }, // Each sample includes reference answer
})
```

```javascript
// Without ground truth (faster)
await synthesize({
  graph,
  synthesizers,
  personas,
  count: 100,
  config: { generateGroundTruth: false }, // Only generates questions
})
```

--------------------------------

### Modification Methods

Source: https://open-evals.com/docs/core/dataset

Methods for adding samples to the dataset.

```APIDOC
## Modification Methods

* `add(sample)` - Add a single sample
* `addMany(samples)` - Add multiple samples
```

--------------------------------

### Attaching Metadata

Source: https://open-evals.com/docs/core/evaluate

Include custom metadata with your evaluation runs to track experiments, compare results, or store contextual information.

```APIDOC
## Attaching Metadata

```typescript
await evaluate(dataset, metrics, {
  metadata: {
    experiment_id: 'exp-001',
    model: 'gpt-4o-mini',
    dataset_version: '2.0',
    timestamp: Date.now(),
    notes: 'Testing with updated prompts',
  },
})
```

This metadata is associated with the evaluation results, aiding in organization and comparison of different runs.
```

--------------------------------

### Transformation Methods

Source: https://open-evals.com/docs/core/dataset

Methods for transforming and manipulating the dataset, returning new datasets.

```APIDOC
## Transformation Methods

* `filter(callback)` - Filter samples (returns new dataset)
* `map(callback)` - Transform samples (returns new dataset)
* `slice(start, end?)` - Extract range (returns new dataset)
* `sample(size)` - Get random subset (returns new dataset)
* `shuffle()` - Randomize order (returns new dataset)
* `split(ratio)` - Split into two datasets
```

--------------------------------

### Create and Use ChunkNode

Source: https://open-evals.com/docs/generator/knowledge-graph/usage

Represents a piece of a document. Instantiate with an ID, content, and optional metadata. Properties like embedding and relationships are accessible.

```javascript
import { ChunkNode } from '@open-evals/generator'

const chunk = new ChunkNode('chunk-1', 'TypeScript is a typed superset...', {
  documentId: 'typescript-guide',
})

// Access properties
console.log(chunk.embedding) // Set by embed transform
console.log(chunk.relationships) // Map of related nodes
```

--------------------------------

### evaluate()

Source: https://open-evals.com/docs/core/evaluate

Evaluates all samples in a dataset with the provided metrics. This function is suitable for comparing different models or configurations by evaluating datasets tailored for each.

```APIDOC
## evaluate()

### Description
Evaluates all samples in a dataset with the provided metrics.

### Function Signature
```typescript
function evaluate<T extends readonly Metric[]>(
  dataset: EvaluationDataset,
  metrics: T,
  config?: {
    concurrency?: number
    throwOnError?: boolean
    metadata?: Record<string, unknown>
  }
): Promise<EvaluationResult<T>>
```

### Parameters
- **dataset** (EvaluationDataset) - The dataset to evaluate.
- **metrics** (T) - An array of metrics to apply to the dataset.
- **config?** (object) - Optional configuration for the evaluation.
  - **config.concurrency?** (number) - The maximum number of concurrent evaluations to run.
  - **config.throwOnError?** (boolean) - If true, the evaluation will throw an error if any sample fails.
  - **config.metadata?** (Record<string, unknown>) - Additional metadata to attach to the evaluation results.
```

--------------------------------

### Save Knowledge Graph

Source: https://open-evals.com/docs/generator/quickstart

Persist the knowledge graph to a JSON file for later reuse. Use 'JSON.stringify' to convert the graph to a JSON string before writing.

```javascript
import { writeFile, readFile } from 'fs/promises'

// Save knowledge graph
await writeFile('knowledge-graph.json', JSON.stringify(graph.toJSON()))
```

--------------------------------

### Build Knowledge Graph from Documents

Source: https://open-evals.com/docs/generator

Transforms documents into a knowledge graph by chunking, embedding, and detecting relationships. Requires document content and an embedding model.

```typescript
import {
  graph,
  DocumentNode,
  chunk,
  embed,
  relationship,
} from '@open-evals/generator'
import { RecursiveCharacterSplitter } from '@open-evals/rag'
import { openai } from '@ai-sdk/openai'

const documents = [new DocumentNode('typescript-guide.md', content, {})]

const knowledgeGraph = await transform(graph(documents))
  .pipe(chunk(new RecursiveCharacterSplitter()))
  .pipe(embed(openai.embedding('text-embedding-3-small')))
  .pipe(relationship())
  .apply()
```

--------------------------------

### Integrate Metrics with Evalite

Source: https://open-evals.com/docs/metrics

Use the `toEvaliteScorer` utility to integrate metrics with the Evalite framework for experiment tracking and analysis. This allows leveraging Evalite's features with production-ready metrics.

```javascript
import { evalite } from 'evalite'
import { Faithfulness, toEvaliteScorer } from '@open-evals/metrics'
import { openai } from '@ai-sdk/openai'

evalite('My Evaluation', {
  data: async () => [{ input: { query: '...', retrievedContexts: ['...'] } }],
  task: async (sample) => 'Your LLM response',
  scorers: [
    toEvaliteScorer(new Faithfulness({ model: openai('gpt-4.1-mini') })),
  ],
})
```

--------------------------------

### Transform Pipeline for Document Processing

Source: https://open-evals.com/docs/generator

Demonstrates a fluent API for processing documents through a series of transforms, including summarization, chunking, embedding, and relationship detection.

```typescript
await transform(graph)
  .pipe(summarize(llm)) // Generate summaries
  .pipe(chunk(splitter)) // Split into chunks
  .pipe(embed(embedModel)) // Create embeddings
  .pipe(relationship()) // Detect relationships
  .apply()
```

--------------------------------

### Save and Load Knowledge Graph to JSON

Source: https://open-evals.com/docs/generator/knowledge-graph/advanced

Persist a knowledge graph to a JSON file and load it back. This is useful for caching expensive operations.

```typescript
import { writeFile, readFile } from 'fs/promises'
import { KnowledgeGraph } from '@open-evals/generator'

// Save to JSON
const json = graph.toJSON()
await writeFile('graph.json', JSON.stringify(json, null, 2))

// Load from JSON
const loaded = await readFile('graph.json', 'utf-8')
const restoredGraph = KnowledgeGraph.fromJSON(JSON.parse(loaded))
```

--------------------------------

### evaluateSample()

Source: https://open-evals.com/docs/core/evaluate

Evaluates a single sample with the provided metrics. This is useful for testing individual data points or when you need fine-grained control over the evaluation process.

```APIDOC
## evaluateSample()

### Description
Evaluates a single sample with the provided metrics.

### Function Signature
```typescript
function evaluateSample(
  sample: EvaluationSample,
  metrics: Metric[],
  config?: {
    throwOnError?: boolean
    metadata?: Record<string, unknown>
  }
): Promise<SampleResult>
```

### Parameters
- **sample** (EvaluationSample) - The single sample to evaluate.
- **metrics** (Metric[]) - An array of metrics to apply to the sample.
- **config?** (object) - Optional configuration for the evaluation.
  - **config.throwOnError?** (boolean) - If true, the evaluation will throw an error if the sample fails.
  - **config.metadata?** (Record<string, unknown>) - Additional metadata to attach to the sample result.
```

--------------------------------

### Generation Methods

Source: https://open-evals.com/docs/core/dataset

Methods for generating responses for all samples in the dataset.

```APIDOC
## Generation Methods

* `generate(generator, config?)` - Generate responses for all samples
  * `generator: (sample: EvaluationSample) => Promise<string> | string` - Function to generate response for each sample
  * `config.concurrency?: number` - Number of samples to process in parallel (default: 10)
  * Returns a new `EvaluationDataset` with generated responses
```

--------------------------------

### Understanding Evaluation Statistics

Source: https://open-evals.com/docs/core/evaluate

The `statistics` object in the evaluation result provides aggregated data, including averages for each metric, the total number of samples, and the total number of metrics used.

```json
{
  "dataset": EvaluationDataset,  // Your original dataset
  "statistics": {
    "averages": {                 // Average score for each metric
      "faithfulness": 0.87,
      "factual_correctness": 0.92
    },
    "totalSamples": 100,          // Number of samples evaluated
    "totalMetrics": 2             // Number of metrics used
  }
}
```
Tokens:
Raw