Evalite

https://v1.evalite.dev
Admin
Evalite is a TypeScript-based test runner that simplifies the evaluation of AI-powered applications,...
Tokens:42,507
Snippets:411
Trust Score:6.6
Update:5 months ago
Show doc for...
Tokens:
Raw
### Configure Setup Files

Source: https://v1.evalite.dev/api/define-config

Provide an array of file paths to be executed before tests run. This is useful for custom environment setup. This example specifies a single setup file.

```typescript
export default defineConfig({
  setupFiles: ["./custom-setup.ts"],
});
```

--------------------------------

### Complete Evalite Configuration Example

Source: https://v1.evalite.dev/api/define-config

Use this example to set up persistent storage, server configuration, test execution parameters, UI preferences, setup files, caching, and watch mode triggers.

```typescript
import { defineConfig } from "evalite/config";
import { createSqliteStorage } from "evalite/sqlite-storage";


export default defineConfig({
  // Persistent storage
  storage: () => createSqliteStorage("./evalite.db"),


  // Server configuration
  server: {
    port: 3006,
  },


  // Quality threshold
  scoreThreshold: 75,


  // Test execution
  testTimeout: 60000,
  maxConcurrency: 50,
  trialCount: 1,


  // UI preferences
  hideTable: false,


  // Setup
  setupFiles: ["./test-setup.ts"],


  // Caching
  cache: true,


  // Watch mode triggers
  forceRerunTriggers: ["src/**/*.ts", "prompts/**/*"],
});

```

--------------------------------

### Complete Multi-Modal Evalite Example

Source: https://v1.evalite.dev/api/evalite-file

This example demonstrates a comprehensive Evalite setup, including mixing file references (`EvaliteFile.fromPath()`) and Buffers in data, reporting traces with files, and defining columns with files. It showcases Evalite's flexibility in handling various media types.

```typescript
import { evalite, EvaliteFile } from "evalite";
import { readFileSync } from "fs";
import { reportTrace } from "evalite/traces";


evalite("Multi-Modal Analysis", {
  data: async () => {
    return [
      {
        // Mix of file references and buffers
        input: {
          image: EvaliteFile.fromPath("./images/cat.jpg"),
          audio: readFileSync("./audio/meow.mp3"),
        },
        expected: "A cat meowing",
      },
    ];
  },
  task: async (input) => {
    // Trace with file
    reportTrace({
      input: input.image,
      output: "Processing...",
    });


    const result = await analyzeMultiModal(input);


    return result;
  },
  columns: ({ output }) => [
    {
      label: "Visualization",
      value: readFileSync("./viz.png"),
    },
  ],
  scorers: [
    {
      name: "Match",
      scorer: ({ output, expected }) => {
        return output === expected ? 1 : 0;
      },
    },
  ],
});
```

--------------------------------

### Install Evalite Beta

Source: https://v1.evalite.dev/

Install the beta version of Evalite using pnpm.

```bash
pnpm add evalite@beta
```

--------------------------------

### Install Evalite and Vitest

Source: https://v1.evalite.dev/guides/quickstart

Install the necessary packages for Evalite and Vitest using pnpm.

```bash
pnpm add -D evalite vitest
```

--------------------------------

### Flexible Mode Example

Source: https://v1.evalite.dev/api/scorers/tool-call-accuracy

Example demonstrating the 'flexible' mode for toolCallAccuracy, which allows tool calls in any order.

```APIDOC
toolCallAccuracy({
  actualCalls: [
    { toolName: "createTask", input: { title: "Buy milk" } },
    { toolName: "getTasks" }, // Order swapped
  ],
  expectedCalls: [
    { toolName: "getTasks" },
    { toolName: "createTask", input: { title: "Buy milk" } },
  ],
  mode: "flexible",
});
// Score: 1.0 (still perfect - order doesn't matter)
```

--------------------------------

### Exact Mode Example

Source: https://v1.evalite.dev/api/scorers/tool-call-accuracy

Example demonstrating the 'exact' mode for toolCallAccuracy, which enforces strict ordering and argument matching.

```APIDOC
toolCallAccuracy({
  actualCalls: [
    { toolName: "getTasks" },
    { toolName: "createTask", input: { title: "Buy milk" } },
  ],
  expectedCalls: [
    { toolName: "getTasks" },
    { toolName: "createTask", input: { title: "Buy milk" } },
  ],
  mode: "exact",
});
// Score: 1.0 (perfect match)
```

--------------------------------

### Example: Tracing LLM Calls in Evalite

Source: https://v1.evalite.dev/tips/vercel-ai-sdk

This example demonstrates how `wrapAISDKModel` automatically traces all LLM calls made through the AI SDK, including prompts, responses, token usage, and timing information. Traces appear in the Evalite UI under each test case.

```typescript
import { openai } from "@ai-sdk/openai";
import { streamText } from "ai";
import { evalite } from "evalite";
import { wrapAISDKModel } from "evalite/ai-sdk";


evalite("Test Capitals", {
  data: async () => [
    {
      input: `What's the capital of France?`,
      expected: "Paris",
    },
    {
      input: `What's the capital of Germany?`,
      expected: "Berlin",
    },
  ],
  task: async (input) => {
    const result = streamText({
      model: wrapAISDKModel(openai("gpt-4o-mini")),
      system: `Answer the question concisely.`,
      prompt: input,
    });


    // All calls are automatically traced
    return await result.text;
  },
  scorers: [
    {
      name: "Exact Match",
      scorer: ({ output, expected }) => (output === expected ? 1 : 0),
    },
  ],
});

```

--------------------------------

### Configuration Priority Example

Source: https://v1.evalite.dev/api/run-evalite

Demonstrates how function arguments override configuration file settings. Options are merged with function arguments having the highest priority.

```typescript
export default defineConfig({
  scoreThreshold: 70,
  hideTable: true,
});


// script.ts
await runEvalite({
  mode: "run-once-and-exit",
  scoreThreshold: 80, // Overrides config (80 used)
  // hideTable not specified, uses config (true)
});
```

--------------------------------

### Install SQLite Peer Dependency

Source: https://v1.evalite.dev/guides/storage

Install the better-sqlite3 package, which is a required peer dependency for using SQLite storage with Evalite.

```bash
pnpm add -D better-sqlite3
```

--------------------------------

### Custom Storage Configuration

Source: https://v1.evalite.dev/api/run-evalite

Example demonstrating how to configure Evalite to use a custom SQLite storage backend.

```typescript
import { runEvalite } from "evalite/runner";
import { createSqliteStorage } from "evalite/sqlite-storage";


const storage = createSqliteStorage("./evalite.db");


await runEvalite({
  mode: "run-once-and-exit",
  storage,
});
```

--------------------------------

### Evaluate RAG Noise Sensitivity

Source: https://v1.evalite.dev/api/scorers/noise-sensitivity

Use this example to evaluate the noise sensitivity of your RAG system. It configures the evalite function with sample data, a task to simulate AI output, and the noiseSensitivity scorer. Ensure you have the necessary SDKs installed.

```typescript
import {
  openai
} from "@ai-sdk/openai";
import {
  evalite
} from "evalite";
import {
  noiseSensitivity
} from "evalite/scorers";


evalite("RAG Noise Sensitivity", {
  data: [
    {
      input: "What is the capital of France?",
      expected: {
        reference: "Paris is the capital of France.",
        groundTruth: [
          "Paris is the capital and largest city of France. It is located in the north-central part of the country.",
          "Lyon is the third-largest city in France and an important cultural center.",
          "Marseille is a major French port city on the Mediterranean coast.",
        ],
      },
    },
  ],
  task: async () => {
    return "Lyon is the capital of France. Paris is the largest city in France.";
  },
  scorers: [
    {
      scorer: ({ input, output, expected }) =>
        noiseSensitivity({
          question: input,
          answer: output,
          reference: expected.reference,
          groundTruth: expected.groundTruth,
          model: openai("gpt-4o-mini"),
          mode: "relevant",
        }),
    },
  ],
});
```

--------------------------------

### CI/CD Path-Based Deployment with Evalite

Source: https://v1.evalite.dev/tips/run-evals-on-ci-cd

Example of exporting Evalite UI with a dynamic base path for deployment to S3/CloudFront, enabling unique paths per run.

```bash
- name: Export UI with base path
  run: |
    RUN_PATH="/evals-${{ github.run_id }}"
    npx evalite export --basePath="$RUN_PATH" --output=./ui-export

- name: Upload to S3
  run: |
    aws s3 sync ./ui-export s3://my-bucket/evals-${{ github.run_id }}/
    echo "View at: https://my-domain.com/evals-${{ github.run_id }}"
```

--------------------------------

### Watch All Evals

Source: https://v1.evalite.dev/api/cli

Starts the watch mode for all evaluation files in the project.

```bash
# Watch all evals
evalite watch
```

--------------------------------

### Basic Scorer Example

Source: https://v1.evalite.dev/api/create-scorer

Demonstrates how to create a simple 'Exact Match' scorer using createScorer() and then use it within an evalite() call.

```APIDOC
## Basic Scorer
```javascript
import { createScorer, evalite } from "evalite";


const exactMatch = createScorer({
  name: "Exact Match",
  scorer: ({ output, expected }) => {
    return output === expected ? 1 : 0;
  },
});


evalite("My Eval", {
  data: [{ input: "Hello", expected: "Hi" }],
  task: async (input) => callLLM(input),
  scorers: [exactMatch],
});
```
```

--------------------------------

### Example Usage of Tool Call Accuracy

Source: https://v1.evalite.dev/api/scorers/tool-call-accuracy

Demonstrates how to integrate the `toolCallAccuracy` scorer within the `evalite` framework to evaluate an AI's tool-calling capabilities. This example sets up a task that generates text with tool calls and then scores the output against expected tool calls.

```typescript
import {
  openai
} from "@ai-sdk/openai";
import { generateText, tool } from "ai";
import { evalite } from "evalite";
import { toolCallAccuracy } from "evalite/scorers/deterministic";
import { z } from "zod";


evalite("Tool Calls", {
  data: [
    {
      input: "What calendar events do I have today?",
      expected: [
        {
          toolName: "getCalendarEvents",
          input: {
            date: "2024-04-27",
          },
        },
      ],
    },
  ],
  task: async (input) => {
    const result = await generateText({
      model: openai("gpt-4o-mini"),
      prompt: input,
      tools: {
        getCalendarEvents: tool({
          description: "Get calendar events",
          inputSchema: z.object({
            date: z.string(),
          }),
          execute: async ({ date }) => {
            return `You have a meeting with John on ${date}`;
          },
        }),
      },
      system: `You are a helpful assistant. Today's date is 2024-04-27.`,
    });

    return result.toolCalls;
  },
  scorers: [
    {
      scorer: ({ output, expected }) => {
        return toolCallAccuracy({
          actualCalls: output,
          expectedCalls: expected,
        });
      },
    },
  ],
});

```

--------------------------------

### Reusable Scorers Example

Source: https://v1.evalite.dev/api/create-scorer

Demonstrates creating a library of reusable scorers, such as 'hasEmoji' and a factory function 'containsKeyword', and then importing and using them in an evaluation.

```APIDOC
## Reusable Scorers
Create a library of scorers to reuse across evals:
```javascript
import { createScorer } from "evalite";


export const hasEmoji = createScorer({
  name: "Has Emoji",
  scorer: ({ output }) => (/\p{Emoji}/u.test(output) ? 1 : 0),
});


export const containsKeyword = (keyword) =>
  createScorer({
    name: `Contains "${keyword}"`, 
    scorer: ({ output }) => (output.includes(keyword) ? 1 : 0),
  });


// my-eval.eval.ts
import { evalite } from "evalite";
import { hasEmoji, containsKeyword } from "./scorers";


evalite("My Eval", {
  data: [{ input: "Hello" }],
  task: async (input) => callLLM(input),
  scorers: [hasEmoji, containsKeyword("greeting")],
});
```
```

--------------------------------

### Usage Examples

Source: https://v1.evalite.dev/api/run-evalite

Illustrates common use cases for the runEvalite function, including setting up CI/CD scripts, development workflows, and custom storage configurations.

```APIDOC
## Usage Examples

### Basic CI/CD Script
```javascript
import { runEvalite } from "evalite/runner";

async function runTests() {
  try {
    await runEvalite({
      mode: "run-once-and-exit",
      scoreThreshold: 75,
      outputPath: "./results.json",
    });
    console.log("All evals passed!");
  } catch (error) {
    console.error("Evals failed:", error);
    process.exit(1);
  }
}

runTests();
```

### Development Script
```javascript
import { runEvalite } from "evalite/runner";

// Run specific eval in watch mode
await runEvalite({
  mode: "watch-for-file-changes",
  path: "chat.eval.ts",
  hideTable: true,
});
```

### Custom Storage
```javascript
import { runEvalite } from "evalite/runner";
import { createSqliteStorage } from "evalite/sqlite-storage";

const storage = createSqliteStorage("./evalite.db");

await runEvalite({
  mode: "run-once-and-exit",
  storage,
});
```
```

--------------------------------

### Using the Answer Relevancy Scorer

Source: https://v1.evalite.dev/api/scorers/answer-relevancy

Demonstrates how to integrate the answerRelevancy scorer into an Evalite evaluation. This example uses OpenAI models for both the main task and the scorer's requirements. Ensure you have the necessary SDKs installed.

```typescript
import {
  openai
} from "@ai-sdk/openai";
import {
  evalite
} from "evalite";
import {
  answerRelevancy
} from "evalite/scorers";


evalite("Answer Relevancy", {
  data: [
    {
      input: "What is the capital of France?",
    },
    {
      input: "Who invented the telephone?",
    },
    {
      input: "What are the health benefits of exercise?",
    },
  ],
  task: async (input) => {
    if (input.includes("capital of France")) {
      return "Paris is the capital of France. It's known for the Eiffel Tower and the Louvre Museum.";
    } else if (input.includes("telephone")) {
      return "Alexander Graham Bell is credited with inventing the telephone in 1876.";
    } else if (input.includes("health benefits")) {
      return "I don't know about that topic.";
    }
    return "I'm not sure about that.";
  },
  scorers: [
    {
      scorer: ({ input, output }) =>
        answerRelevancy({
          question: input,
          answer: output,
          model: openai("gpt-4o-mini"),
          embeddingModel: openai.embedding("text-embedding-3-small"),
        }),
    },
  ],
});

```

--------------------------------

### Complete Example with AI SDK Integration

Source: https://v1.evalite.dev/api/report-trace

Combines automatic AI SDK tracing with manual traces using reportTrace() for custom steps like intent extraction and response formatting. Demonstrates wrapping an AI SDK model.

```typescript
import { evalite } from "evalite";
import { reportTrace } from "evalite/traces";
import { wrapAISDKModel } from "evalite/ai-sdk";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";


const model = wrapAISDKModel(openai("gpt-4"));


evalite("Research Agent", {
  data: [
    {
      input: "What is the capital of France?",
      expected: "Paris",
    },
  ],
  task: async (input) => {
    // Step 1: Extract intent (manually traced)
    const intent = await extractIntent(input);
    reportTrace({
      input: { query: input },
      output: { intent },
    });


    // Step 2: Generate response (automatically traced via AI SDK)
    const result = await generateText({
      model,
      prompt: `Answer this question: ${input}`,
    });


    // Step 3: Format result (manually traced)
    const formatted = formatResponse(result.text);
    reportTrace({
      input: { raw: result.text },
      output: { formatted },
    });


    return formatted;
  },
  scorers: [
    {
      name: "Exact Match",
      scorer: ({ output, expected }) => {
        return output === expected ? 1 : 0;
      },
    },
  ],
});


async function extractIntent(query: string) {
  // Custom intent extraction logic
  return "question";
}


function formatResponse(text: string) {
  // Custom formatting logic
  return text.trim();
}
```

--------------------------------

### Basic CI/CD Script with Threshold and Output

Source: https://v1.evalite.dev/api/run-evalite

A complete CI/CD script example that sets a score threshold and specifies an output path for results. Includes error handling.

```typescript
import { runEvalite } from "evalite/runner";


async function runTests() {
  try {
    await runEvalite({
      mode: "run-once-and-exit",
      scoreThreshold: 75,
      outputPath: "./results.json",
    });
    console.log("All evals passed!");
  } catch (error) {
    console.error("Evals failed:", error);
    process.exit(1);
  }
}


runTests();
```

--------------------------------

### Configure Custom Storage with defineConfig

Source: https://v1.evalite.dev/api/define-config

Define a custom storage backend using `createSqliteStorage` for persistent storage. This example configures SQLite storage with a specified database file.

```typescript
import { defineConfig } from "evalite/config";
import { createSqliteStorage } from "evalite/sqlite-storage";


export default defineConfig({
  storage: () => createSqliteStorage("./custom.db"),
});
```

--------------------------------

### Pass Vite Configuration

Source: https://v1.evalite.dev/api/define-config

Integrate existing Vite/Vitest configuration by passing it to the `viteConfig` option. This example imports and uses a local `vite.config.ts` file.

```typescript
import { defineConfig } from "evalite/config";
import viteConfig from "./vite.config.ts";


export default defineConfig({
  viteConfig: viteConfig,
});
```

--------------------------------

### Complete RAG System Evaluation with Evalite and Vercel AI SDK

Source: https://v1.evalite.dev/tips/vercel-ai-sdk

This example shows a full RAG system evaluation. Wrap your AI SDK model once and reuse it across evaluations. Caching is enabled by default for faster iteration during development.

```typescript
import {
  openai
} from "@ai-sdk/openai";
import { generateText } from "ai";
import { evalite } from "evalite";
import { faithfulness } from "evalite/scorers";
import { wrapAISDKModel } from "evalite/ai-sdk";


// Wrap once, use everywhere
const model = wrapAISDKModel(openai("gpt-4o-mini"));


evalite("RAG System", {
  data: async () => [
    {
      input: "What is Evalite?",
      expected: {
        groundTruth: ["Evalite is a tool for testing LLM applications."],
      },
    },
  ],
  task: async (input) => {
    // Both calls are traced and cached
    const context = await generateText({
      model,
      prompt: `Retrieve context for: ${input}`,
    });


    const result = await generateText({
      model,
      prompt: `Answer using context: ${context.text}\n\nQuestion: ${input}`,
    });


    return result.text;
  },
  scorers: [
    {
      scorer: ({ input, output, expected }) =>
        // Scorer LLM calls are also cached
        faithfulness({
          question: input,
          answer: output,
          groundTruth: expected.groundTruth,
          model,
        }),
    },
  ],
});

```

--------------------------------

### Watch Specific Eval File

Source: https://v1.evalite.dev/api/cli

Starts the watch mode for a single, specified evaluation file.

```bash
# Watch specific eval
evalite watch example.eval.ts
```

--------------------------------

### Async Scorer Example

Source: https://v1.evalite.dev/api/create-scorer

Shows how to define an asynchronous scorer, like an 'LLM Judge', which uses an external API (e.g., OpenAI) to evaluate the output quality.

```APIDOC
## Async Scorer
Scorers can be async for LLM-based evaluation:
```javascript
const llmScorer = createScorer({
  name: "LLM Judge",
  description: "Uses GPT-4 to evaluate output quality",
  scorer: async ({ output, expected }) => {
    const response = await openai.chat.completions.create({
      model: "gpt-4",
      messages: [
        {
          role: "system",
          content: "Rate the output quality from 0 to 1.",
        },
        {
          role: "user",
          content: `Output: ${output}\nExpected: ${expected}`,
        },
      ],
    });


    const score = parseFloat(response.choices[0].message.content);
    return score;
  },
});
```
```

--------------------------------

### createScorer with Description and Length Check

Source: https://v1.evalite.dev/api/create-scorer

Example of creating a scorer that includes a description and checks if the output string meets a minimum length requirement.

```typescript
createScorer({
  name: "Length Check",
  description: "Checks if output is at least 10 characters",
  scorer: ({ output }) => (output.length >= 10 ? 1 : 0),
});
```

--------------------------------

### Greeting Generator Example with Evalite and OpenAI

Source: https://v1.evalite.dev/api/evalite

This snippet sets up an Evalite test for a greeting generator. It defines the data, the task using OpenAI's chat completions, and custom scoring logic. Ensure you have the 'evalite' library and OpenAI API access configured.

```javascript
import { evalite } from "evalite";
import { exactMatch } from "evalite/scorers";


evalite("Greeting Generator", {
  data: async () => {
    return [
      { input: "Hello", expected: "Hi there!" },
      { input: "Good morning", expected: "Good morning to you!" },
      { input: "Howdy", expected: "Howdy partner!" },
    ];
  },
  task: async (input) => {
    const response = await openai.chat.completions.create({
      model: "gpt-4",
      messages: [
        {
          role: "system",
          content: "Generate a friendly greeting response.",
        },
        { role: "user", content: input },
      ],
    });
    return response.choices[0].message.content;
  },
  scorers: [
    {
      scorer: ({ output, expected }) =>
        exactMatch({ actual: output, expected }),
    },
  ],
  columns: ({ output }) => [{ label: "Length", value: output.length }],
});

```

--------------------------------

### Run Evalite Dev Script

Source: https://v1.evalite.dev/guides/quickstart

Execute the `eval:dev` script to start Evalite, which will run your evals, save results, and launch a UI for viewing.

```bash
pnpm run eval:dev
```

--------------------------------

### Implement Custom Postgres Storage

Source: https://v1.evalite.dev/api/storage

Create a class that implements the Evalite.Storage interface for PostgreSQL. This example shows the structure for handling runs, suites, evals, scores, and traces, along with connection management.

```typescript
import type { Evalite } from "evalite/types";


export class PostgresStorage implements Evalite.Storage {
  constructor(private connectionString: string) {}


  runs = {
    async create(opts: Evalite.Storage.Runs.CreateOpts) {
      // Insert run into Postgres
      // Return Evalite.Storage.Entities.Run
    },
    async getMany(opts?: Evalite.Storage.Runs.GetManyOpts) {
      // Query runs from Postgres
      // Return Evalite.Storage.Entities.Run[]
    },
  };


  suites = {
    async create(opts: Evalite.Storage.Suites.CreateOpts) {
      // ...
    },
    async update(opts: Evalite.Storage.Suites.UpdateOpts) {
      // ...
    },
    async getMany(opts?: Evalite.Storage.Suites.GetManyOpts) {
      // ...
    },
  };


  evals = {
    async create(opts: Evalite.Storage.Evals.CreateOpts) {
      // ...
    },
    async update(opts: Evalite.Storage.Evals.UpdateOpts) {
      // ...
    },
    async getMany(opts?: Evalite.Storage.Evals.GetManyOpts) {
      // ...
    },
  };


  scores = {
    async create(opts: Evalite.Storage.Scores.CreateOpts) {
      // ...
    },
    async getMany(opts?: Evalite.Storage.Scores.GetManyOpts) {
      // ...
    },
  };


  traces = {
    async create(opts: Evalite.Storage.Traces.CreateOpts) {
      // ...
    },
    async getMany(opts?: Evalite.Storage.Traces.GetManyOpts) {
      // ...
    },
  };


  async close() {
    // Close database connection
  }


  async [Symbol.asyncDispose]() {
    await this.close();
  }
}


// Factory function
export const createPostgresStorage = (
  connectionString: string
): PostgresStorage => {
  return new PostgresStorage(connectionString);
};

```

--------------------------------

### Watch Evalite Evals

Source: https://v1.evalite.dev/api/cli

Monitors evaluation files for changes and re-runs them automatically. Starts a UI server at `http://localhost:3006`.

```bash
evalite watch
evalite watch path/to/eval.eval.ts
```

--------------------------------

### Scorer with Metadata Example

Source: https://v1.evalite.dev/api/create-scorer

Illustrates creating a 'Length Check' scorer that returns both a score and metadata containing the output's length and defined min/max lengths.

```APIDOC
## Scorer with Metadata
```javascript
const lengthChecker = createScorer({
  name: "Length Check",
  description: "Validates output length is within acceptable range",
  scorer: ({ output }) => {
    const length = output.length;
    const isValid = length >= 10 && length <= 100;


    return {
      score: isValid ? 1 : 0,
      metadata: {
        length,
        minLength: 10,
        maxLength: 100,
      },
    };
  },
});
```
```

--------------------------------

### Define Evalite Configuration

Source: https://v1.evalite.dev/api/define-config

Use defineConfig to create your Evalite configuration object in `evalite.config.ts`. This example sets timeout, concurrency, and score threshold.

```typescript
import { defineConfig } from "evalite/config";


export default defineConfig({
  testTimeout: 60000,
  maxConcurrency: 100,
  scoreThreshold: 80,
});
```

--------------------------------

### Development Script to Run Specific Eval

Source: https://v1.evalite.dev/api/run-evalite

Example of a development script that runs a specific eval file in watch mode and hides the detailed table.

```typescript
import { runEvalite } from "evalite/runner";


// Run specific eval in watch mode
await runEvalite({
  mode: "watch-for-file-changes",
  path: "chat.eval.ts",
  hideTable: true,
});
```

--------------------------------

### Inline Scorers Example

Source: https://v1.evalite.dev/api/create-scorer

Shows how to define scorers inline directly within the `scorers` array of an `evalite()` call, which is equivalent to using `createScorer()` for single-use scorers.

```APIDOC
## Inline Scorers
You can also define scorers inline without `createScorer()`:
```javascript
evalite("My Eval", {
  data: [{ input: "Hello", expected: "Hi" }],
  task: async (input) => callLLM(input),
  scorers: [
    // Inline scorer (same shape as createScorer opts)
    {
      name: "Exact Match",
      scorer: ({ output, expected }) => (output === expected ? 1 : 0),
    },
  ],
});
```
```

--------------------------------

### Compare Models with Evalite

Source: https://v1.evalite.dev/tips/comparing-different-approaches

Compares different language models (e.g., GPT-4o mini, GPT-4o, Claude Sonnet) on a given dataset. Ensure necessary AI SDKs and Evalite are installed.

```typescript
import { evalite } from "evalite";
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";
import { exactMatch } from "evalite/scorers";


evalite.each([
  { name: "GPT-4o mini", input: { model: "gpt-4o-mini", temp: 0.7 } },
  { name: "GPT-4o", input: { model: "gpt-4o", temp: 0.7 } },
  { name: "Claude Sonnet", input: { model: "claude-3-5-sonnet", temp: 1.0 } },
])("Compare models", {
  data: async () => [
    { input: "What's the capital of France?", expected: "Paris" },
    { input: "What's the capital of Germany?", expected: "Berlin" },
  ],
  task: async (input, variant) => {
    return generateText({
      model: openai(variant.model),
      temperature: variant.temp,
      prompt: input,
    });
  },
  scorers: [
    {
      scorer: ({ output, expected }) =>
        exactMatch({ actual: output, expected }),
    },
  ],
});

```

--------------------------------

### Using the Faithfulness Scorer

Source: https://v1.evalite.dev/api/scorers/faithfulness

Demonstrates how to integrate the faithfulness scorer into an Evalite evaluation. Requires importing necessary modules and defining the task, expected output, and scorer configuration. The example uses OpenAI's GPT-4o-mini model via the AI SDK.

```typescript
import {
  openai
} from "@ai-sdk/openai";
import {
  evalite
} from "evalite";
import {
  faithfulness
} from "evalite/scorers";


evalite("RAG Faithfulness", {
  data: [
    {
      input: "What programming languages does John know?",
      expected: {
        groundTruth: [
          "John is a software engineer at XYZ Corp. He specializes in backend development using Python and Go. He has 5 years of experience in the industry.",
        ],
      },
    },
  ],
  task: async () => {
    return "John knows Python, Go, and JavaScript. He also invented TypeScript and works at Google.";
  },
  scorers: [
    {
      scorer: ({ input, output, expected }) =>
        faithfulness({
          question: input,
          answer: output,
          groundTruth: expected.groundTruth,
          model: openai("gpt-4o-mini"),
        }),
    },
  ],
});
```

--------------------------------

### Wrap Vercel AI SDK Model with Evalite

Source: https://v1.evalite.dev/api/ai-sdk

Wrap a Vercel AI SDK model to enable automatic tracing and caching of all LLM calls. This setup is used within an Evalite task.

```typescript
import { openai } from "@ai-sdk/openai";
import { generateText } from "ai";
import { evalite } from "evalite";
import { wrapAISDKModel } from "evalite/ai-sdk";


// Wrap the model
const model = wrapAISDKModel(openai("gpt-4o-mini"));


evalite("My Eval", {
  data: [{ input: "Hello", expected: "Hi" }],
  task: async (input) => {
    // All calls are automatically traced and cached
    const result = await generateText({
      model,
      prompt: input,
    });


    return result.text;
  },
});

```

--------------------------------

### Define an Eval with Data, Task, and Scorers

Source: https://v1.evalite.dev/guides/what-is-evalite

This example demonstrates how to define an eval using Evalite. It includes a dataset, a task function that processes input, and scorers to evaluate the task's output against expected results. Ensure `evalite` and `exactMatch` are imported.

```typescript
import { evalite } from "evalite";
import { exactMatch } from "evalite/scorers";


evalite("My Eval", {
  // 1. A set of data to test
  data: [{ input: "Hello", expected: "Hello World!" }],
  // 2. The task to perform, usually to call a LLM.
  task: async (input) => {
    return input + " World!";
  },
  // 3. Optionally, some scorers to score the eval
  scorers: [
    // For instance, exactMatch checks if the output
    // matches the expected value exactly
    {
      scorer: ({ output, expected }) =>
        exactMatch({ actual: output, expected }),
    },
  ],
});
```

--------------------------------

### Using Exact Match Scorer in Evalite

Source: https://v1.evalite.dev/api/scorers/exact-match

This example demonstrates how to integrate the exactMatch scorer within the Evalite framework to compare an AI's generated output against a predefined reference string. It's suitable for testing exact matches of phrases or structured data.

```javascript
import { evalite } from "evalite";
import { exactMatch } from "evalite/scorers/deterministic";


evalite("Exact Match", {
  data: [
    {
      input: "What is the capital of France?",
      expected: {
        reference: "Paris is the capital of France",
      },
    },
  ],
  task: async (input) => {
    return "Paris is the capital of France";
  },
  scorers: [
    {
      scorer: ({ output, expected }) =>
        exactMatch({
          actual: output,
          expected: expected.reference,
        }),
    },
  ],
});
```

--------------------------------

### Faithfulness Scorer Usage

Source: https://v1.evalite.dev/api/scorers/faithfulness

This example demonstrates how to use the faithfulness scorer within the Evalite framework. It sets up a RAG faithfulness evaluation by providing an input question, expected ground truth, and a task that simulates an AI's response. The faithfulness scorer is then configured with the question, answer, ground truth, and a language model for evaluation.

```APIDOC
## Example

```typescript

import { openai } from "@ai-sdk/openai";
import { evalite } from "evalite";
import { faithfulness } from "evalite/scorers";


evalite("RAG Faithfulness", {
  data: [
    {
      input: "What programming languages does John know?",
      expected: {
        groundTruth: [
          "John is a software engineer at XYZ Corp. He specializes in backend development using Python and Go. He has 5 years of experience in the industry.",
        ],
      },
    },
  ],
  task: async () => {
    return "John knows Python, Go, and JavaScript. He also invented TypeScript and works at Google.";
  },
  scorers: [
    {
      scorer: ({ input, output, expected }) =>
        faithfulness({
          question: input,
          answer: output,
          groundTruth: expected.groundTruth,
          model: openai("gpt-4o-mini"),
        }),
    },
  ],
});

```
```

--------------------------------

### Using Levenshtein Scorer in Evalite

Source: https://v1.evalite.dev/api/scorers/levenshtein

Demonstrates how to use the Levenshtein scorer within the Evalite framework to compare an actual output against an expected reference. This example shows a case with a minor typo in the output.

```typescript
import { evalite } from "evalite";
import { levenshtein } from "evalite/scorers/deterministic";


evalite("Levenshtein", {
  data: [
    {
      input: "What is the capital of France?",
      expected: {
        reference: "Paris",
      },
    },
  ],
  task: async (input) => {
    return "Pari"; // Typo - missing 's'
  },
  scorers: [
    {
      scorer: ({ output, expected }) =>
        levenshtein({
          actual: output,
          expected: expected.reference,
        }),
    },
  ],
});
```

--------------------------------

### Tool Call Accuracy Metadata (Exact Mode)

Source: https://v1.evalite.dev/api/scorers/tool-call-accuracy

Provides an example of the metadata returned by the `toolCallAccuracy` scorer when operating in 'exact' mode. This metadata includes counts of exact matches, name-only matches, and penalties for incorrect or missing calls, along with totals for reference and output calls.

```json
{
  mode: "exact",
  exactMatches: number,
  nameOnlyMatches: number,
  wrongOrMissing: number,
  totals: {
    reference: number,
    output: number
  }
}

```

--------------------------------

### Preview Static Evalite UI Locally

Source: https://v1.evalite.dev/tips/run-evals-on-ci-cd

Serves the exported static UI bundle from the './evalite-export' directory for local preview.

```bash
npx serve -s ./evalite-export
```

--------------------------------

### Local Testing for Path-Based Deployment

Source: https://v1.evalite.dev/tips/run-evals-on-ci-cd

Demonstrates how to test Evalite's path-based deployment locally by creating a matching directory structure and serving it.

```bash
# Export with base path
evalite export --basePath=/evals-123

# Create matching directory structure
mkdir -p /tmp/test/evals-123
cp -r evalite-export/* /tmp/test/evals-123/

# Serve and visit http://localhost:3000/evals-123
npx serve /tmp/test
```

--------------------------------

### createScorer with Metadata

Source: https://v1.evalite.dev/api/create-scorer

Example of a scorer that returns both a score and metadata, in this case, the word count of the output.

```typescript
createScorer({
  name: "Word Count",
  scorer: ({ output }) => {
    const wordCount = output.split(" ").length;
    return {
      score: wordCount >= 10 ? 1 : 0,
      metadata: { wordCount },
    };
  },
});
```

--------------------------------

### Watch with Hidden Table

Source: https://v1.evalite.dev/api/cli

Starts the watch mode with the detailed table output hidden, which can be useful for debugging with `console.log`.

```bash
# Watch with hidden table (useful for debugging with console.log)
evalite watch --hideTable
```

--------------------------------

### Get Suites for a Run

Source: https://v1.evalite.dev/api/storage

Fetch all suites associated with a specific run by using the `run_id` query parameter in `storage.suites.getMany`.

```typescript
const suites = await storage.suites.getMany({ run_id: runId });

```

--------------------------------

### Scorer with Metadata and Range Check

Source: https://v1.evalite.dev/api/create-scorer

Example of a scorer that validates output length against a specified range and returns score and metadata.

```typescript
const lengthChecker = createScorer({
  name: "Length Check",
  description: "Validates output length is within acceptable range",
  scorer: ({ output }) => {
    const length = output.length;
    const isValid = length >= 10 && length <= 100;


    return {
      score: isValid ? 1 : 0,
      metadata: {
        length,
        minLength: 10,
        maxLength: 100,
      },
    };
  },
});
```

--------------------------------

### Set Evalite Server Port

Source: https://v1.evalite.dev/api/define-config

Configure the port for the Evalite UI server. This example sets the server port to 8080.

```typescript
export default defineConfig({
  server: {
    port: 8080,
  },
});
```

--------------------------------

### Evalite with Files on Disk

Source: https://v1.evalite.dev/tips/images-and-media

Shows how to use EvaliteFile.fromPath to reference files directly on disk, enabling display in the UI without reading them into memory.

```javascript
import { EvaliteFile, evalite } from "evalite";


evalite("My Eval", {
  data: [
    {
      input: EvaliteFile.fromPath("path/to/file.jpg"),
    },
  ],
  task: async (input) => {
    console.log(input.path); // "path/to/file.jpg"
  },
  scorers: [],
});
```

--------------------------------

### createScorer with Exact Match

Source: https://v1.evalite.dev/api/create-scorer

Example of creating a simple scorer that checks for exact string equality between the output and expected values.

```typescript
createScorer({
  name: "Exact Match",
  scorer: ({ output, expected }) => (output === expected ? 1 : 0),
});
```

--------------------------------

### Manage Storage Lifecycle with `await using`

Source: https://v1.evalite.dev/api/storage

Utilize the `await using` syntax to manage the lifecycle of storage instances, ensuring proper cleanup. Implement `[Symbol.asyncDispose]()` in your custom storage class for automatic resource management.

```typescript
import { createSqliteStorage } from "evalite/sqlite-storage";


await using storage = createSqliteStorage("./evalite.db");


// Use storage...
// Automatically closed when leaving scope

```

--------------------------------

### Create SQLite Storage Backend

Source: https://v1.evalite.dev/api/storage

Use `createSqliteStorage` to set up persistent storage for evaluation results using a SQLite database file. This backend automatically manages the schema and tracks run history.

```typescript
import {
  defineConfig
} from "evalite/config";
import {
  createSqliteStorage
} from "evalite/sqlite-storage";


export default defineConfig({
  storage: () => createSqliteStorage("./evalite.db"),
});
```

--------------------------------

### Run Evalite Once and Serve UI

Source: https://v1.evalite.dev/api/run-evalite

Use 'run-once-and-serve' mode to run evaluations once and keep the UI server open without watching for changes.

```typescript
import { runEvalite } from "evalite/runner";


// Run once and keep UI open
await runEvalite({
  mode: "run-once-and-serve",
});
```

--------------------------------

### Set Maximum Concurrency

Source: https://v1.evalite.dev/api/define-config

Specify the maximum number of test cases to run in parallel. This example sets the maximum concurrency to 100.

```typescript
export default defineConfig({
  maxConcurrency: 100, // Run up to 100 tests in parallel
});
```

--------------------------------

### Get Evals with Scores

Source: https://v1.evalite.dev/api/storage

Retrieve evaluations for a given suite using `suite_id`, and then fetch the scores associated with those evaluations using `eval_id`.

```typescript
const evals = await storage.evals.getMany({ suite_id: suiteId });
const scores = await storage.scores.getMany({ eval_id: evalId });

```

--------------------------------

### Approve Native Module Builds with pnpm

Source: https://v1.evalite.dev/guides/storage

If using pnpm, approve native module builds to allow better-sqlite3 to compile its native bindings. This is necessary to avoid errors related to missing .node files.

```bash
pnpm approve-builds
```

--------------------------------

### Works With All AI SDK Methods

Source: https://v1.evalite.dev/api/ai-sdk

Illustrates the compatibility of `wrapAISDKModel` with various Vercel AI SDK methods.

```APIDOC
## Works With All AI SDK Methods

### Description
`wrapAISDKModel` works with all Vercel AI SDK methods, including `generateText`, `streamText`, and methods for structured output.

### Generate Example
```javascript
import { generateText } from "ai";
import { openai } from "@ai-sdk/openai";
import { wrapAISDKModel } from "evalite/ai-sdk";

const result = await generateText({
  model: wrapAISDKModel(openai("gpt-4")),
  prompt: "Hello",
});
```

### Stream Example
```javascript
import { streamText } from "ai";
import { openai } from "@ai-sdk/openai";
import { wrapAISDKModel } from "evalite/ai-sdk";

const result = await streamText({
  model: wrapAISDKModel(openai("gpt-4")),
  prompt: "Hello",
});

const text = await result.text;
```

### Structured Output Example
```javascript
import { generateText, Output } from "ai";
import { z } from "zod";
import { openai } from "@ai-sdk/openai";
import { wrapAISDKModel } from "evalite/ai-sdk";

const result = await generateText({
  model: wrapAISDKModel(openai("gpt-4")),
  output: Output.object({ schema: z.object({ name: z.string() }) }),
  prompt: "Generate a person",
});

const person = result.output; // { name: string }
```

### Streaming Structured Output Example
```javascript
import { streamText, Output } from "ai";
import { z } from "zod";
import { openai } from "@ai-sdk/openai";
import { wrapAISDKModel } from "evalite/ai-sdk";

const result = streamText({
  model: wrapAISDKModel(openai("gpt-4")),
  output: Output.object({ schema: z.object({ name: z.string() }) }),
  prompt: "Generate a person",
});

const person = await result.output; // { name: string }
```
```

--------------------------------

### Serve Specific Eval Results

Source: https://v1.evalite.dev/api/cli

Runs a specific evaluation file once and serves its results in the UI.

```bash
# Serve specific eval results
evalite serve example.eval.ts
```

--------------------------------

### Run Evalite in Development Watch Mode

Source: https://v1.evalite.dev/api/run-evalite

Use 'watch-for-file-changes' mode for development. It automatically re-runs evaluations when files change and starts the UI server.

```typescript
import { runEvalite } from "evalite/runner";


// Development mode with watch
await runEvalite({
  mode: "watch-for-file-changes",
});
```

--------------------------------

### Run All Evals

Source: https://v1.evalite.dev/api/cli

Executes all evaluation files in the project.

```bash
# Run all evals
evalite run
```

--------------------------------

### Get Latest Run

Source: https://v1.evalite.dev/api/storage

Retrieve the most recent run by querying for a single run with `limit: 1`. Access the latest run object from the returned array.

```typescript
const runs = await storage.runs.getMany({ limit: 1 });
const latestRun = runs[0];

```

--------------------------------

### Wrap AI SDK Model for Tracing and Caching

Source: https://v1.evalite.dev/tips/vercel-ai-sdk

Wrap your AI SDK models with `wrapAISDKModel` to enable tracing and caching. This single wrapper provides both automatic tracing and intelligent caching of LLM responses.

```typescript
import { openai } from "@ai-sdk/openai";
import { wrapAISDKModel } from "evalite/ai-sdk";


const model = wrapAISDKModel(openai("gpt-4o-mini"));

```

--------------------------------

### Report Trace with Timestamps

Source: https://v1.evalite.dev/api/report-trace

Capture and report trace data including start and end timestamps for performance analysis. Requires manual timing using performance.now().

```typescript
const start = performance.now();
const result = await callLLM(input);
const end = performance.now();


reportTrace({
  input,
  output: result,
  start,
  end,
});
```
Tokens:
Raw