Vally

https://microsoft.github.io/vally
Admin
Vally is a modular platform for agent evaluation, allowing users to define stimuli, capture...
Tokens:61,000
Snippets:772
Trust Score:9.7
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Vally Core Library

Source: https://microsoft.github.io/vally/get-started/install

Install the core Vally package for programmatic use in your projects.

```bash
npm install @microsoft/vally
```

--------------------------------

### Start the Vally Results API Server

Source: https://microsoft.github.io/vally/reference/results-api

Use the `vally serve` command followed by the directory containing Vally results to start the API server.

```bash
vally serve ./vally-results

```

--------------------------------

### Known-domains File Format Example

Source: https://microsoft.github.io/vally/reference/cli/lint

This example demonstrates the format for a known-domains file, including core domains and path-scoped entries. Lines starting with '#' are comments, and blank lines are ignored.

```text
# Core domains
microsoft.com
github.com


# Path-scoped
github.com/dotnet/runtime

```

--------------------------------

### Install Vally CLI

Source: https://microsoft.github.io/vally/get-started/install

Install the Vally command-line tool globally using npm.

```bash
npm install -g @microsoft/vally-cli
```

--------------------------------

### Allowlist File Format Examples

Source: https://microsoft.github.io/vally/reference/cli/lint

Examples demonstrating the format for the allowlist file used with external dependency checks. Each entry specifies the type, skill name, and a detail like a script path or tool reference.

```shell
# Allow the setup script in my-skill
script:my-skill:scripts/setup.ps1
```

```shell
# Allow the INVOKES pattern in nullable-ref skill
invokes:nullable-ref
```

```shell
# Allow a specific tool reference
tool-ref:my-skill:#tool:web/fetch
```

--------------------------------

### SKILL.md Structure Example

Source: https://microsoft.github.io/vally/reference/skill-md

Defines the basic structure of a SKILL.md file including frontmatter, usage, examples, and references.

```yaml
---
name: my-skill-name
description: A concise description of what this skill does.
---


## Usage


Explain when and how to use this skill.


## Examples


- "Example prompt 1"
- "Example prompt 2"


## References


- [Guide](references/guide.md)
- [API docs](references/api.md)
```

--------------------------------

### Complete Vally Evaluation Example

Source: https://microsoft.github.io/vally/reference/eval-spec

A comprehensive example of a Vally evaluation configuration, including skill definition, environment setup, stimuli with prompts and graders, and scoring.

```yaml
name: test-writer-eval
description: Evaluates the test-writer skill
version: "1.0"
type: capability


environment:
  skills:
    - ./my-skill # Skill directory (containing SKILL.md)


defaults:
  runs: 3
  timeout: 2m
  model: gpt-5.5
  executor: copilot-sdk

stimuli:
  - name: basic-test-generation
    prompt: |
      Write unit tests for this function:
      function add(a, b) { return a + b; }
      Save the tests to add.test.js.
    graders:
      - type: file-exists
        name: "test file was created"
        config:
          path: "add.test.js"
      - type: output-contains
        config:
          substring: "test"
    constraints:
      max_turns: 10
      expect_tools: ["write_file"]


  - name: edge-case-empty-input
    prompt: "Write tests for a function that takes no arguments."
    graders:
      - type: file-exists
        config:
          path: "*.test.js"

scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7

```

--------------------------------

### Verify Vally CLI Installation

Source: https://microsoft.github.io/vally/get-started/install

Check if the Vally CLI is installed correctly by running the version command. If you installed using npx, prefix commands with npx.

```bash
vally --version
```

```bash
npx vally --version
```

--------------------------------

### Serve with custom options

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts the HTTP server with custom options for port, host, and CORS settings.

```APIDOC
## Serve with custom options

### Description
Starts the HTTP server with custom options for port, host, and CORS settings.

### Method
`vally serve [directory] [options]`

### Parameters
#### Path Parameters
- **directory** (string) - Optional - Path to eval output directory. Optional when using `--store`.
#### Query Parameters
- **--port** (number) - Optional - Port to listen on. Default: `3200`.
- **--host** (string) - Optional - Host to bind to. Use `0.0.0.0` to expose externally. Default: `127.0.0.1`.
- **--cors** (boolean) - Optional - Enable CORS for cross-origin browser requests. Default: `false`.
- **--allowed-host** (string) - Optional - Additional `Host` header value to accept (repeatable). See Security.
- **--store** (string) - Optional - Persist data to a SQLite file (enables historical mode).

### Examples
```bash
vally serve ./results --port 8080
vally serve ./results --host 0.0.0.0 --cors
```

```

--------------------------------

### Update CI with Vally Commands

Source: https://microsoft.github.io/vally/guides/migrating

Replace existing eval CI steps with Vally commands. This example shows installing the CLI, running lint, and evaluating an eval spec.

```bash
- run: npm install -g @microsoft/vally-cli
- run: vally lint .
- run: vally eval --eval-spec eval.yaml --output-dir ./results
```

--------------------------------

### Vally YAML Configuration Example

Source: https://microsoft.github.io/vally/reference/vally-yaml

Defines paths, environments with commands and servers, and test suites with filters for Vally projects.

```yaml
paths:
  skills: [skills/]
  evals: [evals/]
  results: vally-results/
  evalFilenames: ["eval.yaml", "eval.yml", "*.eval.yaml"]


environments:
  default-workspace:
    skills: [skills/copilot]
    commands: [npm install]
    mcpServers:
      db:
        type: stdio
        command: db-serve
        args: ["--port", "5432"]


suites:
  ci-gate:
    description: "Fast checks for every PR"
    filter:
      priority: [p0, p1]


  safety:
    description: "All safety evals"
    evals: ["evals/safety/**/*.eval.yaml"]


  safety-p0:
    description: "Critical safety evals only"
    evals: ["evals/safety/**/*.eval.yaml"]
    filter: { priority: p0 }
```

--------------------------------

### Node.js Script Example (GraderResult JSON Mode)

Source: https://microsoft.github.io/vally/reference/graders/program

Example of a Node.js script returning a GraderResult JSON object.

```yaml
# Node.js script returning a GraderResult JSON
- type: program
  config:
    program: "node"
    args: ["graders/score.js"]
    timeout: 120s
```

--------------------------------

### Serve from directory

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts an HTTP server to serve evaluation results from a specified directory. This mode is in-memory and does not persist data.

```bash
vally serve ./results
```

```bash
vally serve ./vally-results/
```

--------------------------------

### Vally CLI Grade Workflow Example

Source: https://microsoft.github.io/vally/reference/cli/grade

Demonstrates the typical command-line workflow for using the Vally CLI to re-grade evaluation outcomes after iterating on graders.

```bash
vally grade --eval-spec eval.yaml < outcomes.jsonl
```

--------------------------------

### Matching Arguments Example

Source: https://microsoft.github.io/vally/reference/graders/tool-calls

Illustrates using the 'args' field to match arbitrary call arguments, shown with a 'query' argument.

```yaml
args: { query: "vally" }

```

--------------------------------

### Basic Environment Configuration

Source: https://microsoft.github.io/vally/reference/eval-spec

Defines skills, files to copy, setup commands, Git worktree, and MCP servers for an evaluation.

```yaml
environment:
  skills:
    - ./path/to/my-skill # Skill directory (containing SKILL.md)
  files:
    - src: fixtures/input.txt
      dest: input.txt
    - src: fixtures/test-data # directories are copied recursively
      dest: test-data
  commands:
    - npm install
  git:
    type: worktree
    ref: v2.1.0
    source: ../my-repo
  mcpServers:
    db:
      type: stdio
      command: db-serve
      args: ["--port", "5432"]
    api:
      type: http
      url: http://localhost:3000/mcp

```

--------------------------------

### Python Script Example

Source: https://microsoft.github.io/vally/reference/graders/program

Example of a Python script used as a program grader, inspecting the workspace.

```yaml
# Python script that inspects the workspace
- type: program
  config:
    program: "python"
    args: ["graders/validate_schema.py"]
```

--------------------------------

### Skill Activation Event Example

Source: https://microsoft.github.io/vally/reference/trajectory-format

Indicates that a specific skill was loaded by the agent, including its name, path, and allowed tools.

```json
{
  "type": "skill_activation",
  "timestamp": "...",
  "data": {
    "name": "test-writer",
    "path": "/skills/test-writer/SKILL.md",
    "pluginName": "copilot",
    "allowedTools": ["write_file", "read_file"]
  }
}
```

--------------------------------

### Serve with custom port

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts an HTTP server on a custom port, overriding the default port 3200.

```bash
vally serve ./vally-results/ --port 8080
```

--------------------------------

### Command Execution Evidence Examples

Source: https://microsoft.github.io/vally/reference/graders/run-command

Shows examples of command execution outcomes. A '✔' indicates success with an expected exit code, while a '✘' indicates failure with an unexpected exit code.

```text
✔ Command 'npm test' exited with code 0 (expected 0)
✘ Command 'npm test' exited with code 1 (expected 0)
```

--------------------------------

### Tool Calls Grader Configuration Example

Source: https://microsoft.github.io/vally/reference/graders/tool-calls

Example configuration for the tool calls grader, specifying required, disallowed, and sequential tool calls.

```yaml
graders:
  - type: tool-calls
    config:
      required:
        - create
        - name: bash
          command: "npm test"
      disallowed:
        - name: view
          path: "secret\.env$"
      sequence:
        - name: validate
        - name: create

```

--------------------------------

### Query API with curl

Source: https://microsoft.github.io/vally/reference/cli/serve

Examples of using curl to interact with the REST API endpoints exposed by the `serve` command. These examples demonstrate fetching a list of runs and run-specific ranking data.

```bash
curl http://127.0.0.1:3200/api/runs
curl http://127.0.0.1:3200/api/runs/<run-id>/ranking
```

--------------------------------

### Serve from an existing database

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts a server using data from an existing SQLite database file. This mode enables historical data exploration without needing a directory of results.

```APIDOC
## Serve from an existing database

### Description
Starts an HTTP server using data from an existing SQLite database file. This mode enables historical data exploration without needing a directory of results.

### Method
`vally serve --store <path>`

### Parameters
#### Query Parameters
- **--store** (string) - Required - Path to the SQLite database file.

### Examples
```bash
vally serve --store eval-history.db
```

```

--------------------------------

### Passing Custom Environment Variables

Source: https://microsoft.github.io/vally/reference/graders/program

Example demonstrating how to pass custom environment variables to the grader program.

```yaml
# Pass custom environment variables to the grader
- type: program
  config:
    program: python
    args: [-m, my_checker]
    env:
      API_ENDPOINT: "https://test.example.com"
      EXPECTED_STATUS: "active"
```

--------------------------------

### Workspace Diff Status Examples

Source: https://microsoft.github.io/vally/reference/graders/diff-empty

These examples illustrate the possible outcomes when using the diff-empty grader, indicating whether the workspace diff is empty, not empty, or unavailable.

```text
✔ Workspace diff is empty

```

```text
✘ Workspace diff is not empty — the stimulus changed the workspace

```

```text
✘ No workspace diff is available for this run. …

```

--------------------------------

### Program Grader Configuration Example

Source: https://microsoft.github.io/vally/reference/graders/program

This snippet shows the basic configuration for the program grader, specifying the program to run, arguments, and timeout.

```yaml
graders:
  - type: program
    config:
      program: "python"
      args: ["graders/check_output.py"]
      timeout: 60s
      sub_path: "src/app"
```

--------------------------------

### Serve from a directory

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts a server that provides a REST API and web dashboard for exploring eval results from a specified directory. This mode is in-memory and does not persist data.

```APIDOC
## Serve from a directory

### Description
Starts an HTTP server that provides a REST API and built-in web dashboard for exploring eval results from a specified directory. This mode is in-memory and does not persist data.

### Method
`vally serve [directory]`

### Parameters
#### Path Parameters
- **directory** (string) - Required - Path to eval output directory.

### Examples
```bash
vally serve ./results
```

```

--------------------------------

### Program Grader Evidence Examples

Source: https://microsoft.github.io/vally/reference/graders/program

These examples illustrate common messages seen in grader evidence, covering successful execution, failures, timeouts, and output errors.

```text
✔ Grader exited successfully
✘ Grader exited with exit code 1
✘ Grader timed out
✘ Failed to start grader program: ENOENT (spawn)
✘ Grader returned unparseable JSON output on stdout: SyntaxError: …
✘ Grader output did not match schema
```

--------------------------------

### Bash Script Example (Exit-Code Mode)

Source: https://microsoft.github.io/vally/reference/graders/program

Example of a Bash script using the program grader in exit-code mode for simple pass/fail.

```yaml
# Bash script using exit-code mode
- type: program
  config:
    program: "bash"
    args: ["graders/check.sh"]
    shell: true
```

--------------------------------

### Serve from database

Source: https://microsoft.github.io/vally/reference/cli/serve

Starts an HTTP server that serves evaluation results directly from an existing SQLite database file. This enables historical mode, preserving data across sessions.

```bash
vally serve --store eval-history.db
```

```bash
vally serve ./vally-results/ --store eval-history.db
```

--------------------------------

### Markdown File References

Source: https://microsoft.github.io/vally/reference/skill-md

Illustrates how Markdown links are parsed as file references, with examples of valid and ignored links.

```markdown
[Guide](references/guide.md)
[API docs](references/api.md#section)
[External](https://example.com) <!-- ignored: external URL -->
```

--------------------------------

### Serve evaluation results from a database

Source: https://microsoft.github.io/vally/reference/cli/ingest

After ingesting results into a specific database file, this command can be used to serve that data. Note that this example uses `./vally-results/` as the directory, which might be a placeholder and should align with the directory used for ingestion if serving the same data.

```bash
# Then serve from the database
vally serve ./vally-results/ --store my-evals.db
```

--------------------------------

### Panel grader output example

Source: https://microsoft.github.io/vally/reference/graders/panel

Example console output showing the aggregated panel result and individual judge scores. The aggregate score is used for pass/fail decisions.

```text
✔ panel  PASS via mean aggregation across 3 judge(s):
         claude-sonnet-4.6=0.80, gpt-5.5=0.60, gpt-5.5-mini=0.80 → 0.73
  ✔ panel/claude-sonnet-4.6  Clear, mentions sorted input...
  ✔ panel/gpt-5.5             Concise but skips the sorted-input precondition.
  ✔ panel/gpt-5.5-mini        Good explanation of halving...

```

--------------------------------

### Define a Stimulus with Graders and Constraints

Source: https://microsoft.github.io/vally/concepts/how-it-works

Example of a Stimulus defining a prompt, grading criteria, and resource constraints.

```yaml
stimuli:
  - name: basic-test-generation
    prompt: |
      Write unit tests for this function:
      function add(a, b) { return a + b; }
    graders:
      - type: file-exists
        config:
          path: "add.test.js"
      - type: output-contains
        config:
          substring: "test"
    constraints:
      max_turns: 10
      max_tokens: 5000
```

--------------------------------

### Configure Vally Eval Suite

Source: https://microsoft.github.io/vally/guides/migrating

Set up your eval.yaml to combine stimuli and configure scoring. This example sets default runs, timeout, model, and a scoring threshold.

```yaml
name: migrated-eval-suite
type: capability
defaults:
  runs: 3
  timeout: 120s
  model: gpt-5.5
stimuli:
  # All your converted test cases
scoring:
  threshold: 0.7
```

--------------------------------

### Running Vally Evaluation Suites in CI

Source: https://microsoft.github.io/vally/guides/authoring-eval-suites

Integrate Vally evaluation suites into your CI pipeline using GitHub Actions. This example demonstrates how to check out code and run specific suites like 'safety' and 'fast'.

```yaml
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run safety suite
        run: vally eval --suite safety

      - name: Run fast suite
        run: vally eval --suite fast
```

--------------------------------

### Token Usage Event Example

Source: https://microsoft.github.io/vally/reference/trajectory-format

Logs LLM token counts from a single API call, including input, output, and cached tokens.

```json
{
  "type": "token_usage",
  "timestamp": "2025-01-15T10:30:00.500Z",
  "data": {
    "inputTokens": 1500,
    "outputTokens": 350,
    "model": "gpt-5.5",
    "cacheReadTokens": 200,
    "cacheWriteTokens": 0
  }
}
```

--------------------------------

### Panel Grader Configuration Example

Source: https://microsoft.github.io/vally/reference/graders/panel

Configure the panel grader with specific models, aggregation method, scoring scale, and a passing threshold. The 'prompt' field allows for optional extra instructions for judges.

```yaml
graders:
  - type: panel
    config:
      models: [claude-sonnet-4.6, gpt-5.5, gpt-5.5-mini]
      aggregation: mean
      scoring: scale_1_5
      threshold: 0.5
      prompt: "Optional extra instructions for every judge."

```

--------------------------------

### Minimal Experiment File Example

Source: https://microsoft.github.io/vally/reference/experiment-file

A basic experiment file declaring a comparison between two models across specified evaluation files. Use this structure to define experiment name, evals, varying parameters, baseline, and specific variants.

```yaml
name: model-comparison
evals:
  - evals/auth/eval.yaml
  - evals/search/eval.yaml

vary: [/defaults/model]

baseline: gpt-5.5
variants:
  gpt-5.5:
    overrides:
      model: gpt-5.5
  claude-sonnet-4.6:
    overrides:
      model: claude-sonnet-4.6

```

--------------------------------

### Ingest directory into a database and serve

Source: https://microsoft.github.io/vally/reference/cli/serve

Ingests evaluation results from a directory into a SQLite database and then starts an HTTP server to serve the data. This allows for persistent storage and historical analysis.

```APIDOC
## Ingest directory into a database and serve

### Description
Ingests evaluation results from a directory into a SQLite database and then starts an HTTP server to serve the data. This allows for persistent storage and historical analysis.

### Method
`vally serve [directory] --store <path>`

### Parameters
#### Path Parameters
- **directory** (string) - Required - Path to eval output directory.
#### Query Parameters
- **--store** (string) - Required - Path to the SQLite database file.

### Examples
```bash
vally serve ./results --store eval-history.db
```

```

--------------------------------

### Tool Call Event Example

Source: https://microsoft.github.io/vally/reference/trajectory-format

Represents an event where the agent invokes a tool, including the tool name, call ID, and arguments.

```json
{
  "type": "tool_call",
  "timestamp": "2025-01-15T10:30:00.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "arguments": { "path": "add.test.js", "content": "..." }
  }
}
```

--------------------------------

### Grader Model Selection Example

Source: https://microsoft.github.io/vally/reference/graders/prompt

Demonstrates how grader-specific configurations can override global defaults. The first grader uses 'o3', while the second implicitly uses the default judge model.

```yaml
stimuli:
  - name: test-case
    graders:
      - type: prompt
        config:
          model: o3 # this specific grader uses o3
      - type: prompt # this one uses gpt-5.5 (from judge_model)
```

--------------------------------

### Define Strict and Broad Filters in Vally Suite

Source: https://microsoft.github.io/vally/guides/authoring-eval-suites

Use these examples to understand how to define strict (AND logic) and broad (OR logic) filters within your Vally evaluation suites. Ensure tag values are strings.

```yaml
# Strict: must be p0 AND auth AND free — very narrow
strict:
  filter:
    priority: p0
    area: auth
    cost: free


# Broad: p0 OR p1 OR p2 — very wide
broad:
  filter:
    priority: [p0, p1, p2]
```

--------------------------------

### Max-Repeat Grader Use Cases

Source: https://microsoft.github.io/vally/reference/graders/max-repeat

Examples demonstrating various configurations for the max-repeat grader. These include setting a general maximum repeat count, focusing on specific tools, matching based on name or name-arguments, and applying different thresholds per tool.

```yaml
# Fail if the agent re-runs the same action (same output) more than 3 times in a row
- type: max-repeat
  config:
    max: 3

```

```yaml
# Only watch bash; fail on 3+ identical bash invocations in a row (ignore arguments/output)
- type: max-repeat
  config:
    max: 2
    match: name
    tools: ["^bash$"]

```

```yaml
# Fail if the agent calls the same tool with the same arguments repeatedly
- type: max-repeat
  config:
    max: 2
    match: name-args

```

```yaml
# Different thresholds per tool: stricter on web_search than on bash
- type: max-repeat
  config:
    max: 2
    tools: ["^web_search$"]
- type: max-repeat
  config:
    max: 5
    tools: ["^bash$"]

```

--------------------------------

### Specify backend for trial execution

Source: https://microsoft.github.io/vally/reference/cli/experiment

Use the `--backend` option to specify the backend for running trials. This can be a built-in name like `local` or a plugin package/path.

```bash
vally experiment run <experiment-file> --backend <name-or-package>
```

--------------------------------

### Calculate Aggregate Score with Weights

Source: https://microsoft.github.io/vally/concepts/scoring

Example of calculating an aggregate score when graders have different weights. This example shows a file-exists grader scoring 1.0 and an output-contains grader scoring 0.0.

```plaintext
0.7 × 1.0 + 0.3 × 0.0 = 0.70 → equals 0.7 → PASS
```

--------------------------------

### Create a Skill Directory

Source: https://microsoft.github.io/vally/get-started/run-your-first-eval

Use this command to create a new directory for your skill if you don't have one already.

```bash
mkdir -p my-skill && cd my-skill
```

--------------------------------

### Turn Start and End Events

Source: https://microsoft.github.io/vally/reference/trajectory-format

Marks the beginning and end of a conversation turn, identified by a unique turn ID.

```json
{ "type": "turn_start", "timestamp": "...", "data": { "turnId": "turn-1" } }
```

```json
{ "type": "turn_end",   "timestamp": "...", "data": { "turnId": "turn-1" } }
```

--------------------------------

### Error Event Example

Source: https://microsoft.github.io/vally/reference/trajectory-format

Logs an error that occurred during the agent's run, including a message, type, and error code.

```json
{
  "type": "error",
  "timestamp": "...",
  "data": {
    "message": "Request timed out",
    "type": "TimeoutError",
    "code": 408
  }
}
```

--------------------------------

### Basic usage of vally compare

Source: https://microsoft.github.io/vally/reference/cli/compare

The basic usage of the `vally compare` command. It can take an experiment directory or two specific run paths for baseline and treatment.

```bash
vally compare <experiment-dir> [options]
vally compare --baseline <path> --treatment <path> [options]
```

--------------------------------

### Minimal .vally.yaml Configuration

Source: https://microsoft.github.io/vally/reference/vally-yaml

Sets the directory for discovering eval files. Place this file at your project root.

```yaml
paths:
  evals: evals/

```

--------------------------------

### Tool Result Event Example

Source: https://microsoft.github.io/vally/reference/trajectory-format

Represents an event where a tool returns its result, indicating success or failure and the returned data.

```json
{
  "type": "tool_result",
  "timestamp": "2025-01-15T10:30:01.000Z",
  "data": {
    "toolName": "write_file",
    "toolCallId": "call_abc123",
    "success": true,
    "result": "File written successfully"
  }
}
```

--------------------------------

### Diff-Contains Grader Configuration

Source: https://microsoft.github.io/vally/reference/graders/diff-contains

Example configuration for the diff-contains grader, specifying a regex pattern to match against the unified diff text.

```yaml
graders:
  - type: diff-contains
    config:
      pattern: "def divide"
```

--------------------------------

### Grade with Verbose Output

Source: https://microsoft.github.io/vally/reference/cli/grade

Use the --verbose flag to get detailed output during the grading process. This is useful for debugging.

```bash
# Grade with verbose output
vally grade --eval-spec eval.yaml --verbose < results/outcomes.jsonl
```

--------------------------------

### Vally Experiment Output Directory Structure

Source: https://microsoft.github.io/vally/reference/cli/experiment

Explains the directory structure created by Vally for experiment runs, including timestamped directories and per-variant subdirectories containing results.

```bash
<output-dir>/
└── 2026-06-09T20-41-10-885Z/         # timestamped run directory
    ├── report.md                      # experiment-level markdown report
    ├── baseline/                      # one subdirectory per variant
    │   ├── results.jsonl              # trial-result records + experiment block
    │   └── run-summary.jsonl
    └── treatment/
        ├── results.jsonl
        └── run-summary.jsonl

```

--------------------------------

### Run Every Variant of an Experiment

Source: https://microsoft.github.io/vally/reference/cli/experiment

Execute all defined variants within an experiment configuration file.

```bash
vally experiment run experiments/skill-comparison.yaml
```

--------------------------------

### Basic Test Generation - Passing Scenario

Source: https://microsoft.github.io/vally/reference/cli/grade

Illustrates the output format when a basic test generation passes all graders. It shows which files were found and that the expected output was contained.

```text
━━━ basic-test-generation ━━━
✅ basic-test-generation (2/2 graders passed)
  ✓ [file-exists] Files matching 'add.test.js' found: add.test.js
  ✓ [output-contains] 'test' found in output


Score: 100.0% | PASSED

```

--------------------------------

### Use Bash or PowerShell

Source: https://microsoft.github.io/vally/reference/graders/tool-calls

Allows the agent to use either 'bash' or 'powershell' by matching tool names using a regular expression. This provides flexibility in shell selection.

```yaml
- type: tool-calls
  config:
    required:
      - name: "^(bash|powershell)$"
```

--------------------------------

### Example Grader Evidence

Source: https://microsoft.github.io/vally/guides/debugging-evals

Examine the `evidence` field in grader results to understand why an eval passed or failed. This often directly indicates the problem.

```text
✘ file-exists     No files matching 'add.test.js' found
✔ output-contains 'test' found in output
```

--------------------------------

### Multi-Turn Conversation Stimulus

Source: https://microsoft.github.io/vally/reference/eval-spec

Configures a stimulus for multi-turn conversations using the 'turns' field instead of 'prompt'. Includes an example grader for checking output.

```yaml
stimuli:
  - name: recall-across-turns
    turns:
      - "Remember this code name: ALPHA-7."
      - "What was the code name I gave you?"
    graders:
      - type: output-contains
        turn: 1
        config:
          substring: "ALPHA-7"
```

--------------------------------

### GraderResult JSON Schema Example

Source: https://microsoft.github.io/vally/reference/graders/program

This JSON object conforms to the GraderResult schema, used for returning custom scores, evidence, and metadata from a grader program.

```json
{
  "name": "my-custom-check",
  "passed": true,
  "score": 0.85,
  "evidence": "14 of 16 assertions passed",
  "kind": "code"
}
```

--------------------------------

### Configure Eval Discovery Paths

Source: https://microsoft.github.io/vally/concepts/how-it-works

Customize directories and filenames for discovering eval files.

```yaml
paths:
  evals: [evals/, tests/evals/] # scan multiple directories
  evalFilenames: ["eval.yaml", "*.eval.yaml"] # custom filename patterns
```

--------------------------------

### Get Metrics

Source: https://microsoft.github.io/vally/reference/results-api

Retrieves definitions for all available metrics. Each metric includes its ID, unit, direction, and aggregation type, which can be used by agents to interpret values.

```APIDOC
## GET /api/metrics

### Description
Retrieves definitions for all available metrics.

### Method
GET

### Endpoint
/api/metrics

### Response
#### Success Response (200)
- **id** (string) - The unique identifier for the metric.
- **unit** (string) - The unit of measurement for the metric (e.g., ratio, ms, count, tokens).
- **direction** (string) - Indicates whether higher or lower values are better (higher_is_better, lower_is_better, or neutral).
- **aggregation** (string) - The aggregation type used for the metric (e.g., rate, mean, sum, p50, p95).

#### Response Example
{
  "metrics": [
    {
      "id": "latency",
      "unit": "ms",
      "direction": "lower_is_better",
      "aggregation": "p95"
    }
  ]
}
```

--------------------------------

### Search with Specific Query

Source: https://microsoft.github.io/vally/reference/graders/tool-calls

Ensures the 'web_search' tool is called with the argument 'query' set to 'release notes'. This demonstrates how to match specific arguments for tool calls.

```yaml
- type: tool-calls
  config:
    required:
      - name: "^web_search$"
        args:
          query: "release notes"
```

--------------------------------

### Example grader metadata

Source: https://microsoft.github.io/vally/reference/graders/metric-thresholds

This JSON object shows the structure of metadata included with every grader result, indicating the computed value and the configured maximum threshold.

```json
{
  "value": 1500,
  "max": 50000
}

```

--------------------------------

### Pass arguments to the backend

Source: https://microsoft.github.io/vally/reference/cli/experiment

Use `--backend-args` to pass key-value arguments to the selected backend. Repeated keys are collected as arrays. Requires `--backend` to be set.

```bash
vally experiment run <experiment-file> --backend-args <key=value>
```

--------------------------------

### Use Executor Plugin from CLI

Source: https://microsoft.github.io/vally/guides/writing-custom-executors

Load custom executor plugins using the `--executor-plugin` flag with either an npm package name or a local path. Multiple plugins can be loaded for different eval specs.

```bash
# npm package
vally eval --executor-plugin @myorg/vally-executor-custom --eval-spec eval.yaml


# local path (useful during development)
vally eval --executor-plugin ./my-executor --eval-spec eval.yaml


# multiple executor plugins — useful when a single invocation runs several
# eval specs (e.g., via -e A.yaml -e B.yaml or a suite) that select
# different executors via their `defaults.executor` field
vally eval --executor-plugin @myorg/exec-a --executor-plugin @myorg/exec-b -e A.yaml -e B.yaml
```

--------------------------------

### Scoring Configuration Example

Source: https://microsoft.github.io/vally/concepts/how-it-works

Defines how grader results are aggregated into a final score using weights and a pass threshold. This configuration is typically part of an evaluation specification.

```yaml
scoring:
  weights:
    file-exists: 0.7
    output-contains: 0.3
  threshold: 0.7
```

--------------------------------

### Convert Test Case to Vally Stimulus

Source: https://microsoft.github.io/vally/guides/migrating

Map a test case to a Vally stimulus in eval.yaml. This example shows how to define a stimulus with a prompt and a file-existence grader.

```yaml
stimuli:
  - name: test-generation
    prompt: "Write tests for the add() function"
    graders:
      - type: file-exists
        config:
          path: "add.test.js"
```

--------------------------------

### Start with a single-judge baseline

Source: https://microsoft.github.io/vally/reference/graders/panel

This is the baseline configuration for a single-judge evaluation using the 'prompt' grader. It sets up default executors and runs, and defines stimuli with prompts and rubrics.

```yaml
defaults:
  executor: copilot-sdk
  runs: 1


stimuli:
  - name: explain-binary-search
    prompt: "Explain how binary search works in 3 sentences."
    rubric:
      - "Mentions sorted input is required"
      - "Describes halving the search range"
      - "Stays at roughly 3 sentences"
    graders:
      - type: prompt

```

--------------------------------

### ToolMatch Object Configuration

Source: https://microsoft.github.io/vally/reference/graders/tool-calls

Demonstrates the structure of a ToolMatch object for detailed tool argument filtering.

```yaml
name: string
command: string
path: string
args: Record<string, string>
result: string
min_count: number
final: boolean
at_step: number
before_step: number

```

--------------------------------

### Compute Multi-Trial Metrics

Source: https://microsoft.github.io/vally/reference/scoring-functions

Computes various multi-trial metrics from a list of trial pass/fail results. The example shows how to use it with a boolean array representing trial outcomes.

```typescript
function computeMultiTrialMetrics(trialPassed: boolean[]): MultiTrialMetrics;
```

```typescript
computeMultiTrialMetrics([true, true, false, true, true]);
// → { perTrialPassRate: 0.8, passAtK: 0.99, passToTheK: 0.33, k: 5 }
```

--------------------------------

### Tool Count Grader Configuration in eval.yaml

Source: https://microsoft.github.io/vally/guides/writing-custom-graders

Example of how to configure and use the 'tool-count' grader within an eval.yaml file, specifying minimum and maximum allowed tool calls.

```yaml
graders:
  - type: tool-count
    config:
      min: 1
      max: 10
```

--------------------------------

### Run an experiment

Source: https://microsoft.github.io/vally/reference/cli/experiment

Execute an experiment defined in a YAML file. This command resolves and runs the specified experiment, comparing different variants.

```bash
vally experiment run <experiment-file> [options]
```

--------------------------------

### Vally Score Report Structure

Source: https://microsoft.github.io/vally/concepts/scoring

This is an example of a Vally score report, detailing overall scores, stimulus-specific scores, pass rates, and flakiness for different test cases.

```text
SkillScore: test-writer
├── overallScore: 0.85           ← average across stimuli
├── passed: true                 ← overall score meets the pass threshold
│
├── basic-test-generation
│   ├── aggregateScore: 0.95     ← this stimulus scores well
│   ├── pass@3: 100%             ← agent can definitely do this
│   ├── pass^3: 85.7%            ← and does it reliably
│   └── flaky: false             ← consistent results
│
└── edge-case-handling
    ├── aggregateScore: 0.75     ← this stimulus is weaker
    ├── pass@3: 97.5%            ← agent CAN do it
    ├── pass^3: 42.2%            ← but not reliably
    └── flaky: true (33.3%)      ← 1 in 3 trials fails

```

--------------------------------

### Configure output-matches grader

Source: https://microsoft.github.io/vally/reference/graders/output-matches

Use this configuration to set up the output-matches grader with a specific regex pattern.

```yaml
graders:
  - type: output-matches
    config:
      pattern: "function\s+\w+\("

```

--------------------------------

### Grade trajectories from stdin or file

Source: https://microsoft.github.io/vally/reference/cli/grade

Use this command to grade trajectories. Input can be piped from `vally eval` or provided as a file. Options can be specified to customize the grading process.

```bash
vally eval --output jsonl | vally grade --eval-spec eval.yaml [options]
vally grade --eval-spec eval.yaml [options] < outcomes.jsonl
```

--------------------------------

### Outer Loop Grader Configuration

Source: https://microsoft.github.io/vally/concepts/grader-taxonomy

Include prompt graders (LLM judges) and increase runs for statistical confidence in the outer loop. This example uses 5 runs and gpt-5.5.

```yaml
defaults:
  runs: 5
  judge_model: gpt-5.5
stimuli:
  - name: quality-check
    prompt: "..."
    graders:
      - type: file-exists
        config: { path: "*.test.js" }
      - type: prompt # llm, high cost
        config:
          prompt: "Are the tests comprehensive?"

```

--------------------------------

### Matrix Configuration for Cross-Product Variants

Source: https://microsoft.github.io/vally/reference/experiment-file

Define axes with paths and values in a `matrix` block to automatically expand into multiple variants. This simplifies managing experiments with varying parameters like models and skill sets.

```yaml
name: model-x-skills
evals:
  - evals/auth/eval.yaml

matrix:
  model:
    path: /defaults/model
    values: [gpt-5.5, claude-sonnet-4.6]
  skills:
    path: /environment/skills
    values:
      - none: []
      - matched: ["plugins/${eval.parent}/skills/*"]


baseline:
  model: gpt-5.5
  skills: none

```

--------------------------------

### Typical Skill Directory Structure

Source: https://microsoft.github.io/vally/reference/skill-md

Shows the standard directory layout for a Copilot skill, including the SKILL.md file, optional references, and evaluation files.

```directory
my-skill/
├── SKILL.md              # Required: skill definition
├── references/           # Optional: supporting files
│   ├── guide.md
│   ├── api-reference.md
│   └── examples/
│       └── usage.py
└── eval.yaml             # Optional: evaluation stimuli
```

--------------------------------

### Direct exec with explicit args

Source: https://microsoft.github.io/vally/reference/graders/run-command

Execute a command with explicit arguments, ensuring cross-platform compatibility and reliable exit codes. Verifies stdout contains 'OK'.

```yaml
- type: run-command
  config:
    command: "node"
    args: ["validate.js", "--strict"]
    expected_exit_code: 0
    stdout_contains: "OK"
```

--------------------------------

### Vally Development Loop Workflow

Source: https://microsoft.github.io/vally/concepts/why-vally

Vally supports the entire agent development lifecycle, from initial coding and local linting to thorough evaluations, automated CI gating, and continuous regression tracking in production.

```text
Write code  →  Lint locally  →  Run evals  →  Gate in CI  →  Track regressions
               (instant)       (thorough)    (automated)     (nightly)
```

--------------------------------

### SKILL.md Output with Skill Assessment

Source: https://microsoft.github.io/vally/reference/cli/lint

Presents the output from linting a SKILL.md file, detailing the status of found skills and their checks.

```bash
Found 2 skill(s): auth-helper, code-reviewer


✔ auth-helper
  ✔ spec-compliance (4/4 checks passed)
  ✔ valid-refs (3 references, 0 broken)


✘ code-reviewer
  ✘ spec-compliance (3/4 checks passed)
    ✗ name-format: Name must be kebab-case
  ✔ valid-refs


1/2 skills passed
```
Tokens:
Raw