### Promptfoo Configuration with Custom Providers

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Example promptfoo configuration demonstrating how to reference custom providers using file:// URLs. Supports direct file references, wildcard patterns, and directory watching.

```yaml
providers:
  - file://custom_provider.py
  - id: file://providers/my_provider.js
```

```yaml
providers:
  - file://providers/*.py          # All Python files in providers/
  - file://lib/**/*.js            # All JS files recursively in lib/
```

```yaml
providers:
  - file://providers/             # Watch entire directory
```

--------------------------------

### Promptfoo Configuration with File Watching

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Example promptfoo configuration showcasing file watching for providers, prompts, test data, and validators using file:// URLs and wildcard patterns. The action automatically triggers evaluations when these files change.

```yaml
# promptfooconfig.yaml
providers:
  - file://providers/**/*.py      # Watch all Python files recursively
  
prompts:
  - file://prompts/               # Watch entire prompts directory

tests:
  - vars:
      context: file://data/*.json # Watch all JSON files in data/
    assert:
      - type: javascript
        value: file://validators/ # Watch all files in validators/
```

--------------------------------

### Run All Build and Package Commands

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Execute build, lint, package, and tests sequentially. Ensure all steps are completed before committing.

```bash
npm run all
```

--------------------------------

### Configure Promptfoo Action Inputs

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Configure the promptfoo-action with required and optional inputs for LLM provider keys and behavior flags.

```yaml
# .github/workflows/prompt-eval.yml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    # --- Required ---
    github-token: ${{ secrets.GITHUB_TOKEN }}   # Posts PR comments / workflow summaries
    config: 'promptfooconfig.yaml'              # Path to promptfoo config

    # --- LLM provider keys (can also be set as env vars) ---
    openai-api-key:      ${{ secrets.OPENAI_API_KEY }}
    anthropic-api-key:   ${{ secrets.ANTHROPIC_API_KEY }}
    azure-api-key:       ${{ secrets.AZURE_OPENAI_API_KEY }}
    huggingface-api-key: ${{ secrets.HF_API_TOKEN }}
    mistral-api-key:     ${{ secrets.MISTRAL_API_KEY }}
    groq-api-key:        ${{ secrets.GROQ_API_KEY }}
    cohere-api-key:      ${{ secrets.COHERE_API_KEY }}

    # --- Optional behaviour flags ---
    prompts: |            # Glob patterns for prompt files to watch (one per line)
      prompts/**/*.txt
      prompts/**/*.md
    promptfoo-version: 'latest'   # Pin a specific promptfoo release
    working-directory: '.'        # Run promptfoo from a sub-directory
    cache-path: '.promptfoo-cache'
    no-share: false               # Set true to disable shareable result URLs
    use-config-prompts: false     # Use prompts declared inside the config file
    env-files: '.env,.env.local'  # Comma-separated .env files to load
    fail-on-threshold: 80         # Fail if pass rate drops below 80%
    max-concurrency: 4            # Max parallel LLM API calls
    no-table: false               # Suppress ASCII table in logs
    no-progress-bar: false        # Suppress progress bar in logs
    no-cache: false               # Bypass promptfoo's disk cache
    disable-comment: false        # Do not post PR comments
    force-run: false              # Evaluate even if no files changed

```

--------------------------------

### Run Promptfoo Evaluation and Display Cache Metrics

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Runs a promptfoo evaluation using the promptfoo-action and then displays the cache size and file count from the action's outputs. This helps in monitoring cache performance.

```yaml
- name: Run evaluation
  id: eval
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    cache-path: '.promptfoo-cache'

- name: Display cache metrics
  run: |
    echo "Cache size: ${{ steps.eval.outputs.cache-size-mb }}MB"
    echo "Cache files: ${{ steps.eval.outputs.cache-file-count }}"

```

--------------------------------

### Run Biome Lint and Format

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Apply code formatting and linting rules using Biome. This ensures code quality and consistency.

```bash
npm run biome
```

--------------------------------

### Loading .env Files for Environment Variables

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This configuration shows how to load one or more .env files before running the evaluation. Later files in the list override earlier ones, which is helpful for managing environment-specific variables.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    env-files: '.env,.env.test.local'
    # Variables from .env.test.local override those in .env
    # Useful for Next.js apps or multi-environment setups
```

--------------------------------

### Promptfoo Action with GitHub Actions Caching

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Set up GitHub Actions caching for promptfoo evaluations to improve performance and reduce costs. This configuration includes caching the promptfoo cache directory and using a cache key based on file hashes.

```yaml
name: 'Prompt Evaluation with Caching'
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v5
        with:
          fetch-depth: 0  # Required for git diff comparisons

      # IMPORTANT: Use actions/cache@v4 or later (required after Feb 1, 2025)
      - name: Cache promptfoo evaluations
        uses: actions/cache@v4
        with:
          path: |
            ~/.promptfoo/cache
            .promptfoo-cache
          # Cache key includes content hash for automatic invalidation
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          config: 'promptfooconfig.yaml'
          cache-path: '.promptfoo-cache'  # Local cache directory
```

--------------------------------

### Format Code with Biome

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Apply code formatting rules to the project. This ensures consistent code style across the repository.

```bash
npm run format
```

--------------------------------

### Run Promptfoo Evaluation with API Key Validation and Sharing

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Configure the promptfoo-action to run an evaluation while validating the `PROMPTFOO_API_KEY`. If valid, results are shared, producing a shareable URL. For self-hosted instances, override the API host using `PROMPTFOO_REMOTE_API_BASE_URL`.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  env:
    PROMPTFOO_API_KEY: ${{ secrets.PROMPTFOO_API_KEY }}
    # Self-hosted: override the API host
    # PROMPTFOO_REMOTE_API_BASE_URL: https://promptfoo.my-company.com
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config:       'promptfooconfig.yaml'
```

--------------------------------

### Environment Variable API Keys

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This workflow demonstrates using environment variables for API keys, which are automatically forwarded to promptfoo. This is useful when the same keys are needed across multiple steps. Requires GitHub token.

```yaml
name: Prompt Evaluation

on:
  pull_request:
    paths: ['prompts/**']

env:
  OPENAI_API_KEY:    ${{ secrets.OPENAI_API_KEY }}
  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4
      - uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
          # No need to repeat API keys here — env vars above are inherited
```

--------------------------------

### Prompt Evaluation on Push Event

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Set up a GitHub Actions workflow to automatically evaluate prompts on every push to the main branch. Requires `actions: write` permission for workflow summaries.

```yaml
name: 'Prompt Evaluation - Push'
on:
  push:
    branches:
      - main
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: write # Required for workflow summaries
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2 # Need at least 2 commits for comparison

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@main
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
```

--------------------------------

### Run Promptfoo Evaluation with Caching

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Execute a promptfoo evaluation using the promptfoo-action, specifying a cache path for the action's internal cache. This complements the GitHub Actions cache for efficient evaluations.

```yaml
  # Layer 2: Action configures promptfoo's internal cache (TTL, size limits)
  - name: Run promptfoo evaluation
    id: eval
    uses: promptfoo/promptfoo-action@v1
    with:
      github-token: ${{ secrets.GITHUB_TOKEN }}
      config:       'promptfooconfig.yaml'
      cache-path:   '.promptfoo-cache'   # Tells the action where to store the cache
```

--------------------------------

### Load .env Files in Promptfoo Action

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configure the promptfoo-action to load environment variables from specified .env files before running evaluations. This is useful for frameworks like Next.js that use .env files for configuration.

```yaml
name: 'Prompt Evaluation'
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
          env-files: '.env,.env.test.local' # Load multiple .env files
```

--------------------------------

### Set Environment Variables for Promptfoo Action

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configure environment variables at the job or workflow level to pass API keys to promptfoo. Action inputs take precedence over environment variables.

```yaml
env:
  OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

steps:
  - uses: promptfoo/promptfoo-action@v1
    with:
      github-token: ${{ secrets.GITHUB_TOKEN }}
      config: 'promptfooconfig.yaml'
```

--------------------------------

### Weekly Cache Rotation with actions/cache

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configures weekly cache rotation using the actions/cache@v4 action. It specifies the cache path, a key that includes the OS, prompt file hashes, and the weekly rotation key, and sets up restore keys for fallback.

```yaml
- name: Cache with weekly rotation
  uses: actions/cache@v4
  with:
    path: ~/.promptfoo/cache
    # Weekly rotation ensures fresh results
    key: promptfoo-${{ runner.os }}-${{ hashFiles('prompts/**') }}-${{ steps.cache-key.outputs.week }}
    restore-keys: |
      promptfoo-${{ runner.os }}-${{ hashFiles('prompts/**') }}-

```

--------------------------------

### Custom Provider Dependency Detection Configuration

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This YAML configuration for promptfoo demonstrates how file:// references in providers, prompts, and test variables are automatically tracked. Changes to these files trigger evaluations.

```yaml
# promptfooconfig.yaml — all file:// references are auto-tracked
providers:
  - file://providers/**/*.py      # Glob: triggers if any .py file in providers/ changes
  - id: file://lib/my_provider.js # Single file

prompts:
  - file://prompts/               # Directory: triggers if anything inside changes

tests:
  - vars:
      context: file://data/*.json # Glob: triggers on any JSON change in data/
    assert:
      - type: javascript
        value: file://validators/check.js
```

--------------------------------

### Run All Tests with Coverage

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Execute all Jest tests and generate a coverage report. Verify test suites from both branches were preserved after merges.

```bash
npm test
```

--------------------------------

### Configure GitHub Actions Cache for Promptfoo

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Set up GitHub Actions cache to persist promptfoo's internal disk cache across workflow runs. This helps avoid redundant LLM API calls. Ensure to use actions/cache@v4 or later.

```yaml
steps:
  - uses: actions/checkout@v4
    with:
      fetch-depth: 0

  # Layer 1: Persist promptfoo's disk cache across workflow runs
  - name: Cache promptfoo evaluations
    uses: actions/cache@v4   # Must be v4+ (required after Feb 1, 2025)
    with:
      path: |
        ~/.promptfoo/cache
        .promptfoo-cache
      # Hash-based key ensures automatic invalidation on prompt changes
      key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-${{ github.sha }}
      restore-keys: |
        ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-
        ${{ runner.os }}-promptfoo-
```

--------------------------------

### Enable Sharing with API Key Authentication

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configures the promptfoo-action to share evaluation results online, requiring a PROMPTFOO_API_KEY for authentication. This ensures fast failure for invalid credentials and clear error messages.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  env:
    PROMPTFOO_API_KEY: ${{ secrets.PROMPTFOO_API_KEY }}
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'

```

--------------------------------

### Configure Cache Environment Variables

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Sets environment variables to configure promptfoo caching behavior within a CI environment. This includes enabling the cache, setting the type to disk, defining the cache path, and setting a Time-To-Live (TTL) and maximum size.

```yaml
- name: Configure cache environment
  run: |
    echo "PROMPTFOO_CACHE_ENABLED=true" >> $GITHUB_ENV
    echo "PROMPTFOO_CACHE_TYPE=disk" >> $GITHUB_ENV
    echo "PROMPTFOO_CACHE_PATH=$HOME/.promptfoo/cache" >> $GITHUB_ENV
    echo "PROMPTFOO_CACHE_TTL=86400" >> $GITHUB_ENV  # 1 day for CI
    echo "PROMPTFOO_CACHE_MAX_SIZE=52428800" >> $GITHUB_ENV  # 50MB

```

--------------------------------

### Force Promptfoo Evaluations in GitHub Actions

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configure the promptfoo-action to force evaluations to run regardless of file changes by setting the `force-run` option to `true`.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    force-run: true
```

--------------------------------

### Run Specific Test File

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Execute a single test file for focused debugging. Useful when addressing specific test failures.

```bash
npm test -- __tests__/main.test.ts
```

--------------------------------

### Configure Pull Request Evaluation Workflow

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Set up a GitHub Actions workflow to automatically evaluate prompts on pull requests. Ensure correct permissions and checkout history.

```yaml
# .github/workflows/pr-eval.yml
name: Prompt Evaluation

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write   # Required to post PR comments

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0     # Full history needed for git diff

      - name: Cache promptfoo evaluations
        uses: actions/cache@v4
        with:
          path: |
            ~/.promptfoo/cache
            .promptfoo-cache
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token:   ${{ secrets.GITHUB_TOKEN }}
          config:         'promptfooconfig.yaml'
          prompts: |
            prompts/**/*.txt
          cache-path:     '.promptfoo-cache'
          fail-on-threshold: 75   # Fail the check if < 75% tests pass

```

--------------------------------

### Enable Debug Mode for Cache Troubleshooting

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Enables debug mode for the promptfoo-action to help troubleshoot cache issues by showing cache hits and misses. This is a simple configuration change within the action's `with` parameters.

```yaml
- uses: promptfoo/promptfoo-action@v1
  with:
    debug: true

```

--------------------------------

### Push Evaluation Workflow

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This workflow runs automatically on pushes to the main branch, evaluating prompts based on changes in the 'prompts' directory. It requires OpenAI API key and GitHub token secrets.

```yaml
name: Prompt Evaluation - Push

on:
  push:
    branches: [main]
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: write   # Required for workflow summary

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 2   # Needs at least 2 commits for before/after diff

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token:   ${{ secrets.GITHUB_TOKEN }}
          config:         'promptfooconfig.yaml'
```

--------------------------------

### Display Promptfoo Cache Statistics

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Output cache statistics provided by the promptfoo-action as workflow step outputs. This helps monitor cache usage and effectiveness.

```bash
  # Layer 3: Use cache metrics outputs
  - name: Display cache stats
    run: |
      echo "Cache size : ${{ steps.eval.outputs.cache-size-mb }} MB"
      echo "Cache files: ${{ steps.eval.outputs.cache-file-count }}"
```

--------------------------------

### Minimize Console Output in CI

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Reduces console output in CI environments by setting `no-table: true` and `no-progress-bar: true` in the action configuration. This is useful for cleaner logs.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    no-table: true
    no-progress-bar: true

```

--------------------------------

### Manual Trigger Evaluation Workflow

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This workflow allows manual triggering of prompt evaluations. It supports optional inputs for specifying files to evaluate and the base commit for comparison. API keys are passed via secrets.

```yaml
# .github/workflows/manual-eval.yml
name: Prompt Evaluation - Manual

on:
  workflow_dispatch:
    inputs:
      files:
        description: 'Files to evaluate (leave empty to auto-detect)'
        required: false
        type: string
      base:
        description: 'Base branch/commit to compare against'
        required: false
        default: 'HEAD~1'
        type: string

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: write

    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token:   ${{ secrets.GITHUB_TOKEN }}
          config:         'promptfooconfig.yaml'
          # Alternatively, pass files/base directly as action inputs:
          # workflow-files: |
          #   prompts/system.txt
          #   prompts/user.txt
          # workflow-base: 'main'
```

--------------------------------

### Run Tests in Watch Mode

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Continuously run tests as code changes. This facilitates rapid development and testing cycles.

```bash
npm test -- --watch
```

--------------------------------

### Alternative Action Inputs for Manual Trigger

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Specify files and base branch directly as action inputs when triggering evaluations manually. Requires `actions: write` permission for workflow summaries.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@main
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    workflow-files: |
      prompts/prompt1.txt
      prompts/prompt2.txt
    workflow-base: 'main'
```

--------------------------------

### Manual Prompt Evaluation Trigger

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Configure a GitHub Actions workflow to manually trigger prompt evaluations using `workflow_dispatch`. Results are displayed in the workflow summary.

```yaml
name: 'Prompt Evaluation - Manual'
on:
  workflow_dispatch:
    inputs:
      files:
        description: 'Files to evaluate (leave empty to auto-detect)'
        required: false
        type: string
      base:
        description: 'Base branch/commit to compare against'
        required: false
        default: 'HEAD~1'
        type: string

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      actions: write # Required for workflow summaries
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # Fetch all history for comparisons

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@main
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
```

--------------------------------

### Custom Provider Dependency Detection Workflow

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This GitHub Actions workflow snippet shows that no special configuration is needed for dependency detection; it's automatic. The action triggers evaluations when specified files or directories change.

```yaml
# Workflow: no special config needed — dependency detection is automatic
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    # Evaluation triggers when prompts/, providers/, data/, or validators/ change
```

--------------------------------

### Prompt Evaluation on Pull Request

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Integrate promptfoo-action into a GitHub Actions workflow to evaluate prompts on pull requests. Includes optional caching for performance.

```yaml
name: 'Prompt Evaluation'
on:
  pull_request:
    paths:
      - 'prompts/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read # Required for actions/checkout
      pull-requests: write # Ability to post comments on Pull Requests
    steps:
      # Required for promptfoo-action's git usage
      - uses: actions/checkout@v4

      # This cache is optional, but you'll save money and time by setting it up!
      # IMPORTANT: Use actions/cache@v4 or later (required after Feb 1, 2025)
      - name: Set up promptfoo cache
        uses: actions/cache@v4
        with:
          path: |
            ~/.promptfoo/cache
            .promptfoo-cache
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-${{ github.sha }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}-
            ${{ runner.os }}-promptfoo-

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          openai-api-key: ${{ secrets.OPENAI_API_KEY }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'
          cache-path: '.promptfoo-cache'
```

--------------------------------

### Fail on Threshold Configuration

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

This configuration sets a 'fail-on-threshold' value for the promptfoo action. The workflow step will fail if the evaluation pass rate falls below the specified percentage, acting as a merge gate.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token:      ${{ secrets.GITHUB_TOKEN }}
    openai-api-key:    ${{ secrets.OPENAI_API_KEY }}
    config:            'promptfooconfig.yaml'
    fail-on-threshold: 80   # Fail if fewer than 80% of test cases pass

# If 15/20 tests pass (75%), the action exits with:
# Error: Evaluation success rate (75.00%) is below the required threshold (80%)
```

--------------------------------

### Force Promptfoo Evaluation Run

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Execute a promptfoo evaluation, bypassing file-change detection. This ensures the evaluation runs regardless of whether prompt or configuration files have been modified.

```yaml
- name: Run promptfoo evaluation (forced)
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config:       'promptfooconfig.yaml'
    force-run:    true   # Evaluate even if no files changed
```

--------------------------------

### Persist Promptfoo Evaluation Results as Artifact

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Upload the `output.json` file generated by `promptfoo eval` as a GitHub Actions artifact. This preserves evaluation results beyond the default URL expiration, using `actions/upload-artifact@v4`.

```yaml
      - name: Upload evaluation results
        uses: actions/upload-artifact@v4
        if: always()   # Upload even when tests fail
        with:
          name:           promptfoo-results
          path:           output.json
          retention-days: 90
```

--------------------------------

### Rebuild Dist Files After Merge

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Regenerate distribution files after resolving merge conflicts. This ensures the build artifacts are up-to-date.

```bash
git add .
git commit
npm ci
npm run build
npm run package
git add dist/
git commit -m "fix: rebuild dist files after merge"
git push origin feature/branch-name
```

--------------------------------

### Cache Rotation Key Generation

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Generates a cache rotation key based on the current week. This is used to ensure cache freshness by rotating the cache weekly.

```yaml
- name: Get cache rotation key
  id: cache-key
  run: echo "week=$(date +%Y-W%U)" >> $GITHUB_OUTPUT
```

--------------------------------

### Persist Evaluation Results as GitHub Artifact

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Uploads the `output.json` file containing evaluation results as a GitHub Action artifact. This preserves results for up to 90 days, even if the evaluation fails.

```yaml
jobs:
  evaluate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Run promptfoo evaluation
        uses: promptfoo/promptfoo-action@v1
        with:
          github-token: ${{ secrets.GITHUB_TOKEN }}
          config: 'promptfooconfig.yaml'

      - name: Upload results
        uses: actions/upload-artifact@v4
        if: always()  # Upload even if evaluation fails
        with:
          name: promptfoo-results
          path: output.json
          retention-days: 90

```

--------------------------------

### Merge Main into Feature Branch

Source: https://github.com/promptfoo/promptfoo-action/blob/main/AGENTS.md

Integrate changes from the main branch into a feature branch. Be mindful of conflict resolution for specific files.

```bash
git checkout feature/branch-name
git pull origin feature/branch-name
git merge main
```

--------------------------------

### Promptfoo Action Error Codes

Source: https://context7.com/promptfoo/promptfoo-action/llms.txt

Internal error taxonomy for Promptfoo Action. Use these codes to understand and handle specific execution failures.

```typescript
// Internal error taxonomy (ErrorCodes):
// NO_PULL_REQUEST          — pull_request event but no PR in context
// INVALID_GIT_REF          — ref starts with "-" (option injection attempt)
// INVALID_THRESHOLD        — fail-on-threshold outside 0-100
// THRESHOLD_NOT_MET        — pass rate below required threshold
// PROMPTFOO_EXECUTION_FAILED — npx promptfoo eval exited non-zero
// INVALID_OUTPUT_FILE      — output.json missing or malformed
// ENV_FILE_NOT_FOUND       — specified .env file does not exist
// ENV_FILE_LOAD_ERROR      — .env file exists but failed to parse
// INVALID_CONFIGURATION    — max-concurrency < 1
// AUTH_FAILED              — PROMPTFOO_API_KEY invalid or network error

// Example: invalid threshold
// Error: fail-on-threshold must be a number between 0 and 100
// Help:   Please provide a valid percentage value, e.g., 80 for 80% success rate

// Example: .env file not found
// Error: Environment file /workspace/.env.missing not found
// Help:   Make sure the file path is correct relative to /workspace
```

--------------------------------

### Disable Sharing of Evaluation Results

Source: https://github.com/promptfoo/promptfoo-action/blob/main/README.md

Explicitly disables the sharing of promptfoo evaluation results by setting the `no-share` option to `true` in the action configuration. This ensures results only appear in logs.

```yaml
- name: Run promptfoo evaluation
  uses: promptfoo/promptfoo-action@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    config: 'promptfooconfig.yaml'
    no-share: true

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.