SkillEvaluator

https://docs.nvidia.com/skills/skillevaluator
Admin
SkillEvaluator is an open-source, multi-tier evaluation framework for testing and validating AI...
Tokens:64,148
Snippets:586
Trust Score:10
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install with selected extras using uv

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Installs the tool with specific extras to manage the footprint. Base checks are included by default, while security scanners or specific tiers can be added.

```bash
uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git"            # base checks; scanner checks report INCOMPLETE
uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git"  # + bundled Python security scanners
uv tool install --python 3.13 "skillevaluator[tier2,tier3] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
```

--------------------------------

### Install with pip alternative

Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md

Alternative to `uv sync` using pip. Installs the package in editable mode with all extras and dev dependencies.

```bash
pip install -e '.[all,dev]'
```

--------------------------------

### Editable install with pip

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Sets up a virtual environment and installs the package in editable mode using pip. Requires Python 3.12 or 3.13 on the system path.

```bash
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e ".[all]"
```

--------------------------------

### OpenAI-compatible setup

Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md

Configure a custom OpenAI-compatible server by providing the base URL, model name, and API key.

```bash
export SKILL_EVAL_LLM_PROVIDER=openai-compatible
export SKILL_EVAL_LLM_BASE_URL='http://your-server:8000/v1'
export SKILL_EVAL_LLM_MODEL='your-model-name'
export SKILL_EVAL_LLM_API_KEY='your-key'
```

--------------------------------

### OpenAI setup

Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md

Configure the OpenAI provider by setting the provider variable and your API key.

```bash
export SKILL_EVAL_LLM_PROVIDER=openai
export OPENAI_API_KEY='sk-...'
```

--------------------------------

### Install with uv

Source: https://docs.nvidia.com/skills/skillevaluator/quickstart.md

Installs the CLI with every Python feature using uv, which provisions Python 3.13 automatically.

```bash
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
```

--------------------------------

### Editable install with selected extras using pip

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Installs the package in editable mode with specific extras.

```bash
python -m pip install -e .
python -m pip install -e ".[security]"
python -m pip install -e ".[all,dev]"
```

--------------------------------

### Anthropic setup

Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md

Configure the Anthropic provider by setting the provider variable and your API key.

```bash
export SKILL_EVAL_LLM_PROVIDER=anthropic
export ANTHROPIC_API_KEY='sk-ant-...'
```

--------------------------------

### Install fern-api CLI

Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md

Install the Fern CLI globally. Requires Node.js 22+ and npm 10+.

```bash
npm install -g fern-api
```

--------------------------------

### Install and run SkillEvaluator quality check

Source: https://docs.nvidia.com/skills/skillevaluator

Installs the tool using uv and executes a quality check on a local skill directory without requiring an API key.

```bash
uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
skillevaluator quality-check ./my-skill
```

--------------------------------

### Verify editable installation

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Confirms the command resolves correctly within the environment.

```bash
uv run skillevaluator --version   # uv sync path
skillevaluator --version          # inside the activated pip venv
```

--------------------------------

### Sync dev environment with uv

Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md

Installs all dependencies and extras using uv with Python 3.13. This is the recommended way to sync the development environment.

```bash
uv sync --python 3.13 --all-extras
```

--------------------------------

### Amazon Bedrock setup

Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md

Configure the Amazon Bedrock provider. Ensure you authenticate using the standard AWS credential chain.

```bash
export SKILL_EVAL_LLM_PROVIDER=bedrock
export AWS_REGION=us-west-2   # default when unset
# Authenticate with the standard AWS credential chain
# (environment variables, shared profile, or an assumed role).
```

--------------------------------

### Preview docs with live reload

Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md

Start a local docs preview with live reload. The site serves at http://localhost:3000 and reloads on changes to .mdx files or fern/docs.yml.

```bash
fern docs dev
```

--------------------------------

### Upgrade to the latest commit with uv

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Reruns the installation with the --reinstall flag to update to the latest commit.

```bash
uv tool install --python 3.13 --reinstall "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git"
```

--------------------------------

### Start a local model server with Ollama in Docker

Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md

Commands to run the Ollama container and pull the necessary models for embeddings and chat analysis.

```bash
docker run -d --name ollama -p 11434:11434 ollama/ollama
docker exec ollama ollama pull nomic-embed-text   # embeddings
docker exec ollama ollama pull qwen2.5:0.5b       # chat LLM for dedup analysis
```

--------------------------------

### Run doctor readiness check

Source: https://docs.nvidia.com/skills/skillevaluator/agents-and-sandboxes.md

Runs the doctor command to verify the setup before the first run. Exits nonzero if any check fails. Accepts --agent-model and --verify-models flags.

```bash
skillevaluator doctor --agents codex --env-mode docker
```

--------------------------------

### Choose an agent harness

Source: https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation.md

Demonstrates how to select different agent harnesses for evaluation in a Docker environment.

```bash
skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker
skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker
```

--------------------------------

### Install external security scanners

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Commands to install Semgrep, Gitleaks, and SkillSpector on macOS or via uv for isolated environments.

```bash
brew install semgrep gitleaks                                  # macOS
uv tool install semgrep                                       # isolated Python tool environment
uv tool install git+https://github.com/NVIDIA/SkillSpector.git
```

--------------------------------

### Clone and sync from source

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Clones the repository and uses uv to create an environment from the lockfile for development.

```bash
git clone https://github.com/NVIDIA/SkillEvaluator.git
cd SkillEvaluator
uv sync --python 3.13 --all-extras
```

--------------------------------

### Build and run with Docker

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Builds the Docker image and runs a validation against a sample skill. Note that external scanner executables are not bundled.

```bash
docker build -t skillevaluator:local .
docker run --rm skillevaluator:local --version
docker run --rm \
  -v "$PWD/tests/fixtures/skills/simple:/workspace/skills/simple:ro" \
  skillevaluator:local validate /workspace/skills/simple --no-llm --no-dedup
```

--------------------------------

### Verify install

Source: https://docs.nvidia.com/skills/skillevaluator/quickstart.md

Confirms the install. You should see `skillevaluator, version 0.2.1` (or newer). If the shell can't find `skillevaluator`, run `uv tool update-shell` and open a new terminal.

```bash
skillevaluator --version
```

--------------------------------

### evals/EVAL.md

Source: https://docs.nvidia.com/skills/skillevaluator/eval-datasets.md

Example structure for the EVAL.md file used to provide guidance for dataset generation.

```markdown
## Questions
- Sample user prompts to use as eval questions

## Behaviors
- Expected agent behaviors to verify

## Notes
- General context and constraints
```

--------------------------------

### Promote and verify benchmark files

Source: https://docs.nvidia.com/skills/skillevaluator/benchmark-rollout.md

Copy the benchmark file to the skills directory, run the public benchmark check script, and verify changes using git diff.

```bash
cp ./benchmark-backfill/example-skill/BENCHMARK.md \
  ./skills/example-skill/BENCHMARK.md

python scripts/ci/check_public_benchmarks.py --require-files ./skills
git diff --check
git diff -- ./skills/example-skill/BENCHMARK.md
```

--------------------------------

### Update shell PATH for uv tools

Source: https://docs.nvidia.com/skills/skillevaluator/installation.md

Run this command if the shell cannot find the skillevaluator executable after installation.

```bash
uv tool update-shell
```
Tokens:
Raw