### Install with selected extras using uv Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Installs the tool with specific extras to manage the footprint. Base checks are included by default, while security scanners or specific tiers can be added. ```bash uv tool install --python 3.13 "skillevaluator @ git+https://github.com/NVIDIA/SkillEvaluator.git" # base checks; scanner checks report INCOMPLETE uv tool install --python 3.13 "skillevaluator[security] @ git+https://github.com/NVIDIA/SkillEvaluator.git" # + bundled Python security scanners uv tool install --python 3.13 "skillevaluator[tier2,tier3] @ git+https://github.com/NVIDIA/SkillEvaluator.git" ``` -------------------------------- ### Install with pip alternative Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md Alternative to `uv sync` using pip. Installs the package in editable mode with all extras and dev dependencies. ```bash pip install -e '.[all,dev]' ``` -------------------------------- ### Editable install with pip Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Sets up a virtual environment and installs the package in editable mode using pip. Requires Python 3.12 or 3.13 on the system path. ```bash python3 -m venv .venv source .venv/bin/activate python -m pip install -e ".[all]" ``` -------------------------------- ### OpenAI-compatible setup Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md Configure a custom OpenAI-compatible server by providing the base URL, model name, and API key. ```bash export SKILL_EVAL_LLM_PROVIDER=openai-compatible export SKILL_EVAL_LLM_BASE_URL='http://your-server:8000/v1' export SKILL_EVAL_LLM_MODEL='your-model-name' export SKILL_EVAL_LLM_API_KEY='your-key' ``` -------------------------------- ### OpenAI setup Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md Configure the OpenAI provider by setting the provider variable and your API key. ```bash export SKILL_EVAL_LLM_PROVIDER=openai export OPENAI_API_KEY='sk-...' ``` -------------------------------- ### Install with uv Source: https://docs.nvidia.com/skills/skillevaluator/quickstart.md Installs the CLI with every Python feature using uv, which provisions Python 3.13 automatically. ```bash uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git" ``` -------------------------------- ### Editable install with selected extras using pip Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Installs the package in editable mode with specific extras. ```bash python -m pip install -e . python -m pip install -e ".[security]" python -m pip install -e ".[all,dev]" ``` -------------------------------- ### Anthropic setup Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md Configure the Anthropic provider by setting the provider variable and your API key. ```bash export SKILL_EVAL_LLM_PROVIDER=anthropic export ANTHROPIC_API_KEY='sk-ant-...' ``` -------------------------------- ### Install fern-api CLI Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md Install the Fern CLI globally. Requires Node.js 22+ and npm 10+. ```bash npm install -g fern-api ``` -------------------------------- ### Install and run SkillEvaluator quality check Source: https://docs.nvidia.com/skills/skillevaluator Installs the tool using uv and executes a quality check on a local skill directory without requiring an API key. ```bash uv tool install --python 3.13 "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git" skillevaluator quality-check ./my-skill ``` -------------------------------- ### Verify editable installation Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Confirms the command resolves correctly within the environment. ```bash uv run skillevaluator --version # uv sync path skillevaluator --version # inside the activated pip venv ``` -------------------------------- ### Sync dev environment with uv Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md Installs all dependencies and extras using uv with Python 3.13. This is the recommended way to sync the development environment. ```bash uv sync --python 3.13 --all-extras ``` -------------------------------- ### Amazon Bedrock setup Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md Configure the Amazon Bedrock provider. Ensure you authenticate using the standard AWS credential chain. ```bash export SKILL_EVAL_LLM_PROVIDER=bedrock export AWS_REGION=us-west-2 # default when unset # Authenticate with the standard AWS credential chain # (environment variables, shared profile, or an assumed role). ``` -------------------------------- ### Preview docs with live reload Source: https://docs.nvidia.com/skills/skillevaluator/developer-guide.md Start a local docs preview with live reload. The site serves at http://localhost:3000 and reloads on changes to .mdx files or fern/docs.yml. ```bash fern docs dev ``` -------------------------------- ### Upgrade to the latest commit with uv Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Reruns the installation with the --reinstall flag to update to the latest commit. ```bash uv tool install --python 3.13 --reinstall "skillevaluator[all] @ git+https://github.com/NVIDIA/SkillEvaluator.git" ``` -------------------------------- ### Start a local model server with Ollama in Docker Source: https://docs.nvidia.com/skills/skillevaluator/configuration.md Commands to run the Ollama container and pull the necessary models for embeddings and chat analysis. ```bash docker run -d --name ollama -p 11434:11434 ollama/ollama docker exec ollama ollama pull nomic-embed-text # embeddings docker exec ollama ollama pull qwen2.5:0.5b # chat LLM for dedup analysis ``` -------------------------------- ### Run doctor readiness check Source: https://docs.nvidia.com/skills/skillevaluator/agents-and-sandboxes.md Runs the doctor command to verify the setup before the first run. Exits nonzero if any check fails. Accepts --agent-model and --verify-models flags. ```bash skillevaluator doctor --agents codex --env-mode docker ``` -------------------------------- ### Choose an agent harness Source: https://docs.nvidia.com/skills/skillevaluator/tier3-live-evaluation.md Demonstrates how to select different agent harnesses for evaluation in a Docker environment. ```bash skillevaluator tier3 evaluate ./my-skill --agents opencode --env-mode docker skillevaluator tier3 evaluate ./my-skill --agents codex --env-mode docker skillevaluator tier3 evaluate ./my-skill --agents claude-code --env-mode docker ``` -------------------------------- ### Install external security scanners Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Commands to install Semgrep, Gitleaks, and SkillSpector on macOS or via uv for isolated environments. ```bash brew install semgrep gitleaks # macOS uv tool install semgrep # isolated Python tool environment uv tool install git+https://github.com/NVIDIA/SkillSpector.git ``` -------------------------------- ### Clone and sync from source Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Clones the repository and uses uv to create an environment from the lockfile for development. ```bash git clone https://github.com/NVIDIA/SkillEvaluator.git cd SkillEvaluator uv sync --python 3.13 --all-extras ``` -------------------------------- ### Build and run with Docker Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Builds the Docker image and runs a validation against a sample skill. Note that external scanner executables are not bundled. ```bash docker build -t skillevaluator:local . docker run --rm skillevaluator:local --version docker run --rm \ -v "$PWD/tests/fixtures/skills/simple:/workspace/skills/simple:ro" \ skillevaluator:local validate /workspace/skills/simple --no-llm --no-dedup ``` -------------------------------- ### Verify install Source: https://docs.nvidia.com/skills/skillevaluator/quickstart.md Confirms the install. You should see `skillevaluator, version 0.2.1` (or newer). If the shell can't find `skillevaluator`, run `uv tool update-shell` and open a new terminal. ```bash skillevaluator --version ``` -------------------------------- ### evals/EVAL.md Source: https://docs.nvidia.com/skills/skillevaluator/eval-datasets.md Example structure for the EVAL.md file used to provide guidance for dataset generation. ```markdown ## Questions - Sample user prompts to use as eval questions ## Behaviors - Expected agent behaviors to verify ## Notes - General context and constraints ``` -------------------------------- ### Promote and verify benchmark files Source: https://docs.nvidia.com/skills/skillevaluator/benchmark-rollout.md Copy the benchmark file to the skills directory, run the public benchmark check script, and verify changes using git diff. ```bash cp ./benchmark-backfill/example-skill/BENCHMARK.md \ ./skills/example-skill/BENCHMARK.md python scripts/ci/check_public_benchmarks.py --require-files ./skills git diff --check git diff -- ./skills/example-skill/BENCHMARK.md ``` -------------------------------- ### Update shell PATH for uv tools Source: https://docs.nvidia.com/skills/skillevaluator/installation.md Run this command if the shell cannot find the skillevaluator executable after installation. ```bash uv tool update-shell ```