### Getting Started Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Commands to clone the repository, install dependencies, build, and run tests. ```bash git clone https://github.com/microsoft/vscode-chat-customizations-evaluation.git cd vscode-chat-customizations-evaluation npm install npm run build npm test ``` -------------------------------- ### Waza Run Command Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Example of how to run waza with an eval file, context directory, and output file. ```bash waza run --context-dir --output ``` -------------------------------- ### Example WAZA Command Configuration Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Examples of how to configure the `chatCustomizationsEvaluations.waza.command` setting to specify the WAZA executable path. ```json "chatCustomizationsEvaluations.waza.command": "waza" ``` ```json "chatCustomizationsEvaluations.waza.command": "/usr/local/bin/waza" ``` ```json "chatCustomizationsEvaluations.waza.command": "C:\\tools\\waza.exe" ``` -------------------------------- ### Example Results Filename Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md An example of how a WAZA results file might be named, including a timestamp. ```text my-skill-2026-05-05T13-39-38-888Z.json ``` -------------------------------- ### Prompt Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'prompt' grader type. ```yaml - type: prompt name: quality-judge config: model: gpt-4o-mini prompt: | Evaluate task completion quality. If requirements are met, call set_waza_grade_pass. Otherwise call set_waza_grade_fail with reasons. ``` -------------------------------- ### Program Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'program' grader type. ```yaml - type: program name: custom-policy-checks config: command: "bash" args: ["./validators/check-output.sh"] timeout: 60 ``` -------------------------------- ### File Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'file' grader type. ```yaml - type: file name: report-file-created config: must_exist: - "artifacts/report.json" ``` -------------------------------- ### Behavior Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'behavior' grader type. ```yaml - type: behavior name: token-budget config: max_tokens: 20000 max_duration_ms: 120000 max_tool_calls: 10 ``` -------------------------------- ### Action Sequence Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'action_sequence' grader type. ```yaml - type: action_sequence name: deployment-workflow config: matching_mode: in_order_match expected_actions: - "bash" - "edit" - "bash" - "report_progress" ``` -------------------------------- ### Diff Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'diff' grader type. ```yaml - type: diff name: expected-config-edits config: expected_files: - path: "src/config.json" snapshot: "snapshots/config.json" - path: "README.md" contains: - "+## Installation" - "-pip install" ``` -------------------------------- ### Example WAZA Task YAML Fields Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md This table outlines the possible fields for a WAZA task YAML configuration, including their requirements and descriptions. ```yaml id: Unique task identifier used in output JSON. name: Task display name shown in reports. description: What this task is testing. tags: Labels for filtering/grouping. group: Optional group name used in grouped summaries. enabled: When false, task is skipped/ignored. inputs: Prompt and optional context/files provided to model. inputs.prompt: Main user prompt for this test. inputs.context: Structured key/value context payload for the run. inputs.files: Fixture files copied into run workspace. expected: High-level expectations (triggering, content, behavior). expected.should_trigger: Whether skill should trigger for this prompt. expected.output_contains: Strings that must appear in final output. expected.output_not_contains: Strings that must not appear in final output. expected.outcomes: Expected semantic outcomes (task-specific semantics). expected.behavior: Behavior limits such as tool calls/duration. graders: Task-specific validators in addition to global graders. graders[].type: Same documented types as eval-level graders (including the not-implemented entries). graders[].name: Task-level grader identifier in output validations. graders[].config: Type-specific grader configuration for this task only. hooks: Per-task pre/post execution commands where supported. ``` -------------------------------- ### Text Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'text' grader type. ```yaml - type: text name: no-runtime-errors config: regex_not_match: - "(?i)error|exception|traceback" ``` -------------------------------- ### Code Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'code' grader type. ```yaml - type: code name: has-output config: assertions: - "len(output) > 20" ``` -------------------------------- ### JSON Schema Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'json_schema' grader type. ```yaml - type: json_schema name: valid-structured-output config: schema: type: object required: ["summary", "confidence"] properties: summary: type: string confidence: type: number ``` -------------------------------- ### Skill Invocation Grader Example Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Configuration for the 'skill_invocation' grader type. ```yaml - type: skill_invocation name: orchestration-flow config: required_skills: - "azure-prepare" - "azure-deploy" mode: in_order allow_extra: true ``` -------------------------------- ### trigger grader configuration Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Example configuration for a trigger grader that specifies a skill path, mode, and threshold. ```yaml - type: trigger name: deploy-trigger config: skill_path: "skills/my-skill/SKILL.md" mode: positive threshold: 0.6 ``` -------------------------------- ### tool_constraint grader configuration Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Example configuration for a tool_constraint grader that expects the 'azd up' bash command and rejects 'rm -rf'. ```yaml - type: tool_constraint name: tool-guardrails config: expect_tools: - tool: "bash" command_pattern: "azd\s+up" reject_tools: - tool: "bash" command_pattern: "rm\s+-rf" ``` -------------------------------- ### eval.yaml pseudo structure Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Pseudo structure for an eval.yaml file, outlining the main sections and their purpose. ```yaml name: my-skill-eval description: Behavior-focused evaluation for my skill. skill: my-skill version: "1.0" config: trials_per_task: 1 timeout_seconds: 300 parallel: false executor: copilot-sdk model: claude-sonnet-4.6 metrics: - name: task_completion weight: 0.7 threshold: 0.8 description: Overall completion quality target. - name: efficiency weight: 0.3 threshold: 0.7 description: Token/runtime quality target. graders: - type: behavior name: token-budget config: max_tokens: 20000 max_duration_ms: 120000 tasks: - "tasks/*.yaml" ``` -------------------------------- ### task YAML pseudo structure Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/docs/WAZA-USER-GUIDE.md Pseudo structure for a task YAML file, including id, name, description, tags, inputs, expected outcomes, and graders. ```yaml id: positive-trigger-001 name: Positive Trigger 1 description: Ensure the skill triggers and produces expected behavior. tags: - trigger - happy-path inputs: prompt: "Generate a Python function normalize_email(email: str) -> str" files: - path: fixtures/sample.py context: scenario: basic expected: should_trigger: true output_contains: - "normalize_email" output_not_contains: - "as an ai" outcomes: - type: task_completed behavior: max_tool_calls: 0 graders: - type: text name: has-python-shape config: regex_match: - "(?i)def\s+normalize_email\s*\(" ``` -------------------------------- ### Running the Extension Locally Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Instructions to launch the Extension Development Host and test the extension. ```plaintext 1. Open the project in VS Code 2. Press `F5` to launch the Extension Development Host 3. In the new VS Code window, open any `.prompt.md`, `.system.md`, or `.agent.md` file 4. Diagnostics will appear automatically ``` -------------------------------- ### Build Commands Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md List of available build and test commands. ```plaintext - `npm run compile` — Build server only - `npm run build` — Build server + client - `npm test` — Run tests (vitest) - `npx vitest` — Run tests in watch mode - `npm run lint` — Run ESLint ``` -------------------------------- ### Running Tests Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Commands to run all tests or tests in watch mode. ```bash # Run all tests npm test # Run tests in watch mode npx vitest ``` -------------------------------- ### Development Workflow Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Steps for making changes, building, testing, and debugging the extension. ```plaintext 1. **Make changes** to the server (`src/`) or client (`client/src/`) 2. **Build** with `npm run build` 3. **Test** with `npm test` 4. **Debug** in VS Code by pressing `F5` (launches Extension Development Host) ``` -------------------------------- ### Project Structure Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Overview of the project's directory structure. ```plaintext chat-customizations-evaluations/ ├── src/ # Language server source │ ├── server.ts # Server entry point │ ├── types.ts # Shared type definitions │ ├── analyzers/ │ │ └── llm.ts # LLM-powered semantic analysis │ └── __tests__/ # Unit tests ├── client/ # VS Code extension client │ ├── src/extension.ts # Extension entry point │ └── package.json # Extension manifest & configuration └── vitest.config.ts # Test configuration ``` -------------------------------- ### Watch Mode Source: https://github.com/microsoft/vscode-chat-customizations-evaluation/blob/main/CONTRIBUTING.md Commands to run server and client in watch mode for faster iteration. ```bash # Terminal 1: Watch server changes npm run watch # Terminal 2: Watch client changes cd client && npm run watch ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.