Skillgym

https://incubator.callstack.com/skillgym/docs
Admin
Skillgym is an open-source tool designed to catch skill regressions, compare the behavior of...
Tokens:1,108
Snippets:17
Trust Score:10
Update:4 months ago
Show doc for...
Tokens:
Raw
### Install SkillGym

Source: https://incubator.callstack.com/skillgym

Install SkillGym as a development dependency in your project.

```bash
npm install --save-dev skillgym
```

--------------------------------

### Configure SkillGym Runners

Source: https://incubator.callstack.com/skillgym

Configure different agents and their models for running skill tests. This example shows 'my-agent' using OpenCode and 'copilot-main' using Copilot.

```typescript
// skillgym.config.ts
import type { SkillgymConfig } from "skillgym";

const config: SkillgymConfig = {
  runners: {
    "my-agent": {
      agent: {
        type: "opencode",
        model: "openai/gpt-4o",
      },
    },
    "copilot-main": {
      agent: {
        type: "copilot",
        model: "gpt-5.4-mini",
      },
    },
  },
};

export default config;
```

--------------------------------

### Run SkillGym Tests

Source: https://incubator.callstack.com/skillgym

Execute skill tests using the skillgym CLI. Specify the path to your test suite.

```bash
skillgym run ./examples/basic-suite.ts
```

--------------------------------

### SkillGym Assertions for Skill Behavior

Source: https://incubator.callstack.com/skillgym

Use TypeScript assertions to verify skill loading, file reads, and command execution order. This snippet checks if 'agent-device' skill was loaded, if 'bootstrap-install.md' was read, and the order of 'open', 'snapshot', and 'close' commands.

```typescript
import { assert } from "skillgym";

// assert the skill was loaded
assert.skills.has(report, "agent-device");

// assert the reference was loaded
assert.fileReads.includes(report, /bootstrap-install.md$/);

// assert commands ran in the right order
assert.commands.before(
  report,
  /^agent-devices+open/,
  /^agent-devices+snapshot/
);
assert.commands.before(
  report,
  /^agent-devices+snapshot/,
  /^agent-devices+close/
);
```

--------------------------------

### Define a SkillGym Test Suite

Source: https://incubator.callstack.com/skillgym

This snippet defines a test suite for SkillGym using TypeScript. It includes importing necessary modules, defining a test case with a prompt and an assertion function, and exporting the suite.

```typescript
// skillgym/my-suite.ts
import { assert, type Case } from "skillgym";

const suite: Case[] = [
  {
    id: "agent-device-settings-check",
    prompt: [
      "Use agent-device to open Settings,",
      'verify "Privacy" is visible with snapshot,',
      "then close the session.",
    ].join(" "),
    async assert(report) {
      assert.skills.has(report, "agent-device");
      assert.commands.before(
        report,
        /^agent-devices+open/,
        /^agent-devices+snapshot/
      );
      assert.commands.before(
        report,
        /^agent-devices+snapshot/,
        /^agent-devices+close/
      );
    },
  },
];

export default suite;
```

--------------------------------

### Update Snapshots for Token Usage

Source: https://incubator.callstack.com/skillgym

Update the baseline snapshots for token usage. This command helps catch regressions in agent cost by setting a new token count baseline.

```bash
npx skillgym run ./suite.ts --update-snapshots
```

--------------------------------

### Configure Isolated Workspace

Source: https://incubator.callstack.com/skillgym

Define an isolated workspace configuration for skill executions. This ensures a clean environment for reproducibility, specifying the mode, template directory, and bootstrap commands.

```typescript
export const workspace = {
  mode: "isolated",
  templateDir: "./fixtures/base",
  bootstrap: {
    command: "npm",
    args: ["install"],
  },
};
```
Tokens:
Raw