CoEvoSkills

https://github.com/zhang-henry/coevoskills
Admin
CoEvoSkills is a self-evolving framework that enables LLM agents to autonomously construct and...
Tokens:141,306
Snippets:1,971
Trust Score:8.6
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Defining Examples in Instructions

Source: https://github.com/zhang-henry/coevoskills/blob/main/meta_skills/skill-creator/SKILL.md

Pattern for providing input-output examples to guide model behavior.

```markdown
## Commit message format
**Example 1:**
Input: Added user authentication with JWT tokens
Output: feat(auth): implement JWT-based authentication
```

--------------------------------

### Install CoEvoSkills environment

Source: https://github.com/zhang-henry/coevoskills/blob/main/README.md

Clone the repository, sync dependencies with uv, and initialize the environment configuration.

```bash
git clone https://github.com/Zhang-Henry/CoEvoSkills.git
cd CoEvoSkills

uv sync --locked
cp .env.example .env
docker info
```

--------------------------------

### Install Flash Attention 2

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/simpo-code-reproduction/environment/SimPO/README.md

Installs the Flash Attention 2 library without build isolation.

```shell
python -m pip install flash-attn --no-build-isolation
```

--------------------------------

### Install Alignment Handbook Dependencies

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/simpo-code-reproduction/environment/SimPO/README.md

Clones the alignment-handbook repository and installs its package dependencies.

```shell
git clone https://github.com/huggingface/alignment-handbook.git
cd ./alignment-handbook/
python -m pip install .
```

--------------------------------

### Configure OpenAI Codex Subscription

Source: https://github.com/zhang-henry/coevoskills/blob/main/docs/provider_setup.md

Install and authenticate the Codex CLI for ChatGPT subscription access.

```bash
npm install -g @openai/codex
codex login
codex login status

export CODEX_CHATGPT_AUTH_FILE="$HOME/.codex/auth.json"
export CODEX_CLI_BINARY="$(command -v codex)"
export CODEX_REASONING_EFFORT="low"
```

--------------------------------

### Example problem.json structure

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/jax-computing-basics/instruction.md

The configuration file format used to define task IDs, descriptions, input paths, and output paths.

```json
[
    {
      "id": a unique problem id,
      "description": a language description of the task,
      "input": input data path, for example: "data/x.npy",
      "output": output result path, for example: "basic_reduce.npy"
    },
    ...,
]
```

--------------------------------

### Run SimPO Task End-to-End

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/simpo-code-reproduction/evo-simpo-loss/SKILL.md

Executes the full task pipeline including package installation, code injection, and validation.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-simpo-loss/scripts')
from run_task import run_simpo_task

# Run the complete task end-to-end
run_simpo_task(
    project_dir='/root/SimPO',
    output_path='/root/loss.npz',
    python_info_path='/root/python_info.txt',
)
```

--------------------------------

### Index Tutorial Videos

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/video-tutorial-indexer/evo-video-indexer/SKILL.md

Demonstrates the full workflow from audio extraction to JSON index generation. Requires the scripts directory to be in the Python path.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-video-indexer/scripts')
from transcribe import extract_audio, transcribe_audio, save_transcript
from align_chapters import align_chapters_to_transcript, validate_chapters, generate_index

# Step 1: Extract audio
audio_path = extract_audio('/root/tutorial_video.mp4', '/tmp/audio.wav')

# Step 2: Transcribe
segments = transcribe_audio(audio_path, model_name='base')
save_transcript(segments, '/root/transcript_segments.json')

# Step 3: Define chapter titles (from task)
chapter_titles = [
    "What we'll do",
    "How we'll get there",
    # ... all chapter titles ...
]

# Step 4: Align chapters to transcript
chapters = align_chapters_to_transcript(chapter_titles, segments, first_time=0)

# Step 5: Validate
errors = validate_chapters(chapters, duration=1382, expected_count=29)
if errors:
    print("Validation errors:", errors)

# Step 6: Generate output
generate_index(chapters, title='In-Depth Floor Plan Tutorial Part 1',
               duration=1382, output_path='/root/tutorial_index.json')
```

--------------------------------

### Initialize Environment and Import Skills

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/travel-planning/solution/solve.ipynb

Configures the system path to include necessary skill modules and imports the required classes.

```python
import sys

sys.path.append("../environment/skills/search_cities/scripts")
sys.path.append("../environment/skills/search_accommodations/scripts")
sys.path.append("../environment/skills/search_restaurants/scripts")
sys.path.append("../environment/skills/search_attractions/scripts")
sys.path.append("../environment/skills/search_driving_distance/scripts")

from search_accommodations import Accommodations
from search_attractions import Attractions
from search_cities import Cities
from search_driving_distance import GoogleDistanceMatrix
from search_restaurants import Restaurants
```

--------------------------------

### Verify project build and quality

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/spring-boot-jakarta-migration/instruction.md

Run these Maven commands to ensure the project compiles correctly and passes all existing unit tests after migration.

```bash
mvn clean compile
```

```bash
mvn test
```

--------------------------------

### Grading JSON Schema Structure

Source: https://github.com/zhang-henry/coevoskills/blob/main/meta_skills/skill-creator/references/schemas.md

Example of the grading.json output format generated by the grader agent.

```json
{
  "expectations": [
    {
      "text": "The output includes the name 'John Smith'",
      "passed": true,
      "evidence": "Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'"
    },
    {
      "text": "The spreadsheet has a SUM formula in cell B10",
      "passed": false,
      "evidence": "No spreadsheet was created. The output was a text file."
    }
  ],
  "summary": {
    "passed": 2,
    "failed": 1,
    "total": 3,
    "pass_rate": 0.67
  },
  "execution_metrics": {
    "tool_calls": {
      "Read": 5,
      "Write": 2,
      "Bash": 8
    },
    "total_tool_calls": 15,
    "total_steps": 6,
    "errors_encountered": 0,
    "output_chars": 12450,
    "transcript_chars": 3200
  },
  "timing": {
    "executor_duration_seconds": 165.0,
    "grader_duration_seconds": 26.0,
    "total_duration_seconds": 191.0
  },
  "claims": [
    {
      "claim": "The form has 12 fillable fields",
      "type": "factual",
      "verified": true,
      "evidence": "Counted 12 fields in field_info.json"
    }
  ],
  "user_notes_summary": {
    "uncertainties": ["Used 2023 data, may be stale"],
    "needs_review": [],
    "workarounds": ["Fell back to text overlay for non-fillable fields"]
  },
  "eval_feedback": {
    "suggestions": [
      {
        "assertion": "The output includes the name 'John Smith'",
        "reason": "A hallucinated document that mentions the name would also pass"
      }
    ],
    "overall": "Assertions check presence but not correctness."
  }
}
```

--------------------------------

### Calculate PPSETMODE ioctl number

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/syzkaller-ppdev-syzlang/syzkaller-ppdev-syzlang/SKILL.md

Example calculation for the PPSETMODE ioctl using the _IOW macro formula.

```text
0x40000000 | (4 << 16) | (0x70 << 8) | 0x80 = 0x40047080 = 1074032768
```

--------------------------------

### Dialogue Parser Usage

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/dialogue-parser/evo-dialogue-parser/SKILL.md

Example of running the end-to-end parsing pipeline and validating the resulting graph structure.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-dialogue-parser/scripts')
from utils import run_end_to_end, parse_script, validate_graph

# End-to-end: parse script and write outputs
graph = run_end_to_end('/app/script.txt', '/app/dialogue.json', '/app/dialogue.dot')

# Validate
is_valid, issues = validate_graph(graph)
assert is_valid, f"Validation failed: {issues}"
print("All validations passed")
```

--------------------------------

### Normalize defect logs with Python

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/manufacturing-codebook-normalization/evo-defect-normalizer/SKILL.md

Demonstrates how to import the normalizer and validator, process logs using default or custom configurations, and validate the resulting output.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-defect-normalizer/scripts')
from normalizer import normalize_logs
from validator import validate_output

data_dir = '/app/data'
output_path = '/app/output/solution.json'

# Use defaults
records = normalize_logs(data_dir, output_path)

# Or override policy
records = normalize_logs(data_dir, output_path, config={
    'unknown_threshold': 15,
    'conf_min': 0.45,
})

print(f"Processed {len(records)} records")

errors, warnings, stats = validate_output(output_path, data_dir)
print(f"Stats: {stats}")
if not errors:
    print("VALIDATION PASSED")
```

--------------------------------

### Register and run a custom agent adapter

Source: https://github.com/zhang-henry/coevoskills/blob/main/README.md

Commands to prepare and execute a task using a custom agent registered via an import path.

```bash
export RUN_ID=custom-agent-smoke

uv run python scripts/prepare_tasks.py \
  --condition skill-only \
  --skill-set validated \
  --tasks 3d-scan-calc

MODEL=<provider>/<model-id> \
AGENT=my-agent \
  scripts/run_condition.sh skill-only \
  --only-tasks 3d-scan-calc \
  --agent-import-path my-agent=my_package.agents:MyAgent
```

--------------------------------

### RTTM Diarization Output Format

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/speaker-diarization-subtitles/instruction.md

The RTTM file records speaker segments with start times and durations.

```text
SPEAKER input 1 0.804000 0.860000 <NA> <NA> spk00 <NA> <NA>
SPEAKER input 1 23.820000 1.170000 <NA> <NA> spk01 <NA> <NA>
```

--------------------------------

### Launch SimPO Training Scripts

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/simpo-code-reproduction/environment/SimPO/README.md

Commands to initiate training for different model architectures using accelerate and DeepSpeed configurations. Adjust hardware-specific parameters like num_processes and batch size as needed.

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/mistral-7b-base-simpo.yaml
```

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/mistral-7b-instruct-simpo.yaml
```

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/llama-3-8b-base-simpo.yaml
```

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/llama-3-8b-instruct-simpo.yaml
```

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file accelerate_configs/deepspeed_zero3.yaml scripts/run_simpo.py training_configs/llama-3-8b-instruct-simpo-v2.yaml
```

--------------------------------

### Transcribe video with word-level timestamps

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/video-filler-word-remover/evo-filler-word-detector/SKILL.md

Uses Whisper to generate a list of words with start and end times.

```python
from transcribe import transcribe_video
words = transcribe_video("/root/input.mp4", model_name="base", cache_path="/root/cache/transcript.json")
# Returns: [{"word": "hello", "start": 0.0, "end": 0.5}, ...]
```

--------------------------------

### Execute Supply-Side Model Workflow

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/shock-analysis-supply/evo-supply-shock/SKILL.md

Initializes the environment, runs the model builder, and validates the resulting Excel workbook.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-supply-shock/scripts')
from data_collection import collect_pwt_data, collect_weo_data, collect_ecb_cfc_data
from excel_builder import build_supply_model
from validator import validate_workbook

# End-to-end entry point
build_supply_model(
    template_path='/root/test-supply.xlsx',
    output_path='/root/test-supply.xlsx'
)

# Validate
validate_workbook('/root/test-supply.xlsx')
```

--------------------------------

### Example TPP Task Entry

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/pddl-tpp-planning/instruction.md

JSON structure defining the domain, problem, and output paths for a TPP task.

```json
[
  {
  "id": "problem_id",
  "domain": ".../xxx.pddl",
  "problem": ".../yyy.pddl",
  "plan_output": "xxx/problem_id.txt"
  },
  ...,
]
```

--------------------------------

### Example Wyckoff Analysis Output Format

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/crystallographic-wyckoff-position-analysis/instruction.md

The expected structure of the returned dictionary, including multiplicities and rational-formatted coordinates.

```json
{
    "wyckoff_multiplicity_dict": {"a": 4, "c": 8},
    "wyckoff_coordinates_dict": {
        "a": ["0", "1/2", "1/2"],
        "c": ["3/8", "1/9", "8/9"]
    }
}
```

--------------------------------

### Organize Documents Workflow

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/organize-messy-files/evo-doc-organizer/SKILL.md

Demonstrates the full pipeline for organizing documents from a source directory into categorized output folders and validating the results.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-doc-organizer/scripts')
from utils import organize_documents, validate_organization

# Organize all documents from source into category folders
source_dir = '/root/papers/all'
output_dir = '/root/papers'

assignments = organize_documents(source_dir, output_dir)

# Validate the organization
validate_organization(source_dir, output_dir)
```

--------------------------------

### Implement Solution Script

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/crystallographic-wyckoff-position-analysis/evo-wyckoff/SKILL.md

Shows how to import and utilize the analysis function within a custom solution script.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-wyckoff/scripts')
from utils import analyze_wyckoff_position_multiplicities_and_coordinates

# The function is directly importable and usable
# Example: result = analyze_wyckoff_position_multiplicities_and_coordinates('/root/cif_files/FeS2_mp-226.cif')
```

--------------------------------

### Identify Malicious JavaScript Payload

Source: https://github.com/zhang-henry/coevoskills/blob/main/tasks/fix-druid-loophole-cve/instruction.md

Example of an authenticated request using a JavaScript filter to execute arbitrary system commands.

```http
POST /druid/indexer/v1/sampler HTTP/1.1
Content-Type: application/json

{
  "type": "index",
  "spec": {
    "dataSchema": {
      "transformSpec": {
        "filter": {
          "type": "javascript",
          "function": "function(){java.lang.Runtime.getRuntime().exec('malicious_command');}",
          "": {
            "enabled": true
          }
        }
      }
    }
  }
}
```

--------------------------------

### Implement interaction Hamiltonian in PIQS

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/background_docs/quantum-numerical-simulation/qutip-piqs-tutorial-conventions.md

Use this specific form for the interaction Hamiltonian to maintain consistency with the PIQS Superradiance example.

```python
h_int = g * tensor(a + a.dag(), jx)
```

--------------------------------

### Execute Java Build Fix Pipeline

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/fix-build-google-auto/evo-java-build-fixer/SKILL.md

Demonstrates how to import the skill utilities and run the end-to-end build fix pipeline, followed by output validation.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-java-build-fixer/scripts')
from utils import run_end_to_end, validate_output

# Run the full pipeline
success = run_end_to_end(base_path='/home/travis/build/failed', build_timeout=300)
print(f"Build fix {'succeeded' if success else 'may need manual review'}")

# Validate outputs
valid = validate_output(base_path='/home/travis/build/failed')
print(f"Validation: {'PASS' if valid else 'FAIL'}")
```

--------------------------------

### Run Optimization Script

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/react-performance-debugging/evo-nextjs-perf-optimizer/SKILL.md

Initializes the optimizer environment and executes the optimization and validation routines.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-nextjs-perf-optimizer/scripts')
from optimizer import run_optimization, validate_optimization

run_optimization('/app')
validate_optimization('/app')
```

--------------------------------

### Detect and Verify Java Environment

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/background_docs/fix-druid-loophole-cve/druid-cve-2021-25646-background.md

Set the JAVA_HOME environment variable based on the installed JDK and verify the versions of Java and Maven.

```bash
# Detect JAVA_HOME from the installed JDK
export JAVA_HOME=$(dirname $(dirname $(readlink -f $(which javac))))
# Verify
java -version
mvn -v
```

--------------------------------

### Execute PDDL Planning Tasks

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/virtualhome-agent-planning/evo-pddl-airport-planner/SKILL.md

Demonstrates how to solve all tasks defined in a problem.json file and validate the generated plans.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-pddl-airport-planner/scripts')
from solve import solve_all_tasks, validate_plans

# Solve all tasks from problem.json
results = solve_all_tasks('/app/problem.json', base_dir='/app')

# Validate outputs
valid = validate_plans('/app/problem.json', base_dir='/app')
assert valid, "Some plans failed validation"
print("All tasks solved successfully!")
```

--------------------------------

### Execute End-to-End Fuzzing Workflow

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/setup-fuzzing-py/evo-python-fuzzing/SKILL.md

Run the complete fuzzing pipeline including discovery, analysis, driver generation, setup, execution, and validation.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-python-fuzzing/scripts')
from utils import run_all, validate_all

# End-to-end: discover, analyze, generate, setup, fuzz, validate
results = run_all(
    base_dir='/app',
    libraries_file='/app/libraries.txt',
    timeout=10
)

# Validate all artifacts exist and logs are valid
all_ok = validate_all('/app', '/app/libraries.txt')
print(f"All OK: {all_ok}")
```

--------------------------------

### run_end_to_end(input_path, output_path=None)

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/weighted-gdp-calc/evo-gdp-weighted-mean/SKILL.md

Executes the complete workflow: discovers layout, writes all formulas, saves, recalculates with LibreOffice, and validates.

```APIDOC
## run_end_to_end(input_path, output_path=None)

### Description
Executes the complete workflow: discovers layout, writes all formulas, saves, recalculates with LibreOffice, and validates.

### Parameters
- **input_path** (str) - Required - Path to the input Excel file.
- **output_path** (str) - Optional - Path to save the processed Excel file.
```

--------------------------------

### Run the Web-to-Audiobook Pipeline

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/pg-essay-to-audiobook/evo-web-tts-audiobook/SKILL.md

Orchestrates the full conversion process from a list of URLs to a single MP3 file, followed by a duration validation check.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-web-tts-audiobook/scripts')
from utils import run_pipeline, validate_audio

essays = [
    {'url': 'http://paulgraham.com/ds.html', 'title': 'Do Things that Don\'t Scale'},
    {'url': 'http://paulgraham.com/foundermode.html', 'title': 'Founder Mode'},
]

output_path = '/root/audiobook.mp3'
run_pipeline(essays, output_path, voice='en-US-AriaNeural')

# Validate
duration = validate_audio(output_path, min_duration_seconds=60)
print(f"Validated: {duration:.0f}s")
```

--------------------------------

### HTTP Body Field Boundary Matching with PCRE

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/suricata-custom-exfil/evo-suricata-http-exfil/SKILL.md

Use this PCRE pattern to ensure field names are matched only at the start of the body or after an ampersand separator.

```text
pcre:"/(?:^|&)fieldname=VALUE_PATTERN/"
```

--------------------------------

### Initialize Google Vertex AI Credentials

Source: https://github.com/zhang-henry/coevoskills/blob/main/docs/provider_setup.md

Create Application Default Credentials using the gcloud CLI.

```bash
gcloud auth application-default login
gcloud auth application-default print-access-token >/dev/null
```

--------------------------------

### Generate and Validate Suricata Rules

Source: https://github.com/zhang-henry/coevoskills/blob/main/artifacts/skills/suricata-custom-exfil/evo-suricata-http-exfil/SKILL.md

Example usage of the generate_and_validate function to create a rule and verify it against pcaps. Ensure the path to the skill scripts is correctly added to sys.path.

```python
import sys
sys.path.insert(0, '/app/environment/skills/evo-suricata-http-exfil/scripts')
from utils import generate_and_validate

# Example: detect a hypothetical beacon pattern on /api/v3/heartbeat
# with header X-App-Type: beacon, body containing token= (40+ alphanums)
# and checksum= (32 hex chars). Adapt all parameters from your task instruction.
#
# Note: body_fields[].regex is the VALUE pattern only; the function
# automatically adds (?:^|&)fieldname= boundary logic.
success, details = generate_and_validate(
    rules_path='/root/local.rules',
    suricata_yaml='/root/suricata.yaml',
    pcaps_dir='/root/pcaps/',
    sid=1000099,
    msg='Hypothetical beacon detection',
    method='POST',
    uri_path='/api/v3/heartbeat',
    header_name='X-App-Type',
    header_value='beacon',
    body_fields=[
        {'name': 'token', 'regex': '[A-Za-z0-9]{40,}'},
        {'name': 'checksum', 'regex': '[0-9a-f]{32}(&|$)'},
    ],
)

for line in details:
    print(line)

if success:
    print('\nAll validations passed!')
else:
    print('\nSome validations failed - check details above.')
    sys.exit(1)
```
Tokens:
Raw