Weco

https://docs.weco.ai
Admin
Weco is a platform for self-improving code that automates code optimization with AI, iteratively...
Tokens:27,249
Snippets:282
Trust Score:7.5
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Example Dependencies

Source: https://docs.weco.ai/examples/fraud-detection

Clone the repository and install the specific dependencies required for the fraud detection example.

```shell
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/fraud-detection
pip install --upgrade -r requirements.txt
```

--------------------------------

### Clone the example repository

Source: https://docs.weco.ai/using-weco/integrations/langfuse

Download the ZephHR example project to get started with the LangFuse integration.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/langfuse-zeph-hr-qa
```

--------------------------------

### Launch visual setup wizard

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Starts the Weco optimization process using the browser-based wizard for visual configuration.

```bash
weco run --eval-backend langsmith
```

--------------------------------

### Install Dependencies

Source: https://docs.weco.ai/examples/triton-optimization

Required packages for running the optimization example.

```bash
pip install numpy torch triton
```

--------------------------------

### Install from Source

Source: https://docs.weco.ai/examples/model-development

Installation method for contributors or those modifying the Weco source code.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli
pip install -e .
```

--------------------------------

### Start a run with an agent

Source: https://docs.weco.ai/using-weco/steerability

Example prompt for an agent to initiate a run using the Weco skill.

```text
Use the Weco skill to maximize ROC-AUC on the tabular classification task in train.py.
```

--------------------------------

### Setup Weco Skill for Cursor

Source: https://docs.weco.ai/reference/manual-installation

Installs the Weco skill and adds a global Cursor rule for project-wide availability.

```bash
weco setup cursor
```

--------------------------------

### Clone the Weco example repository

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Download the ZephHR QA agent example project to begin the optimization tutorial.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/langsmith-zeph-hr-qa
```

--------------------------------

### Set up virtual environment and dependencies

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Initialize a Python virtual environment and install required dependencies for the project.

```bash
python -m venv .venv
source .venv/bin/activate
pip install openai langsmith
```

```cmd
python -m venv .venv
.venv\Scripts\activate
pip install openai langsmith
```

```powershell
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install openai langsmith
```

--------------------------------

### Install Weco CLI

Source: https://docs.weco.ai/using-weco/integrations/langfuse

Install the Weco CLI with LangFuse support using uv.

```bash
uv tool install 'weco[langfuse]'
```

--------------------------------

### Install Weco CLI from Source

Source: https://docs.weco.ai/reference/manual-installation

Clone the repository and install in editable mode for development or contribution.

```bash
git clone https://github.com/wecoai/weco-cli.git
cd weco-cli
pip install -e .
```

--------------------------------

### Create the LangSmith dataset

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Run the setup script to initialize the zephhr-qa dataset.

```bash
python setup_dataset.py
```

--------------------------------

### Install Dependencies

Source: https://docs.weco.ai/examples/pytorch-optimization

Install the required PyTorch dependency for the optimization scripts.

```shell
pip install torch
```

--------------------------------

### Install Weco CLI via uv

Source: https://docs.weco.ai/reference/manual-installation

Installation method using the uv package manager.

```bash
uv tool install weco
```

--------------------------------

### Install Python Dependencies

Source: https://docs.weco.ai/examples/model-development

Installs the necessary machine learning libraries for the project.

```bash
pip install pandas numpy scikit-learn torch xgboost lightgbm catboost
```

--------------------------------

### Install development dependencies

Source: https://docs.weco.ai/resources/contributing

Installs the project in editable mode with development dependencies enabled.

```bash
pip install -e ".[dev]"
```

--------------------------------

### Install Dependencies

Source: https://docs.weco.ai/examples/prompt-engineering

Install required Python packages for the optimization and evaluation scripts.

```shell
pip install openai datasets
```

--------------------------------

### Install Weco CLI on Windows

Source: https://docs.weco.ai/reference/manual-installation

Installation method for Windows environments using PowerShell.

```powershell
irm https://weco.ai/install.ps1 | iex
```

--------------------------------

### Run Optimization on Multiple Files

Source: https://docs.weco.ai/reference/optimize-with-cli

Example of running an optimization across multiple source files.

```bash
weco run --sources model.py utils.py config.py \
     --eval-command "python evaluate.py" \
     --metric accuracy \
     --goal maximize
```

--------------------------------

### Install Weco CLI

Source: https://docs.weco.ai/examples/extract-line-plot

Installation commands for the Weco CLI across different environments.

```bash
curl -fsSL https://weco.ai/install.sh | sh
```

```bash
uv tool install weco
```

```powershell
irm https://weco.ai/install.ps1 | iex
```

```bash
pip install weco
```

```bash
git clone https://github.com/wecoai/weco-cli.git
cd weco-cli
pip install -e .
```

--------------------------------

### Set up Weco via AI coding assistant

Source: https://docs.weco.ai/quickstart

Use this prompt with an AI coding assistant to automatically install the CLI, add the Weco skill, and begin identifying optimization opportunities.

```text
Set up Weco in this project and find something worth optimizing:

1. Install the Weco CLI:  curl -fsSL https://weco.ai/install.sh | sh
2. Add the Weco skill: weco setup
3. Log in: weco login
4. Use the Weco skill to identify optimization opportunities in this codebase.
```

--------------------------------

### Steer an existing run

Source: https://docs.weco.ai/using-weco/steerability

Example prompt for an agent to branch from the current best node to test a new idea.

```text
This run has plateaued around 0.81 ROC-AUC. There's a paper, "TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second" (arXiv:2207.01848), that does strong zero-training inference on tabular data. Branch from the current best and ensemble it with the model we have, give it 15 steps.
```

--------------------------------

### Install Weco CLI via pip

Source: https://docs.weco.ai/reference/manual-installation

Standard Python package installation. Using a virtual environment is recommended.

```bash
pip install weco
```

--------------------------------

### Install Weco CLI via Shell

Source: https://docs.weco.ai/reference/manual-installation

Standard installation method for Unix-based systems using a shell script.

```bash
curl -fsSL https://weco.ai/install.sh | sh
```

--------------------------------

### Install Weco skills

Source: https://docs.weco.ai/reference/cli/agent

Commands to install the Weco skill into supported AI coding assistants.

```bash
# Interactive picker (defaults to all supported tools)
weco setup

# Or target a specific tool
weco setup claude-code
weco setup cursor
weco setup codex
weco setup openclaw

# Install for every supported tool at once
weco setup all
```

--------------------------------

### Define Prompt Optimization Guidelines for AIME

Source: https://docs.weco.ai/examples/prompt-engineering

A markdown-formatted guide providing context and constraints for optimizing prompts targeting GPT-4.1-mini.

```markdown
# Weco Prompt Optimization Guidelines for AIME (Targeting GPT-4.1)

## 1. Goal

Your objective is to modify the the `optimize.py` file to improve the `accuracy` metric when solving AIME math problems. The modifications should leverage the capabilities of the target model, **GPT-4.1**.

## 2. Files and Workflow

*   **Target File for Modification:** `optimize.py`. *   **Evaluation Script:** `eval.py`. This script:
    *   Defines the actual LLM used for solving (`MODEL_TO_USE`, which is set to `gpt-4.1-mini` in this context).
    *   Calls `optimize.solve(problem, model_name="gpt-4.1-mini").
    *   Parses the output from `optimize.solve`. **Crucially, it expects the final 3-digit answer (000-999) to be enclosed in `\boxed{XXX}`.** For example: `\boxed{042}`. Your prompt modifications *must* ensure the model consistently produces this format for the final answer.
    *   Compares the extracted answer to the ground truth and prints the `accuracy:` metric, which Weco uses for guidance.

## 3. Target Model: GPT-4.1-mini

You are optimizing the prompt for `gpt-4.1-mini`. Based on its characteristics, consider the following:

*   **Strengths:**
    *   **Significantly Improved Instruction Following:** GPT-4.1-mini is better at adhering to complex instructions, formats, and constraints compared to previous models. This is key for AIME where precision is vital. It excels on hard instruction-following tasks.
    *   **Stronger Coding & Reasoning:** Its improved coding performance (e.g., SWE-bench) suggests enhanced logical reasoning capabilities applicable to mathematical problem-solving.
    *   **Refreshed Knowledge:** Knowledge cutoff is June 2024.
*   **Considerations:**
    *   **Literal Interpretation:** GPT-4.1-mini can be more literal. Prompts should be explicit and specific about the desired reasoning process and output format. Avoid ambiguity.

## 4. Optimization Strategies (Focus on `PROMPT_TEMPLATE` in `optimize.py`)

The primary goal is to enhance the model's reasoning process for these challenging math problems. Focus on Chain-of-Thought (CoT) designs within the `PROMPT_TEMPLATE`.

**Ideas to Explore:**
You don't have to implement all of them, but the following ideas might be helpful:
*   **Workflow Patterns** try to use some of the following patterns:
    *  **Linear**: Linear workflow, standarded CoT E.g. considering the following thinking steps (you don't have to include all of them), "1. Understand the problem constraints. 2. Identify relevant theorems/formulas. 3. Formulate a plan. 4. Execute calculations step-by-step. 5. Verify intermediate results. 6. State the final answer in the required format."
    *  **List Candidates**: You can ask the model to propose a few solutions in a particular step and pick the best solution. You can potentially also set the criterias in the prompt.
    *   **Code** Use pesudo code to define even more complex workflows with loops, conditional statement, or go to statement.
*   **Other CoT Techniques:**
    *   Self-Correction/Reflection
    *   Plan Generation
    *   Debate, simulating multiple characters
    *   Tree of thought
*   **Few-Shot Examples:** You *could* experiment with adding 1-2 high-quality AIME problem/solution examples directly into the `PROMPT_TEMPLATE` string (similar to how Weco attempted in one of the runs). Ensure the examples clearly show the desired reasoning style and the final `\boxed{XXX}` format.
*   **Play with format:** The way you format the prompt. Markdown, xml, json, code or natural language. Similarly for the thinking tokens themselves you can also try out different formats.

## 5. Constraints
*   **Ensure the final output reliably contains `\boxed{XXX}` as the evaluation script depends on it.**
```

--------------------------------

### Example PyTorch Module

Source: https://docs.weco.ai/reference/optimize-with-cli

A sample PyTorch module structure that can be targeted for optimization.

```python
# File: module.py

import torch
import torch.nn as nn

class Model(nn.Module):
    """
    Model that performs a matrix multiplication, division, summation, and scaling.
    """

    def __init__(self, input_size, hidden_size, scaling_factor):
        super(Model, self).__init__()
        self.weight = nn.Parameter(torch.randn(hidden_size, input_size))
        self.scaling_factor = scaling_factor

    def forward(self, x):
        x = torch.matmul(x, self.weight.T)
        x = x / 2
        x = torch.sum(x, dim=1, keepdim=True)
        x = x * self.scaling_factor
        return x
```

--------------------------------

### Initialize and Log External Runs via CLI

Source: https://docs.weco.ai/reference/cli/tracking

Use the observe init command to start a run and log commands to record metrics and status updates. These commands are fire-and-forget and will not interrupt the execution loop.

```bash
# Initialize a run (prints run ID to stdout)
WECO_RUN_ID=$(weco observe init --name "my-experiment" --metric val_bpb --goal min --source train.py)

# Log experiment results
weco observe log --run-id "$WECO_RUN_ID" --step 0 --description "baseline" --metrics '{"val_bpb": 2.36}' --source train.py
weco observe log --run-id "$WECO_RUN_ID" --step 1 --description "increase batch size" --metrics '{"val_bpb": 2.26}' --source train.py
weco observe log --run-id "$WECO_RUN_ID" --step 2 --status failed --description "OOM" --metrics '{"val_bpb": 0.0}' --source train.py
```

--------------------------------

### Fan out multiple branches

Source: https://docs.weco.ai/using-weco/steerability

Example prompt for an agent to initiate multiple parallel branches from the current best node.

```text
Try three ideas in parallel from the current best, 10 steps each:
target encoding for the high-cardinality categoricals, a CatBoost model, and stacking the current model with logistic regression on out-of-fold predictions.
```

--------------------------------

### Start a Weco optimization run from the CLI

Source: https://docs.weco.ai/using-weco/dashboard

Commands to initiate an optimization run using the Weco CLI across different operating systems.

```bash
weco run --source optimize.py \
     --eval-command "python evaluate.py" \
     --metric accuracy --goal maximize
```

```batch
weco run --source optimize.py ^
     --eval-command "python evaluate.py" ^
     --metric accuracy --goal maximize
```

```powershell
weco run --source optimize.py `
     --eval-command "python evaluate.py" `
     --metric accuracy --goal maximize
```

--------------------------------

### Example Prompts for Weco Skill

Source: https://docs.weco.ai/reference/manual-installation

Natural language prompts to trigger Weco optimization within an AI assistant.

```text
Use Weco to make this function faster.
```

```text
Optimize module.py for accuracy using evaluate.py as the benchmark.
```

--------------------------------

### Steer the agent via dashboard chat

Source: https://docs.weco.ai/using-weco/claude-in-dashboard

Example prompt to steer an active agent run from the dashboard interface.

```text
Branch from the best node and add engineered interaction features for the top-importance columns, 12 steps.
```

--------------------------------

### Setup Weco Skill for Claude Code

Source: https://docs.weco.ai/reference/manual-installation

Configures the Weco skill file for use within Claude Code.

```bash
weco setup claude-code
```

--------------------------------

### Derive a new branch via CLI

Source: https://docs.weco.ai/using-weco/steerability

Illustrative example of the underlying command used to create a new branch from a parent run.

```bash
# Illustrative only: your agent runs this for you (see below).
weco run derive <PARENT_RUN_ID> \
     --from-step best \
     --steps 15 \
     --additional-instructions "Ensemble TabPFN (arXiv:2207.01848) with the current model."
```

--------------------------------

### weco run

Source: https://docs.weco.ai/reference/cli/optimizing

Starts an optimization run for the specified source file(s) using the provided evaluation command and configuration.

```APIDOC
## weco run

### Description
Starts the optimization process for one or more source files. The CLI modifies the files during evaluation and can automatically apply the best-performing version upon completion.

### Required Arguments
- **-s, --source** (string) - Path to a single source code file to be optimized.
- **--sources** (list) - Paths to multiple source code files (up to 10).
- **-c, --eval-command** (string) - Command to run for evaluating the code.
- **-m, --metric** (string) - The name of the metric to optimize.
- **-g, --goal** (string) - Optimization goal: 'maximize' or 'minimize'.

### Optional Arguments
- **-n, --steps** (integer) - Number of optimization steps (default: 100).
- **-M, --model** (string) - LLM model identifier.
- **-i, --additional-instructions** (string) - Instructions or path to an instruction file.
- **-l, --log-dir** (string) - Directory for logging (default: .runs/).
- **--save-logs** (boolean) - Save execution output for each step.
- **--eval-timeout** (integer) - Timeout in seconds for evaluation.
- **--apply-change** (boolean) - Automatically apply the best solution.
- **--require-review** (boolean) - Require manual review of proposed changes.
- **--eval-backend** (string) - Backend for evaluation: 'shell', 'langsmith', or 'langfuse'.
- **--output** (string) - Output mode: 'rich' or 'plain'.
- **--daemon** (boolean) - Detach the run to the background.
```

--------------------------------

### Launch Claude Code with Weco

Source: https://docs.weco.ai/reference/cli/agent

Commands to start a Claude Code session bridged to the Weco dashboard with various configuration options.

```bash
# Launch Claude Code with a local TUI bridged to the dashboard
weco start claude

# Seed the first turn with a prompt
weco start claude -p "Use Weco to make forward() faster in model.py"

# Headless: no local UI, stream only to the dashboard (good for background sessions)
weco start claude --headless --allow-tools -p "Optimize the API handler for latency"

# Bill LLM usage through your Weco credit wallet instead of your local Claude auth
weco start claude --billing weco

# Forward extra flags to the underlying claude CLI (after a -- separator)
weco start claude -- --model claude-opus-4-8
```

--------------------------------

### Clone the repository

Source: https://docs.weco.ai/resources/contributing

Initializes the local environment by cloning the repository and navigating into the project directory.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli
```

--------------------------------

### Prepare Data and Evaluate Baseline

Source: https://docs.weco.ai/examples/fraud-detection

Scripts to download the competition data, build the train/val split, and verify the baseline performance.

```shell
python prepare_data.py
```

```shell
python evaluate.py
# auc_roc: 0.909132
```

--------------------------------

### Prepare Dataset

Source: https://docs.weco.ai/examples/extract-line-plot

Clone the repository and generate the subset of data for evaluation.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/extract-line-plot
uv run --with huggingface_hub python prepare_data.py
```

--------------------------------

### Clone Repository

Source: https://docs.weco.ai/examples/model-development

Initializes the project directory by cloning the Weco CLI repository.

```bash
git clone https://github.com/WecoAI/weco-cli.git
cd weco-cli/examples/spaceship-titanic
```

--------------------------------

### Create Evaluation Script

Source: https://docs.weco.ai/examples/pytorch-optimization

A script to benchmark the baseline model and load modules dynamically.

```python
import time
import sys
import os
import pathlib
import importlib
import traceback
import torch
import torch.nn as nn


########################################################
# Baseline
########################################################
class Model(nn.Module):
    """
    Model that performs a matrix multiplication, division, summation, and scaling.
    """

    def __init__(self, input_size, hidden_size, scaling_factor):
        super(Model, self).__init__()
        self.weight = nn.Parameter(torch.randn(hidden_size, input_size))
        self.scaling_factor = scaling_factor

    def forward(self, x):
        """
        Args:
            x (torch.Tensor): Input tensor of shape (batch_size, input_size).
        Returns:
            torch.Tensor: Output tensor of shape (batch_size, hidden_size).
        """
        x = torch.matmul(x, self.weight.T)
        x = x / 2
        x = torch.sum(x, dim=1, keepdim=True)
        x = x * self.scaling_factor
        return x


########################################################
# Weco Solution
########################################################
def load_module_from_path(module_path: str, add_to_sys_modules: bool = False):
    # Clean out all old compiled extensions to prevent namespace collisions during build
    module_path = pathlib.Path(module_path)
    name = module_path.stem
    spec = importlib.util.spec_from_file_location(name, module_path)
    mod = importlib.util.module_from_spec(spec)  # type: ignore
    if add_to_sys_modules:
        sys.modules[name] = mod
    spec.loader.exec_module(mod)  # type: ignore
    return mod


########################################################
# Benchmark
########################################################
os.environ["MAX_JOBS"] = "1"  # number of workers for building with ninja


def get_inputs(B, N, device):
    return torch.randn(B, N, device=device, dtype=torch.float32)

# NOTE: We included this custom benchmark function to avoid adding the triton dependency
# and allow users to run the example on CPUs. However, if you have an NVIDIA GPU,
# you can use the triton.testing.do_bench function instead.
# Refer to examples/triton/evaluate.py for an example.
# triton.testing.do_bench documentation: https://triton-lang.org/main/python-api/generated/triton.testing.do_bench.html
@torch.no_grad()
def bench(f, inputs, n_warmup, n_rep):
    device_type = inputs.device.type

    # warm up
    for _ in range(n_warmup):
        f(inputs)  # noqa
    if device_type == "cuda":
        torch.cuda.synchronize()
    elif device_type == "mps":
        torch.mps.synchronize()

    # benchmark
    t_avg = 0.0
    for _ in range(n_rep):
        # time forward pass
        start_time = time.time()
        f(inputs)
        t_avg += time.time() - start_time

        # Synchronize after each iteration
        if device_type == "cuda":
            torch.cuda.synchronize()
        elif device_type == "mps":
            torch.mps.synchronize()

    t_avg /= n_rep * 1e-3
    return t_avg


if __name__ == "__main__":
    import argparse
```

--------------------------------

### Initialize an Experiment Run

Source: https://docs.weco.ai/using-weco/track-any-loop

Create a new run and capture the run ID to associate subsequent logs.

```bash
WECO_RUN_ID=$(weco observe init \
  --name "my-experiment" \
  --metric val_bpb \
  --goal min \
  --source train.py)
```

--------------------------------

### Weco Evaluation Log Output

Source: https://docs.weco.ai/examples/extract-line-plot

Example of the log output generated during the evaluation phase, showing progress and the final metric.

```text
[setup] evaluating 100 samples using gpt-4o-mini …
[progress] 5/100 done, avg score: 0.3120, elapsed 12.3s
[progress] 10/100 done, avg score: 0.3280, elapsed 24.1s
...
accuracy: 0.3420
```

--------------------------------

### Define a Code Evaluator

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Create a Python function that accepts a run and an example, returning a dictionary with a key, score, and optional comment.

```python
def my_evaluator(run, example) -> dict:
    # run.outputs contains the target function's output
    # example.outputs contains the expected output from the dataset
    return {"key": "my_evaluator", "score": 1.0, "comment": "Passed"}
```

--------------------------------

### Configure environment variables

Source: https://docs.weco.ai/using-weco/integrations/langfuse

Set the necessary API keys and base URL for LangFuse and OpenAI integration.

```bash
export OPENAI_API_KEY="sk-..."
export LANGFUSE_SECRET_KEY="sk-lf-..."
export LANGFUSE_PUBLIC_KEY="pk-lf-..."
export LANGFUSE_BASE_URL="https://cloud.langfuse.com"  # or https://us.cloud.langfuse.com
```

--------------------------------

### Create the Evaluation Script

Source: https://docs.weco.ai/examples/triton-optimization

This script loads a candidate model from a specified path, verifies its output against a baseline model, and benchmarks performance using Triton's do_bench.

```python
import sys
import pathlib
import importlib
import importlib.util
import traceback
import torch
import torch.nn as nn
from triton.testing import do_bench


########################################################
# Baseline
########################################################
class Model(nn.Module):
    """
    Simple model that performs a Swish activation.
    """

    def __init__(self):
        super(Model, self).__init__()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        """
        Applies Swish activation to the input tensor.

        Args:
            x (torch.Tensor): Input tensor of any shape.

        Returns:
            torch.Tensor: Output tensor with Swish applied, same shape as input.
        """
        return x * torch.sigmoid(x)


########################################################
# Benchmark
########################################################
def load_module_from_path(module_path: str, add_to_sys_modules: bool = False):
    # Clean out all old compiled extensions to prevent namespace collisions during build
    module_path = pathlib.Path(module_path)
    name = module_path.stem
    spec = importlib.util.spec_from_file_location(name, module_path)
    mod = importlib.util.module_from_spec(spec)  # type: ignore
    if add_to_sys_modules:
        sys.modules[name] = mod
    spec.loader.exec_module(mod)  # type: ignore
    return mod


def get_inputs(batch_size, dim, device):
    return torch.randn(batch_size, dim, device=device, dtype=torch.float32)


if __name__ == "__main__":
    import argparse

    parser = argparse.ArgumentParser()
    parser.add_argument("--path", type=str, required=True)
    args = parser.parse_args()

    # benchmarking parameters
    n_correctness_trials = 10
    correctness_tolerance = 1e-5
    warmup_ms = 100
    rep_ms = 500

    # input parameters
    batch_size = 2000
    dim = 16384

    # load solution module
    try:
        torch.manual_seed(0)
        solution_module = load_module_from_path(args.path, add_to_sys_modules=False)
        solution_model = solution_module.Model().to("cuda")
        assert isinstance(solution_model, nn.Module)
    except Exception:
        print(f"Candidate module initialization failed: {traceback.format_exc()}")
        exit(1)

    torch.manual_seed(0)
    baseline_model = Model().to("cuda")

    # measure correctness
    max_diff_avg = 0
    for _ in range(n_correctness_trials):
        inputs = get_inputs(batch_size=batch_size, dim=dim, device="cuda")
        with torch.no_grad():
            optimized_output = solution_model(inputs)
            if torch.isnan(optimized_output).any():
                print("Incorrect solution: NaN detected in optimized model output")
            if torch.isinf(optimized_output).any():
                print("Incorrect solution: Inf detected in optimized model output")
            baseline_output = baseline_model(inputs)
            max_diff_avg += torch.max(torch.abs(optimized_output - baseline_output))
    max_diff_avg /= n_correctness_trials
    print(f"max float diff between values of baseline and optimized model: {max_diff_avg}")
    if max_diff_avg > correctness_tolerance:
        print("Incorrect solution: max float diff is too high")

    # measure performance
    inputs = get_inputs(batch_size=batch_size, dim=dim, device="cuda")
    t_avg_baseline = do_bench(lambda: baseline_model(inputs), warmup=warmup_ms, rep=rep_ms)
    print(f"baseline time: {t_avg_baseline:.2f}ms")
    t_avg_optimized = do_bench(lambda: solution_model(inputs), warmup=warmup_ms, rep=rep_ms)
    print(f"optimized time: {t_avg_optimized:.2f}ms")
    print(f"speedup: {t_avg_baseline / t_avg_optimized:.2f}x")
```

--------------------------------

### weco observe init

Source: https://docs.weco.ai/reference/cli/tracking

Initializes a new experiment run and returns a unique run ID.

```APIDOC
## weco observe init

### Description
Initializes a new experiment run in the Weco dashboard. This command outputs the generated run ID to stdout, which is required for subsequent logging operations.

### Parameters
- **--name** (string) - Required - The name of the experiment.
- **--metric** (string) - Required - The primary metric to track.
- **--goal** (string) - Required - The optimization goal (e.g., min or max).
- **--source** (string) - Required - The source file or script associated with the run.
```

--------------------------------

### Configure environment variables

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Set the necessary API keys for OpenAI and LangSmith to enable integration functionality.

```bash
export OPENAI_API_KEY="sk-..."
export LANGCHAIN_API_KEY="lsv2_..."
```

--------------------------------

### Track Python Loop Experiments

Source: https://docs.weco.ai/using-weco/track-any-loop

Use WecoObserver to initialize a run and log metrics for each iteration of a loop. Ensure the source code files are accessible for reading during the logging process.

```python
from weco.observe import WecoObserver

obs = WecoObserver()
run = obs.create_run(
    name="sweep v3",
    source_code={"train.py": open("train.py").read()},
    primary_metric="val_bpb",
    maximize=False,
)

for i, result in enumerate(experiments):
    run.log_step(
        step=i,
        status="completed" if result.kept else "failed",
        description=result.description,
        metrics={"val_bpb": result.val_bpb, "memory_gb": result.memory_gb},
        code={"train.py": open("train.py").read()},
    )
```

--------------------------------

### Run optimization via CLI

Source: https://docs.weco.ai/using-weco/integrations/langsmith

Executes the optimization process with all parameters explicitly defined in the command line.

```bash
weco run --source agent.py \
  --eval-backend langsmith \
  --langsmith-dataset zephhr-qa \
  --langsmith-splits opt \
  --langsmith-target agent:answer_hr_question \
  --langsmith-evaluators evaluators:json_schema_validity evaluators:conciseness \
  --langsmith-dashboard-evaluators helpfulness correctness \
  --langsmith-metric-function evaluators:qa_score \
  --additional-instructions optimizer_exemplars.md \
  --metric qa_score --goal maximize --steps 10
```

--------------------------------

### Configure Automatic Top-up

Source: https://docs.weco.ai/reference/cli/credits

Enables or disables automatic credit purchases when the balance falls below a threshold.

```bash
# Enable automatic top-up
weco credits autotopup --enable

# Disable automatic top-up  
weco credits autotopup --disable
```

--------------------------------

### weco observe init

Source: https://docs.weco.ai/using-weco/track-any-loop

Initializes a new external run for tracking. This command prints the run ID to stdout, which can be captured for subsequent logging operations.

```APIDOC
## weco observe init

### Description
Create a new external run for tracking.

### Arguments
- **-s, --source** (string) - Required - Single source code file to track.
- **--sources** (string) - Required - Multiple source code files to track.
- **--metric** (string) - Required - Primary metric name (e.g. val_bpb).
- **-g, --goal** (string) - Optional - maximize/max or minimize/min (default: minimize).
- **--name** (string) - Optional - Run name.
- **-i, --additional-instructions** (string) - Optional - Instructions for the run.
```

--------------------------------

### Manage Review Mode Workflow

Source: https://docs.weco.ai/reference/cli/optimizing

Commands to initiate a run with review requirements, list pending nodes, revise code, and submit for evaluation.

```bash
# Start a run that waits for your approval at each step
weco run --source model.py \
     --eval-command "python eval.py" \
     --metric accuracy \
     --goal maximize \
     --require-review

# From another terminal: list nodes awaiting approval (includes the proposed code)
weco run review <run_id>

# Optionally replace a pending node's code with your own edit
weco run revise <run_id> --node <node_id> --source model.py

# Approve: evaluate the pending node and unlock the next proposal
weco run submit <run_id> --node <node_id>
```

--------------------------------

### Run Weco Optimization

Source: https://docs.weco.ai/reference/optimize-with-cli

Commands to initiate an optimization run, formatted for different operating systems.

```bash
weco run --source <path-to-your-code-to-optimize> \
     --eval-command <command-to-run-the-evaluation-script> \
     --metric <metric-name> \
     --goal <maximize/minimize> \
     --steps <number-of-optimization-iterations> \
     --additional-instructions <additional-instructions-for-the-LLM-to-follow>
```

```cmd
weco run --source <path-to-your-code-to-optimize> ^
     --eval-command <command-to-run-the-evaluation-script> ^
     --metric <metric-name> ^
     --goal <maximize/minimize> ^
     --steps <number-of-optimization-iterations> ^
     --additional-instructions <additional-instructions-for-the-LLM-to-follow>
```

```powershell
weco run --source <path-to-your-code-to-optimize> `
     --eval-command <command-to-run-the-evaluation-script> `
     --metric <metric-name> `
     --goal <maximize/minimize> `
     --steps <number-of-optimization-iterations> `
     --additional-instructions <additional-instructions-for-the-LLM-to-follow>
```

--------------------------------

### Benchmark PyTorch Model Correctness and Performance

Source: https://docs.weco.ai/examples/pytorch-optimization

This script initializes a model, compares its output against a baseline for correctness, and measures execution time to calculate speedup.

```python
    parser = argparse.ArgumentParser()
    parser.add_argument("--path", type=str, required=True)
    parser.add_argument("--device", default="cpu", type=str)
    args = parser.parse_args()

    # benchmark parameters
    n_correctness_trials = 10
    correctness_tolerance = 1e-5
    n_warmup = 1000
    n_rep = 5000

    # init and input parameters
    batch_size, input_size, hidden_size, scaling_factor = 128, 10, 20, 1.5

    # load solution module
    try:
        torch.manual_seed(0)
        solution_module = load_module_from_path(args.path, add_to_sys_modules=False)
        solution_model = solution_module.Model(input_size, hidden_size, scaling_factor).to(args.device)
        assert isinstance(solution_model, nn.Module)
        assert hasattr(solution_model, "forward")
    except Exception:
        print(f"Candidate module initialization failed: {traceback.format_exc()}")
        exit(1)

    torch.manual_seed(0)
    baseline_model = Model(input_size, hidden_size, scaling_factor).to(args.device)

    # measure correctness
    max_diff_avg = 0
    for _ in range(n_correctness_trials):
        inputs = get_inputs(batch_size, input_size, args.device)
        optimized_output = solution_model(inputs)
        if torch.isnan(optimized_output).any():
            print("Incorrect solution: NaN detected in optimized model output")
        if torch.isinf(optimized_output).any():
            print("Incorrect solution: Inf detected in optimized model output")
        baseline_output = baseline_model(inputs)
        max_diff_avg += torch.max(torch.abs(optimized_output - baseline_output))
    max_diff_avg /= n_correctness_trials
    print(f"max float diff between values of baseline and optimized model: {max_diff_avg}")
    if max_diff_avg > correctness_tolerance:
        print("Incorrect solution: max float diff is too high")

    # measure performance
    inputs = get_inputs(batch_size, input_size, args.device)
    t_avg_baseline = bench(baseline_model, inputs, n_warmup, n_rep)
    print(f"baseline time: {t_avg_baseline:.2f}ms")
    t_avg_optimized = bench(solution_model, inputs, n_warmup, n_rep)
    print(f"optimized time: {t_avg_optimized:.2f}ms")
    print(f"speedup: {t_avg_baseline / t_avg_optimized:.2f}x")
```

--------------------------------

### Authenticate the CLI

Source: https://docs.weco.ai/reference/cli/auth

Initiates the device authorization flow to authenticate the CLI with a Weco account.

```bash
weco login
```

--------------------------------

### Create Evaluation Script

Source: https://docs.weco.ai/examples/cuda-optimization

Script to evaluate the performance of the model using Triton benchmarking tools.

```python
import sys
import os
import shutil
import pathlib
import importlib
import importlib.util
import traceback
import torch
import torch.nn as nn
import torch.nn.functional as F
import math
from triton.testing import do_bench


########################################################
# Baseline
########################################################
class Model(nn.Module):
    """
    A vanilla multi-head masked self-attention layer with a projection at the end.
    """

    def __init__(self, n_embd, n_head, attn_pdrop, resid_pdrop, max_seqlen):
        super().__init__()
        assert n_embd % n_head == 0
        # key, query, value projections for all heads, but in a batch
        self.c_attn = nn.Linear(n_embd, 3 * n_embd)
        # output projection
        self.c_proj = nn.Linear(n_embd, n_embd)
        # regularization
        self.attn_dropout = nn.Dropout(attn_pdrop)
        self.resid_dropout = nn.Dropout(resid_pdrop)
        # causal mask to ensure that attention is only applied to the left in the input sequence
        self.register_buffer("bias", torch.tril(torch.ones(max_seqlen, max_seqlen)).view(1, 1, max_seqlen, max_seqlen))
        self.n_head = n_head
        self.n_embd = n_embd
```

--------------------------------

### Run Weco Optimization

Source: https://docs.weco.ai/using-weco/eval-scripts

Execute the Weco agent with a specified source file, evaluation command, and optimization goal.

```bash
weco run --source file_to_optimize.py --eval-command "python evaluate.py" --metric accuracy --goal maximize
```

--------------------------------

### Log Experiment Results

Source: https://docs.weco.ai/using-weco/track-any-loop

Record metrics and status for specific steps within an existing run.

```bash
# Log baseline result (step 0)
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 0 \
  --status completed \
  --description "baseline" \
  --metrics '{"val_bpb": 2.366, "memory_gb": 0.0}' \
  --source train.py

# Log an experiment
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 1 \
  --status completed \
  --description "increase batch size to 32K" \
  --metrics '{"val_bpb": 2.261, "memory_gb": 0.0}' \
  --source train.py

# Log a failed experiment
weco observe log \
  --run-id "$WECO_RUN_ID" \
  --step 2 \
  --status failed \
  --description "double model depth (OOM)" \
  --metrics '{"val_bpb": 0.0, "memory_gb": 0.0}' \
  --source train.py
```

--------------------------------

### Define Baseline Model

Source: https://docs.weco.ai/examples/pytorch-optimization

Create a PyTorch module that performs matrix multiplication, division, summation, and scaling.

```python
import torch
import torch.nn as nn


class Model(nn.Module):
    """
    Model that performs a matrix multiplication, division, summation, and scaling.
    """

    def __init__(self, input_size, hidden_size, scaling_factor):
        super(Model, self).__init__()
        self.weight = nn.Parameter(torch.randn(hidden_size, input_size))
        self.scaling_factor = scaling_factor

    def forward(self, x):
        """
        Args:
            x (torch.Tensor): Input tensor of shape (batch_size, input_size).
        Returns:
            torch.Tensor: Output tensor of shape (batch_size, hidden_size).
        """
        x = torch.matmul(x, self.weight.T)
        x = x / 2
        x = torch.sum(x, dim=1, keepdim=True)
        x = x * self.scaling_factor
        return x
```

--------------------------------

### Create a feature branch

Source: https://docs.weco.ai/resources/contributing

Creates and switches to a new branch for implementing specific features.

```bash
git checkout -b feature/your-feature-name
```

--------------------------------

### Run Baseline Evaluation

Source: https://docs.weco.ai/examples/extract-line-plot

Execute the evaluation script to measure extraction performance.

```bash
uv run --with openai python eval.py --max-samples 10 --num-workers 4
```

--------------------------------

### Configure Environment Variables

Source: https://docs.weco.ai/examples/extract-line-plot

Set the required OpenAI API key for VLM operations.

```bash
export OPENAI_API_KEY=your_key_here
```
Tokens:
Raw