### Install SWE-smith Development Version from Source

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/installation.md

Clones the SWE-smith repository from GitHub and executes a setup script to install the latest development version. This is suitable for users who want the newest features or to test unreleased changes.

```bash
git clone https://github.com/SWE-bench/SWE-smith
cd SWE-smith
./setup.sh
```

--------------------------------

### Run Repository Installation with Default Python Version

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

Demonstrates the basic usage of `try_install_py` to install a repository with the default Python version (3.10). It takes the target repository, an installation script, and a commit hash as input. Successful execution generates environment and installation log artifacts.

```bash
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84
```

--------------------------------

### Install Pre-commit Hooks for SWE-smith Contribution

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/installation.md

Installs the pre-commit framework, which helps manage and run pre-commit hooks. This is a necessary step for developers who plan to contribute code to the SWE-smith project, ensuring code quality and consistency.

```bash
pre-commit install
```

--------------------------------

### Install Latest Stable SWE-smith Release via Pip

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/installation.md

Installs the most recent stable version of SWE-smith from the Python Package Index (PyPI). This is the recommended method for general users.

```bash
pip install swesmith
```

--------------------------------

### Create Docker Images for Repositories with SWE-smith

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction.md

This command initiates the creation of Docker images for specified repositories. It mirrors the repository and builds a Docker image containing the installed codebase. Options include specifying the repository name, organization, and forcing a rebuild.

```bash
python -m swesmith.build_repo.create_images -r MonkeyType
```

--------------------------------

### Run Model Training (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command executes the script to start the model training process using Torchtune. Before running, ensure the configuration file `config/train/full_ft_qwen_7b.yml` is updated to point to the SFT data file uploaded to Modal.

```bash
./scripts/train.run_ft_torchtune.py
```

--------------------------------

### Add Extra Test Dependencies During Installation

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

Illustrates how to specify additional test dependencies using the `--extra-test-deps` flag. These packages are installed after the main test dependencies, allowing for flexibility when a repository requires extra testing utilities.

```bash
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --extra-test-deps "hypothesis coverage"
```

```bash
python -m pip install hypothesis coverage
```

--------------------------------

### Build Docker Images for Repositories with SWE-smith

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Create isolated execution environments by building Docker images for specified GitHub repositories. This process installs all necessary dependencies. Options include specifying repositories by name, using fuzzy matching, forcing a rebuild, listing available environments, and pushing images to Docker Hub. Pre-built images for the SWE-smith dataset can also be downloaded.

```bash
# Create a Docker image for a specific repository
python -m swesmith.build_repo.create_images -r MonkeyType

# Build images for all repositories matching a pattern (fuzzy match)
python -m swesmith.build_repo.create_images -r django --workers 4

# Force rebuild even if image already exists
python -m swesmith.build_repo.create_images -r pandas -f

# List all available execution environment profiles
python -m swesmith.build_repo.create_images --list-envs

# Build and push to Docker Hub
python -m swesmith.build_repo.create_images -r myrepo --push

# Download pre-built images for the SWE-smith dataset
python -m swesmith.build_repo.download_images
```

--------------------------------

### Create Task Instances with SWE-smith Bash Scripts

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/quickstart.md

This snippet demonstrates the bash commands to create task instances using SWE-smith. It includes steps for generating bugs with an LLM rewrite strategy, collecting all generated patches, validating these patches, gathering valid instances, and finally generating issues from the validated data. Dependencies include Python and the SWE-smith library.

```bash
# Run LM rewrite strategy to produce bugs
python -m swesmith.bug_gen.llm.modify pandas-dev__pandas.95280573 \
    --config_file configs/bug_gen/lm_modify.yml \
    --model claude-3-7-sonnet-20250219 \
    --n_bugs 1 \
    --n_workers=20

# Collect all task instances into a single file for validation
python -m swesmith.bug_gen.collect_patches logs/bug_gen/pandas-dev__pandas.95280573/

# Run validation on the collected task instances
python -m swesmith.harness.valid logs/bug_gen/pandas-dev__pandas.95280573_all_patches.json --workers 8

# Gather valid task instances
python -m swesmith.harness.gather logs/run_validation/pandas_test

# Generate issues for the valid task instances
python -m swesmith.issue_gen.generate \
    --dataset_path logs/run_validation/basic/pandas_test.json \
  --model claude-3-7-sonnet-20250219 \
  --n_workers=1 \
  --config_file configs/issue_gen/ig_v2.yaml \
  --experiment_id ig_v2
```

--------------------------------

### Profile Registry API Usage (Python)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Python code demonstrating how to use the SWE-smith profile registry. This includes loading datasets, retrieving repository profiles, getting containers, executing commands within containers, listing available profiles, cloning repositories, extracting code entities, and building/pushing Docker images.

```python
from swesmith.profiles import registry
from datasets import load_dataset

# Load the SWE-smith dataset from HuggingFace
ds = load_dataset("SWE-bench/SWE-smith", split="train")

# Get a profile by repository name
profile = registry.get("pandas-dev__pandas.95280573")

# Get profile from a task instance
for task in ds:
    rp = registry.get_from_inst(task)

    # Get a Docker container with the task initialized
    container = rp.get_container(task)

    # Execute commands in the container
    result = container.exec_run("pytest tests/", workdir="/testbed")
    print(result.output.decode())

    # Cleanup
    container.stop()
    container.remove()

# List all available profiles
for profile in registry.values():
    print(f"Repository: {profile.repo_name}")
    print(f"Image: {profile.image_name}")
    print(f"Test command: {profile.test_cmd}")

# Clone a repository locally
profile = registry.get("django__django.abc12345")
dest_path, was_cloned = profile.clone()

# Extract code entities (functions, classes) from repository
entities = profile.extract_entities(
    exclude_tests=True,
    max_entities=100
)
for entity in entities:
    print(f"{entity.entity_type}: {entity.name} in {entity.file_path}")

# Build Docker image programmatically
profile.create_mirror()  # Create GitHub mirror repository
profile.build_image()    # Build Docker image
profile.push_image()     # Push to Docker Hub
```

--------------------------------

### Example Valid Task Instance JSON Structure

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/harnesses.md

Illustrates the structure of a validated task instance JSON object. This includes instance details, the bug patch, test case outcomes, and the Docker image name.

```json
{
    "instance_id": <instance_id>,
    "repo": <repo>,
    "patch": <The diff that, when applied, creates the bug>,
    "FAIL_TO_PASS": <List of broken test cases>,
    "PASS_TO_PASS": <List of passing test cases>,
    "image_name": <docker image name>
}
```

--------------------------------

### Configure and Run Smoke Tests for Environment Verification

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

Details how to manage smoke testing after environment installation. It covers the default pytest command, specifying a custom smoke test command with `--smoke-cmd`, and skipping the smoke test entirely using `--skip-smoke`.

```bash
# Default smoke test
pytest -q --maxfail=1
```

```bash
# Custom smoke test command
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --smoke-cmd "python -m pytest tests/test_cli.py -v"
```

```bash
# Skip smoke test
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --skip-smoke
```

--------------------------------

### JavaScript: Google Analytics Initialization

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/blog.html

This JavaScript code initializes Google Analytics tracking for a webpage. It ensures that the 'dataLayer' array is available and then configures the analytics tracker with a specific measurement ID. This is a standard setup for integrating Google Analytics into a website.

```javascript
window.dataLayer = window.dataLayer || [];
function gtag(){
  dataLayer.push(arguments);
} gtag('js', new Date());

gtag('config', 'G-46KMJE3755');
```

--------------------------------

### Install and Test MonkeyType with SWE-Smith (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

This command uses SWE-Smith to clone a specific commit of the MonkeyType repository, create a Python 3.11 conda environment, install the package in editable mode, add extra test dependencies like 'hypothesis', and run a smoke test using pytest. It also forces overwriting and cleans up afterwards.

```bash
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --python-version 3.11 \
    --extra-test-deps "hypothesis" \
    --smoke-cmd "pytest tests/test_cli.py -q" \
    --force
```

--------------------------------

### Specify Python Version for Conda Environment Creation

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

Shows how to use the `--python-version` flag with `try_install_py` to create a conda environment with a specific Python version (e.g., 3.11). This flag influences the `PYTHON_VERSION` environment variable used by the installation script.

```bash
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --python-version 3.11
```

```bash
conda create -n testbed "python=${PYTHON_VERSION}" -yq
```

--------------------------------

### Example Prediction JSON Structure for Evaluation

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/harnesses.md

Defines the expected JSON format for predictions when running the evaluation harness with real model outputs. It includes the instance ID, the proposed fix patch, and the model used.

```json
{
    "instance_id": <instance_id>,
    "patch": <The diff that, when applied, fixes the bug>,
    "model_name_or_path": <The model used to generate the patch>
}
```

--------------------------------

### Get Task Difficulties with SWE-smith Script

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/difficulty_rating.md

This Python script, `get_difficulties.py`, is used to rate the difficulty of task instances from a dataset. It requires the base URL where the model is hosted and the path to the dataset in JSON format. The output is a JSON file mapping task instances to difficulty scores.

```python
python swesmith/train/difficulty_rater/get_difficulties.py \
    --base_url <URL where model is hosted> \
    --dataset_path path/to/dataset.json
```

--------------------------------

### Generate Bugs using LLM Strategies with SWE-smith

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Generate bug-inducing patches by leveraging language models to modify or rewrite code entities. Supports 'LM Modify' to introduce bugs into existing code and 'LM Rewrite' to generate code from scratch. Users can specify the model, configuration files, number of bugs, and workers. Examples include using Anthropic and OpenAI models.

```bash
# Generate bugs using LM Modify strategy (introduce bugs into existing code)
python -m swesmith.bug_gen.llm.modify pandas-dev__pandas.95280573 \
    --config_file configs/bug_gen/lm_modify.yml \
    --model claude-3-7-sonnet-20250219 \
    --n_bugs 1 \
    --n_workers 20

# Generate bugs using LM Rewrite strategy (rewrite functions from scratch)
python -m swesmith.bug_gen.llm.rewrite pandas-dev__pandas.95280573 \
    --model anthropic/claude-3-7-sonnet-20250219 \
    --config_file configs/bug_gen/lm_rewrite.yml \
    --n_workers 4

# Generate bugs using OpenAI models
python -m swesmith.bug_gen.llm.modify tkrajina__gpxpy.09fc46b3 \
    --config_file configs/bug_gen/class_basic.yml \
    --model openai/gpt-4o \
    --n_bugs 3 \
    --max_bugs 100
```

--------------------------------

### Load and Initialize SWE-smith Task Environment (Python)

Source: https://github.com/swe-bench/swe-smith/blob/main/README.md

This Python snippet demonstrates how to load task instances from the SWE-bench/SWE-smith dataset and initialize a Docker container for each task using the swesmith library. It requires the 'swesmith' and 'datasets' libraries.

```python
from swesmith.profiles import registry
from datasets import load_dataset
ds = load_dataset("SWE-bench/SWE-smith", split="train") # Loads all 52k task instances
for task in ds:
    rp = registry.get_from_inst(task)  # Get the RepoProfile for the task
    container = rp.get_container(task) # Returns pointer to a Docker container with the task initialized

    """TODO: Train!"""

```

--------------------------------

### Training SWE-agents: Data Preparation and Fine-tuning (Bash)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Bash commands for preparing data and training SWE-agents. This involves evaluating training instances, collecting resolved trajectories into SFT format, uploading data to a Modal volume, and running fine-tuning scripts using torchtune or unsloth, followed by serving the model.

```bash
# Step 1: Run evaluation on training instances to identify resolved trajectories
python -m swesmith.harness.eval \
    --dataset_path logs/experiments/subset0.json \
    --predictions_path trajectories/user/run_001/preds.json \
    --run_id training_eval \
    --workers 10 \
    --timeout 240

# Step 2: Convert resolved trajectories to SFT format
python -m swesmith.train.traj_mgr.collect_trajs \
    --traj_dir trajectories/user/run_001/ \
    --eval_dir logs/run_evaluation/training_eval/ \
    --style xml \
    --out_dir trajectories_sft/

# Step 3: Upload training data to Modal volume
modal volume put training-data trajectories_sft/ft_xml_training_eval.jsonl

# Step 4: Run fine-tuning with torchtune
./scripts/train.run_ft_torchtune.sh

# Alternative: Run fine-tuning with unsloth
./scripts/train.run_ft_unsloth.sh

# Serve the fine-tuned model for inference
./scripts/train.serve_sglang.sh
```

--------------------------------

### Generate and Validate Bugs at Scale with Modal (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/create_instances.md

This command uses the bug_gen.py script with Modal for scalable cloud execution to generate and validate bugs across multiple repositories. The example shown targets all JavaScript repositories.

```bash
modal run scripts/bug_gen.py --language javascript
```

--------------------------------

### Download Existing SWE-smith Docker Images

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction.md

This command allows you to download pre-built Docker images for repositories included in the SWE-smith dataset. This is useful for quickly accessing environments without building them from scratch.

```bash
python -m swesmith.build_repo.download_images
```

--------------------------------

### Run Docker Image and Test Codebase

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction.md

This command executes a previously built Docker image in an interactive, temporary mode. Once inside the container, you can run the repository's testing suite (e.g., pytest) to verify the codebase's functionality.

```bash
docker run -it --rm swebench/swesmith.x86_64.instagram__monkeytype.70c3acf6
```

--------------------------------

### Host Model with sglang and Run Inference with SWE-agent using Modal

Source: https://github.com/swe-bench/swe-smith/blob/main/swesmith/train/README.md

Hosts a fine-tuned model using sglang for serving and prepares it for inference with SWE-agent. This command also utilizes Modal for execution and specifies model paths and serving names.

```bash
N_HOURS=4 N_GPUS=4 modal run --detach serve_sglang.py --model-path /weights/my-oss-model --served-model-name gpt-4o --tokenizer-path /weights/Qwen/Qwen2.5-Coder-32B-Instruct
```

--------------------------------

### Creating Dataset Subsets (Python)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Python code snippet demonstrating how to load the full SWE-smith dataset from HuggingFace and prepare for creating custom subsets for targeted training.

```python
import json
from datasets import load_dataset

# Load the full SWE-smith dataset
swesmith = load_dataset("SWE-bench/SWE-smith", split="train")

```

--------------------------------

### Serve SGLang Checkpoint with Modal

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/difficulty_rating.md

This command uses Modal to serve a HuggingFace checkpoint with SGLang. It requires specifying the model path, a served model name, and the tokenizer path. The N_HOURS and N_GPUS environment variables control the compute resources allocated.

```bash
N_HOURS=4 N_GPUS=4 modal run --detach swesmith/train/serve_sglang.py \
    --model-path /path/to/checkpoint \
    --served-model-name gpt-4o \
    --tokenizer-path /path/to/Qwen2.5-Coder-32B-Instruct
```

--------------------------------

### Run Fine-Tuning with SWE-smith Dataset using Modal

Source: https://github.com/swe-bench/swe-smith/blob/main/swesmith/train/README.md

Executes the fine-tuning process for a model using a SWE-smith dataset and a specified configuration file. This script leverages Modal for distributed GPU training.

```bash
NGPUS=8 modal run train/run_ft_torchtune.py --config train/config/torchtune.yml
```

--------------------------------

### Run Inference with SFT'ed Model (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command initiates the inference process using the SWE-agent framework and the newly SFT'ed model. Ensure the `scripts/train.serve_sglang.sh` script is updated to point to your SFT'ed model. The script also requires correct Modal URL configuration and allows for specifying the evaluation dataset.

```bash
./agent/_infer_model.sh
```

--------------------------------

### SWE-smith End-to-End Workflow (Bash)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

This bash script outlines the complete SWE-smith pipeline for generating software engineering training data. It covers environment creation, bug generation using LLMs and procedural methods, patch collection and validation, instance gathering, issue text generation, and gold evaluation.

```bash
# 1. Create execution environment
python -m swesmith.build_repo.create_images -r myorg/myrepo -y

# 2. Generate bugs using multiple strategies
python -m swesmith.bug_gen.llm.modify myorg__myrepo.abc12345 \
    --config_file configs/bug_gen/lm_modify.yml \
    --model claude-3-7-sonnet-20250219 \
    --n_bugs 2 \
    --n_workers 10

python -m swesmith.bug_gen.procedural.generate myorg__myrepo.abc12345 \
    --max_bugs 50

# 3. Collect and validate patches
python -m swesmith.bug_gen.collect_patches logs/bug_gen/myorg__myrepo.abc12345

python -m swesmith.harness.valid \
    logs/bug_gen/myorg__myrepo.abc12345_all_patches.json \
    --workers 8

# 4. Gather valid instances
python -m swesmith.harness.gather logs/run_validation/myorg__myrepo.abc12345

# 5. Generate issue text
python -m swesmith.issue_gen.generate \
    --dataset_path logs/task_insts/myorg__myrepo.abc12345.json \
    --config_file configs/issue_gen/ig_v2.yaml \
    --workers 4

# 6. Verify with gold evaluation
python -m swesmith.harness.eval \
    --dataset_path logs/task_insts/myorg__myrepo.abc12345.json \
    --predictions_path gold \
    --run_id sanity
```

--------------------------------

### Upload SFT Data to Modal (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command uploads the generated SFT-formatted trajectory files to a Modal volume. Replace `<volume>` with the actual name of your Modal volume. This step is necessary before configuring the training script.

```bash
modal volume put <volume> trajectories_sft/ft_xml_*.jsonl
```

--------------------------------

### Download Model Checkpoint with Modal

Source: https://github.com/swe-bench/swe-smith/blob/main/swesmith/train/README.md

Downloads a model checkpoint from HuggingFace to a specified target directory using a Modal script. This is the first step in the fine-tuning process.

```bash
modal run download_checkpoint.py --source-repo Qwen/Qwen2.5-7B-Instruct --target-dir /weights/Qwen/Qwen2.5-7B-Instruct
```

--------------------------------

### Control Environment Overwriting and Cleanup for Debugging

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/env_construction_py.md

Explains debugging options for `try_install_py`. The `--force` flag bypasses the overwrite prompt for existing environment files. The `--no_cleanup` flag prevents the script from removing the cloned repository and conda environment after completion, facilitating manual inspection.

```bash
# Force overwrite existing environment file
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --force
```

```bash
# Keep environment and repository for inspection
python -m swesmith.build_repo.try_install_py Instagram/MonkeyType configs/install_repo.sh \
    --commit 70c3acf62950be5dfb28743c7a719bfdecebcd84 \
    --no_cleanup
```

```bash
# Manual inspection after --no_cleanup
conda activate testbed
cd MonkeyType
pytest tests/
```

--------------------------------

### Create SWE-smith Task Instance Subset

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This Python script loads the SWE-smith dataset, filters task instances based on specific criteria (instance ID format, number of fail-to-pass steps), and saves the subset to a JSON file. It requires the 'datasets' library.

```python
import json

from datasets import load_dataset
swesmith = load_dataset("SWE-bench/SWE-smith", split="train")

subset_name = "subset0"
def criteria(task_instance):
    return ".pr_" in task_instance["instance_id"] and \
        len(task_instance["FAIL_TO_PASS"]) <= 5 and \
        len(task_instance["FAIL_TO_PASS"]) >= 2
bugs = [x for x in swesmith if criteria(x)]
print(f"Found {len(bugs)} bugs that match criteria")
with open(f"logs/experiments/{subset_name}.json", "w") as f:
    json.dump(bugs, fp=f, indent=2)
```

--------------------------------

### Download SWE-smith Repository Environments (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/assets.md

This command downloads Docker images for 128 GitHub repositories, which are environments created by SWE-smith. It is executed from the root directory of the SWE-smith project.

```bash
python swesmith/build_repo/download_images.py
```

--------------------------------

### Project and Content Information

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This endpoint provides details about a specific project and its content, including file paths and version identifiers.

```APIDOC
## GET /swe-bench/swe-smith

### Description
Retrieves information about the 'swe-smith' project and its associated content.

### Method
GET

### Endpoint
/swe-bench/swe-smith

### Parameters
#### Query Parameters
- **content** (string) - Optional - A filter to specify the content to retrieve. If not provided, all content for the project is returned.

### Request Example
```bash
GET /swe-bench/swe-smith?content=conan-io__conan.86f29e13.pr_13718.v9ywhphk
```

### Response
#### Success Response (200)
- **Project**: (string) - The name of the project.
- **Content**: (array of strings) - A list of content identifiers within the project.

#### Response Example
```json
{
  "Project": "/swe-bench/swe-smith",
  "Content": [
    "conan-io__conan.86f29e13.pr_13718.v9ywhphk",
    "conan-io__conan.86f29e13.pr_13721.57qwos6s",
    "conan-io__conan.86f29e13.pr_13721.h7usduo2"
  ]
}
```
```

--------------------------------

### Evaluate Model Predictions (Bash)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Commands to run the evaluation harness for SWE-agents. This includes sanity checks with gold patches, evaluating actual model predictions, evaluating specific instances using glob patterns, and optimizing evaluation by focusing on fail-to-pass tests.

```bash
# Run evaluation with gold (correct) patches to sanity check the harness
python -m swesmith.harness.eval \
    --dataset_path logs/task_insts/myrepo.json \
    --predictions_path gold \
    --run_id sanity_check \
    --workers 10

# Evaluate actual model predictions
python -m swesmith.harness.eval \
    --dataset_path logs/task_insts/myrepo.json \
    --predictions_path trajectories/user/run_001/preds.json \
    --run_id model_eval \
    --workers 8 \
    --timeout 240

# Evaluate specific instances using glob patterns
python -m swesmith.harness.eval \
    --dataset_path logs/task_insts/myrepo.json \
    --predictions_path preds.json \
    --run_id partial_eval \
    --instance_ids "pandas-dev__pandas.*"

# Speed up evaluation by only running fail-to-pass tests
python -m swesmith.harness.eval \
    --dataset_path logs/task_insts/myrepo.json \
    --predictions_path preds.json \
    --run_id fast_eval \
    --f2p_only
```

--------------------------------

### Generate Issue Text (Bash)

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Commands for generating GitHub-style issue text using language models or static templates. This includes generating text with Claude, using static templates, extracting text from failed tests, and using original PR issue text for PR mirrors.

```bash
# Generate issue text using Claude
python -m swesmith.issue_gen.generate \
    --dataset_path logs/task_insts/myrepo.json \
    --config_file configs/issue_gen/ig_v2.yaml \
    --workers 4

# Generate static (template-based) issue text
python -m swesmith.issue_gen.get_static logs/task_insts/myrepo.json

# Extract issue text from failed tests
python -m swesmith.issue_gen.get_from_tests logs/task_insts/myrepo.json

# Use original PR issue text (for PR mirrors)
python -m swesmith.issue_gen.get_from_pr logs/task_insts/myrepo.json
```

--------------------------------

### Run SWE-agent Inference Batch with Local Model

Source: https://github.com/swe-bench/swe-smith/blob/main/swesmith/train/README.md

Initiates batch inference using SWE-agent against a locally hosted model. This script requires replacing placeholders for the API base URL and specifies the agent model, instance type, dataset, and split for inference.

```bash
#!/bin/bash

sweagent run-batch \
    --agent.model.api_base <REPLACE WITH MODAL LINK>/v1 \
    --agent.model.api_key swesmith \
    --agent.model.name gpt-4o \
    --instances.type swe_bench \
    --instances.dataset_name jyang20/swebv-mini \
    --instances.split test \
    --config config/anthropic_no_fcalls.yaml
```

--------------------------------

### Submit Evaluation Predictions (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command submits the model's predictions for evaluation against SWE-bench datasets (e.g., `swe-bench_verified`). It requires the path to the predictions JSON file and a run ID. Refer to the `sb-cli` documentation for more details on evaluation submission.

```bash
sb-cli submit swe-bench_verified test \
    --predictions_path trajectories/<username>/<run ID>/preds.json \
    --run_id <run ID>
```

--------------------------------

### Evaluate Training Task Instances (Python)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This Python command evaluates training task instances against generated predictions. It takes the path to the task instance subset and the predictions file as input, along with a run ID and worker count. The output is stored in `logs/run_evaluation/<run ID>/report.json`, indicating successfully resolved instances. It requires the `swesmith` library.

```python
python -m swesmith.harness.eval \
    --dataset_path path/to/subset0.json \
    --predictions_path path/to/trajectories/<username>/<run ID>/preds.json \
    --run_id <run ID> \
    --workers 10 \
    --timeout 240
```

--------------------------------

### BibTeX Citation for SWE-smith

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/getting_started/index.md

This BibTeX entry provides the necessary information to cite the SWE-smith project in academic publications. It includes authors, title, year, arXiv preprint details, and a URL for access.

```bibtex
@misc{yang2025swesmith,
  title={SWE-smith: Scaling Data for Software Engineering Agents},
  author={John Yang and Kilian Lieret and Carlos E. Jimenez and Alexander Wettig and Kabir Khandpur and Yanzhe Zhang and Binyuan Hui and Ofir Press and Ludwig Schmidt and Diyi Yang},
  year={2025},
  eprint={2504.21798},
  archivePrefix={arXiv},
  primaryClass={cs.SE},
  url={https://arxiv.org/abs/2504.21798},
}
```

--------------------------------

### Command-Line Execution for Bug Generation

Source: https://github.com/swe-bench/swe-smith/blob/main/configs/bug_gen/README.md

Shows the command to execute the SWESmith bug generation process. It specifies the target repository, the LLM model, the configuration file, and the number of parallel workers.

```bash
python -m swesmith.bug_gen.llm.modify \
    --repo datamade/usaddress \
    --model openai/gpt-4o \
    --prompt_config configs/bug_gen/func_<your config>.yml \
    --n_workers 4  # 4 parallel queries to LM etc.
```

--------------------------------

### Combine File Functionality in Python (lepture__mistune)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This snippet demonstrates functionality for combining files in Python, as part of the 'lepture__mistune' module. This could be used for merging or processing multiple files.

```python
def combine_file__gvfyaio1(self):
    pass

def combine_file__kci03ucu(self):
    pass

def combine_file__rjfho81s(self):
    pass

def combine_file__uu79o6x7(self):
    pass
```

--------------------------------

### Run Bug Generation Pipeline (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/create_instances.md

Executes the SWE-smith bug generation script for a specified language and repositories. This command initiates both the generation and validation phases of the bug creation process.

```bash
modal run scripts/bug_gen.py --language javascript --repos "owner/repo1,owner/repo2"
```

--------------------------------

### Convert Trajectories to SFT Format (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command converts generated trajectories into a format suitable for Supervised Fine-Tuning (SFT). It requires the directory containing trajectories and the evaluation directory. The output is an `.jsonl` file in the `trajectories_sft/` folder, ready for SFT. It requires the `swesmith` library.

```bash
python -m swesmith.train.traj_mgr.collect_trajs \
    --traj_dir path/to/trajectories/<username>/<run ID>/ \
    --eval_dir logs/run_evaluation/<run ID>/
```

--------------------------------

### Verify Repository Evaluation Sets in Python

Source: https://github.com/swe-bench/swe-smith/blob/main/CONTRIBUTING.md

This Python script verifies that newly added repositories in SWE-smith are correctly associated with their respective evaluation sets. It loads datasets from Hugging Face and compares repository information with the registered profiles to identify any discrepancies in the `eval_sets` property.

```python
from swesmith.profiles import registry
from datasets import load_dataset

sb = set(load_dataset("SWE-bench/SWE-bench_Verified", split="test")["repo"])
sbmm = set(load_dataset("SWE-bench/SWE-bench_Multimodal", split="test")["repo"])
sbml = set(load_dataset("SWE-bench/SWE-bench_Multilingual", split="test")["repo"])

profiles = {
    f"{rp.owner}/{rp.repo}": rp.eval_sets
    for rp in registry.values()
}

for test_repos, test_set in [
    (sb, "SWE-bench/SWE-bench_Verified"),
    (sbmm, "SWE-bench/SWE-bench_Multimodal"),
    (sbml, "SWE-bench/SWE-bench_Multilingual"),
]:
    for repo in test_repos:
        if repo not in profiles:
            continue
        if test_set not in profiles[repo]:
            print(f"Add {test_set} to {repo}'s `eval_sets` property")
```

--------------------------------

### Combine Single-Entity Bugs into Multi-File Bugs with SWE-smith

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Enhance bug complexity by combining validated single-entity bugs into multi-file or multi-module scenarios. This functionality allows for creating more realistic and challenging bug instances by merging a specified number of bugs from the same file or the same module (directory). Parameters control the number of patches to combine, limits per file/module, and the maximum number of combinations generated.

```bash
# Combine 3 bugs from the same file
python -m swesmith.bug_gen.combine.same_file logs/bug_gen/pandas-dev__pandas.95280573 \
    --num_patches 3 \
    --limit_per_file 15 \
    --max_combos 100

# Combine 2 bugs from the same module (directory)
python -m swesmith.bug_gen.combine.same_module logs/bug_gen/pandas-dev__pandas.95280573 \
    --num_patches 2 \
    --limit_per_module 20 \
    --max_combos 200 \
    --depth 2
```

--------------------------------

### Run Evaluation Harness (Sanity Check)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/harnesses.md

Executes the evaluation harness for a sanity check on validated task instances. This command verifies that the testing process for validated instances works as expected using a 'gold' dataset.

```bash
python -m swesmith.harness.eval \
    --dataset_path bugs/task_insts/{repo}.json \
    --predictions_path gold \
    --run_id sanity
```

--------------------------------

### Collect Patches for Validation

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/harnesses.md

Collects all candidate task instances generated by the bug generation process. This command outputs a JSON file containing all candidate task instances for a given repository.

```bash
python -m swesmith.bug_gen.collect_patches logs/bug_gen/<repo>
```

--------------------------------

### Generate Task Instances using SWE-Smith

Source: https://github.com/swe-bench/swe-smith/blob/main/CONTRIBUTING.md

This script generates task instances for the SWE-Smith project. It populates a log folder with generated bugs, which are then filtered and validated. The valid bugs are collected into a JSON file and pushed as branches to the SWE-smith organization repositories and the SWE-bench/SWE-smith Hugging Face dataset.

```python
python -m swesmith.bug_gen.*
python swesmith/harness/valid.py logs/bug_gen/<repo>_all_patches.json
python swesmith/harness/eval.py logs/run_validation/<repo>
python swesmith/harness/gather.py logs/run_validation/<repo>
```

--------------------------------

### Validate Task Instances with SWE-smith Harness

Source: https://context7.com/swe-bench/swe-smith/llms.txt

Validate candidate task instances by executing them against existing tests to determine if they break functionality. The process involves collecting all candidate patches, running the validation harness with specified workers, and optionally redoing validation for existing instances. Finally, valid task instances are gathered, and branches are created for them.

```bash
# Step 1: Collect all candidate patches into a single file
python -m swesmith.bug_gen.collect_patches logs/bug_gen/pandas-dev__pandas.95280573

# Step 2: Run validation on collected patches
python -m swesmith.harness.valid logs/bug_gen/pandas-dev__pandas.95280573_all_patches.json \
    --workers 8

# Redo validation for existing instances
python -m swesmith.harness.valid logs/bug_gen/myrepo_all_patches.json \
    --workers 4 \
    --redo_existing

# Step 3: Gather valid task instances and create branches
python -m swesmith.harness.gather logs/run_validation/pandas-dev__pandas.95280573
```

--------------------------------

### Shuffle Control Flow in Python (lepture__mistune)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This snippet demonstrates shuffling control flow in Python, part of the 'lepture__mistune' module. It can be used for randomization or testing purposes.

```python
def func_pm_ctrl_shuffle__dm21txa2(self):
    pass

def func_pm_ctrl_shuffle__xlb80k6r(self):
    pass

def func_pm_ctrl_shuffle__xvy2mekg(self):
    pass
```

--------------------------------

### Various Operations in pylint-dev/astroid

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This collection of snippets from pylint-dev/astroid represents various operations, including pull request related changes and other internal modifications. These are likely related to abstract syntax tree manipulation for code analysis.

```python
pylint-dev__astroid.b114f6b5.pr_1971.s6z8jazv
pylint-dev__astroid.b114f6b5.pr_1976.9jirhyg4
pylint-dev__astroid.b114f6b5.pr_1977.trb5dnqc
pylint-dev__astroid.b114f6b5.pr_1978.ls5mu3ib
pylint-dev__astroid.b114f6b5.pr_1984.948l3jbz
pylint-dev__astroid.b114f6b5.pr_1986.tdzob2gk
pylint-dev__astroid.b114f6b5.pr_1996.ocz7u07i
pylint-dev__astroid.b114f6b5.pr_2000.j3o55sjz
pylint-dev__astroid.b114f6b5.pr_2001.8hrqczqb
pylint-dev__astroid.b114f6b5.pr_2001.v3ul415i
pylint-dev__astroid.b114f6b5.pr_2002.9356zr3k
pylint-dev__astroid.b114f6b5.pr_2015.tkangqun
pylint-dev__astroid.b114f6b5.pr_2015.x4ukq323
pylint-dev__astroid.b114f6b5.pr_2015.xatmlk75
pylint-dev__astroid.b114f6b5.pr_2043.5lhnomnz
pylint-dev__astroid.b114f6b5.pr_2074.vnxcve0w
pylint-dev__astroid.b114f6b5.pr_2126.sjobw5s4
pylint-dev__astroid.b114f6b5.pr_2139.834j0wgz
pylint-dev__astroid.b114f6b5.pr_2166.g8a2t1jt
pylint-dev__astroid.b114f6b5.pr_2166.zdflwyv3
pylint-dev__astroid.b114f6b5.pr_2181.evbalhlo
pylint-dev__astroid.b114f6b5.pr_2181.oomszeir
pylint-dev__astroid.b114f6b5.pr_2204.4153xydt
pylint-dev__astroid.b114f6b5.pr_2204.em5v697i
pylint-dev__astroid.b114f6b5.pr_2204.muag91lo
pylint-dev__astroid.b114f6b5.pr_2206.mdvpq4ue
pylint-dev__astroid.b114f6b5.pr_2207.8hj5lvfx
pylint-dev__astroid.b114f6b5.pr_2207.o1oyv0cu
pylint-dev__astroid.b114f6b5.pr_2229.9uu2ny21
pylint-dev__astroid.b114f6b5.pr_2254.d7nh5p8f
pylint-dev__astroid.b114f6b5.pr_2257.srb063y2
pylint-dev__astroid.b114f6b5.pr_2259.guif9gxl
pylint-dev__astroid.b114f6b5.pr_2260.ha39vwaj
pylint-dev__astroid.b114f6b5.pr_2263.zfk34ct2
pylint-dev__astroid.b114f6b5.pr_2268.8l673lp8
pylint-dev__astroid.b114f6b5.pr_2277.cleyivb4
pylint-dev__astroid.b114f6b5.pr_2281.5ujez9aw
pylint-dev__astroid.b114f6b5.pr_2289.dhj30ngl
pylint-dev__astroid.b114f6b5.pr_2298.l38t1vba
pylint-dev__astroid.b114f6b5.pr_2307.9413g4d4
pylint-dev__astroid.b114f6b5.pr_2307.af0d1sev
pylint-dev__astroid.b114f6b5.pr_2328.hurkjmzm
pylint-dev__astroid.b114f6b5.pr_2328.ii1hdwom
pylint-dev__astroid.b114f6b5.pr_2328.wx7nctk6
pylint-dev__astroid.b114f6b5.pr_2329.60zlx6gl
pylint-dev__astroid.b114f6b5.pr_2380.zlmwlj9b
pylint-dev__astroid.b114f6b5.pr_2386.adx1nhoh
pylint-dev__astroid.b114f6b5.pr_2399.ubgb2vrl
pylint-dev__astroid.b114f6b5.pr_2410.937cftyp
pylint-dev__astroid.b114f6b5.pr_2410.lmr6k6c7
pylint-dev__astroid.b114f6b5.pr_2410.ycva480t
pylint-dev__astroid.b114f6b5.pr_2436.h02rlamu
pylint-dev__astroid.b114f6b5.pr_2484.rnp498jj
pylint-dev__astroid.b114f6b5.pr_2515.js6oiqwo
pylint-dev__astroid.b114f6b5.pr_2562.dqf4a5ap
pylint-dev__astroid.b114f6b5.pr_2570.mp5zqq5p
pylint-dev__astroid.b114f6b5.pr_2570.y2l3a7rh
pylint-dev__astroid.b114f6b5.pr_2572.3t59gvlx
pylint-dev__astroid.b114f6b5.pr_2573.sdci30pv
pylint-dev__astroid.b114f6b5.pr_2583.r3nl5atj
pylint-dev__astroid.b114f6b5.pr_2586.ay3icbz1
pylint-dev__astroid.b114f6b5.pr_2596.rhs1t9sc
pylint-dev__astroid.b114f6b5.pr_2597.p2ix61re
pylint-dev__astroid.b114f6b5.pr_2598.4oroi415
pylint-dev__astroid.b114f6b5.pr_2624.1ig9s4av
pylint-dev__astroid.b114f6b5.pr_2624.gzzvyhhv
pylint-dev__astroid.b114f6b5.pr_2665.7duc3zeu
```

--------------------------------

### Shuffle Control Flow in Python (keleshev__schema)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This snippet demonstrates shuffling control flow in Python, specifically within the 'keleshev__schema' module. This could be used for testing or randomization of execution paths.

```python
def func_pm_ctrl_shuffle__t2num235(self):
    pass
```

--------------------------------

### Basic Operation in python-hyper/h11

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This snippet represents a 'func_basic' operation within the python-hyper/h11 library. It likely covers fundamental or core functionalities of the HTTP/1.1 protocol implementation. These are essential building blocks for the library's operation.

```python
python-hyper__h11.bed0dd4a.func_basic__go0hr2bm.1k9pp663
python-hyper__h11.bed0dd4a.func_basic__go0hr2bm.3ffyf0or
```

--------------------------------

### Gather Validated Task Instances

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/harnesses.md

Collects all valid task instances from the validation run. Valid instances are those with at least one 'FAIL_TO_PASS' and one 'PASS_TO_PASS' test case. This script also creates a new branch for each valid instance with the patch applied.

```bash
python -m swesmith.harness.gather logs/run_validation/<run_id>
```

--------------------------------

### Combine File Operation in pyparsing

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/assets/traj_ids_swealm32b.txt

This snippet demonstrates a 'combine_file' operation within the pyparsing library. It likely involves combining multiple file-related operations or parsing data from combined files. No specific dependencies are mentioned, and the input/output are inferred from the operation name.

```python
pyparsing__pyparsing.533adf47.combine_file__68skjedt.8viqcipb
pyparsing__pyparsing.533adf47.combine_file__68skjedt.bxj96hoh
pyparsing__pyparsing.533adf47.combine_file__tnpvnxss.sz6qmv92
```

--------------------------------

### Generate Expert Trajectories (Bash)

Source: https://github.com/swe-bench/swe-smith/blob/main/docs/guides/train_swe_agent.md

This bash command is used within the SWE-agent directory to generate expert trajectories. It requires cloning the SWE-agent repository and creating a soft link to the SWE-smith agent folder. The script `_gen_trajs.sh` handles the trajectory generation, and the `--instances.path` argument needs to be adjusted to point to the created subset.

```bash
# Clone SWE-agent and follow installation instructions.
# Create a soft link:
ln -s path/to/SWE-smith/agent/ .

# Run expert trajectory generation:
./agent/_gen_trajs.sh
```