### Create and Activate Conda Environments

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Sets up the main agent environment and an isolated evaluation environment using conda. Ensure you have conda installed and activated.

```bash
conda create -n sci-agent python=3.10 pip setuptools wheel
conda activate sci-agent
pip install -r requirements.txt
export PYTHONPATH=.
```

```bash
conda create -n sci-agent-eval python=3.10 pip setuptools wheel
conda activate sci-agent-eval
pip install pip-tools
conda deactivate
```

--------------------------------

### Install Docker SDK for Python

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Install the Docker SDK for Python, which is required for the dockerized evaluation process.

```bash
pip install docker
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Use these commands to create a new conda environment named 'sci-agent' with Python 3.10 and install necessary packages from requirements.txt.

```bash
conda create -n sci-agent python=3.10 pip setuptools wheel
conda activate sci-agent
pip install -r requirements.txt
```

--------------------------------

### Create Conda Environment for Evaluation

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Set up a separate conda environment 'sci-agent-eval' for evaluation tasks using pip-tools for automatic package updates. Remember to deactivate it after installation.

```bash
conda create -n sci-agent-eval python=3.10 pip setuptools wheel
conda activate sci-agent-eval
pip install pip-tools
conda deactivate
```

--------------------------------

### Compute Scores for Program Evaluation

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Computes semantic similarity (CodeBERTScore) and success rate for a given example against specified program paths. Logs standard output or evaluation details.

```python
valid, cbs, sr, log = compute_scores(
    example,
    eval_program_path="benchmark/eval_programs",
    pred_program_path="pred_programs",
    gold_program_path="benchmark/gold_programs"
)
print(f"Valid: {valid}, CBS: {cbs:.3f}, Success: {sr}, Info: {log[:80]}")
```

--------------------------------

### Configure and Use LLMEngine for Amazon Bedrock

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Initializes the LLMEngine for Amazon Bedrock models. Requires AWS credentials to be configured in ~/.aws/config.

```python
# Amazon Bedrock (llama, mistral, claude, etc.) — requires ~/.aws/config
# ~/.aws/config:
# [default]
# aws_access_key_id = YOUR_AWS_ID
# aws_secret_access_key = YOUR_AWS_KEY
# region = us-west-2

bedrock_engine = LLMEngine("anthropic.claude-3-5-sonnet-20240620-v1:0")
response_text, prompt_tokens, completion_tokens = bedrock_engine.respond(
    messages, temperature=0.2, top_p=0.95
)
```

--------------------------------

### Configure and Use LLMEngine for OpenAI

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Initializes the LLMEngine for OpenAI models. Requires the OPENAI_API_KEY environment variable to be set.

```python
from engine.base_engine import LLMEngine
import os
os.environ["OPENAI_API_KEY"] = "sk-..."

engine = LLMEngine("gpt-4o-2024-08-06")
messages = [{"role": "user", "content": "Write a Python script to compute PCA on a CSV file."}]
response_text, prompt_tokens, completion_tokens = engine.respond(
    messages, temperature=0.2, top_p=0.95
)
print(response_text)
```

--------------------------------

### Benchmark Directory Structure

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

This outlines the expected directory structure after downloading and unzipping the full ScienceAgentBench benchmark. Ensure your downloaded files match this organization for proper usage.

```text
|-- ScienceAgentBench/
|---- benchmark/
|------ datasets/
|-------- ...
|------ eval_programs/
|-------- ...
|------ gold_programs/
|-------- ...
|------ scoring_rubrics/
|-------- ...
|---- engine/
|------ ...
|---- agent.py
|---- ...
```

--------------------------------

### Run Dockerized Code Evaluation

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Execute the evaluation harness using Docker. This requires the Docker SDK, an OpenAI API key, and paths to the benchmark and predicted program files. A log file name and run ID are also mandatory.

```bash
export OPENAI_API_KEY={YOUR_OPENAI_KEY}
python -m evaluation.harness.run_evaluation \
    --benchmark_path benchmark \
    --pred_program_path pred_programs \
    --log_fname self_debug_eval.jsonl \
    --run_id 1
```

--------------------------------

### Initialize ScienceAgent

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Initializes the ScienceAgent with a specified LLM engine, context window limit, and optional flags for self-debugging and domain knowledge.

```python
from agent import ScienceAgent

agent = ScienceAgent(
    llm_engine_name="gpt-4o-2024-08-06",
    context_cutoff=28000,   # max tokens before trimming
    use_self_debug=True,    # enable iterative error-fixing loop
    use_knowledge=True      # inject expert domain knowledge into prompt
)
```

--------------------------------

### Aggregate Benchmark Metrics via CLI

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Aggregates results from multiple independent runs for benchmark-wide averages of success rate, CodeBERTScore, valid program rate, and cost. Requires specifying run logs and evaluation logs.

```bash
# Aggregate 3 independent runs (as recommended in the paper)
python calculate_metrics.py \
    --run_logs  claude_sd_run1.jsonl \
    --run_logs  claude_sd_run2.jsonl \
    --run_logs  claude_sd_run3.jsonl \
    --eval_logs claude_sd_run1_eval.jsonl \
    --eval_logs claude_sd_run2_eval.jsonl \
    --eval_logs claude_sd_run3_eval.jsonl

# Expected output:
# ================ 
# Success Rate:         0.3431
# CodeBERTScore:        0.7812
# Valid Program Rate:   0.6275
# Cost:                 0.0523
# ================ 
```

--------------------------------

### Configure OpenAI API Key

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Set the OPENAI_API_KEY environment variable to authenticate with OpenAI models like gpt-4o.

```bash
export OPENAI_API_KEY={YOUR_OPENAI_KEY}
```

--------------------------------

### Calculate Metrics using Python Script

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Run this Python script to calculate metrics. Provide paths to log files for agent trajectories and evaluation results. The script selects the best trajectory from multiple runs and computes metrics.

```python
python calculate_metrics.py \
    --run_logs {LOG_FNAME_1} \
    --run_logs {LOG_FNAME_2} \
    --run_logs {LOG_FNAME_3} \
    --eval_logs {EVAL_LOG_FNAME_1} \
    --eval_logs {EVAL_LOG_FNAME_2} \
    --eval_logs {EVAL_LOG_FNAME_3}
```

--------------------------------

### Run Dockerized Evaluation

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/docs/20250110_docker/README.md

Execute the agent's generated programs using the containerized evaluation harness. Ensure your OpenAI API key is exported as an environment variable. This command specifies paths for the benchmark, predicted programs, and log file, along with run identifiers and optional parameters for caching, workers, and instance rebuilding.

```shell
export $OPENAI_API_KEY=YOUR_API_KEY
python -m evaluation.harness.run_evaluation \
--benchmark_path benchmark \
--pred_program_path pred_programs \
--log_fname self_debug_eval.jsonl \
--run_id test_run_1 \
--cache_level base \
--max_workers 4 \
--force_rebuild False \
--instance_ids 1 2 3
```

--------------------------------

### Generate System Prompt

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Constructs the complete system prompt by combining task fields and truncates it to fit within the specified token limit.

```python
sys_msg = agent.get_sys_msg(task)
print(sys_msg[:300])
# Output: "You are an expert Python programming assistant...
#  Here's the user request you need to work on: ...
#  You can access the dataset at `benchmark/datasets/dkpes`. ..."
```

--------------------------------

### Run Direct Code Evaluation

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Perform evaluation directly without Docker. This method requires an OpenAI API key for judging output visualizations and a log file name to store results.

```bash
export OPENAI_API_KEY={YOUR_OPENAI_KEY}
python -u run_eval.py \
    --log_fname {EVAL_LOG_FNAME}
```

--------------------------------

### Run Inference with Different Configurations

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Executes inference on the full benchmark dataset using `run_infer.py`. Supports direct prompting, self-debug with domain knowledge, and resuming interrupted runs.

```bash
# Direct prompting (no self-debug, no expert knowledge)
python -u run_infer.py \
    --llm_engine_name gpt-4o-2024-08-06 \
    --log_fname gpt4o_direct.jsonl
```

```bash
# Self-debug + expert domain knowledge
python -u run_infer.py \
    --llm_engine_name anthropic.claude-3-5-sonnet-20240620-v1:0 \
    --log_fname claude_selfdebug_knowledge.jsonl \
    --use_self_debug \
    --use_knowledge
```

```bash
# Resume an interrupted run (reads existing log to skip completed tasks)
python -u run_infer.py \
    --llm_engine_name gpt-4o-2024-08-06 \
    --log_fname gpt4o_direct.jsonl   # existing file detected; resumes from line N+1
```

--------------------------------

### Run Code Generation Agent

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Execute the main inference script to run agents for code generation. Specify the LLM engine and log file name. Optionally enable knowledge or self-debug features.

```bash
python -u run_infer.py \
    --llm_engine_name {MODEL_NAME} \
    --log_fname {LOG_FNAME}
    [--use_knowledge]
    [--use_self_debug]
```

--------------------------------

### Direct Evaluation (Without Docker)

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Performs sequential evaluation of tasks using conda environments. Requires the `sci-agent-eval` conda environment to be set up.

```bash
export OPENAI_API_KEY=sk-...

python -u run_eval.py \
    --log_fname claude_sd_eval.jsonl \
    --pred_program_path pred_programs \
    --gold_program_path benchmark/gold_programs \
    --eval_program_path benchmark/eval_programs \
    --result_path pred_results
```

--------------------------------

### Dockerized Evaluation Harness

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Runs parallel, containerized evaluation of predicted programs using Docker. Requires the Docker SDK for Python. Configuration options include cache levels and worker counts.

```bash
pip install docker  # Docker SDK for Python

export OPENAI_API_KEY=sk-...

# Evaluate all 102 tasks in parallel with 8 workers (~22 min on 16-core machine)
python -m evaluation.harness.run_evaluation \
    --benchmark_path benchmark \
    --pred_program_path pred_programs \
    --log_fname claude_sd_eval.jsonl \
    --run_id run_20241024 \
    --cache_level base \
    --max_workers 8 \
    --force_rebuild False
```

```bash
# Evaluate specific instances only
python -m evaluation.harness.run_evaluation \
    --benchmark_path benchmark \
    --pred_program_path pred_programs \
    --log_fname claude_sd_eval.jsonl \
    --run_id run_debug \
    --instance_ids 1 5 42 \
    --max_workers 4
```

--------------------------------

### Configure AWS Credentials for Bedrock LLMs

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Set up your AWS configuration file (~/.aws/config) with your AWS access key ID, secret access key, and region to use LLMs on Amazon Bedrock.

```ini
[default]
aws_access_key_id = {YOUR_AWS_ID}
aws_secret_access_key = {YOUR_AWS_KEY}
region=us-west-2
```

--------------------------------

### Aggregate Benchmark Metrics Programmatically

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Programmatically aggregates results from multiple runs by selecting the best trajectory per task. Returns a dictionary containing success rate, CodeBERTScore, valid program rate, and cost.

```python
from calculate_metrics import main as calc_metrics

results = calc_metrics(
    run_logs_paths=["run1.jsonl", "run2.jsonl", "run3.jsonl"],
    eval_logs_paths=["eval1.jsonl", "eval2.jsonl", "eval3.jsonl"]
)
print(results)
# {
#   "success_rate": 0.3431,
#   "codebert_score": 0.7812,
#   "valid_program_rate": 0.6275,
#   "cost": 0.0523
# }
```

--------------------------------

### Solve Benchmark Task with ScienceAgent

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Uses the ScienceAgent to generate a Python program for a given task, including its dataset details and domain knowledge. The self-debug loop runs if enabled during agent initialization. The output includes the generated program, conversation history, and API cost.

```python
task = {
    "task_inst": (
        "Use the DKPES dataset to develop a Random Forest classifier predicting "
        "signal inhibition of chemicals. Save test set predictions including the "
        "index and predicted signal inhibition to 'pred_results/dkpes_test_pred.csv'."
    ),
    "dataset_path": "benchmark/datasets/dkpes",
    "dataset_folder_tree": "|-- dkpes/\n|---- dkpes_test.csv\n|---- dkpes_train.csv",
    "dataset_preview": (
        "[START Preview of dkpes/dkpes_train.csv]\n"
        "index,Signal-inhibition,வைக்\n"
        "ZINC04026280,0.24,வைக்\n"
        "[END Preview of dkpes/dkpes_train.csv]"
    ),
    "domain_knowledge": "The decision boundary for signal inhibition is 0.6.",
    "output_fname": "pred_results/dkpes_test_pred.csv"
}

trajectory = agent.solve_task(task, out_fname="pred_programs/pred_dkpes.py")

print(f"Cost: ${trajectory['cost']:.4f}")
print(f"Turns: {len(trajectory['history'])}")
# Expected output file: pred_programs/pred_dkpes.py
# trajectory['history'] contains full multi-turn conversation as role/content dicts
```

--------------------------------

### Extract Predicted Programs from Logs

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Parses agent trajectory JSONL files to extract and save predicted programs as `.py` files. Supports native logs and OpenHands/OpenDevin logs.

```bash
# For ScienceAgentBench native logs
python -u recover_pred_from_log.py \
    --log_fname claude_selfdebug_knowledge.jsonl \
    --pred_program_path pred_programs/
```

```bash
# For OpenHands / OpenDevin logs
python -u recover_pred_from_log.py \
    --log_fname openhands_run.jsonl \
    --pred_program_path pred_programs/ \
    --is_opendevin
# Output: pred_programs/pred_<gold_program_name>.py for each of 102 tasks
# Also prints: Cost: 0.0412 (average per task)
```

--------------------------------

### Score Visualization Outputs with GPT-4o

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Scores visualization outputs by comparing predicted and gold plots using GPT-4o. Requires setting the OPENAI_API_KEY environment variable. Returns a similarity score averaged over 3 samples.

```python
from gpt4_visual_judge import encode_image, score_figure
import os

os.environ["OPENAI_API_KEY"] = "sk-..."

pred_img = encode_image("pred_results/Elk_Analysis.png")
gold_img = encode_image("benchmark/eval_programs/gold_results/Elk_Analysis_gold.png")

full_responses, score = score_figure(pred_img, gold_img)
print(f"Visual similarity score: {score:.1f}/100")
# full_responses: list of 3 GPT-4o judgement strings each ending with [FINAL SCORE]: N

# Using Azure OpenAI instead:
# os.environ["AZURE_OPENAI_KEY"] = "..."
# os.environ["AZURE_OPENAI_ENDPOINT"] = "https://your-resource.openai.azure.com/"
# os.environ["AZURE_OPENAI_API_VERSION"] = "2024-05-01-preview"
# os.environ["AZURE_OPENAI_DEPLOYMENT_NAME"] = "gpt-4o"
```

--------------------------------

### Extract Code Files from Agent Logs

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Use this script to recover generated code files from agent log files. Specify the log file name and optionally indicate if it's for an OpenHands agent.

```bash
python -u recover_pred_from_log.py \
    --log_fname {LOG_FNAME}
    [--is_opendevin]
```

--------------------------------

### Write Program from LLM Output

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

Extracts Python code from an LLM response and saves it to a file. Returns True if the program remains unchanged after a debug iteration, signaling early stopping.

```python
assistant_output = '''
Here is the solution:
```python
import pandas as pd
from sklearn.ensemble import RandomForestClassifier

df_train = pd.read_csv("benchmark/datasets/dkpes/dkpes_train.csv")
X = df_train[["3-Keto", "3-Hydroxy", "TanimotoCombo"]]
y = (df_train["Signal-inhibition"] >= 0.6).astype(int)
clf = RandomForestClassifier().fit(X, y)

df_test = pd.read_csv("benchmark/datasets/dkpes/dkpes_test.csv")
preds = clf.predict(df_test[["3-Keto", "3-Hydroxy", "TanimotoCombo"]])
pd.DataFrame({"index": df_test["index"], "pred": preds}).to_csv(
    "pred_results/dkpes_test_pred.csv", index=False
)
```
'''

unchanged = agent.write_program(assistant_output, "pred_programs/pred_dkpes.py")
print(unchanged)  # False on first write; True if identical to existing file (triggers early stop)
```

--------------------------------

### ScienceAgentBench Citation

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Use this BibTeX entry to cite the ScienceAgentBench paper if you find the code and data useful.

```bibtex
@misc{chen2024scienceagentbenchrigorousassessmentlanguage,
      title={ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery},
      author={Ziru Chen and Shijie Chen and Yuting Ning and Qianheng Zhang and Boshi Wang and Botao Yu and Yifei Li and Zeyi Liao and Chen Wei and Zitong Lu and Vishal Dey and Mingyi Xue and Frazier N. Baker and Benjamin Burns and Daniel Adu-Ampratwum and Xuhui Huang and Xia Ning and Song Gao and Yu Su and Huan Sun},
      year={2024},
      eprint={2410.05080},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.05080},
}
```

--------------------------------

### Set PYTHONPATH for Local Imports

Source: https://github.com/osu-nlp-group/scienceagentbench/blob/main/README.md

Export the PYTHONPATH environment variable to enable relative imports of local programs.

```bash
export PYTHONPATH=.
```

--------------------------------

### Compute Scores Function

Source: https://context7.com/osu-nlp-group/scienceagentbench/llms.txt

The `compute_scores` function within `run_eval.py` calculates evaluation metrics for each task, returning values such as program validity and success rate.

```python
from run_eval import compute_scores

# valid_program: 1 if program ran and produced output, else 0
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.