### Install HumanEval Repository and Dependencies

Source: https://github.com/openai/human-eval/blob/master/README.md

Instructions for setting up the HumanEval environment using Conda and pip. This involves creating a Conda environment with Python 3.7+ and installing the repository from GitHub.

```shell
conda create -n codex python=3.7
conda activate codex
git clone https://github.com/openai/human-eval
pip install -e human-eval
```

--------------------------------

### Evaluate Example Samples with Specific Problem File

Source: https://github.com/openai/human-eval/blob/master/README.md

Example command to evaluate a specific set of samples (`example_samples.jsonl`) using a corresponding problem definition file (`example_problem.jsonl`). Demonstrates how to specify custom problem files.

```shell
$ evaluate_functional_correctness data/example_samples.jsonl --problem_file=data/example_problem.jsonl
Reading samples...
6it [00:00, 3397.11it/s]
Running example suites...
100%|...| 6/6 [00:03<00:00,  1.96it/s]
Writing results to data/example_samples.jsonl_results.jsonl...
100%|...| 6/6 [00:00<00:00, 6148.50it/s]
{'pass@1': 0.4999999999999999}
```

--------------------------------

### Get Help for Evaluation Command

Source: https://github.com/openai/human-eval/blob/master/README.md

Command to display the help message for the `evaluate_functional_correctness` script, showing all available options and arguments.

```shell
$ evaluate_functional_correctness --help
```

--------------------------------

### Generate Code Samples with HumanEval

Source: https://github.com/openai/human-eval/blob/master/README.md

Python script to generate code completion samples using the HumanEval dataset. It reads problems, generates completions for each task, and saves them in JSON Lines format. Requires a `generate_one_completion` function.

```python
from human_eval.data import write_jsonl, read_problems

problems = read_problems()

num_samples_per_task = 200
samples = [
    dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
    for task_id in problems
    for _ in range(num_samples_per_task)
]
write_jsonl("samples.jsonl", samples)
```

--------------------------------

### Load HumanEval Problems

Source: https://context7.com/openai/human-eval/llms.txt

Loads programming problems from the HumanEval dataset. It returns a dictionary where keys are task IDs and values contain the problem's prompt, test cases, and entry point. Supports loading from default or custom file paths.

```python
from human_eval.data import read_problems

# Load the default HumanEval problem set
problems = read_problems()

# Access a specific problem
problem = problems["HumanEval/0"]
print(problem["prompt"])
print(problem["test"])
print(problem["entry_point"])

# Load custom problem file
custom_problems = read_problems("data/example_problem.jsonl")
```

--------------------------------

### Bash: Evaluate Functional Correctness of Code Completions

Source: https://context7.com/openai/human-eval/llms.txt

Command-line tool to evaluate code completions directly from the terminal. Supports specifying sample files, custom k values, multiple workers, timeouts, and custom problem files.

```bash
# Basic evaluation
evaluate_functional_correctness samples.jsonl

# Output:
# Reading samples...
# 32800it [00:01, 23787.50it/s]
# Running test suites...
# 100%|████████| 32800/32800 [16:11<00:00, 33.76it/s]
# Writing results to samples.jsonl_results.jsonl...
# {'pass@1': 0.45, 'pass@10': 0.72, 'pass@100': 0.89}

# Custom k values
evaluate_functional_correctness samples.jsonl --k=1,5,10,25

# More workers and longer timeout
evaluate_functional_correctness samples.jsonl --n_workers=8 --timeout=5.0

# Evaluate with custom problem file
evaluate_functional_correctness data/example_samples.jsonl \
    --problem_file=data/example_problem.jsonl

# Expected output for example: {'pass@1': 0.5}
```

--------------------------------

### Python: Define and Read Sample File Formats for Evaluation

Source: https://context7.com/openai/human-eval/llms.txt

Defines the JSONL format for sample files containing code completions and the structure of the results file after evaluation. The sample file requires 'task_id' and 'completion' fields. The results file adds 'result' and 'passed' fields.

```python
# Format for samples.jsonl (one JSON object per line)
samples = [
    {
        "task_id": "HumanEval/0",
        "completion": "    return a + b\n"
    },
    {
        "task_id": "HumanEval/1",
        "completion": "    return [x for x in lst if x > 0]\n"
    }
]

# After evaluation, results file adds execution details
results = [
    {
        "task_id": "HumanEval/0",
        "completion": "    return a + b\n",
        "result": "passed",
        "passed": True
    },
    {
        "task_id": "HumanEval/1",
        "completion": "    return [x for x in lst if x > 0]\n",
        "result": "failed: AssertionError",
        "passed": False
    }
]
```

--------------------------------

### Python: End-to-End Workflow for Code Completion Evaluation

Source: https://context7.com/openai/human-eval/llms.txt

Demonstrates the complete evaluation process: loading problems, generating completions using a placeholder model, saving samples to a JSONL file, performing functional correctness evaluation, and analyzing the results. This workflow requires functions from human_eval.data and human_eval.evaluation.

```python
from human_eval.data import write_jsonl, read_problems, stream_jsonl
from human_eval.evaluation import evaluate_functional_correctness

# Step 1: Load problems
problems = read_problems()
print(f"Loaded {len(problems)} problems")

# Step 2: Generate completions (implement your model)
def generate_one_completion(prompt):
    # Replace with actual model inference
    # This is a placeholder that returns a simple implementation
    return "    pass\n"

# Step 3: Create multiple samples per task
num_samples_per_task = 200
samples = [
    dict(
        task_id=task_id,
        completion=generate_one_completion(problems[task_id]["prompt"])
    )
    for task_id in problems
    for _ in range(num_samples_per_task)
]

# Step 4: Save samples
write_jsonl("samples.jsonl", samples)
print(f"Generated {len(samples)} samples")

# Step 5: Evaluate functional correctness
results = evaluate_functional_correctness(
    sample_file="samples.jsonl",
    k=[1, 10, 100],
    n_workers=4,
    timeout=3.0
)

# Step 6: Analyze results
print("Evaluation results:")
for metric, value in results.items():
    print(f"  {metric}: {value:.2%}")

# Read detailed results
passed_count = sum(1 for r in stream_jsonl("samples.jsonl_results.jsonl") if r["passed"])
total_count = sum(1 for _ in stream_jsonl("samples.jsonl_results.jsonl"))
print(f"Passed: {passed_count}/{total_count} ({passed_count/total_count:.2%})")

```

--------------------------------

### Evaluate Functional Correctness of Code Samples

Source: https://github.com/openai/human-eval/blob/master/README.md

Command-line tool to evaluate the functional correctness of generated code samples against the HumanEval dataset. It processes sample files and outputs results, including pass@k metrics.

```shell
$ evaluate_functional_correctness samples.jsonl
Reading samples...
32800it [00:01, 23787.50it/s]
Running test suites...
100%|...| 32800/32800 [16:11<00:00, 33.76it/s]
Writing results to samples.jsonl_results.jsonl...
100%|...| 32800/32800 [00:00<00:00, 42876.84it/s]
{'pass@1': ..., 'pass@10': ..., 'pass@100': ...}
```

--------------------------------

### Write Model Completions to JSONL

Source: https://context7.com/openai/human-eval/llms.txt

Writes generated code completions to JSONL format, which can be plain or gzipped. This function is used to save model-generated code samples before they are evaluated. It supports creating new files and appending to existing ones.

```python
from human_eval.data import write_jsonl, read_problems

problems = read_problems()

# Generate completions (placeholder function - implement your model here)
def generate_one_completion(prompt):
    # Your model generates code based on the prompt
    return "    return x + y\n"

# Create samples with 200 completions per task
num_samples_per_task = 200
samples = [
    dict(task_id=task_id, completion=generate_one_completion(problems[task_id]["prompt"]))
    for task_id in problems
    for _ in range(num_samples_per_task)
]

# Write to file (automatically handles compression for .gz extension)
write_jsonl("samples.jsonl", samples)

# Append additional samples
more_samples = [
    dict(task_id="HumanEval/0", completion="    return x + y\n")
]
write_jsonl("samples.jsonl", more_samples, append=True)
```

--------------------------------

### Evaluate Functional Correctness of Code

Source: https://context7.com/openai/human-eval/llms.txt

Evaluates generated code samples against their respective test suites. It executes each completion in a sandboxed environment and calculates pass@k metrics. Supports parallel execution and configurable timeouts.

```python
from human_eval.evaluation import evaluate_functional_correctness

# Evaluate samples with default settings (pass@1, pass@10, pass@100)
results = evaluate_functional_correctness(
    sample_file="samples.jsonl",
    k=[1, 10, 100],
    n_workers=4,
    timeout=3.0,
    problem_file="data/HumanEval.jsonl.gz"
)

print(results)
# Output: {'pass@1': 0.45, 'pass@10': 0.72, 'pass@100': 0.89}

# Custom evaluation with different k values
results = evaluate_functional_correctness(
    sample_file="samples.jsonl",
    k=[1, 5, 10],
    n_workers=8,
    timeout=5.0
)

# Results are written to samples.jsonl_results.jsonl with per-sample details
```

--------------------------------

### Python: Estimate Pass@k Metrics for Code Completions

Source: https://context7.com/openai/human-eval/llms.txt

Calculates unbiased Pass@k estimates from completion results. It measures the probability that at least one of k samples passes all tests. Requires numpy for array manipulation.

```python
import numpy as np
from human_eval.evaluation import estimate_pass_at_k

# Example: 3 problems with varying success rates
num_samples = np.array([100, 100, 100])  # 100 samples per problem
num_correct = np.array([45, 72, 89])     # Correct samples per problem

# Calculate pass@1
pass_at_1 = estimate_pass_at_k(num_samples, num_correct, k=1)
print(f"Pass@1 per problem: {pass_at_1}")
print(f"Average pass@1: {pass_at_1.mean()}")

# Calculate pass@10
pass_at_10 = estimate_pass_at_k(num_samples, num_correct, k=10)
print(f"Average pass@10: {pass_at_10.mean()}")

# Works with uniform samples across problems
num_correct = np.array([20, 35, 50])
pass_rates = estimate_pass_at_k(100, num_correct, k=5)
```

--------------------------------

### Check Correctness of a Single Code Completion

Source: https://context7.com/openai/human-eval/llms.txt

Tests a single code completion against its designated test suite. This function executes the code in an isolated process, providing timeout protection and reliability guards. It returns detailed results including whether the test passed or failed.

```python
from human_eval.execution import check_correctness
from human_eval.data import read_problems

problems = read_problems()

# Test a single completion
problem = problems["HumanEval/0"]
completion = "    return a + b\n"

result = check_correctness(
    problem=problem,
    completion=completion,
    timeout=3.0,
    completion_id=0
)

print(result)
# Output: {
#     'task_id': 'HumanEval/0',
#     'passed': True,
#     'result': 'passed',
#     'completion_id': 0
# }

# Handle different result states
if result['passed']:
    print("Test passed!")
elif result['result'] == 'timed out':
    print("Execution timed out")
else:
    print(f"Test failed: {result['result']}")
```

--------------------------------

### Stream JSONL Data

Source: https://context7.com/openai/human-eval/llms.txt

Parses JSONL files line-by-line as an iterable, efficiently handling both plain and gzipped formats. This is useful for processing large datasets without loading the entire file into memory.

```python
from human_eval.data import stream_jsonl

# Stream samples from a file
for sample in stream_jsonl("samples.jsonl"):
    task_id = sample["task_id"]
    completion = sample["completion"]
    print(f"Processing {task_id}: {completion[:50]}...")

# Works with gzipped files automatically
for problem in stream_jsonl("data/HumanEval.jsonl.gz"):
    print(f"{problem['task_id']}: {problem['entry_point']}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.