### Run RepoBench Experiments Source: https://github.com/leolty/repobench/blob/main/README.md Executes experiments using the `run.py` script with specified model, dataset, and generation parameters. This example demonstrates running a Python experiment with DeepSeek Coder. ```bash CUDA_VISIBLE_DEVICES=0 python run.py --model_name "deepseek-ai/deepseek-coder-1.3b-base" \ --dataset_name "tianyang/repobench_python_v1.1" \ --start_date "2023-12-01" \ --end_date "2023-12-31" \ --language "python" \ --max_token_nums 15800 \ --levels "2k" "4k" "8k" "12k" "16k" \ --temperature 0.2 \ --top_p 0.95 \ --max_new_tokens 128 \ --batch_size 1 ``` -------------------------------- ### Filter Repobench Datasets by Date Range Source: https://context7.com/leolty/repobench/llms.txt Filters a dataset dictionary by a specific start and end date. This is useful for isolating evaluation data to specific time periods. ```python filtered_dataset = filter_dataset_by_date_range( dataset=dataset, start_date="2023-12-01", end_date="2023-12-31" ) ``` -------------------------------- ### Clone RepoBench Repository Source: https://github.com/leolty/repobench/blob/main/README.md Clones the RepoBench repository from GitHub and navigates into the project directory. This is the initial step for setting up the project locally. ```bash git clone https://github.com/Leolty/repobench.git cd repobench ``` -------------------------------- ### Load RepoBench Datasets using HuggingFace Source: https://github.com/leolty/repobench/blob/main/data/README.md Demonstrates how to programmatically load the RepoBench Python and Java datasets directly from the HuggingFace Hub using the datasets library. ```python from datasets import load_dataset # Load the Python dataset python_dataset = load_dataset("tianyang/repobench_python_v1.1") # Load the Java dataset java_dataset = load_dataset("tianyang/repobench_java_v1.1") ``` -------------------------------- ### Load RepoBench Datasets from HuggingFace Source: https://context7.com/leolty/repobench/llms.txt Loads the RepoBench datasets for Python and Java from the HuggingFace Hub. It demonstrates how to access different evaluation splits (cross_file_first, cross_file_random, in_file) and outlines the structure of each data point, including repository name, file paths, context, code snippets, and ground truth. ```python from datasets import load_dataset # Load Python dataset python_dataset = load_dataset("tianyang/repobench_python_v1.1", ignore_verifications=True) # Load Java dataset java_dataset = load_dataset("tianyang/repobench_java_v1.1", ignore_verifications=True) # Access different evaluation settings cross_file_first = python_dataset["cross_file_first"] cross_file_random = python_dataset["cross_file_random"] in_file = python_dataset["in_file"] # Each data point contains: # - repo_name: repository name # - file_path: path of the target file # - context: list of cross-file code snippets with paths # - import_statement: import statements in the target file # - cropped_code: code before the line to predict # - next_line: ground truth (the line to predict) # - level: prompt length category (2k, 4k, 8k, 12k, 16k, 24k, 32k, 64k, 128k) # - created_at: timestamp of repository creation print(f"Cross-file first examples: {len(cross_file_first)}") print(f"Sample data keys: {list(cross_file_first[0].keys())}") ``` -------------------------------- ### Run Code Completion Experiments Source: https://context7.com/leolty/repobench/llms.txt Executes code completion experiments using HuggingFace models on RepoBench datasets. This command-line interface supports batch processing, date range filtering, and selection of multiple prompt length levels. It allows configuration of model parameters such as temperature, top-p, and max new tokens. ```bash # Run experiments with DeepSeek Coder model CUDA_VISIBLE_DEVICES=0 python run.py \ --model_name "deepseek-ai/deepseek-coder-1.3b-base" \ --dataset_name "tianyang/repobench_python_v1.1" \ --start_date "2023-12-01" \ --end_date "2023-12-31" \ --language "python" \ --max_token_nums 15800 \ --levels "2k" "4k" "8k" "12k" "16k" \ --temperature 0.2 \ --top_p 0.95 \ --max_new_tokens 128 \ --batch_size 1 \ --res_dir "./results" # For Java experiments CUDA_VISIBLE_DEVICES=0 python run.py \ --model_name "deepseek-ai/deepseek-coder-1.3b-base" \ --dataset_name "tianyang/repobench_java_v1.1" \ --language "java" \ --levels "2k" "4k" "8k" # Results are saved to: results/{model_name}-{language}/{level}.jsonl # Each line contains: {"idx": 0, "level": "2k", "pred": "predicted line", "gt": "ground truth"} ``` -------------------------------- ### Load RepoBench Python Dataset Source: https://github.com/leolty/repobench/blob/main/README.md Loads the RepoBench v1.1 Python dataset using the Hugging Face `datasets` library. This allows for easy access to the data for running experiments. ```python from datasets import load_dataset dataset = load_dataset("tianyang/repobench_python_v1.1", ignore_verifications=True) ``` -------------------------------- ### Evaluate RepoBench Results Source: https://github.com/leolty/repobench/blob/main/README.md Evaluates the generated code completions using the `eval.py` script. This script calculates performance metrics for a specified results path and language. ```bash python eval.py --path "results/deepseek-coder-1.3b-base-python" --language "python" ``` -------------------------------- ### Run Model Evaluation via CLI Source: https://context7.com/leolty/repobench/llms.txt Executes the evaluation script to compare model predictions against ground truth using metrics like Exact Match and CodeBLEU. ```bash python eval.py --path "results/deepseek-coder-1.3b-base-python" --language "python" python eval.py --path "results/deepseek-coder-1.3b-base-java" --language "java" ``` -------------------------------- ### Construct Prompt for Next-Line Prediction Source: https://context7.com/leolty/repobench/llms.txt Constructs the input prompt for next-line prediction by combining cross-file context snippets with the target file's code. It utilizes a tokenizer to automatically truncate cross-file context to fit within specified token limits, ensuring compatibility with language model context windows. ```python from data.utils import construct_prompt from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("deepseek-ai/deepseek-coder-1.3b-base", trust_remote_code=True) # Sample data point structure data = { "repo_name": "example/repo", "file_path": "src/main.py", "context": [ {"path": "src/utils.py", "snippet": "def helper():\n return 42"}, {"path": "src/config.py", "snippet": "DEBUG = True\nVERSION = '1.0'"} ], "import_statement": "from utils import helper\nfrom config import DEBUG", "cropped_code": "def main():\n result = helper()\n if DEBUG:", "next_line": " print(result)" } # Construct prompt with token limit prompt = construct_prompt( data=data, language="python", # or "java" tokenizer=tokenizer, max_token_nums=15800 # adjust based on model context window ) print(prompt) ``` -------------------------------- ### Cite RepoBench in BibTeX Source: https://github.com/leolty/repobench/blob/main/data/README.md Provides the official BibTeX citation format for researchers using the RepoBench benchmark. ```bibtex @misc{liu2023repobench, title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems}, author={Tianyang Liu and Canwen Xu and Julian McAuley}, year={2024}, url={https://arxiv.org/abs/2306.03091}, booktitle={International Conference on Learning Representations} } ``` -------------------------------- ### Filter Repobench Datasets by Token Levels Source: https://context7.com/leolty/repobench/llms.txt Filters the dataset to include only data points within specific prompt length levels. This allows for targeted evaluation across different context window sizes. ```python from run import filter_dataset_by_levels short_context = filter_dataset_by_levels(dataset=dataset, levels=["2k", "4k"]) long_context = filter_dataset_by_levels(dataset=dataset, levels=["16k", "24k", "32k", "64k", "128k"]) ``` -------------------------------- ### Filter RepoBench Dataset by Date Range Source: https://context7.com/leolty/repobench/llms.txt Filters the RepoBench dataset to include only repositories created within a specified date range. This utility function is useful for creating time-based evaluation splits or analyzing model performance on code from specific periods. ```python from datasets import load_dataset from run import filter_dataset_by_date_range # Load full dataset dataset = load_dataset("tianyang/repobench_python_v1.1", ignore_verifications=True) # Example usage (assuming filter_dataset_by_date_range is defined in run.py and takes dataset and date range as input): # filtered_dataset = filter_dataset_by_date_range(dataset, "2023-10-01", "2023-12-31") # print(f"Filtered dataset size: {len(filtered_dataset)}") ``` -------------------------------- ### Compute Exact Match Score Source: https://context7.com/leolty/repobench/llms.txt Calculates the exact match score by comparing tokenized predictions against ground truth, ignoring extra whitespace. ```python from evaluation.metrics import exact_match_score score = exact_match_score(predictions, ground_truths) ``` -------------------------------- ### Compute Accuracy@k (Python) Source: https://context7.com/leolty/repobench/llms.txt Calculates the retrieval accuracy at k, measuring if the correct code snippet is within the top-k retrieved results for a given query. This is useful for evaluating code retrieval tasks. ```python from evaluation.metrics import accuracy_at_k # Each inner list contains indices of retrieved code snippets (ranked by relevance) prediction_list = [ [3, 1, 4, 2, 0], # Query 1: retrieved indices in order [0, 2, 1, 3, 4], # Query 2 [2, 3, 0, 1, 4], # Query 3 [1, 0, 2, 3, 4] # Query 4 ] # Ground truth: index of the correct code snippet for each query golden_index_list = [3, 0, 1, 2] # Accuracy@1: Is correct answer in top-1? acc_1 = accuracy_at_k(prediction_list, golden_index_list, k=1) print(f"Accuracy@1: {acc_1}") # Output: 0.5 (2/4: queries 1 and 2) # Accuracy@3: Is correct answer in top-3? acc_3 = accuracy_at_k(prediction_list, golden_index_list, k=3) print(f"Accuracy@3: {acc_3}") # Output: 0.75 (3/4: queries 1, 2, and 4) # Accuracy@5: Is correct answer in top-5? acc_5 = accuracy_at_k(prediction_list, golden_index_list, k=5) print(f"Accuracy@5: {acc_5}") # Output: 1.0 (all queries have correct answer in top-5) ``` -------------------------------- ### Compute CodeBLEU Score (Python, Java) Source: https://context7.com/leolty/repobench/llms.txt Calculates the CodeBLEU score, which evaluates code quality by combining n-gram matching, syntax tree similarity, and data flow analysis. It supports multiple programming languages. ```python from evaluation.metrics import codebleu_score predictions = [ "def add(a, b):\n return a + b", "for i in range(10):\n print(i)" ] ground_truths = [ "def add(x, y):\n return x + y", "for j in range(10):\n print(j)" ] # Calculate CodeBLEU for Python code score = codebleu_score( predictions=predictions, ground_truths=ground_truths, language="python", weight=[0.25, 0.25, 0.25, 0.25] # weights for n-gram, weighted n-gram, syntax, dataflow ) print(f"CodeBLEU Score: {score}") # Output: ~0.85 (high similarity despite variable renaming) # For Java code java_preds = ["int sum = a + b;"] java_gts = ["int total = a + b;"] java_score = codebleu_score(java_preds, java_gts, language="java") ``` -------------------------------- ### Extract First Non-Comment Line Source: https://context7.com/leolty/repobench/llms.txt Parses code strings to extract the first line that is not a comment. Supports both Python and Java comment syntax. ```python from run import get_first_line_not_comment # Python example first_line = get_first_line_not_comment(python_code, language="python") # Java example first_line = get_first_line_not_comment(java_code, language="java") ``` -------------------------------- ### Compute Edit Similarity Score Source: https://context7.com/leolty/repobench/llms.txt Calculates the average edit similarity between predictions and ground truth using the Levenshtein ratio, returning a percentage score. ```python from evaluation.metrics import edit_similarity_score score = edit_similarity_score(predictions, ground_truths) ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.