### Copy Example .env File (Editable Install)

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

For editable installations, copy the example .env file to .env and fill in necessary values for API keys and configuration.

```bash
cp bfcl_eval/.env.example .env
# Fill in necessary values in `.env`
```

--------------------------------

### Setup BFCL-v3 Environment

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/README.md

Navigate to the berkeley-function-call-leaderboard directory, create a virtual environment using uv, activate it, and install the project's dependencies.

```bash
cd berkeley-function-call-leaderboard
uv venv
source .venv/bin/activate
uv pip install -e .
```

--------------------------------

### Copy Example .env File (PyPI Install)

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

For PyPI installations, copy the example .env file from the installed package to BFCL_PROJECT_ROOT/.env and fill in necessary values.

```bash
cp $(python -c "import bfcl_eval; print(bfcl_eval.__path__[0])")/.env.example $BFCL_PROJECT_ROOT/.env
# Fill in necessary values in `.env`
```

--------------------------------

### Setup Tau-Bench Environment

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/README.md

Navigate to the tau-bench directory, create a virtual environment using uv, activate it, and install the project's dependencies.

```bash
cd tau-bench
uv venv
source .venv/bin/activate
uv pip install -e .
```

--------------------------------

### Basic Ollama Setup and Aider Launch

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/llms/ollama.md

Steps to pull an Ollama model, start the Ollama server, set the API base URL, and run Aider with a generic model.

```bash
# Pull the model
ollama pull <model>

# Start your ollama server
ollama serve

# In another terminal window...
python -m pip install aider-chat

export OLLAMA_API_BASE=http://127.0.0.1:11434 # Mac/Linux
setx   OLLAMA_API_BASE http://127.0.0.1:11434 # Windows, restart shell after setx

aider --model ollama/<model>
```

--------------------------------

### Copy Example test_case_ids_to_generate.json (PyPI Install)

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

For PyPI installations, copy the example test case IDs file from the installed package to the project root.

```bash
cp $(python -c "import bfcl_eval, pathlib; print(pathlib.Path(bfcl_eval.__path__[0]) / 'test_case_ids_to_generate.json.example')") $BFCL_PROJECT_ROOT/test_case_ids_to_generate.json
```

--------------------------------

### Copy Example test_case_ids_to_generate.json (Editable Install)

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

For editable installations, copy the example test case IDs file to the project root so the CLI can pick it up.

```bash
cp bfcl_eval/test_case_ids_to_generate.json.example ./test_case_ids_to_generate.json
```

--------------------------------

### Install τ-bench from Source

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/tau-bench/README.md

Install the τ-bench package and its dependencies by installing from the source code.

```bash
pip install -e .
```

--------------------------------

### Getting Started with Aider and Sonnet

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_posts/2024-07-01-sonnet-not-lazy.md

Install aider and set the ANTHROPIC_API_KEY environment variable to use Claude 3.5 Sonnet by default. Then, run the aider command to begin.

```bash
$ python -m pip install aider-chat

$ export ANTHROPIC_API_KEY=<key> # Mac/Linux
$ setx   ANTHROPIC_API_KEY <key> # Windows, restart shell after setx

$ aider
```

--------------------------------

### Specific Ollama Model Setup and Aider Launch

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/llms/ollama.md

Instructions for pulling the 'llama3:70b' model, starting the Ollama server, setting the API base URL, and launching Aider with this specific model.

```bash
ollama pull llama3:70b
ollama serve

# In another terminal window...
export OLLAMA_API_BASE=http://127.0.0.1:11434 # Mac/Linux
setx   OLLAMA_API_BASE http://127.0.0.1:11434 # Windows, restart shell after setx

aider --model ollama/llama3:70b 
```

--------------------------------

### Code Infilling Example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/examples/Qwen2.5-Coder.md

Demonstrates how to use Qwen2.5-Coder-32B for code infilling by setting the FIM format in the input text. This example shows a Python library system setup.

```python
input_text = "<|repo_name|>library-system
<|file_sep|>library.py
class Book:
    def __init__(self, title, author, isbn, copies):
        self.title = title
        self.author = author
        self.isbn = isbn
        self.copies = copies

    def __str__(self):
        return f"Title: {self.title}, Author: {self.author}, ISBN: {self.isbn}, Copies: {self.copies}"

class Library:
    def __init__(self):
        self.books = []

    def add_book(self, title, author, isbn, copies):
        book = Book(title, author, isbn, copies)
        self.books.append(book)

    def find_book(self, isbn):
        for book in self.books:
            if book.isbn == isbn:
                return book
        return None

    def list_books(self):
        return self.books

<|file_sep|>student.py
class Student:
    def __init__(self, name, id):
        self.name = name
        self.id = id
        self.borrowed_books = []

    def borrow_book(self, book, library):
        if book and book.copies > 0:
            self.borrowed_books.append(book)
            book.copies -= 1
            return True
        return False

    def return_book(self, book, library):
        if book in self.borrowed_books:
            self.borrowed_books.remove(book)
            book.copies += 1
            return True
        return False

<|file_sep|>main.py
<|fim_prefix|>from library import Library
from student import Student

def main():
    # Set up the library with some books
    library = Library()
    library.add_book("The Great Gatsby", "F. Scott Fitzgerald", "1234567890", 3)
    library.add_book("To Kill a Mockingbird", "Harper Lee", "1234567891", 2)
    
    # Set up a student
    student = Student("Alice", "S1")
    
    # Student borrows a book<|fim_suffix|>
    if student.borrow_book(book, library):
        print(f"{student.name} borrowed {book.title}")
    else:
        print(f"{student.name} could not borrow {book.title}")
        
    # Student returns a book
    if student.return_book(book, library):
        print(f"{student.name} returned {book.title}")
    else:
        print(f"{student.name} could not return {book.title}")
    
    # List all books in the library
    print("All books in the library:")
    for book in library.list_books():
        print(book)

if __name__ == "__main__":
    main()<|fim_middle|>"
model_inputs = tokenizer([input_text], return_tensors="pt").to(device)

# Use `max_new_tokens` to control the maximum output length.
eos_token_ids = [151664, 151662, 151659, 151660, 151661, 151662, 151663, 151664, 151645, 151643]
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=1024, do_sample=False, eos_token_id=eos_token_ids)[0]
# The generated_ids include prompt_ids, so we only need to decode the tokens after prompt_ids.
output_text = tokenizer.decode(generated_ids[len(model_inputs.input_ids[0]):], skip_special_tokens=True)

print(f"Prompt: \n{input_text}\n\nGenerated text: \n{output_text.split('<|file_sep|>')[0]}")

# the expected output as following:
"""
Generated text:
    book = library.find_book("1234567890")
"""

```

--------------------------------

### Install DevQualityEval CLI

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/eval-dev-quality/README.md

Clone the repository, navigate to the directory, and install the command-line interface using Go. This makes the `eval-dev-quality` binary available for use.

```bash
git clone https://github.com/symflower/eval-dev-quality.git
cd eval-dev-quality
go install -v github.com/symflower/eval-dev-quality/cmd/eval-dev-quality
```

--------------------------------

### Install and Launch Aider Browser UI

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_posts/2024-05-02-browser.md

Install the aider-chat package and set your API key. Then, launch the experimental browser UI using the `--browser` flag.

```bash
python -m pip install aider-chat

export OPENAI_API_KEY=<key> # Mac/Linux
setx   OPENAI_API_KEY <key> # Windows, restart shell after setx


aider --browser
```

--------------------------------

### Install Kaggle CLI

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/PlotCraft/README.md

Installs the Kaggle command-line interface for downloading datasets.

```bash
pip install kaggle
```

--------------------------------

### Basic Installation of BFCL

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

Steps to set up a Conda environment, clone the repository, and install the BFCL package in editable mode.

```bash
conda create -n BFCL python=3.10
conda activate BFCL

git clone https://github.com/ShishirPatil/gorilla.git

cd gorilla/berkeley-function-call-leaderboard

pip install -e .
```

--------------------------------

### Install aider and set up API keys

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_posts/2024-07-25-new-models.md

Install the aider chat client and configure environment variables for API keys to use various models. Ensure you are in a git repository before running aider.

```bash
python -m pip install aider-chat

# Change directory into a git repo to work on
cd /to/your/git/repo

export DEEPSEEK_API_KEY=your-key-goes-here
aider --model deepseek/deepseek-coder

export MISTRAL_API_KEY=your-key-goes-here
aider --model mistral/mistral-large-2407

export OPENROUTER_API_KEY=your-key-goes-here
aider --model openrouter/meta-llama/llama-3.1-405b-instruct
aider --model openrouter/meta-llama/llama-3.1-70b-instruct
aider --model openrouter/meta-llama/llama-3.1-8b-instruct
```

--------------------------------

### Install Aider

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_includes/get-started.md

Install the aider-chat package using pip. This is the first step to using Aider.

```bash
python -m pip install aider-chat
```

--------------------------------

### Install and Launch Aider in Browser

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/usage/browser.md

Install Aider using pip and launch the browser UI with the --browser flag. Ensure your LLM API key is set as an environment variable.

```bash
python -m pip install aider-chat

export OPENAI_API_KEY=<key> # Mac/Linux
setx   OPENAI_API_KEY <key> # Windows, restart shell after setx




aider --browser
```

--------------------------------

### Create Flask App with /hello Endpoint

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/examples/hello-world-flask.md

Initializes a Flask application and adds a '/hello' endpoint that returns 'Hello, World!'. Requires Flask to be installed (`pip install flask`).

```python
from flask import Flask

app = Flask(__name__)

@app.route('/hello')
def hello():
    return "Hello, World!"

if __name__ == '__main__':
    app.run()
```

--------------------------------

### Install Aider and Set OpenRouter API Key

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/llms/openrouter.md

Installs the aider-chat package and sets the OPENROUTER_API_KEY environment variable for Mac/Linux and Windows.

```bash
python -m pip install aider-chat

export OPENROUTER_API_KEY=<key> # Mac/Linux
setx   OPENROUTER_API_KEY <key> # Windows, restart shell after setx
```

--------------------------------

### Install Dependencies

Source: https://github.com/qwenlm/qwen3-coder/blob/main/finetuning/dpo/README.md

After activating the Conda environment, run this command to install all necessary project dependencies listed in 'requirements.txt'.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install EvalPlus and Requirements

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/eval_plus/readme.md

Install the EvalPlus package and project-specific requirements. Ensure you have the latest version of EvalPlus.

```bash
pip install evalplus --upgrade
pip install -r requirements.txt
```

--------------------------------

### Install Aider and Run with GPT-4o

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/index.md

Install the aider-chat package and set your OpenAI API key to use GPT-4o for pair programming.

```bash
python -m pip install aider-chat

# Change directory into a git repo
cd /to/your/git/repo

# Work with GPT-4o on your repo
export OPENAI_API_KEY=your-key-goes-here
aider
```

--------------------------------

### Install Pipx and Run Aider on Replit

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/install/pipx.md

On Replit, first install pipx, then use 'pipx run' to execute aider-chat with its arguments. Running aider directly after pipx installation on Replit may cause a missing library error.

```bash
pip install pipx
pipx run aider-chat ...normal aider args...
```

--------------------------------

### Install Aider and Set Environment Variables

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/llms/openai-compat.md

Install the aider-chat package and set the OPENAI_API_BASE and OPENAI_API_KEY environment variables. Restart your shell after setting variables on Windows.

```bash
python -m pip install aider-chat

# Mac/Linux:
export OPENAI_API_BASE=<endpoint>
export OPENAI_API_KEY=<key>

# Windows:
setx OPENAI_API_BASE <endpoint>
setx OPENAI_API_KEY <key>
# ... restart shell after setx commands
```

--------------------------------

### Install and Run Aider via Python Module

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/troubleshooting/aider-not-found.md

Use this command to install aider and then run it, bypassing potential shell PATH issues. This is the recommended method for environments where 'aider' is not found.

```bash
python -m pip install aider-chat
python -m aider
```

--------------------------------

### Install Development Version from GitHub

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/install/optional.md

Upgrade aider to the latest development version by installing directly from its GitHub repository. This ensures you have the most recent features and bug fixes.

```bash
python -m pip install --upgrade git+https://github.com/paul-gauthier/aider.git
```

--------------------------------

### Install Aider and Configure Gemini API Key

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/llms/gemini.md

Install the aider-chat package and set your Gemini API key as an environment variable. This is necessary for aider to authenticate with the Gemini API.

```bash
python -m pip install aider-chat

export GEMINI_API_KEY=<key> # Mac/Linux
setx   GEMINI_API_KEY <key> # Windows, restart shell after setx
```

--------------------------------

### LLM-friendly lint report example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_posts/2024-05-22-linting.md

This example shows how Aider augments raw linter output with AST context for better LLM understanding. It highlights lines with errors within their containing functions, methods, or classes.

```text
app.py:23:36: F821 undefined name 'num'
app.py:41:16: F541 f-string is missing placeholders

app.py:
...⋮...
  6│class LongNum:
  7│    def __init__(self, num):
  8│        """
  9│        Initialize the number.
 10│        """
...⋮...
 19│    def __str__(self):
 20│        """
 21│        Render the number as a string.
 22│        """
 23█        return str(num)
 24│
 25│
 26│@app.route('/subtract/<int:x>/<int:y>')
...⋮...
 38│@app.route('/divide/<int:x>/<int:y>')
 39│def divide(x, y):
 40│    if y == 0:
 41█        return f"Error: Cannot divide by zero"
 42│    else:
 43│        result = x / y
 44│        return str(result)
 45│
...⋮...
```

--------------------------------

### Install Development Version from Local Clone

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/install/optional.md

Install aider in 'editable' mode from a local git clone of the repository. This is useful for developers who want their modifications to take effect immediately without reinstalling.

```bash
python -m pip install -e .
```

--------------------------------

### Setup for Aider Code Editing Benchmark

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/benchmark/README.md

Steps to clone the aider repository, create a scratch directory, clone the exercism repository, copy exercises, and build the Docker container.

```bash
# Clone the aider repo
git clone git@github.com:paul-gauthier/aider.git

# Create the scratch dir to hold benchmarking results inside the main aider dir:
cd aider
mkdir tmp.benchmarks

# Clone the exercism repo
git clone git@github.com:exercism/python.git

# Copy the practice exercises into the benchmark scratch dir
cp -rp python/exercises/practice tmp.benchmarks/exercism-python

# Build the docker container
./benchmark/docker_build.sh
```

--------------------------------

### Run aider locally from source

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/faq.md

Instructions for cloning the aider repository, setting up a virtual environment, installing aider in editable mode, and running the local version.

```bash
# Clone the repository
git clone git@github.com:paul-gauthier/aider.git

# Navigate to the project directory
cd aider

# It's recommended to make a virtual environment

# Install aider in editable/development mode, 
# so it runs from the latest copy of these source files
python -m pip install -e .

# Run the local version of aider
python -m aider
```

--------------------------------

### Aider Startup Announcement Example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/_includes/help.md

This output is printed by Aider at startup and is helpful to include when reporting problems.

```text
Aider v0.37.1-dev
Models: gpt-4o with diff edit format, weak model gpt-3.5-turbo
Git repo: .git with 243 files
Repo-map: using 1024 tokens
```

--------------------------------

### Installing BFCL from PyPI

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

Installs the BFCL evaluation package from PyPI. Ensure you install `bfcl-eval` and not the unrelated `bfcl` project.

```bash
pip install bfcl-eval  # Be careful not to confuse with the unrelated `bfcl` project on PyPI!
```

--------------------------------

### Install aider with pip

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/install.md

Install the aider package using pip. This command installs the latest stable version of aider.

```bash
pip install aider-chat
```

--------------------------------

### Install Required Python Packages

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/base/benchmarks/multiple-eval/README.md

Installs necessary libraries for running the evaluation scripts. Ensure you have Python and pip installed.

```bash
pip install accelerate
pip install attrdict
pip install transformers
pip install pytorch
```

--------------------------------

### Install LiveCodeBench Dependencies

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/livecode_bench/README.md

Installs the necessary Python packages for LiveCodeBench, including flash-attention. Ensure CUDA version is greater than 12.0 before installation.

```bash
pip install -r requirements.txt
pip install flash-attn --no-build-isolation
```

--------------------------------

### Repository Level Code Completion Setup

Source: https://github.com/qwenlm/qwen3-coder/blob/main/examples/Qwen2.5-Coder.md

Prepares input for repository-level code completion by formatting file paths and content with special tokens. This enables the model to understand inter-file relationships.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
device = "cuda" # the device to load the model onto


```

--------------------------------

### Install PlotCraft Environment

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/PlotCraft/README.md

Sets up a Conda environment and installs dependencies for PlotCraft.

```bash
conda create -n plotcraftbench python=3.13
conda activate plotcraftbench
pip install -r requirements.txt
```

--------------------------------

### Voice Command Example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/usage/voice.md

Demonstrates the user interface and interaction flow for the `/voice` command, including visual feedback during recording and the transcription of voice input.

```html
<div class="chat-transcript" markdown="1">

> Aider v0.11.2-dev  
> Added app.py to the chat.  

#### /voice

<blockquote>
<div class="recording-line">
    <p style="display: inline;">Recording, press ENTER when done... <span id="timer">3.5</span>sec</p>
    <div id="progress-bar">
        <div id="progress"></div>
    </div>
</div>
</blockquote>

<style>
    .recording-line > * {
        display: inline-block;
        vertical-align: middle;
    }
    #progress-bar {
        width: 100px;
        height: 20px;
        background-color: #333;
        margin-left: 10px;
    }
    #progress {
        height: 100%;
        background-color: #f0f0f0;
        animation: progress 3s infinite;
    }
    @keyframes progress {
        0% { width: 10%; }
        20% { width: 90%; }
        50% { width: 30%; }
        60% { width: 90%; }
        80% { width: 20%; }
        100% { width: 10%; }

    }
</style>

<script>
window.onload = function() {
    var counter = 0;
    var timer = document.getElementById('timer');
    setInterval(function() {
        counter = (counter + 0.1) % 9;
        timer.textContent = counter.toFixed(1);
    }, 100);
}
</script>

<div style="display: inline-block; position: relative; width: 200px; background-color: #f0f0f0; border-radius: 10px; padding: 10px; color: #333; font-family: 'Comic Sans MS', 'Comic Sans', 'Marker Felt', sans-serif;">
<span style="position: absolute; top: 0; left: 50%; transform: translate(-50%, -100%); width: 0; height: 0; border-left: 10px solid transparent; border-right: 10px solid transparent; border-bottom: 10px solid #f0f0f0;"></span>
“&nbsp;add a factorial endpoint that uses math factorial&nbsp;”
</div>

#### Add a factorial endpoint that uses math.factorial.

To add a factorial endpoint to the Flask application, we need to modify the `app.py` file.

Here are the steps:

1. Add a new route `/fact/<int:x>` to the Flask application.
2. In the function for this route, call `math.factorial(x)` and return the result as a string.

Here is the *edit block* for these changes:

```python
app.py
<<<<<<< HEAD
if __name__ == '__main__':
    print("Starting...")
    app.run()
=======
@app.route('/fact/<int:x>')
def factorial(x):
    result = math.factorial(x)
    return str(result)

if __name__ == '__main__':
    print("Starting...")
    app.run()
>>>>>>> updated
```

> Applied edit to app.py  
> Commit ef9e3e7 aider: Add a factorial endpoint that uses math.factorial.  

</div>
<br/>
<br/>
<br/>
```

--------------------------------

### Install Miniconda

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/base/benchmarks/ExecRepoBench/README.md

Download and install the latest Miniconda3 for Linux x86_64. This is a prerequisite for creating evaluation environments.

```sh
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh
```

--------------------------------

### Benchmark Help Command

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/benchmark/README.md

Displays help information for the benchmark script, listing available arguments for customization.

```bash
./benchmark/benchmark.py --help
```

--------------------------------

### Code Infilling with Qwen2.5-Coder

Source: https://github.com/qwenlm/qwen3-coder/blob/main/examples/Qwen2.5-Coder.md

Demonstrates how to perform code infilling using Qwen2.5-Coder. This snippet shows the process of tokenizing input text, generating code, and printing the output. It includes setup for the tokenizer and model, and the generation process itself.

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-32B")
model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-32B", device_map="auto").eval()

# tokenize the input into tokens
input_text = """<repo_name>library-system
<|file_sep|>library.py
class Book:
    def __init__(self, title, author, isbn, copies):
        self.title = title
        self.author = author
        self.isbn = isbn
        self.copies = copies

    def __str__(self):
        return f"Title: {self.title}, Author: {self.author}, ISBN: {self.isbn}, Copies: {self.copies}"

class Library:
    def __init__(self):
        self.books = []

    def add_book(self, title, author, isbn, copies):
        book = Book(title, author, isbn, copies)
        self.books.append(book)

    def find_book(self, isbn):
        for book in self.books:
            if book.isbn == isbn:
                return book
        return None

    def list_books(self):
        return self.books

<|file_sep|>student.py
class Student:
    def __init__(self, name, id):
        self.name = name
        self.id = id
        self.borrowed_books = []

    def borrow_book(self, book, library):
        if book and book.copies > 0:
            self.borrowed_books.append(book)
            book.copies -= 1
            return True
        return False

    def return_book(self, book, library):
        if book in self.borrowed_books:
            self.borrowed_books.remove(book)
            book.copies += 1
            return True
        return False

<|file_sep|>main.py
from library import Library
from student import Student

def main():
    # Set up the library with some books
    library = Library()
    library.add_book("The Great Gatsby", "F. Scott Fitzgerald", "1234567890", 3)
    library.add_book("To Kill a Mockingbird", "Harper Lee", "1234567891", 2)
    
    # Set up a student
    student = Student("Alice", "S1")
    
    # Student borrows a book
"

model_inputs = tokenizer([input_text], return_tensors="pt")

# Use `max_new_tokens` to control the maximum output length.
eos_token_ids = [151664, 151662, 151659, 151660, 151661, 151662, 151663, 151664, 151645, 151643]
generated_ids = model.generate(model_inputs.input_ids, max_new_tokens=1024, do_sample=False, eos_token_id=eos_token_ids)[0]
# The generated_ids include prompt_ids, so we only need to decode the tokens after prompt_ids.
output_text = tokenizer.decode(generated_ids[len(model_inputs.input_ids[0]):], skip_special_tokens=True)

print(f"Prompt: \n{input_text}\n\nGenerated text: \n{output_text.split('<|file_sep|>')[0]}")
```

--------------------------------

### Repository Level Code Infilling Setup

Source: https://github.com/qwenlm/qwen3-coder/blob/main/examples/Qwen2.5-Coder.md

Illustrates the structure for repository-level code infilling. It shows how to format the input text by concatenating repository name, file separators, file paths, file contents, and the FIM (Fill-in-the-Middle) markers.

```python
input_text = f'''<|repo_name|>{repo_name}
<|file_sep|>{file_path1} 
{file_content1}
<|file_sep|>{file_path2} 
{file_content2}
<|file_sep|>{file_path2} 
<|fim_prefix|>{prefix_code}<|fim_suffix|>{suffix_code}<|fim_middle|>'''
```

--------------------------------

### Sample .env file for Aider Configuration

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/config/dotenv.md

This is a sample .env file that can be used to configure aider. It includes commented-out examples for LLM API keys, model selection, model settings, history file paths, and output preferences. Place this file at the root of your git repo or specify its location using the `aider --env <fname>` command.

```dotenv
##########################################################
# Sample aider .env file.
# Place at the root of your git repo.
# Or use `aider --env <fname>` to specify.
##########################################################

#################
# LLM parameters:
#
# Include xxx_API_KEY parameters and other params needed for your LLMs.
# See https://aider.chat/docs/llms.html for details.

## OpenAI
#OPENAI_API_KEY=

## Anthropic
#ANTHROPIC_API_KEY=

##...

#######
# Main:

## Specify the OpenAI API key
#OPENAI_API_KEY=

## Specify the Anthropic API key
#ANTHROPIC_API_KEY=

## Specify the model to use for the main chat
#AIDER_MODEL=

## Use claude-3-opus-20240229 model for the main chat
#AIDER_OPUS=

## Use claude-3-5-sonnet-20240620 model for the main chat
#AIDER_SONNET=

## Use gpt-4-0613 model for the main chat
#AIDER_4=

## Use gpt-4o-2024-08-06 model for the main chat
#AIDER_4O=

## Use gpt-4o-mini model for the main chat
#AIDER_MINI=

## Use gpt-4-1106-preview model for the main chat
#AIDER_4_TURBO=

## Use gpt-3.5-turbo model for the main chat
#AIDER_35TURBO=

## Use deepseek/deepseek-coder model for the main chat
#AIDER_DEEPSEEK=

#################
# Model Settings:

## List known models which match the (partial) MODEL name
#AIDER_LIST_MODELS=

## Specify the api base url
#OPENAI_API_BASE=

## Specify the api_type
#OPENAI_API_TYPE=

## Specify the api_version
#OPENAI_API_VERSION=

## Specify the deployment_id
#OPENAI_API_DEPLOYMENT_ID=

## Specify the OpenAI organization ID
#OPENAI_ORGANIZATION_ID=

## Specify a file with aider model settings for unknown models
#AIDER_MODEL_SETTINGS_FILE=.aider.model.settings.yml

## Specify a file with context window and costs for unknown models
#AIDER_MODEL_METADATA_FILE=.aider.model.metadata.json

## Verify the SSL cert when connecting to models (default: True)
#AIDER_VERIFY_SSL=true

## Specify what edit format the LLM should use (default depends on model)
#AIDER_EDIT_FORMAT=

## Specify the model to use for commit messages and chat history summarization (default depends on --model)
#AIDER_WEAK_MODEL=

## Only work with models that have meta-data available (default: True)
#AIDER_SHOW_MODEL_WARNINGS=true

## Suggested number of tokens to use for repo map, use 0 to disable (default: 1024)
#AIDER_MAP_TOKENS=

## Control how often the repo map is refreshed (default: auto)
#AIDER_MAP_REFRESH=auto

## Enable caching of prompts (default: False)
#AIDER_CACHE_PROMPTS=false

## Number of times to ping at 5min intervals to keep prompt cache warm (default: 0)
#AIDER_CACHE_KEEPALIVE_PINGS=false

## Multiplier for map tokens when no files are specified (default: 2)
#AIDER_MAP_MULTIPLIER_NO_FILES=true

## Maximum number of tokens to use for chat history. If not specified, uses the model's max_chat_history_tokens.
#AIDER_MAX_CHAT_HISTORY_TOKENS=

## Specify the .env file to load (default: .env in git root)
#AIDER_ENV_FILE=.env

################
# History Files:

## Specify the chat input history file (default: .aider.input.history)
#AIDER_INPUT_HISTORY_FILE=.aider.input.history

## Specify the chat history file (default: .aider.chat.history.md)
#AIDER_CHAT_HISTORY_FILE=.aider.chat.history.md

## Restore the previous chat history messages (default: False)
#AIDER_RESTORE_CHAT_HISTORY=false

## Log the conversation with the LLM to this file (for example, .aider.llm.history)
#AIDER_LLM_HISTORY_FILE=

##################
# Output Settings:

## Use colors suitable for a dark terminal background (default: False)
#AIDER_DARK_MODE=false

## Use colors suitable for a light terminal background (default: False)
#AIDER_LIGHT_MODE=false

## Enable/disable pretty, colorized output (default: True)
#AIDER_PRETTY=true

## Enable/disable streaming responses (default: True)
#AIDER_STREAM=true

## Set the color for user input (default: #00cc00)
#AIDER_USER_INPUT_COLOR=#00cc00

## Set the color for tool output (default: None)
#AIDER_TOOL_OUTPUT_COLOR=

```

--------------------------------

### Download LiveCodeBench Dataset

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/reasoning/livecode_bench_cot/README.md

Use the Hugging Face CLI to download the code generation lite dataset. Ensure you have the CLI installed and configured.

```bash
huggingface-cli download --repo-type dataset livecodebench/code_generation_lite --local-dir code_generation_lite
```

--------------------------------

### Aider Interactive Session Example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/usage.md

An example of an Aider interactive session where the user requests a program to calculate factorials and Aider provides a response.

```bash
$ aider factorial.py

Aider v0.37.1-dev
Models: gpt-4o with diff edit format, weak model gpt-3.5-turbo
Git repo: .git with 258 files
Repo-map: using 1024 tokens
Use /help to see in-chat commands, run with --help to see cmd line args
───────────────────────────────────────────────────────────────────────
> Make a program that asks for a number and prints its factorial

...
```

--------------------------------

### Aider Announce Lines Example

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/troubleshooting/edit-errors.md

Observe the edit format used by the model in the announce lines. This example shows the 'diff' edit format.

```text
Aider v0.50.2-dev
Models: claude-3-5-sonnet-20240620 with ♾️ diff edit format
```

--------------------------------

### Run Base Test with Qwen3-Coder

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/bird-spider/README.md

Execute the base test suite. Provide the model path, tensor-parallel size, and output directory as arguments.

```bash
bash test.sh Qwen/Qwen2.5-Coder-7B-Instruct 2 outputs
```

--------------------------------

### Install WandB for Evaluation Logging

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/tool_calling_eval/berkeley-function-call-leaderboard/README.md

Install the necessary package to log evaluation results to WandB artifacts. Ensure the WANDB_BFCL_PROJECT environment variable is set.

```bash
pip install -e.[wandb]
```

--------------------------------

### Install aider-chat with Pipx

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/aider/aider/website/docs/install/pipx.md

Install aider-chat globally using pipx. This creates an isolated environment for aider, preventing dependency conflicts with your Python projects.

```bash
pipx install aider-chat
```

--------------------------------

### Install Dependencies for BigCodeBench

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/base/readme.md

Installs all necessary Python dependencies for the BigCodeBench evaluation from a requirements file. Ensure the conda environment is activated before running this command.

```bash
pip install -r requirements/bigcodebench-eval.txt
```

--------------------------------

### CodeArena Data Example (JSON)

Source: https://github.com/qwenlm/qwen3-coder/blob/main/qwencoder-eval/instruct/CodeArena/README.md

An example of a data entry from the CodeArena dataset, illustrating a user query and a model's response for a 'validation' task.

```json
{
    "id": "60670a8d9b1e39dd845fb1639d0d8b86",
    "messages": "[{'role': 'user', 'content': 'move gradient chart to the right side\n\n  <CardContent>\n                <Typography color=\"textSecondary\" gutterBottom>\n                  Average Sentiment\n                </Typography>\n                  <Box display=\"flex\" alignItems=\"flex-end\">\n                    <NegativeIcon\n                      style={{ marginRight: \"10px\", color: \"red\" }}
                    />\n                    <Box textAlign=\"center\">\n                      <Typography variant=\"h5\">\n                        {workspaceConversationsDashboardStats.averageSentiment.toFixed(\n                          2\n                        )}\n                      </Typography>\n                      <GaugeContainer\n                        width={100}\n                        height={100}\n                        innerRadius={25} // Adjust for smaller gauge\n                        startAngle={-110}\n                        endAngle={110}\n                        value={\n                          (workspaceConversationsDashboardStats.averageSentiment +\n                            1) *\n                          50\n                        }\n                      >\n                        <GaugeReferenceArc\n                          style={{ stroke: \"url(#gradient)\", strokeWidth: 3 }} // Adjust stroke width\n                        />\n                        <defs>\n                          <linearGradient\n                            id=\"gradient\"\n                            x1=\"0%\"\n                            y1=\"0%\"\n                            x2=\"100%\"\n                            y2=\"0%\"\n                          >\n                            <stop\n                              offset=\"0%\"\n                              style={{ stopColor: \"red\", stopOpacity: 1 }}
                            />\n                            <stop\n                              offset=\"100%\"\n                              style={{ stopColor: \"green\", stopOpacity: 1 }}
                            />\n                          </linearGradient>\n                        </defs>\n                        <GaugePointer />\n                      </GaugeContainer>\n                    </Box>\n                    <PositiveIcon\n                      style={{ marginLeft: \"10px\", color: \"green\" }}
                    />\n                  </Box>\n              </CardContent>\n'}]",
    "gpt-4-turbo-2024-04-09_response": "<CardContent>\n  <Typography color=\"textSecondary\" gutterBottom>\n    Average Sentiment\n  </Typography>\n  <Box display=\"flex\" alignItems=\"flex-end\" justifyContent=\"flex-end\">\n    <NegativeIcon\n      style={{ marginRight: \"10px\", color: \"red\" }}
    />\n    <Box textAlign=\"center\">\n      <Typography variant=\"h5\">\n        {workspaceConversationsDashboardStats.averageSentiment.toFixed(\n          2\n        )}\n      </Typography>\n      <GaugeContainer\n        width={100}\n        height={100}\n        innerRadius={25} // Adjust for smaller gauge\n        startAngle={-110}\n        endAngle={110}\n        value={\n          (workspaceConversationsDashboardStats.averageSentiment +\n            1) *\n          50\n        }\n      >\n        <GaugeReferenceArc\n          style={{ stroke: \"url(#gradient)\", strokeWidth: 3 }} // Adjust stroke width\n        />\n        <defs>\n          <linearGradient\n            id=\"gradient\"\n            x1=\"0%\"\n            y1=\"0%\"\n            x2=\"100%\"\n            y2=\"0%\"\n          >\n            <stop\n              offset=\"0%\"\n              style={{ stopColor: \"red\", stopOpacity: 1 }}
            />\n            <stop\n              offset=\"100%\"\n              style={{ stopColor: \"green\", stopOpacity: 1 }}
            />\n          </linearGradient>\n        </defs>\n        <GaugePointer />\n      </GaugeContainer>\n    </Box>\n    <PositiveIcon\n      style={{ marginLeft: \"10px\", color: \"green\" }}
    />\n  </Box>\n</CardContent>",
    "difficulty": "easy",
    "programming_language": ["JavaScript"]
}
```