### Set up Conda Environment and Install Dependencies

Source: https://github.com/huggingface/open-r1/blob/main/slurm/README.md

Create and activate a Conda environment, then install PyTorch with specific CUDA support and the SGLang library. Adjust the CUDA version (cu124) as needed for your system.

```bash
conda create -n sglang124 python=3.11
conda activate sglang124

pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cu124

pip install sgl-kernel --force-reinstall --no-deps
pip install "sglang[all]>=0.4.2.post4" --find-links https://flashinfer.ai/whl/cu124/torch2.5/flashinfer/
```

--------------------------------

### Install Open R1 and Dependencies

Source: https://context7.com/huggingface/open-r1/llms.txt

Installs the Open R1 project using uv, including base dependencies and optional groups like 'dev'. Logs in to Hugging Face Hub and Weights & Biases for authentication.

```bash
uv venv openr1 --python 3.11 && source openr1/bin/activate
uv pip install vllm==0.8.5.post1
uv pip install setuptools && uv pip install flash-attn --no-build-isolation
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e ".[dev]"
huggingface-cli login
wandb login
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Install the project's development dependencies using `uv pip install -e .[dev]`. Set `GIT_LFS_SKIP_SMUDGE=1` to skip downloading large files during installation.

```shell
GIT_LFS_SKIP_SMUDGE=1 uv pip install -e ".[dev]"
```

--------------------------------

### Install vLLM and FlashAttention

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Install vLLM version 0.8.5.post1 and FlashAttention. Ensure PyTorch version 2.6.0 is used as vLLM binaries are compiled for it.

```shell
uv pip install vllm==0.8.5.post1
```

```shell
uv pip install setuptools && uv pip install flash-attn --no-build-isolation
```

--------------------------------

### Install dependencies for code reward function

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Install the necessary dependencies to enable the 'code' reward function, which is used for executing generated code against test cases.

```shell
uv pip install -e '.[code]'
```

--------------------------------

### Start vLLM Server for Training

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Launch the vLLM server on a specific CUDA device to serve models for training. Ensure the model path is correct.

```shell
CUDA_VISIBLE_DEVICES=0 trl vllm-serve --model Qwen/Qwen2.5-1.5B-Instruct
```

--------------------------------

### Install morphcloud Package

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Install the morphcloud package using pip. This is a prerequisite for using Morph provider services.

```shell
pip install morphcloud
```

--------------------------------

### Launch SFT Training Job on Slurm

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Example command for running Supervised Fine-Tuning (SFT) on a Slurm cluster using 1 node with 8 GPUs. Adjust parameters as needed.

```shell
sbatch --job-name=open_r1 --nodes=1 slurm/train.slurm --model OpenR1-Distill-7B --task sft --config distill --accelerator zero3
```

--------------------------------

### Launch GRPO Training Job on Slurm

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Example command for running GRPO on a Slurm cluster with mixed data and tensor parallelism. This example uses 1+1 nodes.

```shell
sbatch --job-name=open_r1 --nodes=2 slurm/train.slurm --model Qwen2.5-1.5B-Instruct --task grpo --config demo --accelerator zero2 --dp 4 --tp 2
```

--------------------------------

### Start vLLM Server for Synthetic Data Generation

Source: https://context7.com/huggingface/open-r1/llms.txt

This command starts a vLLM server on Slurm nodes for generating synthetic data. It specifies the Hugging Face dataset, model, and output dataset.

```bash
# Start the vLLM server (on 2×8 H100 nodes via Slurm)
sbatch slurm/generate.slurm \
    --hf-dataset AI-MO/NuminaMath-TIR \
    --temperature 0.6 \
    --prompt-column problem \
    --model deepseek-ai/DeepSeek-R1 \
    --hf-output-dataset myorg/r1-math-distilled
```

--------------------------------

### Generate Data with DeepSeek-R1 Model

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Command to generate data using the larger DeepSeek-R1 model, requiring specific dependency installation and SLURM job submission. This example is intended for multi-node, multi-GPU setups.

```shell
pip install https://wheels.vllm.ai/221d388cc5a836fa189305785ed7e887cea8b510/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl --extra-index-url https://download.pytorch.org/whl/cu121
```

```shell
uv pip install "distilabel[vllm,ray,openai]>=1.5.2"
```

```shell
sbatch slurm/generate.slurm     --hf-dataset AI-MO/NuminaMath-TIR     --temperature 0.6     --prompt-column problem     --model deepseek-ai/DeepSeek-R1     --hf-output-dataset username/r1-dataset
```

--------------------------------

### GRPOScriptArguments Configuration Example

Source: https://context7.com/huggingface/open-r1/llms.txt

Configuration for GRPO training using a YAML file, specifying model, dataset, reward functions, and training parameters.

```yaml
model_name_or_path: deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
dataset_name: open-r1/OpenR1-Math-220k
dataset_prompt_column: problem
system_prompt: "You are a helpful AI Assistant..."

reward_funcs:
  - accuracy      # LaTeX math verification
  - format        # <think>/<answer> tag structure
  - tag_count     # Partial credit for correct tag usage

reward_weights: [1.0, 1.0, 1.0]

num_generations: 16
max_completion_length: 2048
learning_rate: 1.0e-06
temperature: 0.7
bf16: true
use_vllm: true
push_to_hub: true
hub_model_id: DeepSeek-R1-Distill-Qwen-1.5B-GRPO

```

--------------------------------

### Check Git LFS Installation

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Verify if Git Large File Storage (LFS) is installed on the system. If not, provide instructions to install it.

```shell
git-lfs --version
```

```shell
sudo apt-get install git-lfs
```

--------------------------------

### GRPO multi-node training on Slurm cluster

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Example Slurm script command for multi-node GRPO training, where one node runs the vLLM server and others run training. This example uses data parallelism.

```shell
sbatch --nodes=2 slurm/train.slurm --model Qwen2.5-1.5B-Instruct --task grpo --config demo --accelerator zero2 --dp 8 --tp 1
```

--------------------------------

### Install IOI Package on Piston Worker

Source: https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md

Send a POST request to a running Piston worker to install the IOI package. Ensure the worker is accessible and the port is correct.

```bash
curl -X POST http://ip-10-53-86-146:1234/api/v2/packages -H "Content-Type: application/json" -d '{"language": "cms_ioi", "version": "1.0.0"}'
```

--------------------------------

### Install CodeForces Package on Piston Worker

Source: https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md

Send a POST request to a running Piston worker to install the CodeForces package. Ensure the worker is accessible and the port is correct.

```bash
curl -X POST http://ip-10-53-86-146:1234/api/v2/packages -H "Content-Type: application/json" -d '{"language": "codeforces", "version": "1.0.0"}'
```

--------------------------------

### Run SGLang Server with SLURM

Source: https://github.com/huggingface/open-r1/blob/main/slurm/README.md

Submit a SLURM job to start the SGLang server. Specify the model path and the Conda environment name for the server.

```bash
sbatch slurm/serve_r1.slurm -m "/fsx/deepseek-r1-checkpoint" -e "sglang124"
```

--------------------------------

### Install IOI Package on Local Docker Worker

Source: https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md

Use curl to install the IOI package on a Piston worker running locally via Docker. Ensure the port matches the one configured in the Docker run command.

```bash
curl -X POST http://localhost:2000/api/v2/packages -H "Content-Type: application/json" -d '{"language": "cms_ioi", "version": "1.0.0"}'
```

--------------------------------

### Get Reward Functions Programmatically

Source: https://context7.com/huggingface/open-r1/llms.txt

Demonstrates how to build GRPOScriptArguments programmatically and retrieve a list of reward function callables using `get_reward_funcs`. This is useful for custom training loops or evaluation.

```python
from open_r1.rewards import get_reward_funcs
from open_r1.configs import GRPOScriptArguments

# Build arguments programmatically
script_args = GRPOScriptArguments(
    dataset_name="open-r1/OpenR1-Math-220k",
    reward_funcs=["accuracy", "format", "tag_count"],
    cosine_min_value_wrong=0.0,
    cosine_max_value_wrong=-0.5,
    cosine_min_value_correct=0.5,
    cosine_max_value_correct=1.0,
    cosine_max_len=1000,
    repetition_n_grams=3,
    repetition_max_penalty=-1.0,
    code_language="python",
    parallel_code_exec_per_proc=2,
    code_provider="e2b",
    code_eval_test_batch_size=1,
    code_eval_scoring_mode="weighted_sum",
    max_completion_len=16384,
    soft_punish_cache=4096,
)

reward_funcs = get_reward_funcs(script_args)
# reward_funcs is a list: [accuracy_reward, format_reward, tag_count_reward]

# Call rewards directly (as GRPOTrainer would)
completions = [[{"role": "assistant", "content": "<think>\n2+2=4\n</think>\n<answer>\n\\boxed{4}\n</answer>"}]]
solution = ["4"]

scores = reward_funcs[0](completions=completions, solution=solution)
# scores -> [1.0]  (correct answer verified via LaTeX math_verify)

```

--------------------------------

### Install CodeForces Package on Local Docker Worker

Source: https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md

Use curl to install the CodeForces package on a Piston worker running locally via Docker. Ensure the port matches the one configured in the Docker run command.

```bash
curl -X POST http://localhost:2000/api/v2/packages -H "Content-Type: application/json" -d '{"language": "codeforces", "version": "1.0.0"}'
```

--------------------------------

### Override Training Configuration Parameters

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Modify training parameters by appending them to the `accelerate launch` command when using a YAML config. This example changes the base model and the Hugging Face Hub model ID.

```shell
# Change the base model to a smaller variant
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml \
    --model_name_or_path Qwen/Qwen3-0.6B-Base \
    --hub_model_id OpenR1-Distill-0.6B \
    --output_dir data/OpenR1-Distill-0.6B
```

--------------------------------

### SFT distillation recipe for Qwen 7B models

Source: https://github.com/huggingface/open-r1/blob/main/README.md

This command initiates the SFT distillation process to reproduce reasoning capabilities. It uses a specific configuration file for distillation.

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/zero3.yaml \
    src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml
```

--------------------------------

### Run MATH-500 Benchmark with LightEval

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this command to reproduce MATH-500 results. Set VLLM_WORKER_MULTIPROC_METHOD to spawn. Adjust NUM_GPUS based on model size. Ensure the model name and output directory are correctly specified.

```shell
export VLLM_WORKER_MULTIPROC_METHOD=spawn
NUM_GPUS=1 # Set to 8 for 32B and 70B models
MODEL=deepseek-ai/{model_name}
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "lighteval|math_500|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Launch Training with Accelerate

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Initiate distributed training using Accelerate. This command specifies the number of processes, configuration files, and the main training script.

```shell
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7 ACCELERATE_LOG_LEVEL=info \
    accelerate launch --config_file recipes/accelerate_configs/zero2.yaml --num_processes=7 \
    src/open_r1/grpo.py --config recipes/Qwen2.5-1.5B-Instruct/grpo/config_demo_code.yaml
```

--------------------------------

### Train Model via Command Line

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Perform Supervised Fine-Tuning (SFT) on a dataset using `accelerate launch` with DeepSpeed ZeRO-3 configuration. This command specifies the model, dataset, training parameters, and output directory.

```shell
# Train via command line
accelerate launch --config_file=recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --model_name_or_path open-r1/Qwen2.5-Math-7B-RoPE-300k \
    --dataset_name open-r1/Mixture-of-Thoughts \
    --dataset_config all \
    --eos_token '<|im_end|>' \
    --learning_rate 4.0e-5 \
    --num_train_epochs 5 \
    --max_seq_length 32768 \
    --per_device_train_batch_size 2 \
    --gradient_checkpointing \
    --bf16 \
    --use_liger_kernel \
    --output_dir data/OpenR1-Distill-7B
```

--------------------------------

### Train Model via YAML Config

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Initiate model training using `accelerate launch` by specifying a YAML configuration file. This method allows for modular configuration of training parameters.

```shell
# Train via YAML config
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml
```

--------------------------------

### Run AIME 2024 Benchmark with LightEval

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this command to reproduce AIME 2024 results. Adjust NUM_GPUS based on model size. Ensure the model name and output directory are correctly specified.

```shell
NUM_GPUS=1 # Set to 8 for 32B and 70B models
MODEL=deepseek-ai/{model_name}
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "lighteval|aime24|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Login to Hugging Face and Weights and Biases

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Authenticate with Hugging Face Hub and Weights and Biases using their respective CLI commands.

```shell
huggingface-cli login
```

```shell
wandb login
```

--------------------------------

### Configure Morph API Key

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Add your Morph API key to the .env file for authentication. Replace 'YOUR_MORPH_API_KEY' with your actual key.

```shell
MORPH_API_KEY="YOUR_MORPH_API_KEY"
```

--------------------------------

### Set Piston Endpoints Environment Variable

Source: https://github.com/huggingface/open-r1/blob/main/slurm/piston/README.md

Export this environment variable to allow the main script to automatically discover Piston workers. This is useful for distributed setups.

```bash
export PISTON_ENDPOINTS=slurm
```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use `uv` to create a new virtual environment named 'openr1' with Python 3.11, activate it, and upgrade pip.

```shell
uv venv openr1 --python 3.11 && source openr1/bin/activate && uv pip install --upgrade pip
```

--------------------------------

### Evaluate Model with make (Data Parallelism)

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use the 'make evaluate' command with Data Parallelism. Specify model, task, parallelism type, and number of GPUs.

```shell
make evaluate MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-32B TASK=aime24 PARALLEL=data NUM_GPUS=8
```

--------------------------------

### Override Weights and Biases Settings

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Customize Weights and Biases logging settings, such as the entity, project, and run name, by appending these arguments to the `accelerate launch` command.

```shell
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml
    --wandb_entity huggingface --wandb_project open-r1 --run_name Qwen2.5-1.5B-GRPO
```

--------------------------------

### Generate Data with Smol Distilled R1 Model

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this Python script to generate data from a smol distilled R1 model. Ensure you have the necessary dependencies installed and replace 'username' with your organization or user name.

```shell
uv pip install "distilabel[vllm]>=1.5.2"
```

```python
from datasets import load_dataset
from distilabel.models import vLLM
from distilabel.pipeline import Pipeline
from distilabel.steps.tasks import TextGeneration

prompt_template = """
You will be given a problem. Please reason step by step, and put your final answer within \boxed{}:
{{ instruction }} """

dataset = load_dataset("AI-MO/NuminaMath-TIR", split="train").select(range(10))

model_id = "deepseek-ai/DeepSeek-R1-Distill-Qwen-7B"  # Exchange with another smol distilled r1

with Pipeline(
    name="distill-qwen-7b-r1",
    description="A pipeline to generate data from a distilled r1 model",
) as pipeline:

    llm = vLLM(
        model=model_id,
        tokenizer=model_id,
        extra_kwargs={
            "tensor_parallel_size": 1,
            "max_model_len": 8192,
        },
        generation_kwargs={
            "temperature": 0.6,
            "max_new_tokens": 8192,
        },
    )
    prompt_column = "problem"
    text_generation = TextGeneration(
        llm=llm, 
        template=prompt_template,
        num_generations=4,
        input_mappings={"instruction": prompt_column} if prompt_column is not None else {}
    )


if __name__ == "__main__":
    distiset = pipeline.run(dataset=dataset)
    distiset.push_to_hub(repo_id="username/numina-deepseek-r1-qwen-7b")

```

--------------------------------

### Evaluate GPQA Diamond with vLLM

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this command to reproduce GPQA Diamond benchmark results with vLLM. Ensure VLLM_WORKER_MULTIPROC_METHOD is set to 'spawn'. Adjust NUM_GPUS based on model size.

```shell
export VLLM_WORKER_MULTIPROC_METHOD=spawn
NUM_GPUS=1 # Set to 8 for 32B and 70B models
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "lighteval|gpqa:diamond|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Supervised Fine-Tuning (SFT) with Accelerate

Source: https://context7.com/huggingface/open-r1/llms.txt

Trains a causal language model using SFTTrainer on a specified dataset. Supports various configurations via command-line arguments or YAML recipes, including model path, dataset, training parameters, and output directory.

```bash
accelerate launch --config_file=recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --model_name_or_path open-r1/Qwen2.5-Math-7B-RoPE-300k \
    --dataset_name open-r1/Mixture-of-Thoughts \
    --dataset_config all \
    --eos_token '<|im_end|>' \
    --learning_rate 4.0e-5 \
    --num_train_epochs 5 \
    --max_seq_length 32768 \
    --per_device_train_batch_size 2 \
    --gradient_checkpointing \
    --bf16 \
    --use_liger_kernel \
    --output_dir data/OpenR1-Distill-7B \
    --push_to_hub
```

```bash
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml
```

```bash
accelerate launch --config_file recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --config recipes/OpenR1-Distill-7B/sft/config_distill.yaml \
    --model_name_or_path Qwen/Qwen3-0.6B-Base \
    --hub_model_id OpenR1-Distill-0.6B \
    --output_dir data/OpenR1-Distill-0.6B
```

--------------------------------

### Tag Count Reward Function

Source: https://context7.com/huggingface/open-r1/llms.txt

Provides partial credit (increments of 0.25) for each correctly placed tag (<think>, </think>, <answer>, </answer>). Useful for guiding models that are still learning the required output structure.

```python
from open_r1.rewards import tag_count_reward

completions = [
    [{"role": "assistant", "content": "<think>\nreasoning\n</think>\n<answer>\n42\n</answer>"}],  # all 4 tags
    [{"role": "assistant", "content": "<think>\nreasoning\n</think>\nno answer tags"}],  # 2 tags
]

rewards = tag_count_reward(completions=completions)
# rewards -> [1.0, 0.5]

```

--------------------------------

### Evaluate LiveCodeBench with vLLM

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Reproduce LiveCodeBench code generation benchmark results using vLLM. Configure NUM_GPUS or data_parallel_size based on model and desired speed. The MODEL variable should be set to 'deepseek-ai/{model_name}'.

```shell
NUM_GPUS=1 # Set to 8 for 32B and 70B models, or data_parallel_size=8 with the smaller models for speed
MODEL=deepseek-ai/{model_name}
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,data_parallel_size=$NUM_GPUS,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "extended|lcb:codegeneration|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Get Soft Overlong Punishment Reward Function

Source: https://context7.com/huggingface/open-r1/llms.txt

Creates a reward function that penalizes completions exceeding a maximum token length. Completions within a cache zone receive a graduated penalty, while those past the limit receive -1.0.

```python
from open_r1.rewards import get_soft_overlong_punishment

soft_penalty = get_soft_overlong_punishment(max_completion_len=2048, soft_punish_cache=512)

# completion_ids are token ID lists (lengths: 1000, 1700, 2200)
completion_ids = [list(range(1000)), list(range(1700)), list(range(2200))]

rewards = soft_penalty(completion_ids=completion_ids)
# rewards -> [0.0, -0.375, -1.0]
# 1000 < 1536 (2048-512) → 0.0
# 1700 in [1536, 2048] → graduated penalty
# 2200 > 2048 → -1.0
```

--------------------------------

### Evaluate Model with make (Single GPU)

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use the 'make evaluate' command for single-GPU model evaluation. Specify the model and task.

```shell
make evaluate MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-32B TASK=aime24
```

--------------------------------

### Run LiveCodeBench Benchmarks with Python Script

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Run LiveCodeBench benchmarks using the provided Python script. Replace {model_id} with the relevant model identifier.

```shell
python scripts/run_benchmarks.py --model-id {model_id}  --benchmarks lcb
```

--------------------------------

### Evaluate Models with LightEval and vLLM

Source: https://context7.com/huggingface/open-r1/llms.txt

Evaluates trained models on standard reasoning benchmarks using LightEval and vLLM. Set the VLLM_WORKER_MULTIPROC_METHOD environment variable and specify model arguments, output directory, and the benchmark task.

```bash
export VLLM_WORKER_MULTIPROC_METHOD=spawn
MODEL=open-r1/OpenR1-Distill-7B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,\n gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

# AIME 2024
lighteval vllm "$MODEL_ARGS" "lighteval|aime24|0|0" \
    --use-chat-template --output-dir $OUTPUT_DIR

# MATH-500
lighteval vllm "$MODEL_ARGS" "lighteval|math_500|0|0" \
    --use-chat-template --output-dir $OUTPUT_DIR

# GPQA Diamond
lighteval vllm "$MODEL_ARGS" "lighteval|gpqa:diamond|0|0" \
    --use-chat-template --output-dir $OUTPUT_DIR

# LiveCodeBench (code generation)
lighteval vllm "$MODEL_ARGS" "extended|lcb:codegeneration|0|0" \
    --use-chat-template --output-dir $OUTPUT_DIR
```

```bash
# Or use the Makefile shorthand with data parallelism
make evaluate MODEL=$MODEL TASK=aime24 PARALLEL=data NUM_GPUS=8
```

--------------------------------

### Evaluate Model with make (Tensor Parallelism)

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use the 'make evaluate' command with Tensor Parallelism. Specify model, task, parallelism type, and number of GPUs.

```shell
make evaluate MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-32B TASK=aime24 PARALLEL=tensor NUM_GPUS=8
```

--------------------------------

### Launch Slurm Jobs for Benchmarks

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Alternatively, launch Slurm jobs using this Python script. Specify the model ID and the benchmark to run.

```shell
python scripts/run_benchmarks.py --model-id {model_id}  --benchmarks aime24
```

```shell
python scripts/run_benchmarks.py --model-id {model_id}  --benchmarks math_500
```

--------------------------------

### Launch IOI Reward Function Training

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Execute the training script for the IOI reward function using Accelerate. This command specifies the number of processes and configuration files.

```shell
ACCELERATE_LOG_LEVEL=info accelerate launch --config_file recipes/accelerate_configs/zero2.yaml \
    --num_processes=7 src/open_r1/grpo.py \
    --config recipes/Qwen2.5-1.5B-Instruct/grpo/config_demo_code_ioi.yaml
```

--------------------------------

### Evaluate Model with Tensor Parallelism using lighteval

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Command to evaluate a large model using lighteval with tensor parallelism across multiple GPUs. Requires specifying the number of GPUs and model details.

```shell
NUM_GPUS=8
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-32B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,tensor_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
TASK=aime24
OUTPUT_DIR=data/evals/$MODEL

export VLLM_WORKER_MULTIPROC_METHOD=spawn
lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Configure E2B API key for sandboxes

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Set up your E2B API key in a .env file to use E2B sandboxes for code execution. This is required for the code reward function.

```dotenv
E2B_API_KEY="e2b_xxx"
```

--------------------------------

### Launch Training Job on Slurm

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this command to submit a training job to a Slurm cluster. Replace placeholders with your specific model, task, config, and accelerator.

```shell
sbatch --job-name=open_r1 --nodes=1 slurm/train.slurm --model {model_name} --task {task} --config {config_suffix} --accelerator {accelerator}
```

--------------------------------

### Fine-tune with custom chat template and EOS token

Source: https://github.com/huggingface/open-r1/blob/main/README.md

This command is used for fine-tuning models like Llama when a custom chat template and EOS token are necessary. The chat template should be provided via a file path.

```shell
accelerate launch --config_file=recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --model_name_or_path meta-llama/Llama-3.2-1B \
    --chat_template "$(cat llama_chat_template.jinja)" \
    --eos_token '<|eot_id|>' \
    --dataset_name open-r1/Mixture-of-Thoughts \
    --dataset_config all \
    --learning_rate 4.0e-5 \
    --num_train_epochs 1 \
    --max_seq_length 32768 \
    --per_device_train_batch_size 16 \
    --gradient_checkpointing \
    --bf16 \
    --use_liger_kernel \
    --output_dir data/Llama-3.2-1B-Open-R1-Distill
```

--------------------------------

### Train OpenR1 Distill 7B Model

Source: https://github.com/huggingface/open-r1/blob/main/recipes/README.md

Use this command to train the OpenR1 Distill 7B model with Zero3 acceleration.

```bash
sbatch --nodes=1 slurm/train.slurm --model OpenR1-Distill-7B --task sft --config distill --accelerator zero3
```

--------------------------------

### Configure IOI Provider as Morph

Source: https://github.com/huggingface/open-r1/blob/main/README.md

In your configuration, set 'ioi_provider' to 'morph' to use Morph as the execution provider for IOI problems.

```yaml
ioi_provider: "morph"
```

--------------------------------

### Evaluate Model on Single GPU with lighteval

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Command to evaluate a model using lighteval on a single GPU. Ensure VLLM worker multiprocessing method is set to 'spawn'.

```shell
export VLLM_WORKER_MULTIPROC_METHOD=spawn
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
OUTPUT_DIR=data/evals/$MODEL

# AIME 2024
TASK=aime24
lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

# MATH-500
TASK=math_500
lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

# GPQA Diamond
TASK=gpqa:diamond
lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR

# LiveCodeBench
lighteval vllm $MODEL_ARGS "extended|lcb:codegeneration|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Build Distilabel Pipeline for Synthetic Data Generation

Source: https://context7.com/huggingface/open-r1/llms.txt

Builds a Ray-based Distilabel pipeline to generate synthetic data from a Hugging Face dataset using a vLLM server. The generated dataset is then pushed to the Hugging Face Hub.

```python
from datasets import load_dataset
from open_r1.generate import build_distilabel_pipeline

dataset = load_dataset("AI-MO/NuminaMath-TIR", split="train").select(range(100))

pipeline = build_distilabel_pipeline(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    base_url="http://localhost:8000/v1",
    prompt_column="problem",
    prompt_template="Please reason step by step, and put your final answer within \boxed{}:\n{{ instruction }}",
    temperature=0.6,
    top_p=0.95,
    max_new_tokens=8192,
    num_generations=4,
    input_batch_size=64,
    client_replicas=2,
)

distiset = pipeline.run(dataset=dataset, use_cache=False)
distiset.push_to_hub("myorg/numina-r1-distilled", private=False)
```

--------------------------------

### build_distilabel_pipeline()

Source: https://context7.com/huggingface/open-r1/llms.txt

Builds a Ray-based Distilabel pipeline that connects to a vLLM server and generates multiple completions per prompt from a Hugging Face dataset. Used to distill reasoning traces from large R1 models.

```APIDOC
## `build_distilabel_pipeline()` — Synthetic Data Generation Pipeline

Builds a Ray-based Distilabel pipeline that connects to a vLLM server and generates multiple completions per prompt from a Hugging Face dataset. Used to distill reasoning traces from large R1 models.

```bash
# Start the vLLM server (on 2×8 H100 nodes via Slurm)
sbatch slurm/generate.slurm \
    --hf-dataset AI-MO/NuminaMath-TIR \
    --temperature 0.6 \
    --prompt-column problem \
    --model deepseek-ai/DeepSeek-R1 \
    --hf-output-dataset myorg/r1-math-distilled
```

```python
from datasets import load_dataset
from open_r1.generate import build_distilabel_pipeline

dataset = load_dataset("AI-MO/NuminaMath-TIR", split="train").select(range(100))

pipeline = build_distilabel_pipeline(
    model="deepseek-ai/DeepSeek-R1-Distill-Qwen-7B",
    base_url="http://localhost:8000/v1",
    prompt_column="problem",
    prompt_template="Please reason step by step, and put your final answer within \boxed{}:\n{{ instruction }}",
    temperature=0.6,
    top_p=0.95,
    max_new_tokens=8192,
    num_generations=4,
    input_batch_size=64,
    client_replicas=2,
)

distiset = pipeline.run(dataset=dataset, use_cache=False)
distiset.push_to_hub("myorg/numina-r1-distilled", private=False)
```
```

--------------------------------

### Set CodeForces Test Cases Folder in .env

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Store the path to the downloaded CodeForces test cases in the .env file for the system to access.

```shell
CF_TESTS_FOLDER=PATH_TO_SAVE_TESTCASES
```

--------------------------------

### Evaluate Model with Data Parallelism using lighteval

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Command to evaluate a model using lighteval with data parallelism across multiple GPUs. Specify the number of GPUs and model details.

```shell
NUM_GPUS=8
MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B
MODEL_ARGS="model_name=$MODEL,dtype=bfloat16,data_parallel_size=$NUM_GPUS,max_model_length=32768,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95}"
TASK=aime24
OUTPUT_DIR=data/evals/$MODEL

lighteval vllm $MODEL_ARGS "lighteval|$TASK|0|0" \
    --use-chat-template \
    --output-dir $OUTPUT_DIR
```

--------------------------------

### Configure IOI Provider as Piston

Source: https://github.com/huggingface/open-r1/blob/main/README.md

In your configuration, set 'ioi_provider' to 'piston' to use Piston as the execution provider for IOI problems.

```yaml
ioi_provider: "piston"
```

--------------------------------

### Launch CodeForces Training Job

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Submit a slurm job for CodeForces training. This command specifies the model, task, configuration, accelerator, and data parallelism settings.

```shell
sbatch --job-name=cf-grpo --nodes=2 slurm/train.slurm --model Qwen2.5-Coder-7B-Instruct --task grpo --config codeforces --accelerator zero3 --dp 8 --tp 1
```

--------------------------------

### GRPO training with vLLM backend (colocate mode)

Source: https://github.com/huggingface/open-r1/blob/main/README.md

This command enables GRPO training on a single node across multiple GPUs using the vLLM backend in 'colocate' mode. This is suitable for smaller models.

```shell
ACCELERATE_LOG_LEVEL=info \
    accelerate launch --config_file recipes/accelerate_configs/zero3.yaml \
    src/open_r1/grpo.py --config recipes/DeepSeek-R1-Distill-Qwen-1.5B/grpo/config_demo.yaml \
    --vllm_mode colocate
```

--------------------------------

### Download CodeForces Test Cases

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Download generated test cases for CodeForces problems. Adjust PATH_TO_SAVE_TESTCASES and --max-workers as needed.

```shell
# change PATH_TO_SAVE_TESTCASES. Increase --max-workers according to your machine's capacity
huggingface-cli download open-r1/codeforces --repo-type=dataset --include='generated_tests/*.parquet' --max-workers=8 --local-dir PATH_TO_SAVE_TESTCASES 
```

--------------------------------

### get_dataset()

Source: https://context7.com/huggingface/open-r1/llms.txt

Loads a single dataset by name or builds a weighted mixture from multiple Hugging Face datasets, with optional subsampling, column selection, shuffling, and train/test splitting.

```APIDOC
## `get_dataset()` — Dataset Loading and Mixture

Loads a single dataset by name or builds a weighted mixture from multiple Hugging Face datasets, with optional subsampling, column selection, shuffling, and train/test splitting.

```python
from open_r1.utils import get_dataset
from open_r1.configs import ScriptArguments

# Single dataset
args = ScriptArguments(
    dataset_name="open-r1/Mixture-of-Thoughts",
    dataset_config="all",
)
dataset = get_dataset(args)
# dataset -> DatasetDict({'train': Dataset(...), 'test': Dataset(...)})

# Weighted dataset mixture (via YAML config or dict)
args_mixture = ScriptArguments(
    dataset_mixture={
        "datasets": [
            {"id": "open-r1/OpenR1-Math-220k", "config": "default", "split": "train",
             "columns": ["problem", "solution"], "weight": 0.7},
            {"id": "open-r1/codeforces-cots",  "config": "default", "split": "train",
             "columns": ["problem", "solution"], "weight": 0.3},
        ],
        "seed": 42,
        "test_split_size": 0.05,
    }
)
mixed_dataset = get_dataset(args_mixture)
# mixed_dataset -> DatasetDict({'train': Dataset(~190k rows), 'test': Dataset(~10k rows)})
```
```

--------------------------------

### Fine-tune Qwen base models with custom EOS token

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Use this command to fine-tune Qwen base models when a custom EOS token is required. Ensure the `eos_token` parameter matches the model's expected token.

```shell
accelerate launch --config_file=recipes/accelerate_configs/zero3.yaml src/open_r1/sft.py \
    --model_name_or_path Qwen/Qwen2.5-1.5B \
    --eos_token '<|im_end|>' \
    --dataset_name open-r1/Mixture-of-Thoughts \
    --dataset_config all \
    --learning_rate 4.0e-5 \
    --num_train_epochs 1 \
    --max_seq_length 32768 \
    --per_device_train_batch_size 16 \
    --gradient_checkpointing \
    --bf16 \
    --use_liger_kernel \
    --output_dir data/Qwen2.5-1.5B-Open-R1-Distill
```

--------------------------------

### Train OlympicCoder 7B Model

Source: https://github.com/huggingface/open-r1/blob/main/recipes/README.md

Command to train the OlympicCoder 7B model using Zero3 acceleration.

```bash
# 7B
sbatch --nodes=1 slurm/train.slurm --model OlympicCoder-7B --task sft --config v00.00 --accelerator zero3
```

--------------------------------

### Configure Router URL in Training YAML

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Add the router URL to your training YAML configuration. Ensure the port number matches the one used when launching the router.

```yaml
# For E2B
e2b_router_url: 1.2.3.4:8000
```

```yaml
# For Morph
morph_router_url: 1.2.3.4:8000
```

--------------------------------

### Load Single or Mixed Hugging Face Datasets

Source: https://context7.com/huggingface/open-r1/llms.txt

Loads a dataset by name or creates a weighted mixture of datasets. Supports subsampling, column selection, shuffling, and train/test splitting.

```python
from open_r1.utils import get_dataset
from open_r1.configs import ScriptArguments

# Single dataset
args = ScriptArguments(
    dataset_name="open-r1/Mixture-of-Thoughts",
    dataset_config="all",
)
dataset = get_dataset(args)
# dataset -> DatasetDict({'train': Dataset(...), 'test': Dataset(...)})

# Weighted dataset mixture (via YAML config or dict)
args_mixture = ScriptArguments(
    dataset_mixture={
        "datasets": [
            {"id": "open-r1/OpenR1-Math-220k", "config": "default", "split": "train",
             "columns": ["problem", "solution"], "weight": 0.7},
            {"id": "open-r1/codeforces-cots",  "config": "default", "split": "train",
             "columns": ["problem", "solution"], "weight": 0.3},
        ],
        "seed": 42,
        "test_split_size": 0.05,
    }
)
mixed_dataset = get_dataset(args_mixture)
# mixed_dataset -> DatasetDict({'train': Dataset(~190k rows), 'test': Dataset(~10k rows)})
```

--------------------------------

### Load Model and Tokenizer from Hugging Face

Source: https://context7.com/huggingface/open-r1/llms.txt

Utility functions to load `AutoModelForCausalLM` and `AutoTokenizer`. Supports various configurations like `torch_dtype`, `attn_implementation`, quantization, and custom chat templates.

```python
from open_r1.utils import get_model, get_tokenizer
from open_r1.configs import SFTConfig
from trl import ModelConfig

model_args = ModelConfig(
    model_name_or_path="Qwen/Qwen2.5-1.5B",
    torch_dtype="bfloat16",
    attn_implementation="flash_attention_2",
    trust_remote_code=False,
)
training_args = SFTConfig(
    output_dir="data/my-model",
    chat_template=None,         # use model's built-in chat template
    gradient_checkpointing=True,
)

tokenizer = get_tokenizer(model_args, training_args)
model = get_model(model_args, training_args)
# model is AutoModelForCausalLM in bf16 with FlashAttention-2 and use_cache=False
```

--------------------------------

### Launch E2B Router Service

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Submit the slurm job to launch the E2B router service on a CPU node. This helps manage rate limiting for sandbox services.

```shell
sbatch slurm/e2b_router.slurm
```

--------------------------------

### Run GPQA Benchmarks with Python Script

Source: https://github.com/huggingface/open-r1/blob/main/README.md

Execute benchmarks for GPQA using the provided Python script. Replace {model_id} with the specific model identifier.

```shell
python scripts/run_benchmarks.py --model-id {model_id}  --benchmarks gpqa
```