### Setup and Installation

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Commands to clone the repository, create a new branch for the task, and install project requirements.

```shell
# After forking...
git clone https://github.com/<YOUR-USERNAME>/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout -b <task-name>
pip install -e ".[dev]"
```

--------------------------------

### Project Setup

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/model_guide.md

Steps to fork the repository, clone it, create a new branch, and install project requirements.

```shell
# After forking...
git clone https://github.com/<YOUR-USERNAME>/lm-evaluation-harness.git
cd lm-evaluation-harness
git checkout -b <model-type>
pip install -e ".[dev]"
```

--------------------------------

### Quick Start Examples

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Examples of common lm-eval commands for listing tasks, basic evaluation, few-shot evaluation, saving results, and using config files.

```bash
# List available tasks
lm-eval ls tasks

# Basic evaluation
lm-eval run --model hf --model_args pretrained=gpt2 --tasks hellaswag

# With few-shot examples
lm-eval run --model hf --model_args pretrained=gpt2 --tasks arc_easy --num_fewshot 5

# Save results and model outputs
lm-eval run --model hf --model_args pretrained=gpt2 --tasks hellaswag --output_path ./results/ --log_samples

# Use a config file
lm-eval run --config eval_config.yaml
```

--------------------------------

### lm-eval run quick examples

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Examples of the lm-eval run command for different scenarios.

```bash
# Basic evaluation with HuggingFace model
lm-eval run --model hf --model_args pretrained=gpt2 dtype=float32 --tasks hellaswag

# Multiple tasks with few-shot examples
lm-eval run --model vllm --model_args pretrained=EleutherAI/gpt-j-6B --tasks arc_easy arc_challenge --num_fewshot 5

# Custom generation parameters
lm-eval run --model hf --model_args pretrained=gpt2 --tasks lambada --gen_kwargs temperature=0.8 top_p=0.95

# Use a YAML configuration file
lm-eval run --config my_config.yaml --tasks mmlu
```

--------------------------------

### Task Configuration Example

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of a task configuration file showing how to set task aliases for better table display.

```yaml
"dataset_name": "abstract_algebra"
"description": "The following are multiple choice questions (with answers) about abstract\n algebra.\n\n"
"include": "_default_template_yaml"
"task": "mmlu_abstract_algebra"
"task_alias": "abstract_algebra"
```

--------------------------------

### List all tasks

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example command to list all available tasks.

```bash
lm-eval ls tasks
```

--------------------------------

### Hardcoded Few-shot Examples YAML

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

YAML snippet for hardcoding few-shot examples using a sampler and a list of samples.

```yaml
fewshot_config:
  sampler: first_n
  samples: [
    {<sample 1>},
    {<sample 2>},
  ]
```

--------------------------------

### Development Setup

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/README.md

Commands to clone the repository and install development dependencies.

```bash
git clone https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e ".[dev,hf]"
```

--------------------------------

### Prompt Template: List of Choices

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of `doc_to_choice` being directly provided as a list of text strings.

```yaml
doc_to_choice: ['No', 'Yes']
```

--------------------------------

### Multiple Choice Task Example (SciQ)

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example YAML configuration for a multiple-choice task, defining how to process text, identify the correct target, and list answer choices.

```yaml
doc_to_text: "{{support.lstrip()}}\nQuestion: {{question}}\nAnswer:" # This is the input portion of the prompt for this doc. It will have " {{choice}}" appended to it as target for each choice in answer_choices.
doc_to_target: 3 # this contains the index into the answer choice list of the correct answer.
doc_to_choice: "{{[distractor1, distractor2, distractor3, correct_answer]}}"
```

--------------------------------

### Multiple Choice Task Example (BoolQ)

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example YAML configuration for a multiple-choice task using a 'label' feature for the target and a simple list for choices.

```yaml
doc_to_text: "{{passage}}\nQuestion: {{question}}?\nAnswer:"
doc_to_target: label
doc_to_choice: ["no", "yes"]
```

--------------------------------

### Full Few-shot Configuration YAML

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Comprehensive YAML configuration for few-shot examples, including sampling strategy, split, hardcoded samples, and formatting overrides.

```yaml
fewshot_config:
  sampler: default        # Sampling strategy: "default" (random) or "first_n"
  split: train            # Dataset split to draw fewshot examples from (overrides fewshot_split)
  samples: [...]          # Hardcoded list of fewshot examples, or a callable returning them
  doc_to_text: "..."      # Override doc_to_text for fewshot examples only
  doc_to_target: "..."    # Override doc_to_target for fewshot examples only
  doc_to_choice: "..."    # Override doc_to_choice for fewshot examples only
  gen_prefix: "Answer:"   # Prefix for assistant response in fewshot examples
  fewshot_delimiter: "\n\n"  # Delimiter between fewshot examples
  target_delimiter: " "      # Delimiter between question and answer
```

--------------------------------

### Prompt Template: Feature as Choices

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example where `doc_to_choice` is set to the name of a dataset feature that is already a list.

```yaml
doc_to_choice: choices
```

--------------------------------

### List only task groups

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example command to list only task groups.

```bash
lm-eval ls groups
```

--------------------------------

### Few-shot Split Configuration YAML

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

YAML snippet for specifying the split to draw few-shot examples from.

```yaml
fewshot_split: <split name to draw fewshot examples from, or `null`>
```

--------------------------------

### Jinja2 Prompt Template: Input Formatting

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of using Jinja2 templating for `doc_to_text` to format input by combining dataset features.

```yaml
doc_to_text: "{{passage}}\nQuestion: {{question}}?\nAnswer:"
```

--------------------------------

### Loading Local JSON Datasets

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Configuration example for loading local JSON files using `dataset_kwargs` with `data_files`.

```yaml
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files: /path/to/my/json
```

--------------------------------

### Loading Previously Downloaded Hugging Face Datasets

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Configuration example for using local Hugging Face datasets by specifying the directory path with `data_dir` under `dataset_kwargs`.

```yaml
dataset_path: hellaswag
dataset_kwargs:
  data_dir: hellaswag_local/
```

--------------------------------

### Jinja2 Prompt Template: Target Formatting

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of using Jinja2 templating for `doc_to_target` to specify the desired output format.

```yaml
doc_to_target: "{{answer}}"
```

--------------------------------

### Evaluating Thinking/Reasoning Models with vllm or sglang backend

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example command to run evaluation with the vllm or sglang backend, enabling thinking mode and specifying the think_end_token as a string.

```bash
lm-eval run --model vllm \
  --model_args pretrained=Qwen/Qwen3-32B,enable_thinking=True,think_end_token="</think>" \
  --tasks gsm8k --apply_chat_template
```

--------------------------------

### Basic Prompt Template: Feature Names

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of a basic prompt template where `doc_to_text` and `doc_to_target` directly use feature names from the dataset.

```yaml
doc_to_text: startphrase
doc_to_target: label
```

--------------------------------

### Evaluating Thinking/Reasoning Models with hf backend

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example command to run evaluation with the hf backend, enabling thinking mode and specifying the think_end_token as a token ID.

```bash
lm-eval run --model hf \
  --model_args pretrained=Qwen/Qwen3-32B,enable_thinking=True,think_end_token=200008 \
  --tasks gsm8k --apply_chat_template
```

--------------------------------

### TemplateAPI Initialization Example

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/API_guide.md

Example of subclassing TemplateAPI and initializing it with custom arguments.

```python
class MyAPIModel(TemplateAPI):
    def __init__(self, **kwargs):
        super().__init__(
            model="my-model",
            base_url="https://api.mymodel.com/v1/completions",
            tokenizer_backend="huggingface",
            num_concurrent=5,
            max_retries=5,
            batch_size=10,
            **kwargs
        )

    # Implement other required methods...

```

--------------------------------

### Validate a task group

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example of validating a task group.

```bash
# Validate a task group
lm-eval validate --tasks mmlu
```

--------------------------------

### Validate multiple tasks

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example of validating multiple tasks.

```bash
# Validate multiple tasks
lm-eval validate --tasks arc_easy,arc_challenge,hellaswag
```

--------------------------------

### Importing Prompt from Promptsource

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

YAML configuration snippet showing how to use prompts from the Promptsource repository.

```yaml
use_prompt: "promptsource:GPT-3 Style"
```

--------------------------------

### Validate a single task

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example of validating a single task.

```bash
# Validate a single task
lm-eval validate --tasks hellaswag
```

--------------------------------

### Install lm-evaluation-harness and Zeno integration

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/examples/visualize-zeno.ipynb

Installs the lm-evaluation-harness and its Zeno visualization capabilities.

```python
# Install this project if you did not already do that. This is all that needs to be installed for you to be able to visualize your data in Zeno!
!pip install -e ..
!pip install -e ..[zeno]
```

--------------------------------

### Dataset Processing Function Example

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Python function to process dataset documents, including an example of transforming fields and creating a new document structure.

```python
def process_docs(dataset: datasets.Dataset) -> datasets.Dataset:
    def _process_doc(doc):
        ctx = doc["ctx_a"] + " " + doc["ctx_b"].capitalize()
        out_doc = {
            "query": preprocess(doc["activity_label"] + ": " + ctx),
            "choices": [preprocess(ending) for ending in doc["endings"]],
            "gold": int(doc["label"]),
        }
        return out_doc

    return dataset.map(_process_doc)
```

--------------------------------

### Install lm-evaluation-harness with W&B support

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/examples/visualize-wandb.ipynb

This command installs the project with the necessary dependencies for Weights and Biases integration.

```bash
# Install this project if you did not already have it.
# This is all that is needed to be installed to start using Weights and Biases

!pip -qq install -e ..[wandb]
```

--------------------------------

### Validate external tasks

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/interface.md

Example of validating tasks from an external path.

```bash
# Validate external tasks
lm-eval validate --tasks my_custom_task --include_path ./custom_tasks
```

--------------------------------

### Run the benchmark

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/lm_eval/tasks/toksuite/README.md

This command demonstrates how to run the TokSuite benchmark for a specified model and output path, with sample logging enabled.

```bash
OUTPUT_PATH="/tmp/lm_eval/toksuite"
mkdir -p $OUTPUT_PATH
lm-eval --model hf --model_args pretrained="toksuite/gpt2,tokenizer=openai-community/gpt2" \
    --tasks toksuite --log_samples \
    --output_path=$OUTPUT_PATH
```

--------------------------------

### Basic Usage Example

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/examples/lm-eval-overview.ipynb

A simple example demonstrating how to run an evaluation using the LM Evaluation Harness.

```python
from lm_eval import evaluator

results = evaluator.simple_evaluate(
    model="hf-causal",
    model_args="pretrained=gpt2",
    tasks=["piqa"],
    batch_size=8,
    device="cuda",
    no_cache=True,
    num_fewshot=0,
)

print(evaluator.make_table(results))
```

--------------------------------

### Task Versioning Metadata

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of adding version information to a task or group configuration file.

```yaml
metadata:
  version: 0
```

--------------------------------

### Example Command

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/examples/lm-eval-overview.ipynb

An example command to run a specific task with the LM Evaluation Harness.

```bash
--tasks demo_mmlu_high_school_geography_function_prompt_2 \
    --limit 10 \
    --output output/demo_mmlu_high_school_geography_function_prompt_2/ \
    --log_samples
```

--------------------------------

### Weights and Biases Integration Example

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/README.md

Example command to run the evaluation harness with Weights and Biases integration enabled.

```bash
lm_eval \
    --model hf \
    --model_args pretrained=microsoft/phi-2,trust_remote_code=True \
    --tasks hellaswag,mmlu_abstract_algebra \
    --device cuda:0 \
    --batch_size 8 \
    --output_path output/phi-2 \
    --limit 10 \
    --wandb_args project=lm-eval-harness-integration \
    --log_samples
```

--------------------------------

### Basic Prompt Template: Hard-coded Target

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/new_task_guide.md

Example of a prompt template where `doc_to_target` is hard-coded to an integer index.

```yaml
doc_to_target: 3
```

--------------------------------

### OpenAICompletionsAPI Subclassing and Initialization

Source: https://github.com/eleutherai/lm-evaluation-harness/blob/main/docs/API_guide.md

Example of subclassing TemplateAPI for OpenAI completions, including registration and initialization with specific defaults.

```python
@register_model("openai-completions")
class OpenAICompletionsAPI(LocalCompletionsAPI):
    def __init__(
        self,
        base_url="https://api.openai.com/v1/completions",
        tokenizer_backend="tiktoken",
        **kwargs,
    ):
        super().__init__(
            base_url=base_url, tokenizer_backend=tokenizer_backend, **kwargs
        )

```