### Install Lighteval for Documentation Building

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with dependencies for building documentation. This requires installing the `docs` extra.

```bash
pip install lighteval[docs]
```

--------------------------------

### Install Lighteval from Source with VLLM Backend

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval from source with the VLLM backend for high-performance inference. This requires installing the `vllm` extra during the source installation.

```bash
pip install -e .[vllm]
```

--------------------------------

### Install Lighteval for Testing

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with testing dependencies. This requires installing the `tests` extra.

```bash
pip install lighteval[tests]
```

--------------------------------

### Install Lighteval with Optimum

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with Optimum support for optimized models. This requires installing the `optimum` extra.

```bash
pip install lighteval[optimum]
```

--------------------------------

### Install Lighteval for Nanotron

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for evaluating Nanotron models. This requires installing the `nanotron` extra.

```bash
pip install lighteval[nanotron]
```

--------------------------------

### Install Lighteval for Quality Development

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with code quality tools for development. This requires installing the `quality` extra.

```bash
pip install lighteval[quality]
```

--------------------------------

### Install Lighteval for Quantization

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for evaluating quantized models. This requires installing the `quantization` extra.

```bash
pip install lighteval[quantization]
```

--------------------------------

### List Available Tasks with Lighteval

Source: https://github.com/huggingface/lighteval/wiki/Available-Tasks

Run this command to get a list of all supported tasks in Lighteval. No additional setup is required beyond installing Lighteval.

```bash
lighteval tasks --list
```

--------------------------------

### Install Lighteval for Adapters

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for evaluating adapter models (PEFT, Delta). This requires installing the `adapters` extra.

```bash
pip install lighteval[adapters]
```

--------------------------------

### Install LightEval and Dependencies

Source: https://context7.com/huggingface/lighteval/llms.txt

Install the basic LightEval package or with optional dependencies for specific backends like VLLM or LiteLLM. Development installation and installation from source are also supported.

```bash
pip install lighteval
```

```bash
pip install lighteval[vllm]
```

```bash
pip install lighteval[litellm]
```

```bash
pip install lighteval[dev]
```

```bash
git clone https://github.com/huggingface/lighteval.git
cd lighteval
pip install -e .
```

--------------------------------

### List All Tasks

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/available-tasks.mdx

Use this command to see all available tasks within LightEval. No setup is required beyond having LightEval installed.

```bash
lighteval tasks list
```

--------------------------------

### Starting a VLLM Server

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-litellm-as-backend.mdx

Command to start a local VLLM server for hosting Hugging Face models. Ensure the host and port are accessible.

```bash
vllm serve HuggingFaceH4/zephyr-7b-beta --host 0.0.0.0 --port 8000
```

--------------------------------

### Install Lighteval for Math Tasks

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for mathematical reasoning tasks. This requires installing the `math` extra.

```bash
pip install lighteval[math]
```

--------------------------------

### Install Lighteval for TensorBoard Logging

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for uploading results to TensorBoard. This requires installing the `tensorboardX` extra.

```bash
pip install lighteval[tensorboardX]
```

--------------------------------

### Install Lighteval with VLLM Backend

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with the VLLM backend for high-performance inference. This requires installing the `vllm` extra.

```bash
pip install lighteval[vllm]
```

--------------------------------

### Install Lighteval with LiteLLM

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with LiteLLM for unified API access. This requires installing the `litellm` extra.

```bash
pip install lighteval[litellm]
```

--------------------------------

### Install Lighteval for Multilingual Support

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for multilingual evaluation. This requires installing the `multilingual` extra.

```bash
pip install lighteval[multilingual]
```

--------------------------------

### Install Lighteval for Trackio Logging

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for logging results to Trackio. This requires installing the `trackio` extra.

```bash
pip install lighteval[trackio]
```

--------------------------------

### Install Lighteval with TGI Backend

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with Text Generation Inference (TGI) API support. This requires installing the `tgi` extra.

```bash
pip install lighteval[tgi]
```

--------------------------------

### Install Lighteval from Source

Source: https://github.com/huggingface/lighteval/wiki/Installation

Clone the repository and install Lighteval in editable mode for development purposes. This allows you to make changes to the source code and have them reflected immediately.

```bash
git clone https://github.com/huggingface/lighteval.git
cd lighteval
pip install -e .
```

--------------------------------

### Install Lighteval for S3 Storage

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for uploading results to S3. This requires installing the `s3` extra.

```bash
pip install lighteval[s3]
```

--------------------------------

### Install Lighteval for Weights & Biases Logging

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for logging results to Weights & Biases. This requires installing the `wandb` extra.

```bash
pip install lighteval[wandb]
```

--------------------------------

### Example: Run Custom Task with Few-Shot Evaluation

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/adding-a-custom-task.mdx

Demonstrates running a custom task with a specified number of few-shot examples. This command targets a specific model and custom task file, applying 3-shot learning.

```bash
# Run a custom task with 3 shot evaluation
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    "myothertask|3" \
    --custom-tasks community_tasks/my_custom_task.py
```

--------------------------------

### Get Help for Lighteval

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/quicktour.mdx

Use the --help flag to get more information about available options for each command.

```bash
lighteval --help
```

--------------------------------

### Install All Development Dependencies for Lighteval

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with all development-related dependencies, including extras for quality, tests, multilingual, math, and VLLM. This requires installing the `dev` extra.

```bash
pip install lighteval[dev]
```

--------------------------------

### Install Lighteval with Extended Tasks

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/installation.mdx

Install Lighteval with support for extended evaluation tasks. This requires installing the `extended_tasks` extra.

```bash
pip install lighteval[extended_tasks]
```

--------------------------------

### Use an Existing TGI Server with Lighteval

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-huggingface-inference-endpoints-or-tgi-as-backend.mdx

Example command to connect to an existing Text Generation Inference (TGI) server for running an evaluation. Provide the path to your TGI server configuration file.

```bash
lighteval endpoint tgi \
    "configs/tgi_server.yaml" \
    gsm8k
```

--------------------------------

### Install Lighteval with Optional Dependencies

Source: https://github.com/huggingface/lighteval/wiki/Installation

Install Lighteval with specific extras for features like accelerate, tgi, nanotron, quantization, adapters, tensorboardX, vllm, or s3. Replace '<group>' with the desired extra name.

```bash
pip install lighteval[<group>]
```

```bash
pip install -e .[<group>]
```

--------------------------------

### Example Task Inspection

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/available-tasks.mdx

An example of how to inspect a specific task, in this case, the 'truthfulqa:mc' task.

```bash
lighteval tasks inspect truthfulqa:mc
```

--------------------------------

### Install Lighteval from PyPI

Source: https://github.com/huggingface/lighteval/wiki/Installation

Use this command to install the latest stable version of Lighteval from the Python Package Index.

```bash
pip install lighteval
```

--------------------------------

### Basic VLLM Backend Usage

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-vllm-as-backend.mdx

Use this command to run Lighteval with VLLM as the backend for a specified model. Ensure VLLM is installed and accessible.

```bash
lighteval vllm \
    "model_name=HuggingFaceH4/zephyr-7b-beta" \
    ifeval
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/huggingface/lighteval/blob/main/README.md

Install development dependencies for basic development, including code quality checks and tests. For GPU/vLLM development and slow tests, use the 'dev-gpu' extra.

```bash
# For basic development (code quality, tests)
pip install -e ".[dev]"

# Or for GPU/vllm development and slow tests
pip install -e ".[dev-gpu]"

pre-commit install
pre-commit run --all-files
```

--------------------------------

### Evaluate GPT-2 with Accelerate

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/quicktour.mdx

Basic usage example to evaluate GPT-2 on the Truthful QA benchmark using the 🤗 Accelerate backend.

```bash
lighteval accelerate \
     "model_name=openai-community/gpt2" \
     truthfulqa:mc
```

--------------------------------

### Deploy a New Inference Endpoint with Lighteval

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-huggingface-inference-endpoints-or-tgi-as-backend.mdx

Example command to deploy a new Hugging Face Inference Endpoint and run an evaluation. Ensure the configuration file path and task name are correct.

```bash
lighteval endpoint inference-endpoint \
    "configs/endpoint_model.yaml" \
    gsm8k
```

--------------------------------

### Lighteval Usage with Configuration File

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-inference-providers-as-backend.mdx

Execute Lighteval evaluations by referencing a configuration file that specifies model and provider details. This is useful for complex setups or repeated configurations.

```bash
lighteval endpoint inference-providers \
    examples/model_configs/inference_providers.yaml \
    gsm8k
```

--------------------------------

### Evaluate Model with Lighteval Python Script

Source: https://github.com/huggingface/lighteval/wiki/Using-the-Python-API

Use this script to set up an evaluation tracker, pipeline parameters, model configuration, and run an evaluation pipeline. Ensure 'accelerate' is installed for distributed training.

```python
import lighteval
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.models.model_config import VLLMModelConfig
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
from lighteval.utils.utils import EnvConfig
from lighteval.utils.imports import is_accelerate_available

if is_accelerate_available():
    from accelerate import Accelerator, InitProcessGroupKwargs
    accelerator = Accelerator(kwargs_handlers=[InitProcessGroupKwargs(timeout=timedelta(seconds=3000))])
else:
    accelerator = None

def main():
    evaluation_tracker = EvaluationTracker(
        output_dir="./results",
        save_details=True,
        push_to_hub=True,
        hub_results_org="your user name",
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.ACCELERATE,
        env_config=EnvConfig(cache_dir="tmp/"),
        # Remove the 2 parameters below once your configuration is tested
        override_batch_size=1,
        max_samples=10 
    )

    model_config = VLLMModelConfig(
            pretrained="HuggingFaceH4/zephyr-7b-beta",
            dtype="float16",
            use_chat_template=True,
    )

    task = "helm|mmlu|5|1"

    pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config,
        custom_task_directory=None, # if using a custom task
    )

    pipeline.evaluate()
    pipeline.save_and_push_results()
    pipeline.show_results()

if __name__ == "__main__":
    main()
```

--------------------------------

### Configure VLLM Backend with Reasoning Tags

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/quicktour.mdx

Example of configuring the VLLM backend for a specific model, including dtype and data parallelism, and using --remove-reasoning-tags with custom reasoning tags.

```bash
lighteval vllm \
    "model_name=mistralai/Magistral-Small-2507,dtype=float16,data_parallel_size=4" \
    aime24 \
    --remove-reasoning-tags \
    --reasoning-tags="[('[THINK]','[/THINK]')]"
```

--------------------------------

### Define a Simple Task with No Subsets

Source: https://github.com/huggingface/lighteval/wiki/Adding-a-Custom-Task

Example of how to create a basic task definition, similar to 'hellaswag', which consists of a single subset.

```python
# This is how you create a simple task (like hellaswag) which has one single subset

```

--------------------------------

### Example LightEval Result File

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/saving-and-reading-results.mdx

This JSON object represents a typical output file from LightEval, detailing the configuration, task-specific settings, and performance metrics of an evaluation run.

```json
{
  "config_general": {
    "lighteval_sha": "203045a8431bc9b77245c9998e05fc54509ea07f",
    "num_fewshot_seeds": 1,
    "max_samples": 1,
    "job_id": "",
    "start_time": 620979.879320166,
    "end_time": 621004.632108041,
    "total_evaluation_time_secondes": "24.752787875011563",
    "model_name": "gpt2",
    "model_sha": "607a30d783dfa663caf39e06633721c8d4cfcd7e",
    "model_dtype": null,
    "model_size": "476.2 MB"
  },
  "results": {
    "gsm8k|0": {
      "em": 0.0,
      "em_stderr": 0.0,
      "maj@8": 0.0,
      "maj@8_stderr": 0.0
    },
    "all": {
      "em": 0.0,
      "em_stderr": 0.0,
      "maj@8": 0.0,
      "maj@8_stderr": 0.0
    }
  },
  "versions": {
    "gsm8k|0": 0
  },
  "config_tasks": {
    "lighteval|gsm8k": {
      "name": "gsm8k",
      "prompt_function": "gsm8k",
      "hf_repo": "gsm8k",
      "hf_subset": "main",
      "metric": [
        {
          "metric_name": "em",
          "higher_is_better": true,
          "category": "3",
          "use_case": "5",
          "sample_level_fn": "compute",
          "corpus_level_fn": "mean"
        },
        {
          "metric_name": "maj@8",
          "higher_is_better": true,
          "category": "5",
          "use_case": "5",
          "sample_level_fn": "compute",
          "corpus_level_fn": "mean"
        }
      ],
      "hf_avail_splits": [
        "train",
        "test"
      ],
      "evaluation_splits": [
        "test"
      ],
      "few_shots_split": null,
      "few_shots_select": "random_sampling_from_train",
      "generation_size": 256,
      "generation_grammar": null,
      "stop_sequence": [
        "Question="
      ],
      "num_samples": null,
      "original_num_docs": 1319,
      "effective_num_docs": 1,
      "must_remove_duplicate_docs": null,
      "version": 0
    }
  },
  "summary_tasks": {
    "gsm8k|0": {
      "hashes": {
        "hash_examples": "8517d5bf7e880086",
        "hash_full_prompts": "8517d5bf7e880086",
        "hash_input_tokens": "29916e7afe5cb51d",
        "hash_cont_tokens": "37f91ce23ef6d435"
      },
      "padded": 0,
      "non_padded": 2,
      "effective_few_shots": 0.0
    }
  },
  "summary_general": {
    "hashes": {
      "hash_examples": "5f383c395f01096e",
      "hash_full_prompts": "5f383c395f01096e",
      "hash_input_tokens": "ac933feb14f96d7b",
      "hash_cont_tokens": "9d03fb26f8da7277"
    },
    "padded": 0,
    "non_padded": 2
  }
}
```

--------------------------------

### Hugging Face Inference Endpoints Configuration Example

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-huggingface-inference-endpoints-or-tgi-as-backend.mdx

Configure model parameters, infrastructure settings, and endpoint details for deploying a new model on Hugging Face Inference Endpoints. Ensure model_name and region are correctly set.

```yaml
model_parameters:
    reuse_existing: false # If true, ignore all params in instance, and don't delete the endpoint after evaluation
    # endpoint_name: "llama-2-7B-lighteval" # Needs to be lowercase without special characters
    model_name: "meta-llama/Llama-2-7b-hf"
    revision: "main"  # Defaults to "main"
    dtype: "float16" # Can be any of "awq", "eetq", "gptq", "4bit" or "8bit" (will use bitsandbytes), "bfloat16" or "float16"
    accelerator: "gpu"
    region: "eu-west-1"
    vendor: "aws"
    instance_type: "nvidia-a10g"
    instance_size: "x1"
    framework: "pytorch"
    endpoint_type: "protected"
    namespace: null # The namespace under which to launch the endpoint. Defaults to the current user's namespace
    image_url: null # Optionally specify the docker image to use when launching the endpoint model. E.g., launching models with later releases of the TGI container with support for newer models.
    env_vars: null # Optional environment variables to include when launching the endpoint. e.g., `MAX_INPUT_LENGTH: 2048`
```

--------------------------------

### VLLM Model Configuration YAML Example

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-vllm-as-backend.mdx

This YAML file defines VLLM model parameters, including model name, revision, parallelism settings, memory utilization, and generation parameters. Adjust `gpu_memory_utilization` and `max_model_length` to mitigate out-of-memory errors.

```yaml
model_parameters:
    model_name: "HuggingFaceTB/SmolLM-1.7B-Instruct"
    revision: "main"
    dtype: "bfloat16"
    tensor_parallel_size: 1
    data_parallel_size: 1
    pipeline_parallel_size: 1
    gpu_memory_utilization: 0.9
    max_model_length: 2048
    swap_space: 4
    seed: 1
    trust_remote_code: True
    add_special_tokens: True
    multichoice_continuations_start_space: True
    pairwise_tokenization: True
    subfolder: null
    generation_parameters:
      presence_penalty: 0.0
      repetition_penalty: 1.0
      frequency_penalty: 0.0
      temperature: 1.0
      top_k: 50
      min_p: 0.0
      top_p: 1.0
      seed: 42
      stop_tokens: null
      max_new_tokens: 1024
      min_new_tokens: 0
```

--------------------------------

### Inference Provider Configuration Example

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-inference-providers-as-backend.mdx

This YAML configuration defines parameters for an LLM inference task, including model name, provider, and generation settings. It can be used with Lighteval for advanced model evaluations.

```yaml
model_parameters:
  model_name: "deepseek-ai/DeepSeek-R1-Distill-Qwen-32B"
  provider: "novita"
  timeout: null
  proxies: null
  parallel_calls_count: 10
  generation_parameters:
    temperature: 0.8
    top_k: 10
    max_new_tokens: 10000
```

--------------------------------

### LightEval Results Table Example

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/inspect-ai.mdx

Example of the per-model results table printed by LightEval after completing tasks.

```text
Completed all tasks in 'lighteval-logs' successfully

|                 Model                 |gpqa|gpqa:diamond|
|---------------------------------------|---:|-----------:|
|vllm/HuggingFaceTB/SmolLM-135M-Instruct|0.01|        0.01|

results saved to lighteval-logs
run "inspect view --log-dir lighteval-logs" to view the results
```

--------------------------------

### Run Multiple Tasks

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/quicktour.mdx

Demonstrates how to run multiple tasks by specifying a file path or a comma-separated list of tasks.

```bash
lighteval accelerate \
     "model_name=openai-community/gpt2" \
     ./path/to/lighteval/examples/tasks/recommended_set.txt
```

```bash
# or, e.g., "truthfulqa:mc|0,gsm8k|3"
```

--------------------------------

### Set WANDB Environment Variable

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/saving-and-reading-results.mdx

Sets the WANDB_PROJECT environment variable for Weights & Biases integration. Ensure W&B is installed (`pip install lighteval[trackio]`).

```bash
export WANDB_PROJECT="lighteval"
```

--------------------------------

### Basic vLLM Integration with Lighteval

Source: https://github.com/huggingface/lighteval/wiki/Use-VLLM-as-backend

Use this command to integrate vLLM as the backend for Lighteval. Specify vLLM and model arguments within `model_args`.

```bash
lighteval accelerate \
    --model_args="vllm,pretrained=HuggingFaceH4/zephyr-7b-beta,dtype=float16" \
    --tasks "leaderboard|truthfulqa:mc|0|0" \
    --output_dir="./evals/"
```

--------------------------------

### Task Format Specification

Source: https://github.com/huggingface/lighteval/wiki/Quicktour

This format is used to specify tasks for evaluation. It includes the suite, task name, number of few-shot examples, and an option to automatically reduce few-shot examples if the prompt is too long.

```bash
{suite}|{task}|{num_few_shot}|{0 or 1 to automatically reduce `num_few_shot` if prompt is too long}
```

--------------------------------

### Basic SGLang Backend Usage

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-sglang-as-backend.mdx

Use this command to run Lighteval with SGLang as the backend. Specify the model name and data type in model_args.

```bash
lighteval sglang \
    "model_name=HuggingFaceH4/zephyr-7b-beta,dtype=float16" \
    truthfulqa:mc
```

--------------------------------

### Assess Few-Shot Impact

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/inspect-ai.mdx

Measure the effect of few-shot examples on model performance by specifying different shot counts.

```bash
lighteval eval hf-inference-providers/openai/gpt-oss-20b "gsm8k|0,gsm8k|5"
```

--------------------------------

### Run Lighteval with TGI or Inference Endpoint Backend

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-huggingface-inference-endpoints-or-tgi-as-backend.mdx

Use this command to initiate an evaluation by specifying the backend type (tgi or inference-endpoint), the path to your configuration file, and any task parameters.

```bash
lighteval endpoint {tgi,inference-endpoint} \
    "/path/to/config/file" \
    <task_parameters>
```

--------------------------------

### Launch Lighteval with Custom Tasks

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/adding-a-new-metric.mdx

Use the `--custom-tasks` flag to specify the path to your custom metric file when launching Lighteval.

```bash
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    "truthfulqa:mc" \
    --custom-tasks path_to_your_metric_file.py
```

--------------------------------

### VLLM Configuration via YAML File

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-vllm-as-backend.mdx

Run Lighteval with VLLM using a YAML configuration file for advanced settings. This allows for detailed specification of model parameters, parallelism, and generation settings.

```bash
lighteval vllm \
    "examples/model_configs/vllm_model_config.yaml" \
    ifeval
```

--------------------------------

### Define Translation Literals for English

Source: https://github.com/huggingface/lighteval/wiki/Contributing-to-multilingual-evaluations

This snippet shows the structure for defining translation literals for a specific language, using English as an example. It includes keywords, punctuation, and enumeration indices.

```python
    Language.ENGLISH: TranslationLiterals(
        language=Language.ENGLISH,
        question_word="question", # Usage: "Question: How are you?"
        answer="answer", # Usage: "Answer: I am fine"
        confirmation_word="right", # Usage: "He is smart, right?"
        yes="yes", # Usage: "Yes, he is"
        no="no", # Usage: "No, he is not"
        also="also", # Usage: "Also, she is smart."
        cause_word="because", # Usage: "She is smart, because she is tall"
        effect_word="therefore", # Usage: "He is tall therefore he is smart"
        or_word="or", # Usage: "He is tall or small"
        true="true", # Usage: "He is smart, true, false or neither?"
        false="false", # Usage: "He is smart, true, false or neither?"
        neither="neither", # Usage: "He is smart, true, false or neither?"
        # Punctuation and spacing: only adjust if your language uses something different than in English
        full_stop=".",
        comma=",",
        question_mark="?",
        exclamation_mark="!",
        word_space=" ",
        sentence_space=" ",
        colon=":",
        # The first characters of your alphabet used in enumerations, if different from English
        indices=["A", "B", "C", ...]
    )
```

--------------------------------

### Initialize Basic EvaluationTracker

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/saving-and-reading-results.mdx

Initializes the EvaluationTracker with basic configuration for local saving and pushing results to the Hugging Face Hub.

```python
from lighteval.logging.evaluation_tracker import EvaluationTracker

tracker = EvaluationTracker(
    output_dir="./results",
    save_details=True,
    push_to_hub=True,
    hub_results_org="your_username",
    public=False
)
```

--------------------------------

### Configure Lighteval Task for Local Data

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/offline-evaluation.md

Sets up a `LightevalTaskConfig` to use local JSONL files for evaluation. Specify the path to your local data file using `hf_data_files` and set `hf_repo` to 'json' for built-in streaming.

```python
local_data = Path(__file__).parent / "samples" / "faq.jsonl"

local_task = LightevalTaskConfig(
    name="faq_eval",
    prompt_function=local_prompt,
    hf_repo="json",  # Built-in streaming loader for json/jsonl files
    hf_subset="default",
    hf_data_files=str(local_data),  # Can also be a dict mapping split names to paths
    evaluation_splits=["train"],
    metrics=[Metrics.ACCURACY],
)
```

--------------------------------

### Initialize Basic Evaluation Tracker

Source: https://context7.com/huggingface/lighteval/llms.txt

Set up a basic local evaluation tracker to save results and details to a specified output directory. This is suitable for local testing and analysis.

```python
from lighteval.logging.evaluation_tracker import EvaluationTracker

# Basic local tracking
tracker = EvaluationTracker(
    output_dir="./results",
    save_details=True,
)
```

--------------------------------

### Compare Providers for a Single Model

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/inspect-ai.mdx

Compare the performance of different providers for the same model by specifying them with colons.

```bash
lighteval eval \
    hf-inference-providers/openai/gpt-oss-20b:fireworks-ai \
    hf-inference-providers/openai/gpt-oss-20b:together \
    hf-inference-providers/openai/gpt-oss-20b:nebius \
    gpqa:diamond
```

--------------------------------

### VLLM Tensor Parallelism Configuration

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-vllm-as-backend.mdx

Configure VLLM to use tensor parallelism across multiple GPUs. Set the VLLM_WORKER_MULTIPROC_METHOD environment variable to 'spawn' for multi-GPU setups. Specify the number of GPUs for tensor parallelism using `tensor_parallel_size`.

```bash
export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm \
    "model_name=HuggingFaceH4/zephyr-7b-beta,tensor_parallel_size=4" \
    ifeval
```

--------------------------------

### Evaluate Models with `lighteval accelerate`

Source: https://context7.com/huggingface/lighteval/llms.txt

Utilize `lighteval accelerate` for evaluations on CPU or GPUs via HuggingFace Accelerate. Supports custom task files, multiple tasks, saving results locally, and pushing to the Hub.

```bash
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    truthfulqa:mc
```

```bash
lighteval accelerate \
    "model_name=openai-community/gpt2" \
    ./path/to/lighteval/examples/tasks/recommended_set.txt
```

```bash
lighteval accelerate \
    "model_name=HuggingFaceH4/zephyr-7b-beta" \
    "truthfulqa:mc|0,gsm8k|3"
```

```bash
lighteval accelerate \
    "model_name=meta-llama/Llama-3-8B-Instruct" \
    gsm8k \
    --output-dir ./results \
    --save-details \
    --push-to-hub \
    --results-org your_username
```

--------------------------------

### Run Lighteval with Model Server

Source: https://github.com/huggingface/lighteval/wiki/Evaluate-the-model-on-a-server-or-container

Execute Lighteval evaluations by pointing to a model configuration file. This command is used when your model is served via a TGI-compatible server.

```bash
lighteval accelerate \
    --model_config_path="/path/to/config/file"\
    --tasks <task parameters> \
    --output_dir output_dir
```

--------------------------------

### VLLM Data Parallelism Configuration

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-vllm-as-backend.mdx

Configure VLLM to use data parallelism when the model fits on a single GPU. Set the VLLM_WORKER_MULTIPROC_METHOD environment variable to 'spawn' for multi-GPU setups. Specify the number of GPUs for data parallelism using `data_parallel_size`.

```bash
export VLLM_WORKER_MULTIPROC_METHOD=spawn && lighteval vllm \
    "model_name=HuggingFaceH4/zephyr-7b-beta,data_parallel_size=4" \
    ifeval
```

--------------------------------

### Run Benchmark and Push to Hub

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/index.mdx

Execute a benchmark evaluation for a specified model and task, saving the results to a bundle directory and pushing them to a Hugging Face Hub repository. Ensure the model and task identifiers are correct.

```bash
lighteval eval "hf-inference-providers/openai/gpt-oss-20b" \
    gpqa:diamond \
    --bundle-dir gpt-oss-bundle \
    --repo-id OpenEvals/evals
```

--------------------------------

### Configure Accelerate for Multi-GPU

Source: https://github.com/huggingface/lighteval/wiki/Quicktour

Run this command to configure Accelerate for multi-GPU usage. This is a prerequisite for distributed evaluation.

```bash
accelerate config
```

--------------------------------

### Configure Simple Lighteval Task

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/adding-a-custom-task.mdx

Creates a configuration for a simple Lighteval task without subsets. This involves specifying the task name, prompt function, dataset repository, evaluation splits, and metrics.

```python
from lighteval.tasks.lighteval_task import LightevalTaskConfig

# This is how you create a simple task (like HellaSwag) which has one single subset
# attached to it, and one evaluation possible.
task = LightevalTaskConfig(
    name="myothertask",
    prompt_function=prompt_fn,  # Must be defined in the file or imported
    hf_repo="your_dataset_repo_on_hf",
    hf_subset="default",
    hf_avail_splits=["train", "test"],
    evaluation_splits=["test"],
    few_shots_split="train",
    few_shots_select="random_sampling_from_train",
    metrics=[metric],  # Select your metric in Metrics
    generation_size=256,
    stop_sequence=["\n", "Question:"]
)
```

--------------------------------

### Dump All Tasks Details

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/available-tasks.mdx

Retrieve detailed information for all tasks. This output can be redirected to a file for later use.

```bash
lighteval tasks dump
```

--------------------------------

### Specify Multiple Tasks for Evaluation

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/using-the-python-api.mdx

Demonstrates how to specify multiple tasks for evaluation, either as a comma-separated string or by loading from a file path.

```python
# Multiple tasks as comma-separated string
tasks = "aime24,aime25"

# Or load from a file
tasks = "./path/to/tasks.txt"

pipeline = Pipeline(
    tasks=tasks,
    # ... other parameters
)
```

--------------------------------

### Compare All Providers for a Model

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/inspect-ai.mdx

A shorthand to compare all available providers for a given model in a single line.

```bash
    hf-inference-providers/openai/gpt-oss-20b:all \
    "lighteval|gpqa:diamond|0"
```

--------------------------------

### Evaluate Nanotron Model with Torchrun

Source: https://github.com/huggingface/lighteval/wiki/Quicktour

Use this command to evaluate a Nanotron model on a single GPU. Ensure `nproc-per-node` matches the parallelism configuration in your `lighteval_config_override_template.yaml` file.

```bash
 torchrun --standalone --nnodes=1 --nproc-per-node=1  \
 src/lighteval/__main__.py nanotron \
 --checkpoint_config_path ../nanotron/checkpoints/10/config.yaml \
 --lighteval_config_path examples/nanotron/lighteval_config_override_template.yaml
```

--------------------------------

### LightevalTask and Configuration

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/package_reference/tasks.mdx

Details on LightevalTask and its configuration class.

```APIDOC
## LightevalTaskConfig

### Description
Configuration class for LightevalTask.

### Autodoc
`tasks.lighteval_task.LightevalTaskConfig`
```

```APIDOC
## LightevalTask

### Description
Core task class for Lighteval.

### Autodoc
`tasks.lighteval_task.LightevalTask`
```

--------------------------------

### Configure Lighteval Task with Custom Subsets

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/adding-a-custom-task.mdx

Defines a base task configuration for tasks with multiple subsets, then creates specific task instances for each subset. This approach is useful for datasets with distinct parts that require separate evaluation configurations.

```python
from lighteval.tasks.lighteval_task import LightevalTaskConfig

SAMPLE_SUBSETS = ["subset1", "subset2", "subset3"]  # List of all the subsets to use for this eval

class CustomSubsetTask(LightevalTaskConfig):
    def __init__(
        self,
        name,
        hf_subset,
    ):
        super().__init__(
            name=name,
            hf_subset=hf_subset,
            prompt_function=prompt_fn,  # Must be defined in the file or imported
            hf_repo="your_dataset_name",
            metrics=[custom_metric],  # Select your metric in Metrics or use your custom_metric
            hf_avail_splits=["train", "test"],
            evaluation_splits=["test"],
            few_shots_split="train",
            few_shots_select="random_sampling_from_train",
            generation_size=256,
            stop_sequence=["\n", "Question:"]
        )

SUBSET_TASKS = [CustomSubsetTask(name=f"task:{subset}", hf_subset=subset) for subset in SAMPLE_SUBSETS]
```

--------------------------------

### Manage LightEval Tasks using Bash

Source: https://context7.com/huggingface/lighteval/llms.txt

Commands to manage evaluation tasks, including listing all available tasks, inspecting a specific task's configuration, and creating new tasks from templates.

```bash
# List all available tasks
lighteval tasks list
```

```bash
# Inspect a specific task configuration and samples
lighteval tasks inspect gsm8k
```

```bash
# Create a new task from template
lighteval tasks create
```

--------------------------------

### Define a Basic Lighteval Task

Source: https://github.com/huggingface/lighteval/wiki/Adding-a-Custom-Task

Use `LightevalTaskConfig` to define a single evaluation task. Ensure the `prompt_function` is defined or imported, and specify metrics and dataset splits.

```python
task = LightevalTaskConfig(
    name="myothertask",
    prompt_function=prompt_fn,  # must be defined in the file or imported from src/lighteval/tasks/tasks_prompt_formatting.py
    suite=["community"],
    hf_repo="",
    hf_subset="default",
    hf_avail_splits=[],
    evaluation_splits=[],
    few_shots_split=None,
    few_shots_select=None,
    metric=[],  # select your metric in Metrics
)
```

--------------------------------

### Basic LiteLLM Endpoint Usage

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/use-litellm-as-backend.mdx

Use this command to evaluate a model via LiteLLM using the OpenAI provider and a specific model.

```bash
lighteval endpoint litellm \
    "provider=openai,model_name=gpt-3.5-turbo" \
    gsm8k
```

--------------------------------

### Evaluate Nanotron Model with Torchrun

Source: https://github.com/huggingface/lighteval/blob/main/docs/source/quicktour.mdx

Use this command to evaluate a Nanotron model on a single GPU. Ensure the `nproc-per-node` argument matches the parallelism configuration in your Lighteval config file.

```bash
torchrun --standalone --nnodes=1 --nproc-per-node=1 \
    src/lighteval/__main__.py nanotron \
    --checkpoint-config-path ../nanotron/checkpoints/10/config.yaml \
    --lighteval-config-path examples/nanotron/lighteval_config_override_template.yaml
```