### Install eva and vLLM

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/large_model_inference.md

Install the necessary packages for eva and vLLM. Refer to the vLLM installation guide for platform-specific instructions.

```bash
pip install "kaiko-eva[all]"
pip install vllm
```

--------------------------------

### Install EVA with Extras

Source: https://context7.com/kaiko-ai/eva/llms.txt

Install EVA with specific extras for vision, language, multimodal tasks, or all functionalities. Verify the installation using the --version flag.

```bash
pip install "kaiko-eva[vision]"

pip install "kaiko-eva[language]"

pip install "kaiko-eva[multimodal]"

pip install "kaiko-eva[all]"

# Verify installation
eva --version
```

--------------------------------

### Install kaiko-eva[language]

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/language/index.md

Install the language package for kaiko-eva. This command is used to set up the necessary dependencies for the language API.

```bash
pip install 'kaiko-eva[language]'
```

--------------------------------

### Load and Instantiate a Backbone Model

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_registry.md

Instantiate a model from the registry using its name. This example loads a ViT-small model with random weights and demonstrates how to get the CLS embedding output.

```python
import torch
from eva.vision.models.networks.backbones.registry import backbone_registry

model = backbone_registry.get("universal/vit_small_patch16_224_random")()
output = model(torch.randn(1, 3, 224, 224))
print(output.shape)
```

--------------------------------

### Install Multimodal Package

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/multimodal/index.md

Install the multimodal package using pip. This command installs the base package with multimodal capabilities.

```bash
pip install 'kaiko-eva[multimodal]'
```

--------------------------------

### Install kaiko-eva with Vision

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/getting-started/installation.md

Install the kaiko-eva package along with the eva-vision extra for vision capabilities. Ensure you have Python 3.10+ and an active virtual environment.

```bash
pip install "kaiko-eva[vision]"
```

--------------------------------

### Install EVA Core and Expanded Versions

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Install the core EVA package or expanded versions with specific functionalities like 'vision', 'language', 'multimodal', or 'all'. Use the git+https URL for the latest version from the main branch.

```sh
pip install kaiko-eva
```

```sh
pip install 'kaiko-eva[vision]'
```

```sh
pip install 'kaiko-eva[language]'
```

```sh
pip install 'kaiko-eva[multimodal]'
```

```sh
pip install 'kaiko-eva[all]'
```

```sh
pip install "kaiko-eva[all] @ git+https://github.com/kaiko-ai/eva.git"
```

--------------------------------

### Install vision Package

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/vision/index.md

Install the `vision`-package using pip. This command installs the necessary dependencies for the vision module.

```bash
pip install 'kaiko-eva[vision]'
```

--------------------------------

### BTCV Dataset Download Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/datasets/btcv.md

This code snippet demonstrates how to download the BTCV dataset during runtime by setting the `download` argument to `True` when initializing the dataset class.

```python
from eva.datasets import BTCV

# Download the BTCV dataset
BTCV(download=True)
```

--------------------------------

### Initialize and Run EVA Pipeline in Python

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Demonstrates how to initialize the Trainer, HeadModule, and DataModule, then fit the pipeline using the core EVA library in Python. This setup is equivalent to the provided YAML configuration.

```python
# main.py
# execute with: `python main.py`

from torch import nn

from eva import core
from eva.vision import datasets, transforms

# initialize trainer
trainer = core.Trainer(max_steps=100)

# initialize model
model = core.HeadModule(
  backbone=nn.Flatten(),
  head=nn.Linear(150528, 4),
  criterion=nn.CrossEntropyLoss(),
)

# initialize data
data = core.DataModule(
  datasets=core.DatasetsSchema(
    train=datasets.BACH(
      root="data/bach",
      split="train",
      download=True,
      transforms=transforms.ResizeAndCrop(),
    ),
  ),
  dataloaders=core.DataloadersSchema(
    train=core.DataLoader(batch_size=32),
  ),
)

# perform fit
pipeline = core.Interface()
pipeline.fit(trainer, model=model, data=data)
```

--------------------------------

### Install Python Dependencies with PDM

Source: https://github.com/kaiko-ai/eva/blob/main/docs/DEVELOPER_GUIDE.md

Installs Python dependencies for development, creating a virtual environment. Ensure your Python version matches NOX requirements.

```bash
PDM_PYTHON=$(pyenv which python) && pdm install -G all -G dev
```

--------------------------------

### Verify EVA Installation

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Check if the EVA installation was successful by running the version command in your terminal.

```sh
eva --version
```

--------------------------------

### Online Segmentation Quick Start

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Execute an online segmentation training pipeline using the fit command. Ensure DOWNLOAD_DATA is set to true and specify the model name and configuration file.

```sh
# define the model backbone
DOWNLOAD_DATA=true \
MODEL_NAME=universal/vit_small_patch16_224_dino \
\
# execute online segmentation training for MoNuSAC Dataset
eşva fit \
    --config https://raw.githubusercontent.com/kaiko-ai/ \
             eva/main/configs/vision/pathology/ \
             online/segmentation/monusac.yaml
```

--------------------------------

### Eva Configuration Example

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

This YAML configuration defines the trainer, model, and data modules for an Eva project. It specifies class paths and initialization arguments for various components.

```yaml
trainer:
  class_path: eva.Trainer
  init_args:
    max_steps: 100
model:
  class_path: eva.HeadModule
  init_args:
    backbone: torch.nn.Flatten
    head:
      class_path: torch.nn.Linear
      init_args:
        in_features: 150528
        out_features: 4
    criterion: torch.nn.CrossEntropyLoss
data:
  class_path: eva.DataModule
  init_args:
    datasets:
      train:
        class_path: eva.vision.datasets.BACH
        init_args:
          root: ./data/bach
          split: train
          download: true
          transforms: eva.vision.transforms.ResizeAndCrop
    dataloaders:
      train:
        batch_size: 32
```

--------------------------------

### Quick Start: Evaluate Qwen2.5-VL 72B

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/large_model_inference.md

Run an evaluation of the Qwen2.5-VL 72B model on the PatchCamelyon dataset using vLLM for distributed inference. This command configures model distribution, accelerator, and batch size.

```bash
MODEL_NAME="alibaba/qwen2-5-vl-72b-instruct-vllm" \
VLLM_TENSOR_PARALLEL_SIZE=4 \
ACCELERATOR=cpu \
NUM_DEVICES=1 \
BATCH_SIZE=256 \
EVA fit --config configs/multimodal/pathology/online/multiple_choice/patch_camelyon.yaml
```

--------------------------------

### Evaluate Large Models via vLLM

Source: https://context7.com/kaiko-ai/eva/llms.txt

Evaluate large models using vLLM for distributed inference. This example demonstrates setting up vLLM with multiple GPUs and configuring batch size.

```bash
pip install vllm

# Evaluate 72B model across 4 GPUs
MODEL_NAME=alibaba/qwen2-5-vl-72b-instruct-vllm \
VLLM_TENSOR_PARALLEL_SIZE=4 \
ACCELERATOR=cpu \
NUM_DEVICES=1 \
BATCH_SIZE=256 \
eva fit --config configs/multimodal/pathology/online/multiple_choice/patch_camelyon.yaml
```

--------------------------------

### Python Class Docstring Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/STYLE_GUIDE.md

Illustrates the docstring format for public classes in Python, explaining the class's purpose and its initialization parameters.

```python
class DataModule(pl.LightningDataModule):
    """DataModule encapsulates all the steps needed to process data.

    It will initialize and create the mapping between dataloaders and
    datasets. During the `prepare_data`, `setup` and `teardown`, the
    datamodule will call the respectively methods from all the datasets,
    given that they are defined.
    """

    def __init__(
        self,
        datasets: schemas.DatasetsSchema | None = None,
        dataloaders: schemas.DataloadersSchema | None = None,
    ) -> None:
        """Initializes the datamodule.

        Args:
            datasets: The desired datasets. Defaults to `None`.
            dataloaders: The desired dataloaders. Defaults to `None`.
        """
        pass
```

--------------------------------

### Python Function with Docstring Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/STYLE_GUIDE.md

Demonstrates a Python function with a comprehensive docstring following Google Python style, including type hints, argument descriptions, return values, and cross-referencing.

```python
def example_function(variable: int, optional: str | None = None) -> str:
    """An example docstring that explains what this functions do.

    Docs sections can be referenced via :ref:`custom text here <anchor-link>`.

    Classes can be referenced via :class:`eva.data.datamodules.DataModule`.

    Functions can be referenced via :func:`eva.data.datamodules.call.call_method_if_exists`.

    Example:

        >>> from torch import nn
        >>> import eva
        >>> eva.models.modules.HeadModule(
        >>>     head=nn.Linear(10, 2),
        >>>     criterion=nn.CrossEntropyLoss(),
        >>> )

    Args:
        variable: A required argument.
        optional: An optional argument.

    Returns:
        A description of the output string.
    """
    pass
```

--------------------------------

### Python Function Docstring with Args, Returns, Raises

Source: https://github.com/kaiko-ai/eva/blob/main/docs/STYLE_GUIDE.md

Provides an example of a Python function docstring for public functions, detailing arguments, return values, and potential exceptions raised.

```python
def fake_datamodule(
    n_samples: int, random: bool = True
) -> eva.data.datamodules.DataModule:
    """Generates a fake DataModule.

    It builds a :class:`eva.data.datamodules.DataModule` by generating
    a fake dataset with generated data while fixing the seed. It can
    be useful for debugging purposes.

    Args:
        n_samples: The number of samples of the generated datasets.
        random: Whether to generated randomly.

    Returns:
        A :class:`eva.data.datamodules.DataModule` with generated random data.

    Raises:
        ValueError: If `n_samples` is `0`.
    """
    pass
```

--------------------------------

### Use Custom Model Wrapper in Configuration

Source: https://context7.com/kaiko-ai/eva/llms.txt

Example of how to use a custom model wrapper in a YAML configuration file. Specify the class path to your custom wrapper and its initialization arguments.

```yaml
# Use in a config
backbone:
  class_path: my_wrapper.MyCustomWrapper
  init_args:
    model_path: /path/to/model.bin
    embedding_dim: 768
```

--------------------------------

### Dataset Organization Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/datasets/msd_task7_pancreas.md

Illustrates the expected directory structure for the training data of the MSD Task 7 Pancreas dataset. Ensure images and labels are placed in their respective 'imagesTr' and 'labelsTr' folders.

```text
Dataset007_Pancreas
├── imagesTr/
│   ├── pancreas_001_0000.nii.gz
│   ├── pancreas_002_0000.nii.gz
│   └── ...
└── labelsTr/
    ├── pancreas_001.nii.gz
    ├── pancreas_002.nii.gz
    └── ...
```

--------------------------------

### Python Multiline Docstring Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/STYLE_GUIDE.md

Shows the correct structure for a multiline docstring in Python, including a summary line and subsequent paragraphs, adhering to PEP-8 and PEP-257.

```python
"""First line of multiline docstring that shouldn't wrap.

Subsequent line or paragraphs.
"""
```

--------------------------------

### End-to-end ResNet18 Training

Source: https://context7.com/kaiko-ai/eva/llms.txt

Example of end-to-end ResNet18 training without a frozen backbone, using environment variables to configure training parameters.

```bash
# End-to-end ResNet18 training (no frozen backbone)
OUTPUT_ROOT=logs/resnet18/bach MAX_STEPS=5000 LR_VALUE=0.01 \
    eva fit --config configs/vision/resnet18/bach.yaml
```

--------------------------------

### List Available Backbone Models

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_registry.md

Use this snippet to retrieve a list of all available model names from the backbone registry. Ensure the 'eva' package is installed.

```python
from eva.vision.models.networks.backbones.registry import backbone_registry

models = backbone_registry.entries()
print(models)
```

--------------------------------

### LiteLLM Model Configuration in Eva

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/pubmedqa_classification.md

Example of how to configure a LiteLLM model within Eva's configuration, specifying the model class path and initialization arguments, including an environment variable for the model name.

```yaml
model:
  class_path: eva.language.models.LiteLLMModel
  init_args:
    model_name_or_path: ${oc.env:MODEL_NAME, anthropic/claude-3-7-sonnet-latest}
```

--------------------------------

### Python Constant Docstring Example

Source: https://github.com/kaiko-ai/eva/blob/main/docs/STYLE_GUIDE.md

Demonstrates how to add docstrings to public constants in Python, placed directly below the constant definition.

```python
SOME_CONSTANT = 3
"""Either a single-line docstring or multiline as per above."""
```

--------------------------------

### Swap Backbone using Environment Variables

Source: https://context7.com/kaiko-ai/eva/llms.txt

Swap backbones without editing configuration files by setting the MODEL_NAME environment variable. This example also shows loading a timm model with a custom checkpoint.

```bash
# Swap backbone without editing configs
MODEL_NAME=pathology/kaiko_vitb8 IN_FEATURES=768 \
eva predict_fit --config configs/vision/pathology/offline/classification/crc.yaml

# Load a timm model with a custom checkpoint
MODEL_NAME=timm/vit_small_patch16_224 \
MODEL_EXTRA_KWARGS='{checkpoint_path: path/to/model.ckpt}' \
    eva predict_fit --config configs/vision/pathology/offline/segmentation/consep.yaml
```

--------------------------------

### Offline Classification Quick Start

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Execute an offline classification pipeline using the predict_fit command. Ensure DOWNLOAD_DATA is set to true and specify the model name and configuration file.

```sh
# set the model architecture
DOWNLOAD_DATA=true \
MODEL_NAME=universal/vit_small_patch16_224_dino \
\
# execute the offline evaluation pipeline with the BACH dataset config
eşva predict_fit \
    --config https://raw.githubusercontent.com/kaiko-ai/ \
             eva/main/configs/vision/pathology/ \
             offline/classification/bach.yaml
```

--------------------------------

### Configure TimmModel for Feature Extraction

Source: https://context7.com/kaiko-ai/eva/llms.txt

Dedicated wrapper for timm models with native support for out_indices for feature map extraction. This example configures a ViT model to return the last feature map and accept variable input sizes.

```yaml
backbone:
  class_path: eva.vision.models.wrappers.TimmModel
  init_args:
    model_name: vit_tiny_patch16_224
    pretrained: true
    out_indices: 1                   # Return last feature map
    model_kwargs:
      dynamic_img_size: true         # Accept variable input sizes
```

--------------------------------

### Eva YAML Configuration for Offline Pathology Classification

Source: https://context7.com/kaiko-ai/eva/llms.txt

A complete YAML configuration file for offline pathology classification (BACH) using Eva. This example details the 'trainer', 'model', and 'data' sections, including settings for runs, directories, callbacks, logging, model architecture, loss functions, optimizers, and dataset configurations. Environment variables are used for dynamic configuration.

```yaml
# configs/vision/pathology/offline/classification/bach.yaml
---
trainer:
  class_path: eva.Trainer
  init_args:
    n_runs: ${oc.env:N_RUNS, 5}          # Statistical robustness — runs N times, reports mean±std
    default_root_dir: logs/${oc.env:MODEL_NAME, dino_vits16}/offline/bach
    max_steps: ${oc.env:MAX_STEPS, 12500}
    checkpoint_type: ${oc.env:CHECKPOINT_TYPE, best}  # "best" or "last"
    accelerator: ${oc.env:ACCELERATOR, auto}
    devices: ${oc.env:NUM_DEVICES, 1}
    callbacks:
      - class_path: eva.callbacks.ConfigurationLogger
      - class_path: lightning.pytorch.callbacks.TQDMProgressBar
      - class_path: lightning.pytorch.callbacks.LearningRateMonitor
        init_args:
          logging_interval: epoch
      - class_path: lightning.pytorch.callbacks.ModelCheckpoint
        init_args:
          filename: best
          save_top_k: 1
          monitor: ${oc.env:MONITOR_METRIC, val/MulticlassAccuracy}
          mode: ${oc.env:MONITOR_METRIC_MODE, max}
      - class_path: lightning.pytorch.callbacks.EarlyStopping
        init_args:
          patience: ${oc.env:PATIENCE, 1250}
          monitor: val/MulticlassAccuracy
          mode: max
      # Backbone lives in the callback for offline mode — computes and caches embeddings
      - class_path: eva.callbacks.ClassificationEmbeddingsWriter
        init_args:
          output_dir: ${oc.env:EMBEDDINGS_ROOT, ./data/embeddings}/${oc.env:MODEL_NAME, dino_vits16}/bach
          dataloader_idx_map:
            0: train
            1: val
          backbone:
            class_path: eva.vision.models.ModelFromRegistry
            init_args:
              model_name: ${oc.env:MODEL_NAME, universal/vit_small_patch16_224_dino}
          overwrite: false
    logger:
      - class_path: lightning.pytorch.loggers.TensorBoardLogger
        init_args:
          save_dir: logs/${oc.env:MODEL_NAME, dino_vits16}/offline/bach
          name: ""

model:
  class_path: eva.HeadModule
  init_args:
    head:
      class_path: torch.nn.Linear
      init_args:
        in_features: ${oc.env:IN_FEATURES, 384}
        out_features: 4                   # Number of BACH classes
    criterion: torch.nn.CrossEntropyLoss
    optimizer:
      class_path: torch.optim.AdamW
      init_args:
        lr: ${oc.env:LR_VALUE, 0.0003}
    metrics:
      common:
        - class_path: eva.metrics.AverageLoss
        - class_path: eva.metrics.MulticlassClassificationMetrics
          init_args:
            num_classes: 4

data:
  class_path: eva.DataModule
  init_args:
    datasets:
      train:
        class_path: eva.datasets.EmbeddingsClassificationDataset  # Load cached embeddings
        init_args:
          root: ./data/embeddings/${oc.env:MODEL_NAME, dino_vits16}/bach
          manifest_file: manifest.csv
          split: train
      val:
        class_path: eva.datasets.EmbeddingsClassificationDataset
        init_args:
          root: ./data/embeddings/${oc.env:MODEL_NAME, dino_vits16}/bach
          manifest_file: manifest.csv
          split: val
      predict:                            # Raw images used only during predict stage
        - class_path: eva.vision.datasets.BACH
          init_args:
            root: ${oc.env:DATA_ROOT, ./data/bach}
            split: train
            download: ${oc.env:DOWNLOAD_DATA, false}
            transforms:
              class_path: eva.vision.data.transforms.common.ResizeAndCrop
              init_args:
                size: ${oc.env:RESIZE_DIM, 224}
                mean: ${oc.env:NORMALIZE_MEAN, [0.485, 0.456, 0.406]}
                std: ${oc.env:NORMALIZE_STD, [0.229, 0.224, 0.225]}
    dataloaders:
      train:
        batch_size: ${oc.env:BATCH_SIZE, 256}
        num_workers: ${oc.env:N_DATA_WORKERS, 4}
        shuffle: true
      val:
        batch_size: ${oc.env:BATCH_SIZE, 256}
        num_workers: 4
      predict:
        batch_size: ${oc.env:PREDICT_BATCH_SIZE, 64}
        num_workers: 4

```

--------------------------------

### View Training Curves with Tensorboard

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/offline_vs_online.md

Launch Tensorboard to visualize training curves from offline evaluations. Point the log directory to the relevant path within the `logs` directory.

```bash
tensorboard --logdir logs/$MODEL_NAME/offline/bach
```

--------------------------------

### Download and Run PathMMU Atlas

Source: https://github.com/kaiko-ai/eva/blob/main/docs/datasets/path_mmu_atlas.md

Set DOWNLOAD_DATA to 'true' to automatically download the dataset. Then, run the eva test command with the specified configuration file.

```bash
DOWNLOAD_DATA="true" eva test --config configs/multimodal/pathology/online/multiple_choice/path_mmu_atlas.yaml
```

--------------------------------

### Run eva Command

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/getting-started/installation.md

Execute the eva command-line tool with a subcommand and configuration file. Refer to the 'How to use eva' documentation for details on subcommands and configurations.

```bash
eva <subcommand> --config <path-to-config-file>
```

--------------------------------

### List Available Backbone Models

Source: https://context7.com/kaiko-ai/eva/llms.txt

Use `backbone_registry.entries()` to get a list of all registered backbone model names. This is useful for exploring available pre-trained models.

```python
from eva.vision.models.networks.backbones.registry import backbone_registry

# List all available models
models = backbone_registry.entries()
# ['universal/vit_small_patch16_224_random', 'universal/vit_small_patch16_224_dino',
#  'pathology/kaiko_vits16', 'pathology/kaiko_vitb8', 'pathology/owkin_phikon',
#  'pathology/mahmood_uni', 'pathology/bioptimus_h_optimus_0', 'radiology/voco_b', ...]

```

--------------------------------

### Load Backbone for Classification

Source: https://context7.com/kaiko-ai/eva/llms.txt

Load a backbone model for classification tasks using `backbone_registry.get()`. The model returns a CLS embedding. Ensure PyTorch is installed and imported.

```python
import torch
model = backbone_registry.get("universal/vit_small_patch16_224_dino")()
output = model(torch.randn(1, 3, 224, 224))
print(output.shape)  # torch.Size([1, 384])

```

--------------------------------

### Evaluate UNI (MahmoodLab) - DINOv2 ViT-L16 (Mass-100k)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Evaluate the UNI FM from MahmoodLab, pretrained on Mass-100k. Access to the model on HuggingFace needs to be requested, and your HF_TOKEN must be provided.

```bash
MODEL_NAME=pathology/mahmood_uni \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=1024 \
HF_TOKEN=<your-huggingace-token-for-downloading-the-model> \

```

```bash
eva predict_fit --config configs/vision/phikon/offline/<task>.yaml
```

--------------------------------

### Load Backbone for Segmentation

Source: https://context7.com/kaiko-ai/eva/llms.txt

Load a backbone model for segmentation tasks by specifying `out_indices`. The model returns a list of feature maps. Ensure PyTorch is installed and imported.

```python
import torch
model = backbone_registry.get("universal/vit_small_patch16_224_dino")(out_indices=2)
outputs = model(torch.randn(1, 3, 224, 224))
for feat in outputs:
    print(feat.shape)
# torch.Size([1, 384, 14, 14])
# torch.Size([1, 384, 14, 14])

```

--------------------------------

### Evaluate DINO ViT-S16 (ImageNet)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Run this command to evaluate a ViT-S16 backbone pretrained with ImageNet weights. Replace <task>.yaml with the appropriate configuration file.

```bash
MODEL_NAME="universal/vit_small_patch16_224_dino" \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=384 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Evaluate DINO ViT-S16 (random weights)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Use this command to evaluate the backbone with randomly initialized weights as a baseline. Ensure the config file is correctly specified.

```bash
MODEL_NAME="universal/vit_small_patch16_224_random" \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=384 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Load Backbone Model with Feature Maps

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_registry.md

For tasks requiring feature maps, use the `out_indices` argument to specify which feature levels to extract. This example retrieves the last two feature maps.

```python
model = backbone_registry.get("universal/vit_small_patch16_224_random")(out_indices=2)
outputs = model(torch.randn(1, 3, 224, 224))
for output in outputs:
    print(output.shape)
```

--------------------------------

### LiteLLMModel

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/language/models/wrappers.md

Documentation for the LiteLLMModel wrapper.

```APIDOC
## LiteLLMModel

This wrapper allows integration with various LLMs through the LiteLLM library.

### Description

Utilize LiteLLM's extensive model support via this wrapper.

### Usage

```python
from eva.language.models.wrappers import LiteLLMModel

model = LiteLLMModel(model_name="gpt-3.5-turbo")
response = model.generate(prompt="What is the capital of France?")
print(response)
```

### Parameters

- **model_name** (str) - Required - The name of the model as supported by LiteLLM.
- **api_key** (str) - Optional - Your API key for the chosen model provider.
- **base_url** (str) - Optional - The base URL for custom endpoints.
```

--------------------------------

### Run ResNet Training and Evaluation

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/evaluate_resnet.md

Execute the training and evaluation process for the ResNet model using Eva's fit command. Customize training parameters like output root, max steps, and learning rate via environment variables.

```bash
OUTPUT_ROOT=logs/resnet/bach \
MAX_STEPS=50 \
LR_VALUE=0.01 \

```

```bash
eva fit --config configs/vision/resnet18/bach.yaml
```

--------------------------------

### Set API Keys for LiteLLM Providers

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/pubmedqa_classification.md

Set environment variables for your API keys before using LiteLLM with providers like OpenAI, Anthropic, or Together.ai.

```bash
export OPENAI_API_KEY=your_openai_api_key
```

```bash
export ANTHROPIC_API_KEY=your_anthropic_api_key
```

```bash
export TOGETHER_API_KEY=your_together_api_key
```

--------------------------------

### Evaluate Phikon-v2 (Owkin) - DINOv2 ViT-L16 (PANCAN-XL)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Use this command to evaluate Owkin's Phikon-v2 FM, trained with DINOv2 on PANCAN-XL. Verify the config path and model accessibility.

```bash
MODEL_NAME=pathology/owkin_phikon_v2 \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=1024 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Run EVA CLI Tool

Source: https://github.com/kaiko-ai/eva/blob/main/README.md

Execute EVA as a CLI tool by specifying the desired command (fit, predict, predict_fit) and providing a configuration file path or URL.

```sh
eva {fit,predict,predict_fit} --config url/or/path/to/the/config.yaml
```

--------------------------------

### Run Complete Offline Workflow with Eva Predict-Fit

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/offline_vs_online.md

Execute a complete offline workflow, including embedding computation and model fitting, with a single `predict_fit` command. This example uses an FM pretrained on ImageNet.

```bash
MODEL_NAME=universal/vit_small_patch16_224_dino \
EMBEDDINGS_ROOT=./data/embeddings/dino_vits16_imagenet \
N_RUNS=2 \
MAX_STEPS=20 \
LR_VALUE=0.1 \
EVA predict_fit --config configs/vision/pathology/offline/classification/bach.yaml
```

--------------------------------

### VllmModel

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/language/models/wrappers.md

Documentation for the VllmModel wrapper.

```APIDOC
## VllmModel

This wrapper interfaces with models served via the vLLM engine.

### Description

Optimized for high-throughput LLM serving with vLLM.

### Usage

```python
from eva.language.models.wrappers import VllmModel

model = VllmModel(model_name="lmsys/vicuna-7b-v1.5")
response = model.generate(prompt="Explain the concept of recursion.")
print(response)
```

### Parameters

- **model_name** (str) - Required - The name or path of the vLLM compatible model.
- **host** (str) - Optional - The host address of the vLLM server.
- **port** (int) - Optional - The port of the vLLM server.
```

--------------------------------

### CoNSeP Dataset Directory Structure

Source: https://github.com/kaiko-ai/eva/blob/main/docs/datasets/consep.md

Illustrates the organization of the CoNSeP dataset files, including training and testing images and labels.

```bash
consep
├── Train
│   ├── Images                 # raw training input images
│   │   ├── train_1.png
│   │   └── ...
│   ├── Labels                 # train segmentation labels        
│   │   ├── train_1.mat
│   │   └── ...
│   ├── Overlay                # train images with bounding boxes, not in use
├── Test
│   ├── Images                 # raw test input images
│   │   ├── test_1.png
│   │   └── ...
│   ├── Labels                 # test segmentation labels        
│   │   ├── test_1.mat
│   │   └── ...
│   ├── Overlay                # test images with bounding boxes, not in use
└── README.txt                 # data description
```

--------------------------------

### Evaluate UNI2-h (MahmoodLab) - DINOv2 ViT-G14

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Command to evaluate the UNI2-h FM from MahmoodLab. Model access requires a request on HuggingFace, and your HF_TOKEN must be supplied.

```bash
MODEL_NAME=pathology/mahmood_uni2_h \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=1536 \
HF_TOKEN=<your-huggingace-token-for-downloading-the-model> \

```

```bash
eva predict_fit --config configs/vision/phikon/offline/<task>.yaml
```

--------------------------------

### Compare Model Sizes Within a Provider

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/pubmedqa_classification.md

Compare the performance of different model sizes from the same API provider. This helps in understanding the trade-offs between model capability and computational cost.

```bash
# Compare model sizes within a provider
MODEL_NAME=anthropic/claude-3-haiku-20240307 eva validate --config configs/language/pubmedqa.yaml
MODEL_NAME=anthropic/claude-3-sonnet-20240229 eva validate --config configs/language/pubmedqa.yaml
```

--------------------------------

### Evaluate Lunit - DINO ViT-S16 (TCGA)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Evaluate using Lunit's DINO ViT-S16 backbone pretrained on TCGA data. Ensure you have downloaded the weights and specify the correct config file.

```bash
MODEL_NAME=pathology/lunit_vits16
NORMALIZE_MEAN="[0.70322989,0.53606487,0.66096631]" \
NORMALIZE_STD="[0.21716536,0.26081574,0.20723464]" \
IN_FEATURES=384 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Run Basic PubMedQA Evaluation with Default Config

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/pubmedqa_classification.md

Perform a basic PubMedQA evaluation using Eva's default configuration, which typically uses LiteLLM with an Anthropic Claude model.

```bash
eva validate --config configs/language/pubmedqa.yaml
```

--------------------------------

### TimmModel Wrapper

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/vision/models/wrappers.md

Documentation for the TimmModel wrapper, which allows integration with models from the `timm` library.

```APIDOC
## TimmModel

### Description
This wrapper provides access to various pre-trained models available in the `timm` library.

### Usage
```python
from eva.vision.models.wrappers import TimmModel

# Instantiate the wrapper with a specific model name
model_wrapper = TimmModel(model_name="resnet50")

# You can then use this wrapper to load and utilize the specified model.
```

### Parameters
- **model_name** (str) - Required - The name of the model to load from the `timm` library (e.g., "resnet50", "vit_base_patch16_224").
- **pretrained** (bool) - Optional - Whether to load pre-trained weights. Defaults to True.
- **num_classes** (int) - Optional - The number of classes for the final classification layer. If not provided, the default from the `timm` model will be used.

### Methods
- **load_model()**: Loads the specified `timm` model.
- **forward(x)**: Performs a forward pass through the model.

### Response
- **Success Response**: The `TimmModel` object is returned upon successful instantiation.
- **Error Response**: Raises an error if the `model_name` is invalid or if there are issues loading the model weights.
```

--------------------------------

### Pathology FM Benchmarking (Offline)

Source: https://context7.com/kaiko-ai/eva/llms.txt

Use this configuration for offline benchmarking of pathology vision foundation models. Requires Hugging Face access and a token.

```bash
MODEL_NAME=pathology/mahmood_uni \
NORMALIZE_MEAN="[0.485,0.456,0.406]" NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=1024 \
HF_TOKEN=<your-huggingface-token> \

```

--------------------------------

### Run Online Evaluation Workflow

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/offline_vs_online.md

Use this command to perform a complete online evaluation. Ensure environment variables like MODEL_NAME, N_RUNS, MAX_STEPS, and LR_VALUE are set appropriately for your task. The configuration file specifies the task details.

```bash
MODEL_NAME=universal/vit_small_patch16_224_dino \
N_RUNS=1 \
MAX_STEPS=20 \
LR_VALUE=0.1 \

```

```bash
eva fit --config configs/vision/pathology/online/classification/bach.yaml
```

--------------------------------

### Set API Keys for Eva

Source: https://context7.com/kaiko-ai/eva/llms.txt

Set environment variables for API keys before running Eva commands. Ensure you replace 'your_api_key' with your actual credentials.

```bash
export ANTHROPIC_API_KEY=your_anthropic_api_key
export OPENAI_API_KEY=your_openai_api_key
```

--------------------------------

### Evaluate Phikon (Owkin) - iBOT ViT-B16 (TCGA)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Evaluate using Owkin's Phikon FM, trained with iBOT on TCGA data. Ensure the model weights are accessible and the config path is correct.

```bash
MODEL_NAME=pathology/owkin_phikon \
NORMALIZE_MEAN="[0.485,0.456,0.406]" \
NORMALIZE_STD="[0.229,0.224,0.225]" \
IN_FEATURES=768 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Compare Different API Providers

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/tutorials/pubmedqa_classification.md

Run evaluations to compare the performance of different language models from various API providers on the PubMedQA test set. Set the MODEL_NAME environment variable to specify the model to evaluate.

```bash
# Compare different API providers
MODEL_NAME=anthropic/claude-3-sonnet-20240229 eva validate --config configs/language/pubmedqa.yaml
MODEL_NAME=openai/gpt-4o eva validate --config configs/language/pubmedqa.yaml
```

--------------------------------

### Run Free-Form VQA Evaluation

Source: https://context7.com/kaiko-ai/eva/llms.txt

This command executes the free-form VQA evaluation using the specified configuration. Ensure API keys for Anthropic and Google are set as environment variables.

```bash
# Run free-form VQA evaluation
export ANTHROPIC_API_KEY=your_key
export GOOGLE_API_KEY=your_gemini_key  # For judge model

MODEL_NAME=anthropic/claude-3-7-sonnet-20250219 \
DOWNLOAD_DATA=true \
    eva fit --config configs/multimodal/pathology/online/free_form/quilt_vqa.yaml
```

--------------------------------

### Download and Validate PubMedQA Dataset

Source: https://github.com/kaiko-ai/eva/blob/main/docs/datasets/pubmedqa.md

Use this command to download the PubMedQA dataset and validate it with the 'eva' tool. Ensure 'DOWNLOAD_DATA' is set to 'true' and specify the configuration file.

```bash
DOWNLOAD_DATA="true" eva validate --config configs/language/pubmedqa.yaml
```

--------------------------------

### Pre-configured Models

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/multimodal/models/networks.md

Information on pre-configured models available through the Networks API.

```APIDOC
## Pre-configured Models

::: eva.multimodal.models.networks.Claude35Sonnet20240620
::: eva.multimodal.models.networks.Claude37Sonnet20250219
::: eva.multimodal.models.networks.PathoR13b
::: eva.multimodal.models.networks.Qwen25VL7BInstruct
```

--------------------------------

### Run Specific Unit Tests with Nox

Source: https://github.com/kaiko-ai/eva/blob/main/docs/DEVELOPER_GUIDE.md

Executes a specific unit test file using Nox.

```bash
nox -s test -- tests/eva/metrics/test_average_loss.py
```

--------------------------------

### Evaluate Lunit - DINO ViT-S8 (TCGA)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Command to evaluate with Lunit's DINO ViT-S8 backbone pretrained on TCGA. Adjust the config path as needed.

```bash
MODEL_NAME=pathology/lunit_vits8 \
NORMALIZE_MEAN="[0.70322989,0.53606487,0.66096631]" \
NORMALIZE_STD="[0.21716536,0.26081574,0.20723464]" \
IN_FEATURES=384 \

```

```bash
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Evaluate kaiko.ai DINOv2 Midnight-12k (TCGA)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Use this command to evaluate the kaiko.ai DINOv2 Midnight-12k (ViT-G14) model pretrained on TCGA data. Ensure the MODEL_NAME, NORMALIZE_MEAN, NORMALIZE_STD, and IN_FEATURES are set correctly.

```bash
MODEL_NAME=pathology/kaiko_midnight_12k \
NORMALIZE_MEAN="[0.5,0.5,0.5]" \
NORMALIZE_STD="[0.5,0.5,0.5]" \
IN_FEATURES=1536 \
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### ModelFromRegistry

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/multimodal/models/wrappers.md

Represents a wrapper for models loaded from a registry.

```APIDOC
## ModelFromRegistry

### Description
Wrapper for models loaded from a registry.

### Class
`eva.multimodal.models.wrappers.ModelFromRegistry`
```

--------------------------------

### Configure Backbone Model in YAML

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_registry.md

This YAML configuration snippet shows how to specify a backbone model using `eva.vision.models.ModelFromRegistry`. It utilizes environment variables for model name and output indices.

```yaml
backbone:
  class_path: eva.vision.models.ModelFromRegistry
  init_args:
    model_name: ${oc.env:MODEL_NAME, universal/vit_small_patch16_224_dino}
    model_kwargs:
      out_indices: ${oc.env:OUT_INDICES, 1}
```

--------------------------------

### Load ONNX Models

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_wrappers.md

Load `.onnx` model checkpoints using the `ONNXModel` wrapper class. Specify the model path and the target device.

```yaml
class_path: eva.models.wrappers.ONNXModel
init_args:
  path: path/to/model.onnx
  device: cuda
```

--------------------------------

### Load Model from Torch Hub Function

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_wrappers.md

Use `ModelFromFunction` to load models from `torch.hub.load`. If `checkpoint_path` is provided, the model will be initialized with weights from that checkpoint.

```yaml
backbone:
  class_path: eva.models.wrappers.ModelFromFunction
  init_args:
    path: torch.hub.load
    arguments:
      repo_or_dir: facebookresearch/dino:main
      model: dino_vits16
      pretrained: false
    checkpoint_path: path/to/your/checkpoint.torch
```

--------------------------------

### Run Online Classification with Eva

Source: https://context7.com/kaiko-ai/eva/llms.txt

Execute an online classification task using the 'eva fit' command. This command computes backbone embeddings on-the-fly and trains the model in a single step. Ensure environment variables like MODEL_NAME, N_RUNS, MAX_STEPS, and LR_VALUE are set.

```bash
# Online classification — backbone + head in one shot
MODEL_NAME=universal/vit_small_patch16_224_dino \
N_RUNS=2 \
MAX_STEPS=20 \
LR_VALUE=0.1 \
 eva fit --config configs/vision/pathology/online/classification/bach.yaml
```

--------------------------------

### Configure Free-Form VQA with QuiltVQA Dataset

Source: https://context7.com/kaiko-ai/eva/llms.txt

This configuration sets up EVA for free-form Visual Question Answering using the QuiltVQA dataset. It specifies the model, metrics, and data loading parameters, including image transformations and batching.

```yaml
model:
  class_path: eva.multimodal.models.modules.VisionLanguageModule
  init_args:
    model:
      class_path: eva.multimodal.models.wrappers.ModelFromRegistry
      init_args:
        model_name: ${oc.env:MODEL_NAME, anthropic/claude-3-7-sonnet-20250219}
    metrics:
      common:
        - class_path: eva.language.metrics.GEvalCorrectness
          init_args:
            model:
              # A separate LLM acts as a judge to score free-form responses
              class_path: eva.language.models.wrappers.ModelFromRegistry
              init_args:
                model_name: ${oc.env:JUDGE_MODEL_NAME, google/gemini-2.5-flash-lite}
            missing_limit: 5

data:
  class_path: eva.DataModule
  init_args:
    datasets:
      test:
        class_path: eva.multimodal.data.datasets.QuiltVQA
        init_args:
          root: ${oc.env:DATA_ROOT, ./data/quilt_vqa}
          split: test
          download: ${oc.env:DOWNLOAD_DATA, false}
          transforms:
            image:
              class_path: eva.vision.data.transforms.Resize
              init_args:
                size: ${oc.env:RESIZE_DIM, null}
                max_bytes: ${oc.env:IMAGE_MAX_BYTES, null}  # Compress large images
    dataloaders:
      test:
        batch_size: ${oc.env:BATCH_SIZE, 16}
        num_workers: 1
        collate_fn: eva.multimodal.data.dataloaders.text_image_collate
    samplers:
      test:
        class_path: eva.core.data.samplers.RandomSampler
        init_args:
          seed: 42
          num_samples: ${oc.env:NUM_SAMPLES, null}
```

--------------------------------

### Update All Dependencies

Source: https://github.com/kaiko-ai/eva/blob/main/docs/DEVELOPER_GUIDE.md

Updates all project dependencies to their latest compatible versions and regenerates the lock file.

```bash
pdm update
```

--------------------------------

### Evaluate kaiko.ai DINO ViT-S16 (TCGA)

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/replicate_evaluations.md

Use this command to evaluate the kaiko.ai DINO ViT-S16 model pretrained on TCGA data. Ensure the MODEL_NAME, NORMALIZE_MEAN, NORMALIZE_STD, and IN_FEATURES are set correctly.

```bash
MODEL_NAME=pathology/kaiko_vits16 \
NORMALIZE_MEAN="[0.5,0.5,0.5]" \
NORMALIZE_STD="[0.5,0.5,0.5]" \
IN_FEATURES=384 \
eva predict_fit --config configs/vision/pathology/offline/<task>.yaml
```

--------------------------------

### Backbone Registry Operations

Source: https://context7.com/kaiko-ai/eva/llms.txt

Demonstrates how to list available models, load models for classification and segmentation, and register custom models.

```APIDOC
## Backbone Registry Operations

### Description
This section details how to interact with the `backbone_registry` to manage and utilize pre-registered backbone models.

### List Available Models
To see all available pre-registered models:
```python
from eva.vision.models.networks.backbones.registry import backbone_registry

models = backbone_registry.entries()
print(models)
```

### Load Model for Classification
Loads a backbone model suitable for classification, returning a CLS embedding.
```python
import torch
from eva.vision.models.networks.backbones.registry import backbone_registry

# Load a specific model (e.g., for classification)
model = backbone_registry.get("universal/vit_small_patch16_224_dino")()
input_tensor = torch.randn(1, 3, 224, 224)
output = model(input_tensor)
print(output.shape) # Expected: torch.Size([1, 384])
```

### Load Model for Segmentation
Loads a backbone model for segmentation, returning a list of feature maps.
```python
import torch
from eva.vision.models.networks.backbones.registry import backbone_registry

# Load a specific model for segmentation, specifying output indices
model = backbone_registry.get("universal/vit_small_patch16_224_dino")(out_indices=2)
input_tensor = torch.randn(1, 3, 224, 224)
outputs = model(input_tensor)
for feat in outputs:
    print(feat.shape) # Expected: torch.Size([1, 384, 14, 14]) for each feature map
```

### Register Custom Backbone
Allows registration of a custom model under a specified name.
```python
from eva.vision.models.networks.backbones.registry import backbone_registry
import timm
import torch

@backbone_registry.register("custom/my_model")
def my_model(out_indices=None):
    """Returns an nn.Module for feature extraction."""
    # Example using timm to create a backbone
    return timm.create_model("vit_base_patch16_224", pretrained=True, num_classes=0)

# To load the custom model after registration:
# custom_model = backbone_registry.get("custom/my_model")()
```
```

--------------------------------

### eva.data.DataModule

Source: https://github.com/kaiko-ai/eva/blob/main/docs/reference/core/data/datamodules.md

Reference for the core DataModule class in Eva.

```APIDOC
## eva.data.DataModule

### Description
Represents a base class for data modules in Eva, providing a structured way to handle datasets and their associated transformations.

### Usage
This class is intended to be subclassed to create specific data modules for different tasks.
```

--------------------------------

### Load Timm Models with Specific Wrapper

Source: https://github.com/kaiko-ai/eva/blob/main/docs/user-guide/advanced/model_wrappers.md

Use the `TimmModel` wrapper for loading models from the `timm` library. Specify `pretrained=true` and `out_indices` to return feature maps.

```yaml
backbone:
  class_path: eva.vision.models.wrappers.TimmModel
  init_args:
    model_name: vit_tiny_patch16_224
    pretrained: true
    out_indices=1 # to return the last feature map
    model_kwargs:
      dynamic_img_size: true
```