### Install MMEval with all dependencies

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/get_started/installation.md

Install MMEval along with all required dependencies for metrics.

```bash
pip install 'mmeval[all]'
```

--------------------------------

### Install MMEval

Source: https://github.com/open-mmlab/mmeval/blob/main/README.md

Install MMEval using pip. For all dependencies, use the '[all]' option.

```bash
pip install mmeval
```

```bash
pip install 'mmeval[all]'
```

--------------------------------

### Install Documentation Dependencies

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/README.md

Install the Python packages required for building the documentation.

```bash
pip install -r requirements/docs.txt
```

--------------------------------

### Install MMEval

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/get_started/installation.md

Install MMEval using pip. Requires Python 3.6+.

```bash
pip install mmeval
```

--------------------------------

### Install pre-commit hooks

Source: https://github.com/open-mmlab/mmeval/blob/main/CONTRIBUTING.md

Install the pre-commit hooks in the repository. This ensures that code linters and formatters are enforced on every commit.

```shell
pre-commit install
```

--------------------------------

### Install pre-commit hook

Source: https://github.com/open-mmlab/mmeval/blob/main/CONTRIBUTING.md

Install the pre-commit package to manage hooks. This is a prerequisite for setting up the project's commit hooks.

```shell
pip install -U pre-commit
```

--------------------------------

### Install MPI4Py and OpenMPI

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Install the necessary libraries for MPI4Py distributed communication using conda.

```bash
conda install openmpi
conda install mpi4py
```

--------------------------------

### List Available Distributed Backends

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Prints a list of all supported distributed communication backends in MMEval. Ensure MMEval is installed to use this function.

```python
import mmeval

print(mmeval.core.dist.list_all_backends())
```

--------------------------------

### Launch Distributed Evaluation with mpirun

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Use `mpirun` to launch multiple Python processes for distributed evaluation. This example launches 3 processes.

```bash
# Launch 3 processes with mpirun
mpirun -np 3 python3 cifar10_eval_mpi4py.py
```

--------------------------------

### Distributed Evaluation with PyTorch

Source: https://context7.com/open-mmlab/mmeval/llms.txt

This example demonstrates how to perform distributed evaluation using MMEval metrics with PyTorch. It shows how to initialize distributed processes, load data using `DistributedSampler`, and accumulate metric results across multiple ranks.

```APIDOC
## Distributed Evaluation with PyTorch — End-to-End Example
MMEval metrics integrate with `torch.distributed` for multi-GPU evaluation. Set `dist_backend='torch_cpu'` (CPU collectives) or `'torch_cuda'` (NCCL), call `metric.add()` per batch on each rank, then call `metric.compute(size=total_dataset_size)` to synchronize and get results on all ranks.

```python
import torch
import torchvision as tv
from torch.utils.data import DataLoader, DistributedSampler
from mmeval import Accuracy

def eval_fn(rank, process_num):
    torch.distributed.init_process_group(
        backend='gloo',
        init_method='tcp://127.0.0.1:2345',
        world_size=process_num,
        rank=rank,
    )

    dataset = tv.datasets.CIFAR10(
        root='./', train=False, download=True,
        transform=tv.transforms.ToTensor()
    )
    sampler = DistributedSampler(dataset, num_replicas=process_num, rank=rank)
    loader = DataLoader(dataset, batch_size=64, sampler=sampler)
    total_samples = len(dataset)

    model = tv.models.resnet18(num_classes=10).eval()

    # Each rank gets its own metric instance; dist_backend handles sync
    accuracy = Accuracy(topk=(1, 3), dist_backend='torch_cpu')

    with torch.no_grad():
        for images, labels in loader:
            scores = model(images)
            accuracy.add(predictions=scores, labels=labels)

    # compute() all-gathers results, removes padding, broadcasts final metric
    result = accuracy.compute(size=total_samples)
    if rank == 0:
        print(f"Top-1: {result['top1']:.4f}, Top-3: {result['top3']:.4f}")

    accuracy.reset()

if __name__ == '__main__':
    torch.multiprocessing.spawn(eval_fn, nprocs=3, args=(3,))
```
```

--------------------------------

### Basic Multiple Dispatch Example

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/design/multiple_dispatch.md

Demonstrates how to use the @dispatch decorator to define multiple implementations of a function based on the types of its arguments. The correct function is called based on the runtime type of the inputs.

```python
from mmeval.core import dispatch

@dispatch
def compute(x: int, y: int):
    print('this is int')

@dispatch
def compute(x: str, y: str):
    print('this is str')

compute(1, 1)
# this is int

compute('1', '1')
# this is str
```

--------------------------------

### Distributed Accuracy Evaluation with PyTorch

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Example of using MMEval's Accuracy metric with PyTorch's distributed data parallel. Metrics are synchronized across ranks using `dist_backend`. Requires `torch` and `torchvision`.

```python
import torch
import torchvision as tv
from torch.utils.data import DataLoader, DistributedSampler
from mmeval import Accuracy

def eval_fn(rank, process_num):
    torch.distributed.init_process_group(
        backend='gloo',
        init_method='tcp://127.0.0.1:2345',
        world_size=process_num,
        rank=rank,
    )

    dataset = tv.datasets.CIFAR10(
        root='./', train=False, download=True,
        transform=tv.transforms.ToTensor()
    )
    sampler = DistributedSampler(dataset, num_replicas=process_num, rank=rank)
    loader = DataLoader(dataset, batch_size=64, sampler=sampler)
    total_samples = len(dataset)

    model = tv.models.resnet18(num_classes=10).eval()

    # Each rank gets its own metric instance; dist_backend handles sync
    accuracy = Accuracy(topk=(1, 3), dist_backend='torch_cpu')

    with torch.no_grad():
        for images, labels in loader:
            scores = model(images)
            accuracy.add(predictions=scores, labels=labels)

    # compute() all-gathers results, removes padding, broadcasts final metric
    result = accuracy.compute(size=total_samples)
    if rank == 0:
        print(f"Top-1: {result['top1']:.4f}, Top-3: {result['top3']:.4f}")

    accuracy.reset()

if __name__ == '__main__':
    torch.multiprocessing.spawn(eval_fn, nprocs=3, args=(3,))
```

--------------------------------

### Run TensorPack MMEval Evaluation

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/tensorpack/README.md

Execute the evaluation script with a specified model path. Ensure MMEval and MPI4Py are installed.

```bash
python tensorpack_mmeval.py --load <model_path>
```

--------------------------------

### Distributed Backend Management

Source: https://context7.com/open-mmlab/mmeval/llms.txt

MMEval provides functions to list, get, and configure distributed communication backends for synchronizing metrics across processes. Available backends include `non_dist`, `mpi4py`, `torch_cpu`, `torch_cuda`, `tf_horovod`, `paddle_dist`, and `oneflow`.

```APIDOC
## Distributed Backend Management

MMEval provides functions to list, get, and configure distributed communication backends used for synchronizing metrics across processes. Available backends include `non_dist`, `mpi4py`, `torch_cpu`, `torch_cuda`, `tf_horovod`, `paddle_dist`, and `oneflow`.

```python
from mmeval.core.dist import list_all_backends, get_dist_backend, set_default_dist_backend

# List all available backends
backends = list_all_backends()
print(backends)
# ['non_dist', 'mpi4py', 'oneflow', 'tf_horovod', 'torch_cpu', 'torch_cuda', 'paddle_dist']

# Set default distributed backend for all metrics globally
set_default_dist_backend('torch_cpu')

# Get a specific backend instance
backend = get_dist_backend('torch_cpu')
print(backend.is_initialized)  # True if torch.distributed is initialized
print(backend.world_size)       # Number of processes
print(backend.rank)             # Current process rank

# Use per-metric backend override
from mmeval import Accuracy
accuracy = Accuracy(topk=(1, 3), dist_backend='torch_cpu')
# This metric will use torch_cpu for distributed all_gather
```
```

--------------------------------

### Manage Distributed Backends in MMEval

Source: https://context7.com/open-mmlab/mmeval/llms.txt

MMEval provides functions to list, get, and set distributed communication backends. You can set a default backend globally or override it per metric instance.

```python
from mmeval.core.dist import list_all_backends, get_dist_backend, set_default_dist_backend

# List all available backends
backends = list_all_backends()
print(backends)
# ['non_dist', 'mpi4py', 'oneflow', 'tf_horovod', 'torch_cpu', 'torch_cuda', 'paddle_dist']

# Set default distributed backend for all metrics globally
set_default_dist_backend('torch_cpu')

# Get a specific backend instance
backend = get_dist_backend('torch_cpu')
print(backend.is_initialized)  # True if torch.distributed is initialized
print(backend.world_size)       # Number of processes
print(backend.rank)             # Current process rank

# Use per-metric backend override
from mmeval import Accuracy
accuracy = Accuracy(topk=(1, 3), dist_backend='torch_cpu')
# This metric will use torch_cpu for distributed all_gather
```

--------------------------------

### VOCMeanAP with '11points' evaluation mode

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Calculate VOC mAP using the '11points' interpolation method, standard for VOC 2007. This example also shows classwise evaluation and multiprocessing.

```python
from mmeval import VOCMeanAP

# VOC 2007 style (11-point interpolation)
voc_2007 = VOCMeanAP(num_classes=4, iou_thrs=0.5, eval_mode='11points',
                     classwise=True, nproc=4)
```

--------------------------------

### Configure Accuracy Metric in MMCls

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/mmclassification.md

Use this configuration to set the Accuracy metric for validation and testing in MMCls. Ensure MMEval is installed and configured correctly.

```python
val_evaluator = dict(type='Accuracy', topk=(1, ))

test_evaluator = val_evaluator
```

--------------------------------

### Run MMEval Evaluation Script for TensorPack

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/tensorpack.md

Execute the MMEval evaluation script within the TensorPack-FasterRCNN example directory. Replace `<model_path>` with the actual path to your trained model.

```bash
# run evaluation
python tensorpack_mmeval.py --load <model_path>
```

--------------------------------

### Evaluate TensorPack Model Checkpoint

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/tensorpack.md

Use this command to evaluate a trained TensorPack model checkpoint. Ensure the output JSON path, model checkpoint path, and configuration match your training setup.

```bash
./predict.py --evaluate output.json --load /path/to Trained-Model-Checkpoint --config SAME-AS-TRAINING
```

--------------------------------

### MPI4PyDist Implementation of BaseDistBackend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/design/distributed_backend.md

This class implements the distributed communication backend using mpi4py. It provides methods for checking initialization status, getting rank and world size, and performing all-gather and broadcast operations on pickled objects.

```python
from mpi4py import MPI


class MPI4PyDist(BaseDistBackend):
    """A distributed communication backend for mpi4py."""

    @property
    def is_initialized(self) -> bool:
        """Returns True if the distributed environment has been initialized."""
        return 'OMPI_COMM_WORLD_SIZE' in os.environ

    @property
    def rank(self) -> int:
        """Returns the rank index of the current process group."""
        comm = MPI.COMM_WORLD
        return comm.Get_rank()

    @property
    def world_size(self) -> int:
        """Returns the world size of the current process group."""
        comm = MPI.COMM_WORLD
        return comm.Get_size()

    def all_gather_object(self, obj: Any) -> List[Any]:
        """All gather the given object from the current process group and
        returns a list consisting gathered object of each process."""
        comm = MPI.COMM_WORLD
        return comm.allgather(obj)

    def broadcast_object(self, obj: Any, src: int = 0) -> Any:
        """Broadcast the given object from source process to the current
        process group."""
        comm = MPI.COMM_WORLD
        return comm.bcast(obj, root=src)
```

--------------------------------

### PeakSignalNoiseRatio for image fidelity

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Measure image fidelity using PeakSignalNoiseRatio (PSNR). This example uses CHW format input and converts to Y-channel for luminance PSNR calculation. Ensure predictions and ground truths are NumPy arrays.

```python
import numpy as np
from mmeval import PeakSignalNoiseRatio as PSNR

# CHW format, convert to Y-channel for luminance PSNR
psnr = PSNR(input_order='CHW', convert_to='Y', channel_order='rgb')
gts = np.random.randint(0, 256, size=(3, 64, 64)).astype(np.uint8)
preds = np.random.randint(0, 256, size=(3, 64, 64)).astype(np.uint8)
result = psnr(preds, gts)
print(result)  # {'psnr': <float>}
```

--------------------------------

### Build HTML Documentation

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/README.md

Execute the Make command to build the HTML documentation.

```bash
make html
```

--------------------------------

### Open Documentation

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/README.md

Open the generated index.html file in your web browser to view the documentation.

```bash
Open _build/html/index.html with browser
```

--------------------------------

### Navigate to Documentation Directory

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/README.md

Change the current directory to the desired documentation language folder.

```bash
cd docs/en  # or docs/zh_cn
```

--------------------------------

### Instantiate Accuracy Metric

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/get_started/installation.md

Import and instantiate the Accuracy metric from MMEval. This is the first step for using the metric.

```python
from mmeval import Accuracy
import numpy as np

accuracy = Accuracy()
```

--------------------------------

### Evaluate PaddleSeg Model with MMEval (Multi-GPU)

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/paddleseg.md

Run the MMEval evaluation script for PaddleSeg across multiple GPUs. Specify the launcher as 'paddle', the number of processes, and provide the configuration and model paths.

```bash
python ppseg_mmeval.py --config <config_path> --model_path <model_path> --launcher paddle --num_process <num_gpus>
```

--------------------------------

### Initialize Accuracy Metric with Distributed Backend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Instantiates the Accuracy metric and explicitly specifies the distributed communication backend to use. This is an alternative to setting the global default.

```python
from mmeval import Accuracy

# 2. Initialize the evaluation metrics by passing `dist_backend`.
accuracy = Accuracy(dist_backend='torch_cpu')
```

--------------------------------

### Evaluate PaddleSeg Model with MMEval (Single GPU)

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/paddleseg.md

Execute the MMEval evaluation script for PaddleSeg. This script leverages mmeval.MeanIoU for evaluation. Provide the configuration and model paths.

```bash
python ppseg_mmeval.py --config <config_path> --model_path <model_path>
```

--------------------------------

### Launch Multi-GPU Evaluation with Mpirun

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/tensorpack.md

Utilize mpirun to launch the MMEval evaluation script across multiple GPUs for faster processing. Adjust the `-np` value according to your available resources.

```bash
# launch multi-gpus evaluation by mpirun
mpirun -np 8 python tensorpack_mmeval.py --load <model_path>
```

--------------------------------

### Evaluate PaddleSeg Model (Single GPU)

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/examples/paddleseg.md

Use this command to evaluate your PaddleSeg model using the standard PaddleSeg evaluation script. Ensure you have the configuration and model paths ready.

```bash
python val.py --config <config_path> --model_path <model_path>
```

--------------------------------

### Clone MMEval Repository

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/README.md

Clone the MMEval repository to your local machine.

```bash
git clone https://github.com/open-mmlab/mmeval.git
cd mmeval
```

--------------------------------

### Initialize and Use SSIM Metric

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Initialize SSIM metric with specific configurations like input order, color conversion, and channel order. Then, use the instance to compute SSIM between prediction and ground truth tensors.

```python
import numpy as np
from mmeval import StructuralSimilarity as SSIM

# CHW format with Y-channel luminance comparison
ssim = SSIM(input_order='CHW', convert_to='Y', channel_order='rgb')
gts = np.random.randint(0, 256, size=(3, 64, 64)).astype(np.uint8)
preds = np.random.randint(0, 256, size=(3, 64, 64)).astype(np.uint8)
result = ssim(preds, gts)
print(result)  # {'ssim': <float between -1 and 1>}
```

--------------------------------

### COCODetection loading annotations from file

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Initialize COCODetection by providing the path to a COCO annotation JSON file. This allows evaluation using pre-existing annotation datasets.

```python
from mmeval import COCODetection

dataset_meta = {'classes': ('0', '1', '2', '3')}

coco_from_file = COCODetection(
    ann_file='path/to/annotations.json',
    dataset_meta=dataset_meta,
    metric=['bbox', 'segm'],
    iou_thrs=[0.5, 0.75],
)
```

--------------------------------

### TorchCPUDist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

Distributed backend for PyTorch on CPU.

```APIDOC
## TorchCPUDist

### Description
Distributed backend for PyTorch operations on CPU. This backend uses PyTorch's distributed package for communication.

### Methods
- `init_process(backend, **kwargs)`: Initializes the PyTorch distributed process for CPU.
- `destroy_process()`: Destroys the PyTorch distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation using PyTorch CPU backend.
- `all_gather(tensor)`: Performs an all-gather operation using PyTorch CPU backend.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation using PyTorch CPU backend.
- `broadcast(tensor, src)`: Broadcasts a tensor using PyTorch CPU backend.
- `get_rank()`: Gets the PyTorch process rank.
- `get_world_size()`: Gets the PyTorch world size.
- `get_local_rank()`: Gets the PyTorch local rank.
- `get_local_size()`: Gets the PyTorch local world size.
- `new_group(ranks)`: Creates a new PyTorch process group for the specified ranks.
- `get_rank_of_group(group)`: Gets the rank within the specified PyTorch group.
- `get_world_size_of_group(group)`: Gets the world size of the specified PyTorch group.
- `get_local_rank_of_group(group)`: Gets the local rank within the specified PyTorch group.
- `get_local_size_of_group(group)`: Gets the local world size of the specified PyTorch group.
```

--------------------------------

### NonDist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

A backend for non-distributed environments.

```APIDOC
## NonDist

### Description
A backend for non-distributed environments. It provides dummy implementations for distributed operations.

### Methods
- `init_process(backend, **kwargs)`: Initializes the distributed process (no-op).
- `destroy_process()`: Destroys the distributed process (no-op).
- `all_reduce(tensor, op)`: Returns the input tensor.
- `all_gather(tensor)`: Returns the input tensor.
- `reduce_scatter(tensor, op)`: Returns the input tensor.
- `broadcast(tensor, src)`: Returns the input tensor.
- `get_rank()`: Returns 0.
- `get_world_size()`: Returns 1.
- `get_local_rank()`: Returns 0.
- `get_local_size()`: Returns 1.
- `get_group_rank()`: Returns 0.
- `get_group_size()`: Returns 1.
- `new_group(ranks)`: Returns None.
- `get_rank_of_group(group)`: Returns 0.
- `get_world_size_of_group(group)`: Returns 1.
- `get_local_rank_of_group(group)`: Returns 0.
- `get_local_size_of_group(group)`: Returns 1.
```

--------------------------------

### Distribution Utilities

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/zh_cn/api/core.rst

Utilities for managing distributed backends and configurations.

```APIDOC
def list_all_backends() -> list[str]:
    """Lists all available distributed backends."""
    pass

def set_default_dist_backend(backend: str) -> None:
    """Sets the default distributed backend."""
    pass

def get_dist_backend() -> str:
    """Gets the current default distributed backend."""
    pass
```

--------------------------------

### Run PaddleSeg Evaluation

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/paddleseg/README.md

Execute the evaluation script for PaddleSeg models. Specify the configuration and model paths. For multi-GPU evaluation, include the launcher and number of processes.

```bash
python ppseg_mmeval.py --config <config_path> --model_path <model_path>
```

```bash
python ppseg_mmeval.py --config <config_path> --model_path <model_path> --launcher paddle --num_process <num_gpus>
```

--------------------------------

### Prepare Evaluation Model

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Loads a ResNet-18 model from Torchvision, optionally loading pre-trained weights. The model is set to evaluation mode.

```python
import torch
import torchvision as tv

def get_model(pretrained_model_fpath=None):
    model = tv.models.resnet18(num_classes=10)
    if pretrained_model_fpath is not None:
        model.load_state_dict(torch.load(pretrained_model_fpath))
    return model.eval()
```

--------------------------------

### Launch Multi-GPU TensorPack MMEval Evaluation

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/tensorpack/README.md

Initiate distributed evaluation across multiple GPUs using mpirun. Adjust the number of processes (-np) as needed.

```bash
mpirun -np 8 python tensorpack_mmeval.py --load <model_path>
```

--------------------------------

### PaddleDist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

Distributed backend for PaddlePaddle.

```APIDOC
## PaddleDist

### Description
Distributed backend for PaddlePaddle operations. This backend uses PaddlePaddle's distributed communication primitives.

### Methods
- `init_process(backend, **kwargs)`: Initializes the PaddlePaddle distributed process.
- `destroy_process()`: Destroys the PaddlePaddle distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation using PaddlePaddle.
- `all_gather(tensor)`: Performs an all-gather operation using PaddlePaddle.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation using PaddlePaddle.
- `broadcast(tensor, src)`: Broadcasts a tensor using PaddlePaddle.
- `get_rank()`: Gets the PaddlePaddle process rank.
- `get_world_size()`: Gets the PaddlePaddle world size.
- `get_local_rank()`: Gets the PaddlePaddle local rank.
- `get_local_size()`: Gets the PaddlePaddle local world size.
- `new_group(ranks)`: Creates a new PaddlePaddle process group for the specified ranks.
- `get_rank_of_group(group)`: Gets the rank within the specified PaddlePaddle group.
- `get_world_size_of_group(group)`: Gets the world size of the specified PaddlePaddle group.
- `get_local_rank_of_group(group)`: Gets the local rank within the specified PaddlePaddle group.
- `get_local_size_of_group(group)`: Gets the local world size of the specified PaddlePaddle group.
```

--------------------------------

### Run Multi-Process CIFAR-10 Evaluation with torch.distributed

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/cifar10_dist_eval/README.md

Perform CIFAR-10 evaluation using multiple processes with torch.distributed. This is suitable for distributed training environments.

```bash
python cifar10_eval_torch_dist.py
```

--------------------------------

### Prepare Evaluation DataLoader with DistributedSampler

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Loads the CIFAR-10 test dataset and creates a DataLoader. It uses DistributedSampler to ensure data is correctly partitioned across distributed processes.

```python
import torchvision as tv
from torch.utils.data import DataLoader, DistributedSampler

def get_eval_dataloader(rank=0, num_replicas=1):
    dataset = tv.datasets.CIFAR10(
        root='./', train=False, download=True,
        transform=tv.transforms.ToTensor())
    dist_sampler = DistributedSampler(
        dataset, num_replicas=num_replicas, rank=rank)
    data_loader = DataLoader(dataset, batch_size=1, sampler=dist_sampler)
    return data_loader, len(dataset)
```

--------------------------------

### File IO Operations

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/fileio.rst

Provides essential functions for interacting with files and directories.

```APIDOC
## File IO Operations

This section covers core file I/O functions.

### Functions

- **load(filepath, mode='r', **kwargs)**: Loads data from a file.
- **exists(filepath)**: Checks if a file or directory exists.
- **get(filepath, **kwargs)**: Gets the content of a file.
- **get_file_backend(filepath)**: Retrieves the file backend for a given filepath.
- **get_local_path(filepath)**: Gets the local path of a file.
- **get_text(filepath, **kwargs)**: Reads the entire content of a file as text.
- **isdir(filepath)**: Checks if a path is a directory.
- **isfile(filepath)**: Checks if a path is a file.
- **join_path(path1, path2, **kwargs)**: Joins two path components.
- **list_dir_or_file(filepath, **kwargs)**: Lists the contents of a directory or files.

```

--------------------------------

### try_import

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/utils.rst

Attempts to import a module and returns the module object if successful, otherwise returns None. This is useful for optional dependencies.

```APIDOC
## try_import

### Description
Attempts to import a module. Returns the module object if successful, otherwise returns None.

### Signature
```python
try_import(module_name: str)
```

### Parameters
* **module_name** (str) - The name of the module to import.
```

--------------------------------

### Run TensorPack MMEval Evaluation with Specific Configurations

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/tensorpack/README.md

Perform evaluation with custom configurations, such as disabling FPN. This command is useful for reproducing specific benchmark results.

```bash
mpirun -np 8 python tensorpack_mmeval.py --load <model_path> --config MODE_FPN=False
```

--------------------------------

### TorchCUDADist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

Distributed backend for PyTorch on CUDA.

```APIDOC
## TorchCUDADist

### Description
Distributed backend for PyTorch operations on CUDA-enabled GPUs. This backend uses PyTorch's distributed package for communication over CUDA.

### Methods
- `init_process(backend, **kwargs)`: Initializes the PyTorch distributed process for CUDA.
- `destroy_process()`: Destroys the PyTorch distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation using PyTorch CUDA backend.
- `all_gather(tensor)`: Performs an all-gather operation using PyTorch CUDA backend.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation using PyTorch CUDA backend.
- `broadcast(tensor, src)`: Broadcasts a tensor using PyTorch CUDA backend.
- `get_rank()`: Gets the PyTorch process rank.
- `get_world_size()`: Gets the PyTorch world size.
- `get_local_rank()`: Gets the PyTorch local rank.
- `get_local_size()`: Gets the PyTorch local world size.
- `new_group(ranks)`: Creates a new PyTorch process group for the specified ranks.
- `get_rank_of_group(group)`: Gets the rank within the specified PyTorch group.
- `get_world_size_of_group(group)`: Gets the world size of the specified PyTorch group.
- `get_local_rank_of_group(group)`: Gets the local rank within the specified PyTorch group.
- `get_local_size_of_group(group)`: Gets the local world size of the specified PyTorch group.
```

--------------------------------

### Run Single Process CIFAR-10 Evaluation

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/cifar10_dist_eval/README.md

Execute the CIFAR-10 evaluation script in a single process. This is the simplest way to evaluate your model.

```bash
python cifar10_eval.py
```

--------------------------------

### Run Multi-Process CIFAR-10 Evaluation with MPI4Py

Source: https://github.com/open-mmlab/mmeval/blob/main/examples/cifar10_dist_eval/README.md

Evaluate CIFAR-10 models across multiple processes using MPI4Py. This method is useful for HPC environments.

```bash
mpirun -np 3 python3 cifar10_eval_mpi4py.py
```

--------------------------------

### Distribution Utilities

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.rst

Utilities for managing distributed backends and operations within mmeval.

```APIDOC
## Distribution Utilities

### Description
Utilities for managing distributed backends and operations within mmeval.

### Functions

- **list_all_backends**(): Lists all available distribution backends.
- **set_default_dist_backend**(backend_name: str):
  Sets the default distribution backend to use.
- **get_dist_backend**(): Retrieves the currently set default distribution backend.
```

--------------------------------

### OneFlowDist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

Distributed backend for OneFlow.

```APIDOC
## OneFlowDist

### Description
Distributed backend for OneFlow operations. This backend utilizes OneFlow's distributed communication capabilities.

### Methods
- `init_process(backend, **kwargs)`: Initializes the OneFlow distributed process.
- `destroy_process()`: Destroys the OneFlow distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation using OneFlow.
- `all_gather(tensor)`: Performs an all-gather operation using OneFlow.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation using OneFlow.
- `broadcast(tensor, src)`: Broadcasts a tensor using OneFlow.
- `get_rank()`: Gets the OneFlow process rank.
- `get_world_size()`: Gets the OneFlow world size.
- `get_local_rank()`: Gets the OneFlow local rank.
- `get_local_size()`: Gets the OneFlow local world size.
- `new_group(ranks)`: Creates a new OneFlow process group for the specified ranks.
- `get_rank_of_group(group)`: Gets the rank within the specified OneFlow group.
- `get_world_size_of_group(group)`: Gets the world size of the specified OneFlow group.
- `get_local_rank_of_group(group)`: Gets the local rank within the specified OneFlow group.
- `get_local_size_of_group(group)`: Gets the local world size of the specified OneFlow group.
```

--------------------------------

### Distributed Evaluation with MPI4Py Backend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

This Python script demonstrates how to configure MMEval's Accuracy metric to use MPI4Py for distributed communication. Ensure that `get_eval_dataloader` and `get_model` are defined elsewhere.

```python
# cifar10_eval_mpi4py.py

import tqdm
from mpi4py import MPI
import torch
from mmeval import Accuracy


def eval_fn(rank, process_num):
    eval_dataloader, total_num_samples = get_eval_dataloader(rank, process_num)
    model = get_model()
    accuracy = Accuracy(topk=(1, 3), dist_backend='mpi4py')

    with torch.no_grad():
        for images, labels in tqdm.tqdm(eval_dataloader, disable=(rank!=0)):
            predicted_score = model(images)
            accuracy.add(predictions=predicted_score, labels=labels)

    print(accuracy.compute(size=total_num_samples))
    accuracy.reset()


if __name__ == "__main__":
    comm = MPI.COMM_WORLD
    eval_fn(comm.Get_rank(), comm.Get_size())
```

--------------------------------

### MPI4PyDist

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

Distributed backend using MPI4Py.

```APIDOC
## MPI4PyDist

### Description
Distributed backend using MPI4Py. This backend leverages the Message Passing Interface (MPI) for distributed communication.

### Methods
- `init_process(backend, **kwargs)`: Initializes the MPI4Py distributed process.
- `destroy_process()`: Destroys the MPI4Py distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation using MPI.
- `all_gather(tensor)`: Performs an all-gather operation using MPI.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation using MPI.
- `broadcast(tensor, src)`: Broadcasts a tensor using MPI.
- `get_rank()`: Gets the MPI rank.
- `get_world_size()`: Gets the MPI world size.
- `get_local_rank()`: Gets the local MPI rank.
- `get_local_size()`: Gets the local MPI world size.
- `new_group(ranks)`: Creates a new MPI communicator for the specified ranks.
- `get_rank_of_group(group)`: Gets the rank within the specified MPI group.
- `get_world_size_of_group(group)`: Gets the world size of the specified MPI group.
- `get_local_rank_of_group(group)`: Gets the local rank within the specified MPI group.
- `get_local_size_of_group(group)`: Gets the local world size of the specified MPI group.
```

--------------------------------

### TensorBaseDistBackend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

A base class for distributed backends that operate on tensors.

```APIDOC
## TensorBaseDistBackend

### Description
A base class for distributed backends that operate on tensors. It inherits from `BaseDistBackend` and provides tensor-specific operations.

### Methods
- `all_reduce(tensor, op)`: Performs an all-reduce operation on a tensor.
- `all_gather(tensor)`: Performs an all-gather operation on a tensor.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation on a tensor.
- `broadcast(tensor, src)`: Broadcasts a tensor from a source process.
```

--------------------------------

### Implement Simple Accuracy Metric

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/custom_metric.md

Subclass BaseMetric to create a custom Accuracy metric. Override the add method to store intermediate results and compute_metric to calculate the final accuracy.

```python
import numpy as np
from mmeval.core import BaseMetric

class Accuracy(BaseMetric):

    def add(self, predictions, labels):
        self._results.append((predictions, labels))

    def compute_metric(self, results):
        predictions = np.concatenate(
            [res[0] for res in results])
        labels = np.concatenate(
            [res[1] for res in results])
        correct = (predictions == labels)
        accuracy = sum(correct) / len(predictions)
        return {'accuracy': accuracy}
```

--------------------------------

### Set Default Distributed Backend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Sets the global default distributed communication backend for MMEval. This is one of the ways to configure MMEval for distributed operations.

```python
from mmeval.core import set_default_dist_backend
from mmeval import Accuracy

# 1. Set the global default distributed communication backend.
set_default_dist_backend('torch_cpu')
```

--------------------------------

### Implement Custom Accuracy Metric with BaseMetric

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Subclasses of BaseMetric must implement `add()` and `compute_metric()`. The `compute()` method handles distributed synchronization. Use `reset()` to clear accumulated results.

```python
import numpy as np
from mmeval.core import BaseMetric

class MyAccuracy(BaseMetric):
    def add(self, predictions, labels):
        # Store per-batch intermediate results
        self._results.append((np.array(predictions), np.array(labels)))

    def compute_metric(self, results):
        preds = np.concatenate([r[0] for r in results])
        labels = np.concatenate([r[1] for r in results])
        correct = (preds == labels)
        return {'accuracy': float(correct.sum() / len(correct)))}

metric = MyAccuracy()

# Stateless single-call usage
result = metric(predictions=[0, 1, 2, 3], labels=[0, 1, 3, 3])
print(result)  # {'accuracy': 0.75}

# Batch accumulation usage
for batch_idx in range(5):
    preds = np.random.randint(0, 4, size=(32,))
    labels = np.random.randint(0, 4, size=(32,))
    metric.add(preds, labels)

final = metric.compute()
print(final)  # {'accuracy': ~0.25}

metric.reset()  # Clear accumulated results
```

--------------------------------

### Accumulate and Compute Metric

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/get_started/installation.md

Accumulate data from multiple batches using the `add` method and then compute the final metric using `compute`. Suitable for multi-batch scenarios.

```python
for i in range(10):
    labels = np.random.randint(0, 4, size=(100, ))
    predicts = np.random.randint(0, 4, size=(100, ))
    accuracy.add(predicts, labels)

accuracy.compute()
# {'top1': ...}
```

--------------------------------

### BaseMetric Class Diagram

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/design/base_metric.md

This diagram illustrates the structure and key components of the BaseMetric class, including its attributes and methods.

```mermaid
classDiagram
    class BaseMetric
    BaseMetric : +{BaseDistBackend} dist_comm
    BaseMetric : +str dist_collect_mode
    BaseMetric : +dict dataset_meta
    BaseMetric : #list _results
    BaseMetric : +reset()
    BaseMetric : +compute()
    BaseMetric : +{abstractmethod} add()
    BaseMetric : +{abstractmethod} compute_metric()
```

--------------------------------

### AveragePrecision for per-class AP

Source: https://context7.com/open-mmlab/mmeval/llms.txt

To obtain per-class Average Precision (AP) values, instantiate AveragePrecision with `average=None`. This returns a dictionary containing a list of AP values for each class.

```python
import numpy as np
ap_classwise = AveragePrecision(average=None)
preds_np = np.array([[0.9, 0.8, 0.3, 0.2],
                     [0.1, 0.2, 0.2, 0.1],
                     [0.7, 0.5, 0.9, 0.3],
                     [0.8, 0.1, 0.1, 0.2]])
labels_np = np.array([[1, 1, 0, 0],
                      [0, 1, 0, 0],
                      [0, 0, 1, 0],
                      [1, 0, 0, 0]])
result = ap_classwise(preds_np, labels_np)
print(result)  # {'AP_classwise': [100.0, 83.33, 100.0, 0.0]}
```

--------------------------------

### Perplexity

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/metrics.rst

Calculates the perplexity for language models.

```APIDOC
## Perplexity

### Description
Calculates the perplexity for language models.

### Class
Perplexity
```

--------------------------------

### Single Process Evaluation with MMEval Accuracy

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/tutorials/dist_evaluation.md

Performs evaluation on a single process using the MMEval Accuracy metric. It iterates through the DataLoader, accumulates predictions, computes accuracy, and resets the metric.

```python
import tqdm
import torch
from mmeval import Accuracy

eval_dataloader, total_num_samples = get_eval_dataloader()
model = get_model()
# Instantiate `Accuracy` and calculate the top1 and top3 accuracy
accuracy = Accuracy(topk=(1, 3))

with torch.no_grad():
    for images, labels in tqdm.tqdm(eval_dataloader):
        predicted_score = model(images)
        # Accumulate batch data, intermediate results will be saved in
        # `accuracy._results`.
        accuracy.add(predictions=predicted_score, labels=labels)

# Invoke `accuracy.compute` for metric calculation
print(accuracy.compute())
# Invoke `accuracy.reset` to clear the intermediate results saved in
# `accuracy._results`
accuracy.reset()
```

--------------------------------

### Calculate Metric Directly

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/get_started/installation.md

Calculate the metric by directly calling the instantiated metric object with predictions and labels. Useful for single-batch calculations.

```python
labels = np.asarray([0, 1, 2, 3])
preds = np.asarray([0, 2, 1, 3])
accuracy(preds, labels)
# {'top1': 0.5}
```

--------------------------------

### DOTAMeanAP

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/metrics.rst

Calculates the Mean Average Precision (mAP) for the DOTA dataset.

```APIDOC
## DOTAMeanAP

### Description
Calculates the Mean Average Precision (mAP) for the DOTA dataset.

### Class
DOTAMeanAP
```

--------------------------------

### AVAMeanAP

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/metrics.rst

Calculates the Mean Average Precision (mAP) for the AVA dataset.

```APIDOC
## AVAMeanAP

### Description
Calculates the Mean Average Precision (mAP) for the AVA dataset.

### Class
AVAMeanAP
```

--------------------------------

### PCKAccuracy

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/metrics.rst

Calculates the Percentage of Correct Keypoints (PCK) accuracy.

```APIDOC
## PCKAccuracy

### Description
Calculates the Percentage of Correct Keypoints (PCK) accuracy.

### Class
PCKAccuracy
```

--------------------------------

### BaseDistBackend

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/core.dist_backends.rst

The base class for all distributed backends. It defines the common interface for distributed operations.

```APIDOC
## BaseDistBackend

### Description
The base class for all distributed backends. It defines the common interface for distributed operations.

### Methods
- `init_process(backend, **kwargs)`: Initializes the distributed process.
- `destroy_process()`: Destroys the distributed process.
- `all_reduce(tensor, op)`: Performs an all-reduce operation on a tensor.
- `all_gather(tensor)`: Performs an all-gather operation on a tensor.
- `reduce_scatter(tensor, op)`: Performs a reduce-scatter operation on a tensor.
- `broadcast(tensor, src)`: Broadcasts a tensor from a source process.
- `get_rank()`: Gets the rank of the current process.
- `get_world_size()`: Gets the world size.
- `get_local_rank()`: Gets the local rank of the current process.
- `get_local_size()`: Gets the local world size.
- `get_group_rank()`: Gets the rank within a specific group.
- `get_group_size()`: Gets the size of a specific group.
- `new_group(ranks)`: Creates a new process group.
- `get_rank_of_group(group)`: Gets the rank within a given group.
- `get_world_size_of_group(group)`: Gets the world size of a given group.
- `get_local_rank_of_group(group)`: Gets the local rank within a given group.
- `get_local_size_of_group(group)`: Gets the local world size of a given group.
```

--------------------------------

### ProposalRecall

Source: https://github.com/open-mmlab/mmeval/blob/main/docs/en/api/metrics.rst

Calculates the recall of proposed regions in object detection.

```APIDOC
## ProposalRecall

### Description
Calculates the recall of proposed regions in object detection.

### Class
ProposalRecall
```

--------------------------------

### Batch Accumulation for SSIM

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Accumulate SSIM scores over multiple batches using an initialized SSIM metric instance. This is useful for calculating the metric over a dataset incrementally. Supports 'HWC' input order and border cropping.

```python
# HWC format batch accumulation
ssim_batch = SSIM(input_order='HWC', crop_border=4)
for _ in range(5):
    gts = np.random.randint(0, 256, size=(64, 64, 3)).astype(np.uint8)
    preds = np.random.randint(0, 256, size=(64, 64, 3)).astype(np.uint8)
    ssim_batch.add([preds], [gts])
print(ssim_batch.compute())
```

--------------------------------

### COCODetection metric for bounding box evaluation

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Use COCODetection to evaluate bounding box predictions against COCO-formatted ground truths. Initialize with dataset metadata and specify metrics like 'bbox'. Predictions and ground truths should be lists of dictionaries.

```python
import numpy as np
from mmeval import COCODetection

num_classes = 4
dataset_meta = {'classes': tuple([str(i) for i in range(num_classes)])}

coco_metric = COCODetection(
    dataset_meta=dataset_meta,
    metric=['bbox'],
    classwise=False,
)

def gen_bboxes(n, w=256, h=256):
    x = np.random.rand(n) * w
    y = np.random.rand(n) * h
    bw = np.random.rand(n) * (w - x)
    bh = np.random.rand(n) * (h - y)
    return np.stack([x, y, x + bw, y + bh], axis=1)

prediction = {
    'img_id': 1,
    'bboxes': gen_bboxes(10),
    'scores': np.random.rand(10),
    'labels': np.random.randint(0, num_classes, size=(10,)),
}
groundtruth = {
    'img_id': 1,
    'width': 256,
    'height': 256,
    'bboxes': gen_bboxes(5),
    'labels': np.random.randint(0, num_classes, size=(5,)),
    'ignore_flags': np.zeros(5),
}

result = coco_metric(predictions=[prediction], groundtruths=[groundtruth])
print(result)
# {
#   'bbox_mAP': ..., 'bbox_mAP_50': ..., 'bbox_mAP_75': ...,
#   'bbox_mAP_s': ..., 'bbox_mAP_m': ..., 'bbox_mAP_l': ...
# }
```

--------------------------------

### AveragePrecision with PyTorch tensors

Source: https://context7.com/open-mmlab/mmeval/llms.txt

AveragePrecision can directly process PyTorch tensors for predictions and labels. Ensure tensors are on the same device.

```python
import torch
ap = AveragePrecision()
preds_t = torch.tensor([[0.9, 0.8, 0.3, 0.2], [0.1, 0.2, 0.2, 0.1]])
labels_t = torch.tensor([[1, 1, 0, 0], [0, 1, 0, 0]])
ap(preds_t, labels_t)
```

--------------------------------

### VOCMeanAP

Source: https://context7.com/open-mmlab/mmeval/llms.txt

Computes Pascal VOC mean Average Precision (mAP) with configurable IoU thresholds, scale ranges, and average modes ('area' or '11points'). Supports multiprocessing.

```APIDOC
## VOCMeanAP — Pascal VOC Detection mAP

### Description
Computes the Pascal VOC mean Average Precision with configurable IoU thresholds, scale ranges, and the choice of `'area'` (VOC 2012 style) or `'11points'` (VOC 2007 style) average mode. It supports multiprocessing for large datasets.

### Usage
```python
import numpy as np
from mmeval import VOCMeanAP

voc_map = VOCMeanAP(num_classes=4, iou_thrs=[0.5, 0.75], eval_mode='area')

def gen_bboxes(n, w=256, h=256):
    x = np.random.rand(n) * w
    y = np.random.rand(n) * h
    return np.stack([x, y, x + np.random.rand(n) * (w - x),
                     y + np.random.rand(n) * (h - y)], axis=1)

prediction = {
    'bboxes': gen_bboxes(10),
    'scores': np.random.rand(10),
    'labels': np.random.randint(0, 4, size=(10,)),
}
groundtruth = {
    'bboxes': gen_bboxes(8),
    'labels': np.random.randint(0, 4, size=(8,)),
    'bboxes_ignore': gen_bboxes(2),
    'labels_ignore': np.random.randint(0, 4, size=(2,)),
}

result = voc_map(predictions=[prediction], groundtruths=[groundtruth])
print(result)  # {'AP50': ..., 'AP75': ..., 'mAP': ...}

# VOC 2007 style (11-point interpolation)
voc_2007 = VOCMeanAP(num_classes=4, iou_thrs=0.5, eval_mode='11points',
                     classwise=True, nproc=4)
```
```