### Install TorchRL from PyPI

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the TorchRL library using pip. This is the primary method for obtaining the library. For more advanced installation options or nightly builds, refer to the official installation guide.

```bash
pip install torchrl
```

--------------------------------

### Install BenchMARL from GitHub

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the BenchMARL library directly from GitHub using pip. This method allows access to the latest code and configurations. Cloning the repository locally provides further flexibility for modifying configs and scripts.

```bash
pip install benchmarl
```

```bash
git clone https://github.com/facebookresearch/BenchMARL.git
pip install -e BenchMARL
```

--------------------------------

### Install Virtual Environment Dependencies

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Installs the 'vmas' library, which is likely used for multi-agent virtual environments. It also sets up necessary X11 utilities, OpenGL, and XVFB for headless display environments, along with pyvirtualdisplay to manage virtual displays.

```shell
#@title
!pip install vmas
!apt-get update
!apt-get install -y x11-utils python3-opengl xvfb 
!pip install pyvirtualdisplay
import pyvirtualdisplay
display = pyvirtualdisplay.Display(visible=False, size=(1400, 900))
display.start()
```

--------------------------------

### Example Script for Configuring Algorithm

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

A Python script demonstrating how to override algorithm hyperparameters programmatically. This allows for fine-tuning algorithm behavior directly within the script.

```python
# This is a placeholder for the actual script content.
# The actual script would involve loading algorithm configurations and potentially modifying them.
print("Example script for configuring algorithm.")

```

--------------------------------

### BenchMARL Configuration Example

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/sweep/wandb/readme.md

Example of how to configure experiment setup in BenchMARL using Hydra. This YAML file defines default configurations for experiment, algorithm, task, and model, along with a random seed.

```yaml
defaults:
  - experiment: base_experiment
  - algorithm: ippo
  - task: customenv/task_1
  - model: layers/mlp
  - model@critic_model: layers/mlp
  - _self_

seed: 0
```

--------------------------------

### Example Script for Configuring Task

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

A Python script demonstrating how to override task hyperparameters programmatically. This script serves as an example for adjusting task-specific settings.

```python
# This is a placeholder for the actual script content.
# The actual script would involve loading task configurations and potentially modifying them.
print("Example script for configuring task.")

```

--------------------------------

### Install PettingZoo Environment Dependency

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the PettingZoo library with all its dependencies. PettingZoo is a popular toolkit for multi-agent reinforcement learning environments and is an optional dependency for BenchMARL.

```bash
pip install "pettingzoo[all]"
```

--------------------------------

### Launch Experiment from Command Line (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/usage/running.rst

Launches a single experiment by specifying the algorithm and task. This is the most basic way to start an experiment from the command line.

```console
python benchmarl/run.py algorithm=mappo task=vmas/balance
```

--------------------------------

### Install MAgent2 Environment Dependency

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the MAgent2 library from its GitHub repository. MAgent2 is a platform for large-scale multi-agent reinforcement learning research and is an optional dependency for BenchMARL.

```bash
pip install git+https://github.com/Farama-Foundation/MAgent2
```

--------------------------------

### Install Core Dependencies

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Installs PyTorch, torchvision, and the BenchMARL package itself in editable mode. PyTorch is a fundamental deep learning framework, and editable mode allows for direct modification of the installed package.

```shell
#@title
!pip install -U torch torchvision
!pip install -e .
```

--------------------------------

### Load and Validate Configuration from YAML (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Provides Python code examples for directly loading and validating configuration YAML files without using Hydra. This is useful for programmatic configuration management and ensures parameters adhere to defined schemas.

```python
# Example for ExperimentConfig
from benchmarl.experiment import ExperimentConfig

config = ExperimentConfig.get_from_yaml("path/to/your/experiment_config.yaml")

# Example for AlgorithmConfig (replace YourAlgorithmConfig with the actual config class)
# from benchmarl.algorithms.common import YourAlgorithmConfig
# config = YourAlgorithmConfig.get_from_yaml("path/to/your/algorithm_config.yaml")

# Example for TaskConfig (replace YourEnvTask with the actual task class)
# from benchmarl.environments.common import YourEnvTask
# config = YourEnvTask.TASK_NAME.get_from_yaml("path/to/your/task_config.yaml")
```

--------------------------------

### Launch Experiment from Command Line

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Launches a single experiment using the BenchMARL run script. It specifies the algorithm (mappo), task (vmas/balance), maximum number of frames, and disables loggers.

```shell
!python benchmarl/run.py algorithm=mappo task=vmas/balance experiment.max_n_frames=12000 "experiment.loggers=[]"
```

--------------------------------

### Launch Experiment from Python Script

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Launches a single experiment programmatically from a Python script. It configures the task, algorithm, model, and experiment settings, then instantiates and runs the experiment.

```python
from benchmarl.algorithms import MappoConfig
from benchmarl.environments import VmasTask
from benchmarl.experiment import Experiment, ExperimentConfig
from benchmarl.models.mlp import MlpConfig

# Loads from "benchmarl/conf/experiment/base_experiment.yaml"
experiment_config = ExperimentConfig.get_from_yaml()
# Loads from "benchmarl/conf/task/vmas/balance.yaml"
task = VmasTask.BALANCE.get_from_yaml()
# Loads from "benchmarl/conf/algorithm/mappo.yaml"
algorithm_config = MappoConfig.get_from_yaml()
# Loads from "benchmarl/conf/model/layers/mlp.yaml"
model_config = MlpConfig.get_from_yaml()
critic_model_config = MlpConfig.get_from_yaml()

experiment_config.max_n_frames = 12_000
experiment_config.loggers = []

experiment = Experiment(
    task=task,
    algorithm_config=algorithm_config,
    model_config=model_config,
    critic_model_config=critic_model_config,
    seed=0,
    config=experiment_config,
)
experiment.run()
```

--------------------------------

### Launch Multi-Run Experiment from Command Line

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Launches multiple experiments concurrently from the command line using the '-m' flag. It iterates over different algorithms (mappo, qmix, masac), tasks (vmas/balance, vmas/sampling), and seeds (0, 1), while setting experiment parameters.

```shell
!python benchmarl/run.py -m algorithm=mappo,qmix,masac task=vmas/balance,vmas/sampling seed=0,1 experiment.max_n_frames=12000 "experiment.loggers=[]"
```

--------------------------------

### Install MeltingPot Environment Dependency

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the MeltingPot library, developed by DeepMind, which provides a suite of challenging multi-agent scenarios. This is an optional dependency for BenchMARL.

```bash
pip install dm-meltingpot
```

--------------------------------

### Clone BenchMARL Repository

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Clones the BenchMARL GitHub repository to the local file system. This is the initial step to obtain the project's source code.

```shell
#@title
!git clone https://github.com/facebookresearch/BenchMARL
```

--------------------------------

### Example Script for Configuring Experiment

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

A Python script demonstrating how to override experiment hyperparameters programmatically. This approach complements Hydra by allowing configuration adjustments directly within the code.

```python
# This is a placeholder for the actual script content.
# The actual script would involve loading configurations and potentially modifying them.
print("Example script for configuring experiment.")

```

--------------------------------

### W&B Sweep Configuration Example

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/sweep/wandb/readme.md

An example of a Weights & Biases sweep configuration file (`sweepconfig.yaml`). This defines the W&B entity, sweep method (e.g., 'bayes'), the metric to optimize, and the parameters to sweep over, including learning rate and maximum iterations.

```yaml
entity: "ENTITY_NAME"

#options: bayes, random, grid
method: bayes

metric:
  name: eval/agent/reward/episode_reward_mean
  goal: maximize

parameters:
  experiment.lr:
    max: 0.003
    min: 0.000025
    # distribution: uniform

  experiment.max_n_iters:
    value: 321
```

--------------------------------

### Install VMAS Environment Dependency

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Installs the VMAS (Multi-Agent Vectorized Environments) library, which is an optional dependency for BenchMARL. This command uses pip to install the package.

```bash
pip install vmas
```

--------------------------------

### Load Configuration from YAML (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Provides a Python example of directly loading and validating a configuration YAML file without using Hydra. This is useful for programmatic configuration management and ensures parameters adhere to the defined schema.

```python
from benchmarl.conf.config import ComponentConfig

# Assuming 'path/to/your/config.yaml' exists and is valid
config = ComponentConfig.get_from_yaml('path/to/your/config.yaml')
```

--------------------------------

### Run Sequential Benchmark from Python Script

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Executes a benchmark sequentially from a Python script. It defines multiple tasks, algorithms, and seeds, then configures and runs a Benchmark object.

```python
from benchmarl.algorithms import MappoConfig, MasacConfig, QmixConfig
from benchmarl.benchmark import Benchmark
from benchmarl.environments import VmasTask
from benchmarl.experiment import ExperimentConfig
from benchmarl.models.mlp import MlpConfig

# Loads from "benchmarl/conf/experiment/base_experiment.yaml"
experiment_config = ExperimentConfig.get_from_yaml()
# Loads from "benchmarl/conf/task/vmas"
tasks = [VmasTask.BALANCE.get_from_yaml(), VmasTask.SAMPLING.get_from_yaml()]
# Loads from "benchmarl/conf/algorithm"
algorithm_configs = [
    MappoConfig.get_from_yaml(),
    QmixConfig.get_from_yaml(),
    MasacConfig.get_from_yaml(),
]
# Loads from "benchmarl/conf/model/layers"
model_config = MlpConfig.get_from_yaml()
critic_model_config = MlpConfig.get_from_yaml()

experiment_config.max_n_frames = 12_000
experiment_config.loggers = []

benchmark = Benchmark(
    algorithm_configs=algorithm_configs,
    tasks=tasks,
    seeds={0, 1},
    experiment_config=experiment_config,
    model_config=model_config,
    critic_model_config=critic_model_config,
)
benchmark.run_sequential()
```

--------------------------------

### Run Single Training Experiment in BenchMARL (Python)

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

Launches a single training experiment by orchestrating environment setup, data collection, training, and evaluation. It requires configurations for the task, algorithm, model, and experiment itself. Dependencies include benchmarl and torchrl.

```python
from benchmarl.algorithms import MappoConfig
from benchmarl.environments import VmasTask
from benchmarl.experiment import Experiment, ExperimentConfig
from benchmarl.models.mlp import MlpConfig

# Load configurations from YAML files
experiment_config = ExperimentConfig.get_from_yaml()
task = VmasTask.BALANCE.get_from_yaml()
algorithm_config = MappoConfig.get_from_yaml()
model_config = MlpConfig.get_from_yaml()
critic_model_config = MlpConfig.get_from_yaml()

# Create and run the experiment
experiment = Experiment(
    task=task,
    algorithm_config=algorithm_config,
    model_config=model_config,
    critic_model_config=critic_model_config,
    seed=0,
    config=experiment_config,
)
experiment.run()
```

--------------------------------

### Change Directory to BenchMARL

Source: https://github.com/facebookresearch/benchmarl/blob/main/notebooks/run.ipynb

Changes the current working directory to the root of the cloned BenchMARL repository. This is necessary for subsequent commands to locate project files.

```shell
#@title
%cd /content/BenchMARL
```

--------------------------------

### Override Task Hyperparameters via Console

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Illustrates overriding task-specific configurations from the console using Hydra. While not recommended for benchmarking due to reproducibility concerns, this example shows how to change the number of agents for a task.

```console
python benchmarl/run.py task=vmas/balance algorithm=mappo task.n_agents=4
```

--------------------------------

### Initialize W&B Sweep

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/sweep/wandb/readme.md

Command to initialize a Weights & Biases sweep using a specified configuration file. This command starts the sweep process on W&B and provides instructions for running the agents.

```bash
wandb sweep sweepconfig.yaml
```

--------------------------------

### Load, Merge, and Plot MARL Evaluation Results with BenchMARL and marl-eval

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

This Python code demonstrates how to load JSON results from multiple experiment folders using BenchMARL's `load_and_merge_json_dicts` function. It then utilizes the `marl-eval` library to generate various plots, including single task performance, aggregate scores, and performance profiles. Ensure `marl-eval` is installed (`pip install marl-eval`).

```python
from benchmarl.eval_results import load_and_merge_json_dicts
from pathlib import Path

# Load results from experiment folders
experiment_folders = [
    Path("./outputs/mappo_balance_mlp__abc123"),
    Path("./outputs/qmix_balance_mlp__def456"),
    Path("./outputs/masac_balance_mlp__ghi789"),
]

# Merge JSON results for marl-eval plotting
json_dicts = []
for folder in experiment_folders:
    json_file = list(folder.glob("*.json"))[0]
    json_dicts.append(json_file)

merged_data = load_and_merge_json_dicts(json_dicts)

# Use with marl-eval for plotting
# pip install marl-eval
from marl_eval.plotting_tools import (
    plot_single_task,
    plot_aggregate_scores,
    plot_performance_profile,
)

# Generate standard MARL evaluation plots
plot_single_task(
    processed_data=merged_data,
    environment_name="vmas",
    task_name="balance",
    metric_name="mean_episode_return",
)

plot_aggregate_scores(
    processed_data=merged_data,
    environment_name="vmas",
)

plot_performance_profile(
    processed_data=merged_data,
    environment_name="vmas",
)
```

--------------------------------

### Checkpoint and Resume Experiments

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

Illustrates how to configure checkpointing for experiments, including setting the save folder, checkpoint interval, and the number of checkpoints to keep. It also shows how to resume a previously saved experiment from a checkpoint file.

```python
import os
from pathlib import Path
from benchmarl.algorithms import MappoConfig
from benchmarl.environments import VmasTask
from benchmarl.experiment import Experiment, ExperimentConfig
from benchmarl.models.mlp import MlpConfig

# Configure checkpointing
experiment_config = ExperimentConfig.get_from_yaml()
experiment_config.save_folder = Path("./experiments")
experiment_config.checkpoint_interval = 6000  # Save every N frames
experiment_config.checkpoint_at_end = True
experiment_config.keep_checkpoints_num = 3     # Keep last N checkpoints
experiment_config.max_n_iters = 10

# Run initial experiment
experiment = Experiment(
    task=VmasTask.BALANCE.get_from_yaml(),
    algorithm_config=MappoConfig.get_from_yaml(),
    model_config=MlpConfig.get_from_yaml(),
    seed=0,
    config=experiment_config,
)
experiment.run()

# Resume from checkpoint
checkpoint_path = experiment.folder_name / "checkpoints" / f"checkpoint_{experiment.total_frames}.pt"
experiment_config.restore_file = str(checkpoint_path)
experiment_config.save_folder = None  # Use same folder as restore
experiment_config.max_n_iters = 20    # Continue for more iterations

resumed_experiment = Experiment(
    task=VmasTask.BALANCE.get_from_yaml(),
    algorithm_config=MappoConfig.get_from_yaml(),
    model_config=MlpConfig.get_from_yaml(),
    seed=0,
    config=experiment_config,
)
resumed_experiment.run()
```

--------------------------------

### Launch BenchMARL Experiment from Python Script

Source: https://github.com/facebookresearch/benchmarl/blob/main/README.md

Launches a single BenchMARL experiment programmatically using a Python script. It configures the experiment by specifying task, algorithm, model, critic model, seed, and general experiment configurations.

```python
experiment = Experiment(
    task=VmasTask.BALANCE.get_from_yaml(),
    algorithm_config=MappoConfig.get_from_yaml(),
    model_config=MlpConfig.get_from_yaml(),
    critic_model_config=MlpConfig.get_from_yaml(),
    seed=0,
    config=ExperimentConfig.get_from_yaml(),
)
experiment.run()
```

--------------------------------

### Using Algorithm Configurations in BenchMARL (Python)

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

This snippet shows how to use algorithm configuration classes provided by BenchMARL. It demonstrates loading default configurations from YAML and creating custom configurations programmatically for algorithms like MAPPO and MASAC. It also includes checks for algorithm properties like on-policy support and action space compatibility.

```python
from benchmarl.algorithms import (
    MappoConfig,    # Multi-Agent PPO (on-policy, actor-critic)
    IppoConfig,     # Independent PPO (on-policy, actor-critic)
    MaddpgConfig,   # Multi-Agent DDPG (off-policy, actor-critic)
    IddpgConfig,    # Independent DDPG (off-policy, actor-critic)
    MasacConfig,    # Multi-Agent SAC (off-policy, actor-critic)
    IsacConfig,     # Independent SAC (off-policy, actor-critic)
    QmixConfig,     # QMIX (off-policy, value-based)
    VdnConfig,      # VDN (off-policy, value-based)
    IqlConfig,      # Independent Q-Learning (off-policy, value-based)
)

# Load from default YAML
mappo_config = MappoConfig.get_from_yaml()

# Create custom MAPPO config
mappo_config = MappoConfig(
    share_param_critic=True,
    clip_epsilon=0.2,
    entropy_coef=0.01,
    critic_coef=1.0,
    loss_critic_type="l2",
    lmbda=0.9,
    scale_mapping="biased_softplus_1.0",
    use_tanh_normal=True,
)

# Create custom MASAC config
masac_config = MasacConfig(
    share_param_critic=True,
    num_qvalue_nets=2,
    target_entropy="auto",
    alpha_init=1.0,
    alpha_lr=0.001,
)

# Check algorithm capabilities
print(f"MAPPO on_policy: {MappoConfig.on_policy()}")  # True
print(f"MAPPO continuous: {MappoConfig.supports_continuous_actions()}")  # True
print(f"QMIX on_policy: {QmixConfig.on_policy()}")  # False
print(f"QMIX discrete: {QmixConfig.supports_discrete_actions()}")  # True
```

--------------------------------

### Load Algorithm Configuration from YAML (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Shows how to load and validate specific algorithm configurations from YAML files using Python. This approach uses algorithm-specific dataclasses (e.g., `YourAlgorithmConfig`) for robust parameter handling.

```python
# Replace 'YourAlgorithmConfig' with the actual dataclass for the algorithm
# e.g., from benchmarl.algorithm.mappo import MappoConfig

# Assuming 'path/to/your/algorithm.yaml' exists and is valid
algorithm_config = YourAlgorithmConfig.get_from_yaml('path/to/your/algorithm.yaml')
```

--------------------------------

### Run Single Experiment Programmatically (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/usage/running.rst

Loads and runs a single experiment from within a Python script. This provides fine-grained control over experiment setup, including task, algorithm, model, and seed.

```python
from benchmarl.algorithms import MappoConfig
from benchmarl.environments import VmasTask
from benchmarl.experiment import Experiment, ExperimentConfig
from benchmarl.models.mlp import MlpConfig

experiment = Experiment(
   task=VmasTask.BALANCE.get_from_yaml(),
   algorithm_config=MappoConfig.get_from_yaml(),
   model_config=MlpConfig.get_from_yaml(),
   critic_model_config=MlpConfig.get_from_yaml(),
   seed=0,
   config=ExperimentConfig.get_from_yaml(),
)
experiment.run()
```

--------------------------------

### Resume Experiment from Checkpoint (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/features.rst

This command provides a simpler way to resume an experiment directly from a checkpoint file without needing to specify the full configuration. It's useful when you want to continue training exactly where it left off without any configuration changes.

```console
python benchmarl/resume.py ../outputs/2024-09-09/20-39-31/mappo_balance_mlp__cd977b69_24_09_09-20_39_31/checkpoints/checkpoint_100.pt
```

--------------------------------

### Override Algorithm Hyperparameters via Console

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Shows how to override algorithm-specific hyperparameters from the console using Hydra. This example modifies the number of Q-value networks, target entropy, and critic parameter sharing for the MASAC algorithm.

```console
python benchmarl/run.py task=vmas/balance algorithm=masac algorithm.num_qvalue_nets=3 algorithm.target_entropy=auto algorithm.share_param_critic=true
```

--------------------------------

### Run Custom Algorithm (Bash)

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/extending/algorithm/README.md

This command shows how to execute a newly created custom algorithm using the Benchmarl run script. It specifies the algorithm name and the task to be performed.

```bash
python benchmarl/run.py algorithm=customalgorithm task=...

```

--------------------------------

### Configure GNN and Sequence Models

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

Demonstrates how to create configurations for Graph Neural Networks (GNNs) and sequence models. GNN configurations specify graph topology and self-loops, while sequence models can combine multiple layer configurations, including MLPs and GNNs.

```python
from benchmarl.models.gnn import GnnConfig
from benchmarl.models.mlp import MlpConfig
import torch.nn as nn
import torch_geometric

# Create GNN config for graph-based models
gnn_config = GnnConfig(
    topology="full",                         # Graph connectivity
    self_loops=False,
    gnn_class=None,                          # Use default GNN class
)

# Create sequence model combining multiple layers
sequence_config = SequenceModelConfig(
    model_configs=[
        MlpConfig(num_cells=[64], activation_class=nn.Tanh, layer_class=nn.Linear),
        GnnConfig(
            topology="full",
            self_loops=False,
            gnn_class=torch_geometric.nn.conv.GraphConv,
        ),
        MlpConfig(num_cells=[32], activation_class=nn.Tanh, layer_class=nn.Linear),
    ],
    intermediate_sizes=[48, 32],             # Sizes between layers
)
```

--------------------------------

### Launch Multi-Run Experiments from Command Line (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/usage/running.rst

Launches multiple experiments concurrently by specifying comma-separated values for algorithms, tasks, and seeds. This leverages Hydra's multi-run capabilities for efficient benchmarking.

```console
python benchmarl/run.py -m algorithm=mappo,qmix,masac task=vmas/balance,vmas/sampling seed=0,1
```

--------------------------------

### Checkpointing Configuration (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/features.rst

This command illustrates how to configure checkpointing for an experiment. It sets the maximum number of iterations, frames collected per batch, and the interval at which checkpoints should be saved. This is crucial for resuming training or evaluating specific points in an experiment.

```console
python benchmarl/run.py task=vmas/balance algorithm=mappo experiment.max_n_iters=3 experiment.on_policy_collected_frames_per_batch=100 experiment.checkpoint_interval=100
```

--------------------------------

### Execute BenchMARL Experiments via Command Line (Bash)

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

Executes training experiments directly from the command line using Hydra's configuration override syntax. This allows for rapid testing of different algorithm, task, and hyperparameter combinations without code modification. Supports single runs, multi-runs (benchmarks), and overriding specific algorithm parameters.

```bash
# Run a single experiment with MAPPO on the balance task
python benchmarl/run.py algorithm=mappo task=vmas/balance

# Run with custom hyperparameters
python benchmarl/run.py task=vmas/balance algorithm=mappo \
    experiment.lr=0.03 experiment.evaluation=true experiment.train_device="cpu"

# Run multi-runs (benchmarks) with multiple algorithms and tasks
python benchmarl/run.py -m algorithm=mappo,qmix,masac task=vmas/balance,vmas/sampling seed=0,1

# Override algorithm-specific parameters
python benchmarl/run.py task=vmas/balance algorithm=masac \
    algorithm.num_qvalue_nets=3 algorithm.target_entropy=auto algorithm.share_param_critic=true
```

--------------------------------

### Evaluate Experiment via Command Line (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/features.rst

This command-line script automates the evaluation of a saved experiment. It takes the path to a checkpoint file as an argument and initiates the evaluation process, logging the results.

```console
python benchmarl/evaluate.py ../outputs/2024-09-09/20-39-31/mappo_balance_mlp__cd977b69_24_09_09-20_39_31/checkpoints/checkpoint_100.pt
```

--------------------------------

### Load Experiment Configuration from YAML (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Demonstrates how to load and validate experiment configurations directly from a YAML file using Python. This method leverages the `ExperimentConfig` dataclass for type checking and parameter validation.

```python
from benchmarl.experiment import ExperimentConfig

# Assuming 'path/to/your/experiment.yaml' exists and is valid
experiment_config = ExperimentConfig.get_from_yaml('path/to/your/experiment.yaml')
```

--------------------------------

### Configure Loggers via Command Line (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/features.rst

This command demonstrates how to specify loggers for an experiment using command-line arguments. It overrides the default logger configuration and sets 'wandb' as the logger. This is useful for quick experimentation or when modifying configurations directly.

```console
python benchmarl/run.py algorithm=mappo task=vmas/balance "experiment.loggers=[wandb]"
```

--------------------------------

### Implement Custom DQN Algorithm in Python

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

This Python code defines a `CustomDQN` class that extends BenchMARL's `Algorithm` abstract class. It includes methods for defining the loss function, parameters, policy for loss calculation, policy for data collection, and batch processing. This allows for the creation of custom Deep Q-Network (DQN) algorithms tailored to specific reinforcement learning tasks within the BenchMARL environment. Dependencies include `dataclasses`, `typing`, `benchmarl`, `tensordict`, and `torchrl`.

```python
from dataclasses import dataclass, MISSING
from typing import Dict, Iterable, Tuple, Type

from benchmarl.algorithms.common import Algorithm, AlgorithmConfig
from benchmarl.models.common import ModelConfig
from tensordict import TensorDictBase
from tensordict.nn import TensorDictModule, TensorDictSequential
from torchrl.data import CompositeSpec, UnboundedContinuousTensorSpec
from torchrl.modules import EGreedyModule, QValueModule
from torchrl.objectives import DQNLoss, LossModule, ValueEstimators


class CustomDQN(Algorithm):
    def __init__(self, delay_value: bool, loss_function: str, **kwargs):
        super().__init__(**kwargs)
        self.delay_value = delay_value
        self.loss_function = loss_function

    def _get_loss(
        self, group: str, policy_for_loss: TensorDictModule, continuous: bool
    ) -> Tuple[LossModule, bool]:
        loss_module = DQNLoss(
            policy_for_loss,
            delay_value=self.delay_value,
            loss_function=self.loss_function,
            action_space=self.action_spec[group, "action"],
        )
        loss_module.set_keys(
            reward=(group, "reward"),
            action=(group, "action"),
            done=(group, "done"),
            terminated=(group, "terminated"),
            action_value=(group, "action_value"),
            value=(group, "chosen_action_value"),
        )
        loss_module.make_value_estimator(
            ValueEstimators.TD0, gamma=self.experiment_config.gamma
        )
        return loss_module, True  # True = use target network

    def _get_parameters(self, group: str, loss: LossModule) -> Dict[str, Iterable]:
        return {"loss": loss.parameters()}

    def _get_policy_for_loss(
        self, group: str, model_config: ModelConfig, continuous: bool
    ) -> TensorDictModule:
        n_agents = len(self.group_map[group])
        logits_shape = [*self.action_spec[group, "action"].shape,
                        self.action_spec[group, "action"].space.n]

        actor_input_spec = CompositeSpec({
            group: self.observation_spec[group].clone().to(self.device)
        })
        actor_output_spec = CompositeSpec({
            group: CompositeSpec(
                {"action_value": UnboundedContinuousTensorSpec(shape=logits_shape)},
                shape=(n_agents,),
            )
        })

        actor = model_config.get_model(
            input_spec=actor_input_spec,
            output_spec=actor_output_spec,
            agent_group=group,
            input_has_agent_dim=True,
            n_agents=n_agents,
            centralised=False,
            share_params=self.experiment_config.share_policy_params,
            device=self.device,
            action_spec=self.action_spec,
        )
        value_module = QValueModule(
            action_value_key=(group, "action_value"),
            out_keys=[(group, "action"), (group, "action_value"), (group, "chosen_action_value")],
            spec=self.action_spec[group, "action"],
        )
        return TensorDictSequential(actor, value_module)

    def _get_policy_for_collection(
        self, policy_for_loss: TensorDictModule, group: str, continuous: bool
    ) -> TensorDictModule:
        greedy = EGreedyModule(
            annealing_num_steps=self.experiment_config.get_exploration_anneal_frames(self.on_policy),
            action_key=(group, "action"),
            spec=self.action_spec[(group, "action")],
            eps_init=self.experiment_config.exploration_eps_init,
            eps_end=self.experiment_config.exploration_eps_end,
        )
        return TensorDictSequential(*policy_for_loss, greedy)

    def process_batch(self, group: str, batch: TensorDictBase) -> TensorDictBase:
        # Expand shared rewards/dones to agent dimension
        group_shape = batch.get(group).shape
        for key_type in ["done", "terminated", "reward"]:
            nested_key = ("next", group, key_type)
            if nested_key not in batch.keys(True, True):
                batch.set(nested_key, batch.get(("next", key_type)).unsqueeze(-1).expand((*group_shape, 1)))
        return batch

```

--------------------------------

### Run Multiple Experiments Sequentially Programmatically (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/usage/running.rst

Sets up and runs a benchmark consisting of multiple experiments sequentially using the Benchmark class. This is useful for systematically evaluating different combinations of algorithms, tasks, and seeds.

```python
from benchmarl.algorithms import MappoConfig, MasacConfig, QmixConfig
from benchmarl.benchmark import Benchmark
from benchmarl.environments import VmasTask
from benchmarl.experiment import ExperimentConfig
from benchmarl.models.mlp import MlpConfig

benchmark = Benchmark(
   algorithm_configs=[
       MappoConfig.get_from_yaml(),
       QmixConfig.get_from_yaml(),
       MasacConfig.get_from_yaml(),
   ],
   tasks=[
       VmasTask.BALANCE.get_from_yaml(),
       VmasTask.SAMPLING.get_from_yaml(),
   ],
   seeds={0, 1},
   experiment_config=ExperimentConfig.get_from_yaml(),
   model_config=MlpConfig.get_from_yaml(),
   critic_model_config=MlpConfig.get_from_yaml(),
)
benchmark.run_sequential()
```

--------------------------------

### Configure Custom DQN Algorithm in Python

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

This Python code defines `CustomDQNConfig`, a configuration class for the `CustomDQN` algorithm, inheriting from `AlgorithmConfig`. It specifies required parameters like `delay_value` and `loss_function` and declares support for discrete actions only. This configuration is essential for initializing and running the custom DQN algorithm within BenchMARL, ensuring all necessary parameters are provided.

```python
@dataclass
class CustomDQNConfig(AlgorithmConfig):
    delay_value: bool = MISSING
    loss_function: str = MISSING

    @staticmethod
    def associated_class() -> Type[Algorithm]:
        return CustomDQN

    @staticmethod
    def supports_continuous_actions() -> bool:
        return False

    @staticmethod
    def supports_discrete_actions() -> bool:
        return True

    @staticmethod
    def on_policy() -> bool:
        return False

```

--------------------------------

### Configure Ensemble Algorithm in Python

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/ensemble/README.md

This snippet shows how to create an ensemble algorithm configuration in Python. It takes a dictionary mapping group names to specific algorithm configurations (e.g., MaddpgConfig, IsacConfig). Ensure all algorithms within the ensemble share the same policy paradigm (on-policy or off-policy).

```python
from benchmarl.algorithms import EnsembleAlgorithmConfig, IsacConfig, MaddpgConfig

algorithm_config = EnsembleAlgorithmConfig(
    {"agent": MaddpgConfig.get_from_yaml(), "adversary": IsacConfig.get_from_yaml()}
)
```

--------------------------------

### Load Task Configuration from YAML (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/configuring.rst

Illustrates loading task configurations directly from YAML using Python. This method utilizes task-specific enumerations and their `get_from_yaml` method for validation, ensuring correct task parameters.

```python
# Replace 'YourEnvTask.TASK_NAME' with the specific task enumeration
# e.g., from benchmarl.environments.vmas import VmasTask

# Assuming 'path/to/your/task.yaml' exists and is valid
task_config = YourEnvTask.TASK_NAME.get_from_yaml('path/to/your/task.yaml')
```

--------------------------------

### Restore Experiment from Checkpoint (Console)

Source: https://github.com/facebookresearch/benchmarl/blob/main/docs/source/concepts/features.rst

This command shows how to restore a previously saved experiment from a specific checkpoint file. It allows for modifying configuration parameters, such as the task or evaluation settings, while resuming from a saved state. The `restore_file` argument points to the exact checkpoint to load.

```console
python benchmarl/run.py task=vmas/balance algorithm=mappo experiment.max_n_iters=6 experiment.on_policy_collected_frames_per_batch=100 experiment.restore_file="/hydra/experiment/folder/checkpoint/checkpoint_300.pt"
```

--------------------------------

### Add Custom Algorithm to Registry (Python)

Source: https://github.com/facebookresearch/benchmarl/blob/main/examples/extending/algorithm/README.md

This snippet demonstrates how to register a new custom algorithm's configuration with the Benchmarl algorithm registry. It requires defining a `CustomAlgorithmConfig` class and adding it to the `algorithm_config_registry`.

```python
from benchmarl.algorithms.algorithm_config_registry import algorithm_config_registry
from .customalgorithm import CustomAlgorithmConfig

algorithm_config_registry.update({"customalgorithm": CustomAlgorithmConfig})

```

--------------------------------

### Configuring Models in BenchMARL (Python)

Source: https://context7.com/facebookresearch/benchmarl/llms.txt

This snippet demonstrates how to configure neural network models in BenchMARL, including MLP, GNN, and sequence models. It shows loading configurations from YAML and creating custom configurations with specified layer sizes, activation functions, and normalization options.

```python
from torch import nn
from benchmarl.models import MlpConfig, GnnConfig, SequenceModelConfig

# Load MLP config from YAML
mlp_config = MlpConfig.get_from_yaml()

# Create custom MLP config
mlp_config = MlpConfig(
    num_cells=[256, 256],                  # Hidden layer sizes
    layer_class=nn.Linear,                  # Layer type
    activation_class=nn.Tanh,               # Activation function
    activation_kwargs=None,
    norm_class=None,                         # Optional normalization
    norm_kwargs=None,
)
```