### Install All PettingZoo Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/README.md

Installs the base PettingZoo library and all dependencies for every environment family.

```bash
pip install 'pettingzoo[all]'
```

--------------------------------

### Install Make on Linux

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install the 'make' utility on Linux systems using apt. This is a prerequisite for using Make commands.

```bash
sudo apt install make
```

--------------------------------

### Install Tianshou and Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/tianshou/intermediate.md

Install necessary libraries including numpy, pettingzoo, packaging, and tianshou for the tutorial. It's recommended to use a virtual environment.

```text
numpy<2.0.0
pettingzoo[classic]>=1.23.0
packaging>=21.3
tianshou==0.5.0
```

--------------------------------

### Install SuperSuit for Atari Preprocessing

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/environments/atari.md

Install the supersuit library to enable preprocessing for Atari environments.

```bash
pip install supersuit
```

--------------------------------

### Install PettingZoo with Specific Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/content/basic_usage.md

Install PettingZoo with dependencies for a specific family of environments, such as Atari. Use 'all' to install all dependencies.

```bash
pip install 'pettingzoo[atari]'
```

```bash
pip install 'pettingzoo[all]'
```

--------------------------------

### Install Documentation Dependencies and PettingZoo

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/README.md

Install the necessary packages for documentation and the PettingZoo library itself using pip.

```bash
pip install -e .
pip install -r docs/requirements.txt
```

--------------------------------

### Install Classic Environments

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/environments/classic.md

Installs the necessary dependencies for the classic set of PettingZoo environments.

```bash
pip install 'pettingzoo[classic]'
```

--------------------------------

### Install Dependencies for RLlib Poker Tutorial

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/rllib/holdem.md

Install the necessary libraries for the RLlib Poker tutorial. It is recommended to use a virtual environment.

```text
PettingZoo[classic,butterfly]>=1.24.0
Pillow>=9.4.0
ray[rllib]==2.55.0
SuperSuit>=3.9.0
torch>=1.13.1
tensorflow-probability>=0.19.0
```

--------------------------------

### Install Pre-commit

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install the pre-commit framework, which helps automate Git hooks for code quality checks.

```bash
pip install pre-commit
```

--------------------------------

### Install Base PettingZoo

Source: https://github.com/farama-foundation/pettingzoo/blob/main/README.md

Installs the core PettingZoo library. Dependencies for all environment families are not included by default.

```bash
pip install pettingzoo
```

--------------------------------

### Install Dependencies for Connect Four Tutorial

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/sb3/connect_four.md

Install the necessary libraries for the Connect Four tutorial, including PettingZoo, Stable-Baselines3, and SB3-Contrib. It is recommended to use a virtual environment.

```text
pettingzoo[classic]>=1.24.0
stable-baselines3>=2.0.0
sb3-contrib>=2.0.0
```

--------------------------------

### Install Make on Windows

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install the 'make' utility on Windows using Chocolatey. This enables the use of Make commands.

```bash
choco install make
```

--------------------------------

### DQN Training Code Example

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/agilerl/DQN.md

This code snippet shows the main training loop for a DQN agent in AgileRL, including curriculum loading, network and hyperparameter configuration, environment setup, and population initialization for hyperparameter optimization.

```python
if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print("===== AgileRL Curriculum Learning Demo ====")

    for lesson_number in range(1, 5):
        # Load lesson for curriculum
        with open(f"./curriculums/connect_four/lesson{lesson_number}.yaml") as file:
            LESSON = yaml.safe_load(file)

        # Define the network configuration
        NET_CONFIG = {
            "encoder_config": {
                "channel_size": [128],  # CNN channel size
                "kernel_size": [4],  # CNN kernel size
                "stride_size": [1],  # CNN stride size
            },
            "head_config": {
                "hidden_size": [64, 64],  # Actor head hidden size
            },
        }

        # Define the initial hyperparameters
        INIT_HP = {
            "POPULATION_SIZE": 6,
            # "ALGO": "Rainbow DQN",  # Algorithm
            "ALGO": "DQN",  # Algorithm
            "DOUBLE": True,
            # Swap image channels dimension from last to first [H, W, C] -> [C, H, W]
            "BATCH_SIZE": 256,  # Batch size
            "LR": 1e-4,  # Learning rate
            "GAMMA": 0.99,  # Discount factor
            "MEMORY_SIZE": 10000,  # Max memory buffer size
            "LEARN_STEP": 1,  # Learning frequency
            "CUDAGRAPHS": False,  # Use CUDA graphs
            "N_STEP": 1,  # Step number to calculate td error
            "PER": False,  # Use prioritized experience replay buffer
            "ALPHA": 0.6,  # Prioritized replay buffer parameter
            "TAU": 0.01,  # For soft update of target parameters
            "BETA": 0.4,  # Importance sampling coefficient
            "PRIOR_EPS": 0.000001,  # Minimum priority for sampling
            "NUM_ATOMS": 51,  # Unit number of support
            "V_MIN": 0.0,  # Minimum value of support
            "V_MAX": 200.0,  # Maximum value of support
        }

        # Define the connect four environment
        env = connect_four_v3.env()
        env.reset()

        # Configure the algo input arguments
        observation_spaces = [env.observation_space(agent) for agent in env.agents]
        action_spaces = [env.action_space(agent) for agent in env.agents]

        # Warp the environment in the curriculum learning wrapper
        env = CurriculumEnv(env, LESSON)

        # Pre-process dimensions for PyTorch layers
        # We only need to worry about the state dim of a single agent
        # We flatten the 6x7x2 observation as input to the agent"s neural network
        observation_space = observation_space_channels_to_first(
            observation_spaces[0]["observation"]
        )
        action_space = action_spaces[0]

        # Mutation config for RL hyperparameters
        hp_config = HyperparameterConfig(
            lr=RLParameter(min=1e-4, max=1e-2),
            batch_size=RLParameter(min=8, max=64, dtype=int),
            learn_step=RLParameter(
                min=1, max=120, dtype=int, grow_factor=1.5, shrink_factor=0.75
            ),
        )

        # Create a population ready for evolutionary hyper-parameter optimisation
        pop: List[DQN] = create_population(
            INIT_HP["ALGO"],
            observation_space,
            action_spaces[0],
            NET_CONFIG,
            INIT_HP,
            hp_config,
            population_size=INIT_HP["POPULATION_SIZE"],
            device=device,
        )

        # Configure the replay buffer
        memory = ReplayBuffer(
            max_size=INIT_HP["MEMORY_SIZE"],  # Max replay buffer size
            device=device,
        )

        # Instantiate a tournament selection object (used for HPO)
        tournament = TournamentSelection(
            tournament_size=2,  # Tournament selection size
            elitism=True,  # Elitism in tournament selection
            population_size=INIT_HP["POPULATION_SIZE"],  # Population size
            eval_loop=1,  # Evaluate using last N fitness scores
        )

```

--------------------------------

### Install Butterfly Environments

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/environments/butterfly.md

Installs the necessary dependencies for PettingZoo's Butterfly environments.

```bash
pip install 'pettingzoo[butterfly]'
```

--------------------------------

### Install Make on MacOS

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install the 'make' utility on MacOS using Homebrew. The command will be available as 'gmake'.

```bash
brew install make
```

--------------------------------

### Install Testing Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install PettingZoo with testing dependencies and all extras. This command is necessary for running existing tests.

```bash
pip install -e ".[testing,all]"
```

--------------------------------

### Environment Setup for Training

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/tianshou/advanced.md

Initializes vectorized environments for training and testing, and sets random seeds for reproducibility. This is a standard setup for reinforcement learning experiments.

```python
def train_agent(
    args: argparse.Namespace = get_args(),
    agent_learn: Optional[BasePolicy] = None,
    agent_opponent: Optional[BasePolicy] = None,
    optim: Optional[torch.optim.Optimizer] = None,
) -> Tuple[dict, BasePolicy]:
    # ======== environment setup =========
    train_envs = DummyVectorEnv([get_env for _ in range(args.training_num)])
    test_envs = DummyVectorEnv([get_env for _ in range(args.test_num)])
    # seed
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    train_envs.seed(args.seed)
    test_envs.seed(args.seed)
```

--------------------------------

### DQN Training Setup and Imports

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/agilerl/DQN.md

This snippet includes necessary imports and setup for training a DQN agent. It defines constants like `max_episodes` and imports modules for environment interaction, agent management, and visualization.

```python
"""This tutorial shows how to train a DQN agent on the connect four environment, using curriculum learning and self play.

Author: Nick (https://github.com/nicku-a), Jaime (https://github.com/jaimesabalbermudez)
"""

import copy
import os
import random
from collections import deque
from typing import List, Optional, Tuple

```

--------------------------------

### Install Git Hooks

Source: https://github.com/farama-foundation/pettingzoo/blob/main/CONTRIBUTING.md

Install the Git hooks managed by pre-commit. These hooks will run automatically on each commit.

```bash
pre-commit install
```

--------------------------------

### Parallel Environment Usage Example

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/api/parallel.md

Demonstrates how to create, reset, step through, and close a parallel environment using the PettingZoo API.

```APIDOC
## Usage Example

```python
from pettingzoo.butterfly import pistonball_v6
parallel_env = pistonball_v6.parallel_env(render_mode="human")
observations, infos = parallel_env.reset(seed=42)

while parallel_env.agents:
    # this is where you would insert your policy
    actions = {agent: parallel_env.action_space(agent).sample() for agent in parallel_env.agents}

    observations, rewards, terminations, truncations, infos = parallel_env.step(actions)
parallel_env.close()
```
```

--------------------------------

### Collector and Logger Setup

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/tianshou/advanced.md

Configures data collectors for training and testing, initializes a replay buffer, and sets up Tensorboard logging. This prepares the components needed for the training loop.

```python
    # ======== agent setup =========
    policy, optim, agents = get_agents(
        args, agent_learn=agent_learn, agent_opponent=agent_opponent, optim=optim
    )

    # ======== collector setup =========
    train_collector = Collector(
        policy,
        train_envs,
        VectorReplayBuffer(args.buffer_size, len(train_envs)),
        exploration_noise=True,
    )
    test_collector = Collector(policy, test_envs, exploration_noise=True)
    # policy.set_eps(1)
    train_collector.collect(n_step=args.batch_size * args.training_num)

    # ======== tensorboard logging setup =========
    log_path = os.path.join(args.logdir, "tic_tac_toe", "dqn")
    writer = SummaryWriter(log_path)
    writer.add_text("args", str(args))
    logger = TensorboardLogger(writer)
```

--------------------------------

### Capture Stdout Example

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/api/utils.md

Demonstrates how to capture standard output from a block of code using the capture_stdout context manager. The captured output can be retrieved as a string.

```python
from pettingzoo.utils.capture_stdout import capture_stdout
with capture_stdout() as var:
    print("test")
    data = var.getvalue()

data
```

--------------------------------

### PettingZoo and LangChain Integration Example

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/langchain/langchain.md

This is the main integration script. It defines a generic `main` function to run any PettingZoo environment with LangChain agents. Specific game functions like `rock_paper_scissors`, `tic_tac_toe`, and `texas_holdem_no_limit` are provided as examples.

```python
from langchain.chat_models import ChatOpenAI
from pettingzoo_agent import PettingZooAgent

from action_masking_agent import ActionMaskAgent  # isort: skip


def main(agents, env):
    env.reset()

    for name, agent in agents.items():
        agent.reset()

    for agent_name in env.agent_iter():
        observation, reward, termination, truncation, info = env.last()
        obs_message = agents[agent_name].observe(
            observation, reward, termination, truncation, info
        )
        print(obs_message)
        if termination or truncation:
            action = None
        else:
            action = agents[agent_name].act()
        print(f"Action: {action}")
        env.step(action)
    env.close()


def rock_paper_scissors():
    from pettingzoo.classic import rps_v2

    env = rps_v2.env(max_cycles=3, render_mode="human")
    agents = {
        name: PettingZooAgent(name=name, model=ChatOpenAI(temperature=1), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


def tic_tac_toe():
    from pettingzoo.classic import tictactoe_v3

    env = tictactoe_v3.env(render_mode="human")
    agents = {
        name: ActionMaskAgent(name=name, model=ChatOpenAI(temperature=0.2), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


def texas_holdem_no_limit():
    from pettingzoo.classic import texas_holdem_no_limit_v6

    env = texas_holdem_no_limit_v6.env(num_players=4, render_mode="human")
    agents = {
        name: ActionMaskAgent(name=name, model=ChatOpenAI(temperature=0.2), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


if __name__ == "__main__":
    rock_paper_scissors()
    tic_tac_toe()
    texas_holdem_no_limit()

```

--------------------------------

### Install SISL Environments

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/environments/sisl.md

Install the SISL environments and their unique dependencies using pip. This command ensures all necessary packages are included.

```bash
pip install 'pettingzoo[sisl]'
```

--------------------------------

### Interacting with a Custom Parallel Environment

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/content/environment_creation.md

Example code to instantiate, reset, and step through a custom parallel environment. It demonstrates sampling actions and closing the environment.

```python
from . import parallel_rps

env = parallel_rps.parallel_env(render_mode="human")
observations, infos = env.reset(seed=42)

while env.agents:
    # this is where you would insert your policy
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}

    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### ClipOutOfBoundsWrapper Example

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/content/environment_creation.md

Demonstrates how to wrap an environment with ClipOutOfBoundsWrapper to constrain agent actions. Wrapped environments must be reset before use.

```python
from pettingzoo.butterfly import pistonball_v6
from pettingzoo.utils import ClipOutOfBoundsWrapper

env = pistonball_v6.env()
wrapped_env = ClipOutOfBoundsWrapper(env)
# Wrapped environments must be reset before use
wrapped_env.reset()
```

--------------------------------

### MATD3 Training Setup with AgileRL

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/agilerl/MATD3.md

This code sets up the MATD3 algorithm for training in a multi-agent environment. It configures network architecture, initial hyperparameters, and initializes the environment and replay buffer.

```python
"""This tutorial shows how to train an MATD3 agent on the simple speaker listener multi-particle environment.

Authors: Michael (https://github.com/mikepratt1), Nickua (https://github.com/nicku-a), Jaime (https://github.com/jaimesabalbermudez)
"""

import os

import numpy as np
import torch
from agilerl.algorithms.core.registry import HyperparameterConfig, RLParameter
from agilerl.components.multi_agent_replay_buffer import MultiAgentReplayBuffer
from agilerl.hpo.mutation import Mutations
from agilerl.hpo.tournament import TournamentSelection
from agilerl.utils.algo_utils import obs_channels_to_first
from agilerl.utils.utils import create_population, observation_space_channels_to_first
from agilerl.vector.pz_async_vec_env import AsyncPettingZooVecEnv
from mpe2 import simple_speaker_listener_v4
from tqdm import trange

if __name__ == "__main__":
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    print("===== AgileRL Online Multi-Agent Demo =====")

    # Define the network configuration
    NET_CONFIG = {
        "encoder_config": {
            "hidden_size": [32, 32],  # Actor hidden size
        }
    }

    # Define the initial hyperparameters
    INIT_HP = {
        "POPULATION_SIZE": 4,
        "ALGO": "MATD3",  # Algorithm
        # Swap image channels dimension from last to first [H, W, C] -> [C, H, W]
        "CHANNELS_LAST": False,
        "BATCH_SIZE": 32,  # Batch size
        "O_U_NOISE": True,  # Ornstein Uhlenbeck action noise
        "EXPL_NOISE": 0.1,  # Action noise scale
        "MEAN_NOISE": 0.0,  # Mean action noise
        "THETA": 0.15,  # Rate of mean reversion in OU noise
        "DT": 0.01,  # Timestep for OU noise
        "LR_ACTOR": 0.001,  # Actor learning rate
        "LR_CRITIC": 0.001,  # Critic learning rate
        "GAMMA": 0.95,  # Discount factor
        "MEMORY_SIZE": 100000,  # Max memory buffer size
        "LEARN_STEP": 100,  # Learning frequency
        "TAU": 0.01,  # For soft update of target parameters
        "POLICY_FREQ": 2,  # Policy frequnecy
    }

    num_envs = 8
    # Define the simple speaker listener environment as a parallel environment
    env = simple_speaker_listener_v4.parallel_env(continuous_actions=True)
    env = AsyncPettingZooVecEnv([lambda: env for _ in range(num_envs)])
    env.reset()

    # Configure the multi-agent algo input arguments
    observation_spaces = [env.single_observation_space(agent) for agent in env.agents]
    action_spaces = [env.single_action_space(agent) for agent in env.agents]
    if INIT_HP["CHANNELS_LAST"]:
        observation_spaces = [
            observation_space_channels_to_first(obs) for obs in observation_spaces
        ]

    # Append number of agents and agent IDs to the initial hyperparameter dictionary
    INIT_HP["N_AGENTS"] = env.num_agents
    INIT_HP["AGENT_IDS"] = env.agents

    # Mutation config for RL hyperparameters
    hp_config = HyperparameterConfig(
        lr_actor=RLParameter(min=1e-4, max=1e-2),
        lr_critic=RLParameter(min=1e-4, max=1e-2),
        batch_size=RLParameter(min=8, max=512, dtype=int),
        learn_step=RLParameter(
            min=20, max=200, dtype=int, grow_factor=1.5, shrink_factor=0.75
        ),
    )

    # Create a population ready for evolutionary hyper-parameter optimisation
    pop = create_population(
        INIT_HP["ALGO"],
        observation_spaces,
        action_spaces,
        NET_CONFIG,
        INIT_HP,
        hp_config=hp_config,
        population_size=INIT_HP["POPULATION_SIZE"],
        num_envs=num_envs,
        device=device,
    )

    # Configure the multi-agent replay buffer
    field_names = ["state", "action", "reward", "next_state", "done"]
    memory = MultiAgentReplayBuffer(
        INIT_HP["MEMORY_SIZE"],
        field_names=field_names,
        agent_ids=INIT_HP["AGENT_IDS"],
        device=device,
    )

    # Instantiate a tournament selection object (used for HPO)
    tournament = TournamentSelection(
        tournament_size=2,  # Tournament selection size
        elitism=True,  # Elitism in tournament selection
        population_size=INIT_HP["POPULATION_SIZE"],  # Population size
        eval_loop=1,  # Evaluate using last N fitness scores
    )

```

--------------------------------

### DeprecatedModule for Environment Versioning

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/content/environment_creation.md

Illustrates the use of DeprecatedModule to manage environment versioning, guiding users from older to newer versions. This example shows how a deprecated environment is declared.

```python
from pettingzoo.utils.deprecated_module import DeprecatedModule
knights_archers_zombies_v0 = DeprecatedModule("knights_archers_zombies", "v0", "v10")
```

--------------------------------

### PPO Training Loop Setup

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/cleanrl/advanced_PPO.md

Initializes the experiment by parsing arguments, setting up Weights & Biases for tracking, configuring the TensorBoard writer, and applying manual seeding for reproducibility across PyTorch and NumPy.

```python
if __name__ == "__main__":
    # --num-steps 32 --num-envs 6 --total-timesteps 256
    args = parse_args()
    print(args)
    run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}"
    if args.track:
        import wandb

        wandb.init(
            project=args.wandb_project_name,
            entity=args.wandb_entity,
            sync_tensorboard=True,
            config=vars(args),
            name=run_name,
            monitor_gym=True,
            save_code=True,
        )
    writer = SummaryWriter(f"runs/{run_name}")
    writer.add_text(
        "hyperparameters",
        "|param|value|\n|-|-|\n%s"
        % ("\n".join([f"|{key}|{value}|" for key, value in vars(args).items()]))
    )

    # TRY NOT TO MODIFY: seeding
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.backends.cudnn.deterministic = args.torch_deterministic

    device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu")

```

--------------------------------

### Example Custom Parallel Environment Implementation

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/content/environment_creation.md

This Python code defines a custom parallel environment for Rock-Paper-Scissors. It includes the necessary setup for PettingZoo's ParallelEnv, defining observation and action spaces, and implementing rendering and closing logic.

```python
import functools

import gymnasium
import numpy as np
from gymnasium.spaces import Discrete
from gymnasium.utils import seeding

from pettingzoo import ParallelEnv
from pettingzoo.utils import parallel_to_aec, wrappers

ROCK = 0
PAPER = 1
SCISSORS = 2
NO_MOVE = 3
MOVES = ["ROCK", "PAPER", "SCISSORS", "None"]
NUM_ITERS = 100
REWARD_MAP = {
    (ROCK, ROCK): (0, 0),
    (ROCK, PAPER): (-1, 1),
    (ROCK, SCISSORS): (1, -1),
    (PAPER, ROCK): (1, -1),
    (PAPER, PAPER): (0, 0),
    (PAPER, SCISSORS): (-1, 1),
    (SCISSORS, ROCK): (-1, 1),
    (SCISSORS, PAPER): (1, -1),
    (SCISSORS, SCISSORS): (0, 0),
}


def env(render_mode=None):
    """
    The env function often wraps the environment in wrappers by default.
    You can find full documentation for these methods
    elsewhere in the developer documentation.
    """
    internal_render_mode = render_mode if render_mode != "ansi" else "human"
    env = raw_env(render_mode=internal_render_mode)
    # This wrapper is only for environments which print results to the terminal
    if render_mode == "ansi":
        env = wrappers.CaptureStdoutWrapper(env)
    # this wrapper helps error handling for discrete action spaces
    env = wrappers.AssertOutOfBoundsWrapper(env)
    # Provides a wide vareity of helpful user errors
    # Strongly recommended
    env = wrappers.OrderEnforcingWrapper(env)
    return env


def raw_env(render_mode=None):
    """
    To support the AEC API, the raw_env() function just uses the from_parallel
    function to convert from a ParallelEnv to an AEC env
    """
    env = parallel_env(render_mode=render_mode)
    env = parallel_to_aec(env)
    return env


class parallel_env(ParallelEnv):
    metadata = {"render_modes": ["human"], "name": "rps_v2"}

    def __init__(self, render_mode=None):
        """
        The init method takes in environment arguments and should define the following attributes:
        - possible_agents
        - render_mode

        Note: as of v1.18.1, the action_spaces and observation_spaces attributes are deprecated.
        Spaces should be defined in the action_space() and observation_space() methods.
        If these methods are not overridden, spaces will be inferred from self.observation_spaces/action_spaces, raising a warning.

        These attributes should not be changed after initialization.
        """
        self.possible_agents = ["player_" + str(r) for r in range(2)]

        # optional: a mapping between agent name and ID
        self.agent_name_mapping = dict(
            zip(self.possible_agents, list(range(len(self.possible_agents))))
        )
        self.render_mode = render_mode

    # Observation space should be defined here.
    # lru_cache allows observation and action spaces to be memoized, reducing clock cycles required to get each agent's space.
    # If your spaces change over time, remove this line (disable caching).
    @functools.lru_cache(maxsize=None)
    def observation_space(self, agent):
        # gymnasium spaces are defined and documented here: https://gymnasium.farama.org/api/spaces/
        # Discrete(4) means an integer in range(0, 4)
        return Discrete(4)

    # Action space should be defined here.
    # If your spaces change over time, remove this line (disable caching).
    @functools.lru_cache(maxsize=None)
    def action_space(self, agent):
        return Discrete(3, seed=self.np_random_seed)

    def render(self):
        """
        Renders the environment. In human mode, it can print to terminal, open
        up a graphical window, or open up some other display that a human can see and understand.
        """
        if self.render_mode is None:
            gymnasium.logger.warn(
                "You are calling render method without specifying any render mode."
            )
            return

        if len(self.agents) == 2:
            string = "Current state: Agent1: {} , Agent2: {}".format(
                MOVES[self.state[self.agents[0]]], MOVES[self.state[self.agents[1]]]
            )
        else:
            string = "Game over"
        print(string)

    def close(self):
        """
        Close should release any graphical displays, subprocesses, network connections
        or any other environment data which should not be kept around after the
        user is no longer using the environment.
        """
        pass

```

--------------------------------

### Build Documentation Locally

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/README.md

Navigate to the docs directory and use make to build the HTML documentation.

```bash
cd docs
make dirhtml
```

--------------------------------

### General Setup for AgileRL DQN

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/agilerl/DQN.md

Sets up the environment, network configuration, initial hyperparameters, and essential components for training and hyperparameter optimization in AgileRL. This includes device selection, loading curriculum, defining network architecture, and initializing replay buffers and selection algorithms.

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("===== AgileRL Curriculum Learning Demo ====")

lesson_number = 1

# Load lesson for curriculum
with open(f"./curriculums/connect_four/lesson{lesson_number}.yaml") as file:
   LESSON = yaml.safe_load(file)

# Define the network configuration
NET_CONFIG = {
   "encoder_config": {
   "channel_size": [128],  # CNN channel size
   "kernel_size": [4],  # CNN kernel size
   "stride_size": [1],  # CNN stride size
   }
}

# Define the initial hyperparameters
INIT_HP = {
   "POPULATION_SIZE": 6,
   # "ALGO": "Rainbow DQN",  # Algorithm
   "ALGO": "DQN",  # Algorithm
   "DOUBLE": True,
   # Swap image channels dimension from last to first [H, W, C] -> [C, H, W]
   "BATCH_SIZE": 256,  # Batch size
   "LR": 1e-4,  # Learning rate
   "GAMMA": 0.99,  # Discount factor
   "MEMORY_SIZE": 100000,  # Max memory buffer size
   "LEARN_STEP": 1,  # Learning frequency
   "N_STEP": 1,  # Step number to calculate td error
   "PER": False,  # Use prioritized experience replay buffer
   "ALPHA": 0.6,  # Prioritized replay buffer parameter
   "TAU": 0.01,  # For soft update of target parameters
   "BETA": 0.4,  # Importance sampling coefficient
   "PRIOR_EPS": 0.000001,  # Minimum priority for sampling
   "NUM_ATOMS": 51,  # Unit number of support
   "V_MIN": 0.0,  # Minimum value of support
   "V_MAX": 200.0,  # Maximum value of support
   "WANDB": False,  # Use Weights and Biases tracking
}

# Define the connect four environment
env = connect_four_v3.env()
env.reset()

# Configure the algo input arguments
observation_spaces = [env.observation_space(agent)("observation") for agent in env.agents]
action_spaces = [env.action_space(agent) for agent in env.agents]

# Warp the environment in the curriculum learning wrapper
env = CurriculumEnv(env, LESSON)

# RL hyperparameters configuration for mutation during training
 hp_config = HyperparameterConfig(
   lr = RLParameter(min=1e-4, max=1e-2),
   learn_step = RLParameter(min=1, max=120, dtype=int),
   batch_size = RLParameter(
       min=8, max=64, dtype=int
       )
 )

# Create a population ready for evolutionary hyper-parameter optimisation
observation_space = observation_space_channels_to_first(observation_spaces[0])
action_space = action_spaces[0]
pop = create_population(
   INIT_HP["ALGO"],
   observation_space,
   action_space,
   NET_CONFIG,
   INIT_HP,
   hp_config=hp_config,
   population_size=INIT_HP["POPULATION_SIZE"],
   device=device,
)

# Configure the replay buffer
field_names = ["state", "action", "reward", "next_state", "done"]
memory = ReplayBuffer(
   action_dim=action_dim,  # Number of agent actions
   memory_size=INIT_HP["MEMORY_SIZE"],  # Max replay buffer size
   field_names=field_names,  # Field names to store in memory
   device=device,
)

# Instantiate a tournament selection object (used for HPO)
tournament = TournamentSelection(
   tournament_size=2,  # Tournament selection size
   elitism=True,  # Elitism in tournament selection
   population_size=INIT_HP["POPULATION_SIZE"],  # Population size
   evo_step=1,
)  # Evaluate using last N fitness scores

# Instantiate a mutations object (used for HPO)
mutations = Mutations(
   no_mutation=0.2,  # Probability of no mutation
   architecture=0,  # Probability of architecture mutation
   new_layer_prob=0.2,  # Probability of new layer mutation
   parameters=0.2,  # Probability of parameter mutation
   activation=0,  # Probability of activation function mutation
   rl_hp=0.2,  # Probability of RL hyperparameter mutation
   mutation_sd=0.1,  # Mutation strength
   rand_seed=1,
   device=device,
)

# Define training loop parameters
episodes_per_epoch = 10
max_episodes = LESSON["max_train_episodes"]  # Total episodes
max_steps = 500  # Maximum steps to take in each episode
evo_epochs = 20  # Evolution frequency
evo_loop = 50  # Number of evaluation episodes
elite = pop[0]  # Assign a placeholder "elite" agent
epsilon = 1.0  # Starting epsilon value
eps_end = 0.1  # Final epsilon value
eps_decay = 0.9998  # Epsilon decays
opp_update_counter = 0
wb = INIT_HP["WANDB"]

```

--------------------------------

### PPO Training Setup and Initialization

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/cleanrl/implementing_PPO.md

Initializes the Pistonball environment with specified preprocessing (color reduction, resizing, frame stacking), sets up the PPO agent, optimizer, and allocates memory for storing episode data.

```python
if __name__ == "__main__":
    """ALGO PARAMS"""
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    ent_coef = 0.1
    vf_coef = 0.1
    clip_coef = 0.1
    gamma = 0.99
    batch_size = 32
    stack_size = 4
    frame_size = (64, 64)
    max_cycles = 125
    total_episodes = 2

    """ ENV SETUP """
    env = pistonball_v6.parallel_env(
        render_mode="rgb_array", continuous=False, max_cycles=max_cycles
    )
    env = color_reduction_v0(env)
    env = resize_v1(env, frame_size[0], frame_size[1])
    env = frame_stack_v1(env, stack_size=stack_size)
    num_agents = len(env.possible_agents)
    num_actions = env.action_space(env.possible_agents[0]).n
    observation_size = env.observation_space(env.possible_agents[0]).shape

    """ LEARNER SETUP """
    agent = Agent(num_actions=num_actions).to(device)
    optimizer = optim.Adam(agent.parameters(), lr=0.001, eps=1e-5)

    """ ALGO LOGIC: EPISODE STORAGE"""
    end_step = 0
    total_episodic_return = 0
    rb_obs = torch.zeros((max_cycles, num_agents, stack_size, *frame_size)).to(device)
    rb_actions = torch.zeros((max_cycles, num_agents)).to(device)
    rb_logprobs = torch.zeros((max_cycles, num_agents)).to(device)
    rb_rewards = torch.zeros((max_cycles, num_agents)).to(device)
    rb_terms = torch.zeros((max_cycles, num_agents)).to(device)
    rb_values = torch.zeros((max_cycles, num_agents)).to(device)

```

--------------------------------

### Install PettingZoo with Atari Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/README.md

Installs PettingZoo along with the dependencies required for the Atari environment family.

```bash
pip install 'pettingzoo[atari]'
```

--------------------------------

### Initialize PettingZoo Environment

Source: https://github.com/farama-foundation/pettingzoo/blob/main/README.md

Demonstrates how to import and initialize a specific environment from the PettingZoo library. This is the first step to using any PettingZoo environment.

```python
from pettingzoo.butterfly import pistonball_v6
env = pistonball_v6.env()
```

--------------------------------

### Test Documentation Code Examples

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/README.md

Run pytest with the markdown-docs plugin to ensure that code examples within the documentation are functional.

```bash
pytest docs --markdown-docs -m markdown-docs
```

--------------------------------

### Tianshou DQN Agent CLI and Logging Setup

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/tianshou/advanced.md

This Python script sets up a DQN agent for multi-agent reinforcement learning using Tianshou and PettingZoo. It includes argument parsing for training configurations and logging.

```python
"""This is a full example of using Tianshou with MARL to train agents, complete with argument parsing (CLI) and logging.

Author: Will (https://github.com/WillDudley)

Python version used: 3.8.10

Requirements:
pettingzoo == 1.22.0
git+https://github.com/thu-ml/tianshou
"""

import argparse
import os
from copy import deepcopy
from typing import Optional, Tuple

import gymnasium
import numpy as np
import torch
from tianshou.data import Collector, VectorReplayBuffer
from tianshou.env import DummyVectorEnv
from tianshou.env.pettingzoo_env import PettingZooEnv
from tianshou.policy import BasePolicy, DQNPolicy, MultiAgentPolicyManager, RandomPolicy
from tianshou.trainer import offpolicy_trainer
from tianshou.utils import TensorboardLogger
from tianshou.utils.net.common import Net
from torch.utils.tensorboard import SummaryWriter

from pettingzoo.classic import tictactoe_v3


def get_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser()
    parser.add_argument("--seed", type=int, default=1626)
    parser.add_argument("--eps-test", type=float, default=0.05)
    parser.add_argument("--eps-train", type=float, default=0.1)
    parser.add_argument("--buffer-size", type=int, default=20000)
    parser.add_argument("--lr", type=float, default=1e-4)
    parser.add_argument(
        "--gamma", type=float, default=0.9, help="a smaller gamma favors earlier win"
    )
    parser.add_argument("--n-step", type=int, default=3)
    parser.add_argument("--target-update-freq", type=int, default=320)
    parser.add_argument("--epoch", type=int, default=50)
    parser.add_argument("--step-per-epoch", type=int, default=1000)
    parser.add_argument("--step-per-collect", type=int, default=10)
    parser.add_argument("--update-per-step", type=float, default=0.1)
    parser.add_argument("--batch-size", type=int, default=64)
    parser.add_argument(
        "--hidden-sizes", type=int, nargs="*", default=[128, 128, 128, 128]
    )
    parser.add_argument("--training-num", type=int, default=10)
    parser.add_argument("--test-num", type=int, default=10)
    parser.add_argument("--logdir", type=str, default="log")
    parser.add_argument("--render", type=float, default=0.1)
    parser.add_argument(
        "--win-rate",
        type=float,
        default=0.6,
        help="the expected winning rate: Optimal policy can get 0.7",
    )
    parser.add_argument(
        "--watch",
        default=False,
        action="store_true",
        help="no training, " "watch the play of pre-trained models",
    )
    parser.add_argument(
        "--agent-id",
        type=int,
        default=2,
        help="the learned agent plays as the" " agent_id-th player. Choices are 1 and 2.",
    )
    parser.add_argument(
        "--resume-path",
        type=str,
        default="",
        help="the path of agent pth file " "for resuming from a pre-trained agent",
    )
    parser.add_argument(
        "--opponent-path",
        type=str,
        default="",
        help="the path of opponent agent pth file "
        "for resuming from a pre-trained agent",
    )
    parser.add_argument(
        "--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu"
    )
    return parser

def get_args() -> argparse.Namespace:
    parser = get_parser()
    return parser.parse_known_args()[0]

```

--------------------------------

### Install AgileRL and Dependencies

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/tutorials/agilerl/MADDPG.md

Install the required libraries for training MADDPG agents with AgileRL, including PettingZoo for environments and PyTorch for deep learning. Ensure your Python version is compatible.

```text
agilerl==2.2.1; python_version >= '3.10' and python_version < '3.12'
pettingzoo[classic,atari]>=1.23.1
mpe2>=1.0.0
AutoROM>=0.6.1
SuperSuit>=3.9.0
torch>=2.0.1
numpy>=1.24.2
tqdm>=4.65.0
fastrand==1.3.0
gymnasium>=0.28.1
imageio>=2.31.1
Pillow>=9.5.0
PyYAML>=5.4.1
```

--------------------------------

### Load and Use OpenSpiel Backgammon Environment with TerminateIllegalWrapper

Source: https://github.com/farama-foundation/pettingzoo/blob/main/docs/api/wrappers/shimmy_wrappers.md

Shows how to load an OpenSpiel game (e.g., chess) using OpenSpielCompatibilityV0 and further wrap it with PettingZoo's TerminateIllegalWrapper. Includes an agent interaction loop.

```python
from shimmy import OpenSpielCompatibilityV0
from pettingzoo.utils import TerminateIllegalWrapper

env = OpenSpielCompatibilityV0(game_name="chess", render_mode=None)
env = TerminateIllegalWrapper(env, illegal_reward=-1)

env.reset()
for agent in env.agent_iter():
    observation, reward, termination, truncation, info = env.last()
    if termination or truncation:
        action = None
    else:
        action = env.action_space(agent).sample(info["action_mask"])  # this is where you would insert your policy
    env.step(action)
    env.render()
```