### Implement Tianshou Training CLI

Source: https://pettingzoo.farama.org/1.22.3/tutorials/tianshou/advanced

Full example demonstrating argument parsing and environment setup for training agents with Tianshou.

```python
"""This is a full example of using Tianshou with MARL to train agents, complete with argument parsing (CLI) and logging.

Author: Will (https://github.com/WillDudley)

Python version used: 3.8.10

Requirements:
pettingzoo == 1.22.0
git+https://github.com/thu-ml/tianshou
"""

import argparse
import os
from copy import deepcopy
from typing import Optional, Tuple

import gym
import numpy as np
import torch
from tianshou.data import Collector, VectorReplayBuffer
from tianshou.env import DummyVectorEnv
from tianshou.env.pettingzoo_env import PettingZooEnv
from tianshou.policy import BasePolicy, DQNPolicy, MultiAgentPolicyManager, RandomPolicy
from tianshou.trainer import offpolicy_trainer
from tianshou.utils import TensorboardLogger
from tianshou.utils.net.common import Net
from torch.utils.tensorboard import SummaryWriter

from pettingzoo.classic import tictactoe_v3


def get_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser()
    parser.add_argument("--seed", type=int, default=1626)
    parser.add_argument("--eps-test", type=float, default=0.05)
    parser.add_argument("--eps-train", type=float, default=0.1)
    parser.add_argument("--buffer-size", type=int, default=20000)
    parser.add_argument("--lr", type=float, default=1e-4)
    parser.add_argument(
        "--gamma", type=float, default=0.9, help="a smaller gamma favors earlier win"
    )
    parser.add_argument("--n-step", type=int, default=3)
    parser.add_argument("--target-update-freq", type=int, default=320)
    parser.add_argument("--epoch", type=int, default=50)
    parser.add_argument("--step-per-epoch", type=int, default=1000)
    parser.add_argument("--step-per-collect", type=int, default=10)
    parser.add_argument("--update-per-step", type=float, default=0.1)
    parser.add_argument("--batch-size", type=int, default=64)
    parser.add_argument(
        "--hidden-sizes", type=int, nargs="*", default=[128, 128, 128, 128]
    )
    parser.add_argument("--training-num", type=int, default=10)
    parser.add_argument("--test-num", type=int, default=10)
    parser.add_argument("--logdir", type=str, default="log")
    parser.add_argument("--render", type=float, default=0.1)
    parser.add_argument(
        "--win-rate",
        type=float,
        default=0.6,
        help="the expected winning rate: Optimal policy can get 0.7",
    )
    parser.add_argument(
        "--watch",
        default=False,
        action="store_true",
        help="no training, " "watch the play of pre-trained models",
    )
    parser.add_argument(
        "--agent-id",
        type=int,
        default=2,
        help="the learned agent plays as the"
        " agent_id-th player. Choices are 1 and 2.",
    )
    parser.add_argument(
        "--resume-path",
        type=str,
        default="",
        help="the path of agent pth file " "for resuming from a pre-trained agent",
    )
    parser.add_argument(
        "--opponent-path",
        type=str,
        default="",
        help="the path of opponent agent pth file "
        "for resuming from a pre-trained agent",
    )
    parser.add_argument(
        "--device", type=str, default="cuda" if torch.cuda.is_available() else "cpu"
    )
    return parser


def get_args() -> argparse.Namespace:
    parser = get_parser()
    return parser.parse_known_args()[0]

```

--------------------------------

### Install Classic Environments

Source: https://pettingzoo.farama.org/1.22.3/environments/classic

Use this command to install the necessary dependencies for all classic environments.

```bash
pip install pettingzoo[classic]
```

--------------------------------

### Environment Initialization Arguments

Source: https://pettingzoo.farama.org/1.23.0/_modules/pettingzoo/mpe/simple_tag/simple_tag

Example usage of the environment constructor with configuration parameters.

```python
simple_tag_v3.env(num_good=1, num_adversaries=3, num_obstacles=2, max_cycles=25, continuous_actions=False)
```

--------------------------------

### Environment Initialization Arguments

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/mpe/simple_tag/simple_tag

Example of how to instantiate the simple_tag_v3 environment with custom parameters.

```python
simple_tag_v3.env(num_good=1, num_adversaries=3, num_obstacles=2, max_cycles=25, continuous_actions=False, dynamic_rescaling=False)
```

--------------------------------

### Install SuperSuit

Source: https://pettingzoo.farama.org/1.22.3/environments/atari

Install the supersuit library to enable preprocessing for Atari environments.

```bash
pip install supersuit
```

--------------------------------

### Install dependencies for RLLib tutorial

Source: https://pettingzoo.farama.org/1.22.4/tutorials/rllib/holdem

List of required packages for the tutorial. It is recommended to install these in a clean virtual environment.

```text
# not confirmed

pickle5==0.0.11
Pillow==9.3.0
torch==1.13.1
SuperSuit==3.6.0
ray[rllib]==2
pettingzoo==1.22.3
```

--------------------------------

### Install Dependencies

Source: https://pettingzoo.farama.org/1.24.1/tutorials/cleanrl/advanced_PPO

Required packages for running the PPO training script.

```bash
pettingzoo[butterfly,atari,testing]>=1.24.0
SuperSuit>=3.9.0
tensorboard>=2.11.2
torch>=1.13.1

```

--------------------------------

### Install dependencies

Source: https://pettingzoo.farama.org/1.23.0/tutorials/langchain/langchain

Required packages for running the LangChain PettingZoo agent tutorial.

```bash
pettingzoo[classic]
langchain
openai
tenacity
```

--------------------------------

### Install Dependencies

Source: https://pettingzoo.farama.org/1.22.3/tutorials/rllib/holdem

List of required packages for running the Leduc Holdem tutorial.

```text
# not confirmed

pickle5==0.0.11
Pillow==9.3.0
torch==1.13.1
SuperSuit==3.6.0
ray[rllib]==2
pettingzoo==1.22.0
```

--------------------------------

### Install Environment Dependencies

Source: https://pettingzoo.farama.org/1.24.0/tutorials/sb3/waterworld

Required packages for running the Waterworld tutorial.

```text
pettingzoo[sisl]>=1.23.1
stable-baselines3>=2.0.0
supersuit>=3.9.0
pymunk
```

--------------------------------

### PPO Training Script Setup

Source: https://pettingzoo.farama.org/1.24.0/tutorials/cleanrl/advanced_PPO

Initial imports and argument parsing configuration for the PPO training script.

```python
"""Advanced training script adapted from CleanRL's repository: https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo_pettingzoo_ma_atari.py.

This is a full training script including CLI, logging and integration with TensorBoard and WandB for experiment tracking.

Full documentation and experiment results can be found at https://docs.cleanrl.dev/rl-algorithms/ppo/#ppo_pettingzoo_ma_ataripy.

Note: default value for total-timesteps has been changed from 2 million to 8000, for easier testing.

Authors: Costa (https://github.com/vwxyzjn), Elliot (https://github.com/elliottower)
"""

# flake8: noqa

import argparse
import importlib
import os
import random
import time
from distutils.util import strtobool

import gymnasium as gym
import numpy as np
import supersuit as ss
import torch
import torch.nn as nn
import torch.optim as optim
from torch.distributions.categorical import Categorical
from torch.utils.tensorboard import SummaryWriter


def parse_args():
    # fmt: off
    parser = argparse.ArgumentParser()
    parser.add_argument("--exp-name", type=str, default=os.path.basename(__file__).rstrip(".py"),
        help="the name of this experiment")
    parser.add_argument("--seed", type=int, default=1,
        help="seed of the experiment")
    parser.add_argument("--torch-deterministic", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True,
        help="if toggled, `torch.backends.cudnn.deterministic=False`")
    parser.add_argument("--cuda", type=lambda x: bool(strtobool(x)), default=True, nargs="?", const=True,
        help="if toggled, cuda will be enabled by default")
    parser.add_argument("--track", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True,
        help="if toggled, this experiment will be tracked with Weights and Biases")
    parser.add_argument("--wandb-project-name", type=str, default="cleanRL",
        help="the wandb's project name")
    parser.add_argument("--wandb-entity", type=str, default=None,
        help="the entity (team) of wandb's project")
    parser.add_argument("--capture-video", type=lambda x: bool(strtobool(x)), default=False, nargs="?", const=True,
        help="whether to capture videos of the agent performances (check out `videos` folder)")
```

--------------------------------

### Install dependencies for Knights-Archers-Zombies

Source: https://pettingzoo.farama.org/1.24.0/tutorials/sb3/kaz

Required packages for running the PPO training tutorial.

```bash
pettingzoo[butterfly]>=1.23.1
stable-baselines3>=2.0.0
supersuit>=3.9.0
```

--------------------------------

### Install Tianshou and PettingZoo dependencies

Source: https://pettingzoo.farama.org/1.22.3/tutorials/tianshou/beginner

Required packages for running the Tianshou integration tutorial.

```text
pettingzoo[classic]==1.22.1
packaging==21.3
git+https://github.com/WillDudley/tianshou.git
```

--------------------------------

### Install Tianshou and PettingZoo dependencies

Source: https://pettingzoo.farama.org/1.25.0/tutorials/tianshou/beginner

Required packages for running the Tianshou and PettingZoo integration tutorial.

```text
numpy<2.0.0
pettingzoo[classic]>=1.23.0
packaging>=21.3
tianshou==0.5.0
```

--------------------------------

### Install Tianshou Dependencies

Source: https://pettingzoo.farama.org/1.24.1/tutorials/tianshou/advanced

Required packages for running the Tianshou tutorial in a virtual environment.

```text
pettingzoo[classic]==1.23.0
packaging==21.3
tianshou==0.5.0

```

--------------------------------

### Environment Initialization

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/butterfly/knights_archers_zombies/knights_archers_zombies

Example of initializing the Knights Archers Zombies environment with specific configuration arguments.

```python
knights_archers_zombies_v10.env(
  spawn_rate=20,
  num_archers=2,
  num_knights=2,
  max_zombies=10,
  max_arrows=10,
  killable_knights=True,
  killable_archers=True,
  pad_observation=True,
  line_death=False,
  max_cycles=900,
  vector_state=True,
  use_typemasks=False,
  sequence_space=False,
)
```

--------------------------------

### Install AgileRL dependencies

Source: https://pettingzoo.farama.org/1.24.2/tutorials/agilerl/MATD3

List of required packages and minimum versions to run the MATD3 tutorial.

```text
agilerl>=0.1.16
pettingzoo[classic,atari,mpe]>=1.23.1
SuperSuit>=3.9.0
torch>=2.0.1
numpy>=1.24.2
tqdm>=4.65.0
fastrand==1.3.0
gymnasium>=0.28.1
imageio>=2.31.1
Pillow>=9.5.0
PyYAML>=5.4.1
wandb>=0.13.10
```

--------------------------------

### Install AgileRL and Environment Dependencies

Source: https://pettingzoo.farama.org/1.26.0/tutorials/agilerl/MADDPG

List of required packages and version constraints for running the MADDPG tutorial.

```text
agilerl==2.2.1; python_version >= '3.10' and python_version < '3.12'
pettingzoo[classic,atari]>=1.23.1
mpe2>=1.0.0
AutoROM>=0.6.1
SuperSuit>=3.9.0
torch>=2.0.1
numpy>=1.24.2
tqdm>=4.65.0
fastrand==1.3.0
gymnasium>=0.28.1
imageio>=2.31.1
Pillow>=9.5.0
PyYAML>=5.4.1
```

--------------------------------

### Parallel API Usage

Source: https://pettingzoo.farama.org/1.24.1/environments/atari/surround

Example of how to initialize and run the Surround environment using the Parallel API.

```APIDOC
from pettingzoo.atari import surround_v2

env = surround_v2.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}

    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Initialize and Run Parallel Environment

Source: https://pettingzoo.farama.org/1.24.0/environments/atari/quadrapong

Example of initializing and running the Quadrapong environment using the Parallel API.

```python
from pettingzoo.atari import quadrapong_v4

env = quadrapong_v4.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    # this is where you would insert your policy
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}

    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Install Dependencies

Source: https://pettingzoo.farama.org/1.22.3/tutorials/cleanrl/implementing_PPO

Required packages for running the PPO implementation. Use a virtual environment to prevent dependency conflicts.

```text
SuperSuit==3.6.0
torch==1.13.1
pettingzoo==1.22.0
```

--------------------------------

### Parallel API Usage

Source: https://pettingzoo.farama.org/1.24.0/environments/atari/wizard_of_wor

Example of how to initialize and run the Wizard of Wor environment using the Parallel API.

```APIDOC
from pettingzoo.atari import wizard_of_wor_v3

env = wizard_of_wor_v3.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Initialize Training Environment

Source: https://pettingzoo.farama.org/1.24.0/tutorials/cleanrl/advanced_PPO

Sets up seeding, device selection, and experiment tracking with Weights & Biases.

```python
if __name__ == "__main__":
    # --num-steps 32 --num-envs 6 --total-timesteps 256
    args = parse_args()
    print(args)
    run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}"
    if args.track:
        import wandb

        wandb.init(
            project=args.wandb_project_name,
            entity=args.wandb_entity,
            sync_tensorboard=True,
            config=vars(args),
            name=run_name,
            monitor_gym=True,
            save_code=True,
        )
    writer = SummaryWriter(f"runs/{run_name}")
    writer.add_text(
        "hyperparameters",
        "|param|value|\n|-|-|\n%s"
        % ("\n".join([f"|{key}|{value}|" for key, value in vars(args).items()])),
    )

    # TRY NOT TO MODIFY: seeding
    random.seed(args.seed)
    np.random.seed(args.seed)
    torch.manual_seed(args.seed)
    torch.backends.cudnn.deterministic = args.torch_deterministic

    device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu")
```

--------------------------------

### Initialize Tianshou with PettingZoo

Source: https://pettingzoo.farama.org/1.22.3/tutorials/tianshou/beginner

A minimal example demonstrating environment loading, policy definition, and collector execution using PettingZoo and Tianshou.

```python
"""This is a minimal example to show how to use Tianshou with a PettingZoo environment. No training of agents is done here.

Author: Will (https://github.com/WillDudley)

Python version used: 3.8.10

Requirements:
pettingzoo == 1.22.0
git+https://github.com/thu-ml/tianshou
"""

from tianshou.data import Collector
from tianshou.env import DummyVectorEnv, PettingZooEnv
from tianshou.policy import MultiAgentPolicyManager, RandomPolicy

from pettingzoo.classic import rps_v2

if __name__ == "__main__":
    # Step 1: Load the PettingZoo environment
    env = rps_v2.env(render_mode="human")

    # Step 2: Wrap the environment for Tianshou interfacing
    env = PettingZooEnv(env)

    # Step 3: Define policies for each agent
    policies = MultiAgentPolicyManager([RandomPolicy(), RandomPolicy()], env)

    # Step 4: Convert the env to vector format
    env = DummyVectorEnv([lambda: env])

    # Step 5: Construct the Collector, which interfaces the policies with the vectorised environment
    collector = Collector(policies, env)

    # Step 6: Execute the environment with the agents playing for 1 episode, and render a frame every 0.1 seconds
    result = collector.collect(n_episode=1, render=0.1)
```

--------------------------------

### Install dependencies for SB3 Connect Four

Source: https://pettingzoo.farama.org/1.24.0/tutorials/sb3/connect_four

Required packages for running the Connect Four tutorial with Stable Baselines3.

```bash
pettingzoo[classic]>=1.23.1
stable-baselines3>=2.0.0
sb3-contrib>=2.0.0
```

--------------------------------

### Install SISL dependencies

Source: https://pettingzoo.farama.org/1.24.3/environments/sisl

Install the required dependencies for the SISL environments using pip.

```bash
pip install 'pettingzoo[sisl]'
```

--------------------------------

### Install Atari dependencies

Source: https://pettingzoo.farama.org/1.22.4/environments/atari

Install the necessary dependencies for Atari environments using pip.

```bash
pip install pettingzoo[atari]
```

--------------------------------

### Initialize AgileRL DQN Environment and Components

Source: https://pettingzoo.farama.org/1.24.2/tutorials/agilerl/DQN

Sets up the device, network configuration, hyperparameters, environment, population, replay buffer, and evolutionary operators for the training loop.

```python
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("===== AgileRL Curriculum Learning Demo =====")

lesson_number = 1

# Load lesson for curriculum
with open(f"./curriculums/connect_four/lesson{lesson_number}.yaml") as file:
   LESSON = yaml.safe_load(file)

# Define the network configuration
NET_CONFIG = {
   "arch": "cnn",  # Network architecture
   "h_size": [64, 64],  # Actor hidden size
   "c_size": [128],  # CNN channel size
   "k_size": [4],  # CNN kernel size
   "s_size": [1],  # CNN stride size
   "normalize": False,  # Normalize image from range [0,255] to [0,1]
}

# Define the initial hyperparameters
INIT_HP = {
   "POPULATION_SIZE": 6,
   # "ALGO": "Rainbow DQN",  # Algorithm
   "ALGO": "DQN",  # Algorithm
   "DOUBLE": True,
   # Swap image channels dimension from last to first [H, W, C] -> [C, H, W]
   "BATCH_SIZE": 256,  # Batch size
   "LR": 1e-4,  # Learning rate
   "GAMMA": 0.99,  # Discount factor
   "MEMORY_SIZE": 100000,  # Max memory buffer size
   "LEARN_STEP": 1,  # Learning frequency
   "N_STEP": 1,  # Step number to calculate td error
   "PER": False,  # Use prioritized experience replay buffer
   "ALPHA": 0.6,  # Prioritized replay buffer parameter
   "TAU": 0.01,  # For soft update of target parameters
   "BETA": 0.4,  # Importance sampling coefficient
   "PRIOR_EPS": 0.000001,  # Minimum priority for sampling
   "NUM_ATOMS": 51,  # Unit number of support
   "V_MIN": 0.0,  # Minimum value of support
   "V_MAX": 200.0,  # Maximum value of support
   "WANDB": False,  # Use Weights and Biases tracking
}

# Define the connect four environment
env = connect_four_v3.env()
env.reset()

# Configure the algo input arguments
state_dim = [
   env.observation_space(agent)["observation"].shape for agent in env.agents
]
one_hot = False
action_dim = [env.action_space(agent).n for agent in env.agents]
INIT_HP["DISCRETE_ACTIONS"] = True
INIT_HP["MAX_ACTION"] = None
INIT_HP["MIN_ACTION"] = None

# Warp the environment in the curriculum learning wrapper
env = CurriculumEnv(env, LESSON)

# Pre-process dimensions for PyTorch layers
# We only need to worry about the state dim of a single agent
# We flatten the 6x7x2 observation as input to the agent"s neural network
state_dim = np.moveaxis(np.zeros(state_dim[0]), [-1], [-3]).shape
action_dim = action_dim[0]

# Create a population ready for evolutionary hyper-parameter optimisation
pop = initialPopulation(
   INIT_HP["ALGO"],
   state_dim,
   action_dim,
   one_hot,
   NET_CONFIG,
   INIT_HP,
   population_size=INIT_HP["POPULATION_SIZE"],
   device=device,
)

# Configure the replay buffer
field_names = ["state", "action", "reward", "next_state", "done"]
memory = ReplayBuffer(
   action_dim=action_dim,  # Number of agent actions
   memory_size=INIT_HP["MEMORY_SIZE"],  # Max replay buffer size
   field_names=field_names,  # Field names to store in memory
   device=device,
)

# Instantiate a tournament selection object (used for HPO)
tournament = TournamentSelection(
   tournament_size=2,  # Tournament selection size
   elitism=True,  # Elitism in tournament selection
   population_size=INIT_HP["POPULATION_SIZE"],  # Population size
   evo_step=1,
)  # Evaluate using last N fitness scores

# Instantiate a mutations object (used for HPO)
mutations = Mutations(
   algo=INIT_HP["ALGO"],
   no_mutation=0.2,  # Probability of no mutation
   architecture=0,  # Probability of architecture mutation
   new_layer_prob=0.2,  # Probability of new layer mutation
   parameters=0.2,  # Probability of parameter mutation
   activation=0,  # Probability of activation function mutation
   rl_hp=0.2,  # Probability of RL hyperparameter mutation
   rl_hp_selection=[
         "lr",
         "learn_step",
         "batch_size",
   ],  # RL hyperparams selected for mutation
   mutation_sd=0.1,  # Mutation strength
   # Define search space for each hyperparameter
   min_lr=0.0001,
   max_lr=0.01,
   min_learn_step=1,
   max_learn_step=120,
   min_batch_size=8,
   max_batch_size=64,
   arch=NET_CONFIG["arch"],  # MLP or CNN
   rand_seed=1,
   device=device,
)

# Define training loop parameters
episodes_per_epoch = 10
max_episodes = LESSON["max_train_episodes"]  # Total episodes
max_steps = 500  # Maximum steps to take in each episode
evo_epochs = 20  # Evolution frequency
evo_loop = 50  # Number of evaluation episodes
elite = pop[0]  # Assign a placeholder "elite" agent
epsilon = 1.0  # Starting epsilon value
eps_end = 0.1  # Final epsilon value
eps_decay = 0.9998  # Epsilon decays
opp_update_counter = 0
wb = INIT_HP["WANDB"]
```

--------------------------------

### Install MPE Dependencies

Source: https://pettingzoo.farama.org/1.22.3/environments/mpe

Use this command to install the necessary dependencies for all MPE environments.

```bash
pip install pettingzoo[mpe]
```

--------------------------------

### Initialize Opponent Configuration

Source: https://pettingzoo.farama.org/1.25.0/tutorials/agilerl/DQN

Sets up the environment and difficulty level for the opponent.

```python
   def __init__(self, env, difficulty):
      self.env = env.env
      self.difficulty = difficulty
      if self.difficulty == "random":
         self.get_action = self.random_opponent
      elif self.difficulty == "weak":
         self.get_action = self.weak_rule_based_opponent
      else:
         self.get_action = self.strong_rule_based_opponent
      self.num_cols = 7
      self.num_rows = 6
      self.length = 4
      self.top = [0] * self.num_cols
```

--------------------------------

### Install Butterfly dependencies

Source: https://pettingzoo.farama.org/1.22.3/environments/butterfly

Use this command to install the necessary dependencies for all Butterfly environments.

```bash
pip install pettingzoo[butterfly]
```

--------------------------------

### go_v5.env

Source: https://pettingzoo.farama.org/1.22.3/environments/classic/go

Initializes the Go environment with configurable board size and komi compensation.

```APIDOC
## go_v5.env(board_size=19, komi=7.5)

### Description
Initializes the Go environment. The board size determines the dimensions of the grid, and komi provides points to the white player to compensate for moving second.

### Parameters
- **board_size** (int) - Optional - The length of each side of the board (default: 19).
- **komi** (float) - Optional - The number of points given to white to compensate for the disadvantage of moving second (default: 7.5).

### Request Example
```python
from pettingzoo.classic import go_v5
env = go_v5.env(board_size=19, komi=7.5)
```
```

--------------------------------

### Install SISL Environments

Source: https://pettingzoo.farama.org/1.22.3/environments/sisl

Use this command to install the necessary dependencies for the SISL environment suite.

```bash
pip install pettingzoo[sisl]
```

--------------------------------

### Initialize Go Environment

Source: https://pettingzoo.farama.org/1.22.3/environments/classic/go

Configure the board size and komi compensation points during environment initialization.

```python
go_v5.env(board_size = 19, komi = 7.5)
```

--------------------------------

### reset

Source: https://pettingzoo.farama.org/1.23.0/environments/butterfly/knights_archers_zombies

Resets the environment to its starting state.

```APIDOC
## reset(seed=None, options=None)

### Description
Resets the environment to a starting state.
```

--------------------------------

### Import Simple Reference

Source: https://pettingzoo.farama.org/1.23.0/environments/mpe/simple_reference

Import the environment module to initialize the simulation.

```python
from pettingzoo.mpe import simple_reference_v3
```

--------------------------------

### Environment Initialization

Source: https://pettingzoo.farama.org/1.25.0/environments/atari/foozpong

How to initialize the Foozpong environment with specific parameters.

```APIDOC
## Initialization

### Description
Initializes the Foozpong environment.

### Method
`foozpong_v3.env(num_players=4)`

### Parameters
- **num_players** (int) - Optional - Number of players (must be either 2 or 4). Defaults to 4.
```

--------------------------------

### Get State

Source: https://pettingzoo.farama.org/1.23.0/_modules/pettingzoo/butterfly/cooperative_pong/cooperative_pong

Retrieves the global state of the environment.

```python
    def state(self):
        state = self.env.state()
        return state
```

--------------------------------

### raw_env.__init__(board_size, komi, render_mode, screen_height)

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/classic/go/go

Initializes the raw Go environment with specific game parameters.

```APIDOC
## raw_env.__init__(board_size, komi, render_mode, screen_height)

### Description
Initializes the Go environment instance. Sets up the board size, komi, and rendering configurations.

### Parameters
- **board_size** (int) - Optional - The size of the square board (default 19).
- **komi** (float) - Optional - Points given to the second player (default 7.5).
- **render_mode** (str) - Optional - The mode for rendering ('human' or 'rgb_array').
- **screen_height** (int) - Optional - Height of the rendering screen.
```

--------------------------------

### Get Global State

Source: https://pettingzoo.farama.org/1.24.0/_modules/pettingzoo/butterfly/pistonball/pistonball

Returns a global observation of the environment.

```python
    def state(self):
        """Returns an observation of the global environment."""
        state = pygame.surfarray.pixels3d(self.screen).copy()
        state = np.rot90(state, k=3)
        state = np.fliplr(state)
        return state
```

--------------------------------

### Initialize and Run a Classic Environment

Source: https://pettingzoo.farama.org/1.23.0/environments/classic

Example of launching the Texas Holdem environment with random agents and handling action masks.

```python
from pettingzoo.classic import texas_holdem_v4

env = texas_holdem_v4.env(render_mode="human")
env.reset(seed=42)

for agent in env.agent_iter():
    observation, reward, termination, truncation, info = env.last()
    
    if termination or truncation:
        break
    
    mask = observation["action_mask"]
    action = env.action_space(agent).sample(mask)  # this is where you would insert your policy
        
    env.step(action)  
env.close()
```

--------------------------------

### Reset Method

Source: https://pettingzoo.farama.org/1.24.0/_modules/pettingzoo/classic/tictactoe/tictactoe

Resets the environment state to start a new game.

```python
def reset(self, seed=None, options=None):
        # reset environment
        self.board = Board()
```

--------------------------------

### Get Observation Space

Source: https://pettingzoo.farama.org/1.24.1/_modules/pettingzoo/classic/hanabi/hanabi

Retrieves the observation space for a specific agent.

```python
    def observation_space(self, agent):
        return self.observation_spaces[agent]
```

--------------------------------

### Environment initialization and wrappers

Source: https://pettingzoo.farama.org/1.23.0/_modules/pettingzoo/classic/rps/rps

Standard environment entry point and parallel wrapper initialization.

```python
def env(**kwargs):
    env = raw_env(**kwargs)
    env = wrappers.AssertOutOfBoundsWrapper(env)
    env = wrappers.OrderEnforcingWrapper(env)
    return env


parallel_env = parallel_wrapper_fn(env)
```

--------------------------------

### PettingZoo LangChain Integration Example

Source: https://pettingzoo.farama.org/1.23.0/tutorials/langchain/langchain

A complete implementation showing how to initialize agents and run a main loop for various PettingZoo environments.

```python
from langchain.chat_models import ChatOpenAI
from pettingzoo_agent import PettingZooAgent

from action_masking_agent import ActionMaskAgent  # isort: skip


def main(agents, env):
    env.reset()

    for name, agent in agents.items():
        agent.reset()

    for agent_name in env.agent_iter():
        observation, reward, termination, truncation, info = env.last()
        obs_message = agents[agent_name].observe(
            observation, reward, termination, truncation, info
        )
        print(obs_message)
        if termination or truncation:
            action = None
        else:
            action = agents[agent_name].act()
        print(f"Action: {action}")
        env.step(action)
    env.close()


def rock_paper_scissors():
    from pettingzoo.classic import rps_v2

    env = rps_v2.env(max_cycles=3, render_mode="human")
    agents = {
        name: PettingZooAgent(name=name, model=ChatOpenAI(temperature=1), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


def tic_tac_toe():
    from pettingzoo.classic import tictactoe_v3

    env = tictactoe_v3.env(render_mode="human")
    agents = {
        name: ActionMaskAgent(name=name, model=ChatOpenAI(temperature=0.2), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


def texas_holdem_no_limit():
    from pettingzoo.classic import texas_holdem_no_limit_v6

    env = texas_holdem_no_limit_v6.env(num_players=4, render_mode="human")
    agents = {
        name: ActionMaskAgent(name=name, model=ChatOpenAI(temperature=0.2), env=env)
        for name in env.possible_agents
    }
    main(agents, env)


if __name__ == "__main__":
    rock_paper_scissors()
    tic_tac_toe()
    texas_holdem_no_limit()
```

--------------------------------

### Install Dependencies

Source: https://pettingzoo.farama.org/1.24.0/tutorials/rllib/pistonball

List of required packages for running the Pistonball RLlib tutorial.

```text
PettingZoo[classic, butterfly]==1.23.1
Pillow>=9.4.0
ray[rllib]>2.6.2
SuperSuit==3.8.0
torch>=1.13.1
tensorflow-probability>=0.19.0
```

--------------------------------

### Environment Initialization

Source: https://pettingzoo.farama.org/1.22.3/environments/atari/wizard_of_wor

How to import and initialize the Wizard of Wor environment.

```APIDOC
## Import

### Description
Import the Wizard of Wor environment from the PettingZoo Atari module.

### Usage
```python
from pettingzoo.atari import wizard_of_wor_v3
```
```

--------------------------------

### Environment Initialization

Source: https://pettingzoo.farama.org/1.22.3/environments/atari/double_dunk

How to import the Double Dunk environment.

```APIDOC
from pettingzoo.atari import double_dunk_v3
```

--------------------------------

### Get Action Mask

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/classic/tictactoe/tictactoe

Calculates the legal moves for the currently selected agent.

```python
    def _get_mask(self, agent):
        action_mask = np.zeros(9, dtype=np.int8)

        # Per the documentation, the mask of any agent other than the
        # currently selected one is all zeros.
        if agent == self.agent_selection:
            for i in self.board.legal_moves():
                action_mask[i] = 1

        return action_mask
```

--------------------------------

### Physics Setup

Source: https://pettingzoo.farama.org/1.24.1/_modules/pettingzoo/butterfly/pistonball/pistonball

Methods to add walls and the ball to the Pymunk physics space.

```python
    def add_walls(self):
        top_left = (self.wall_width, self.wall_width)
        top_right = (self.screen_width - self.wall_width, self.wall_width)
        bot_left = (self.wall_width, self.screen_height - self.wall_width)
        bot_right = (
            self.screen_width - self.wall_width,
            self.screen_height - self.wall_width,
        )
        walls = [
            pymunk.Segment(self.space.static_body, top_left, top_right, 1),  # Top wall
            pymunk.Segment(self.space.static_body, top_left, bot_left, 1),  # Left wall
            pymunk.Segment(
                self.space.static_body, bot_left, bot_right, 1
            ),  # Bottom wall
            pymunk.Segment(self.space.static_body, top_right, bot_right, 1),  # Right
        ]
        for wall in walls:
            wall.friction = 0.64
            self.space.add(wall)

    def add_ball(self, x, y, b_mass, b_friction, b_elasticity):
        mass = b_mass
        radius = 40
        inertia = pymunk.moment_for_circle(mass, 0, radius, (0, 0))
        body = pymunk.Body(mass, inertia)
        body.position = x, y
        # radians per second
        if self.random_rotate:
            body.angular_velocity = self.np_random.uniform(-6 * math.pi, 6 * math.pi)
        shape = pymunk.Circle(body, radius, (0, 0))
        shape.friction = b_friction
        shape.elasticity = b_elasticity
        self.space.add(body, shape)
        return body
```

--------------------------------

### Initialize Training Parameters and Environment

Source: https://pettingzoo.farama.org/1.24.3/tutorials/agilerl/DQN

Sets up hyperparameters, population placeholders, and environment-specific configurations for the training process.

```python
        # ! NOTE: Uncomment the max_episodes line below to change the number of training episodes. ! #
        # It is deliberately set low to allow testing to ensure this tutorial is sound.
        max_episodes = 10
        # max_episodes = LESSON["max_train_episodes"]  # Total episodes

        max_steps = 500  # Maximum steps to take in each episode
        evo_epochs = 20  # Evolution frequency
        evo_loop = 50  # Number of evaluation episodes
        elite = pop[0]  # Assign a placeholder "elite" agent
        epsilon = 1.0  # Starting epsilon value
        eps_end = 0.1  # Final epsilon value
        eps_decay = 0.9998  # Epsilon decays
        opp_update_counter = 0
        wb = INIT_HP["WANDB"]

        if LESSON["pretrained_path"] is not None:
            for agent in pop:
                # Load pretrained checkpoint
                agent.loadCheckpoint(LESSON["pretrained_path"])
                # Reinit optimizer for new task
                agent.lr = INIT_HP["LR"]
                agent.optimizer = torch.optim.Adam(
                    agent.actor.parameters(), lr=agent.lr
                )

        if LESSON["opponent"] == "self":
            # Create initial pool of opponents
            opponent_pool = deque(maxlen=LESSON["opponent_pool_size"])
            for _ in range(LESSON["opponent_pool_size"]):
                opp = copy.deepcopy(pop[0])
                opp.actor.load_state_dict(pop[0].actor.state_dict())
                opp.actor.eval()
                opponent_pool.append(opp)

        # Perform buffer and agent warmups if desired
        if LESSON["buffer_warm_up"]:
            warm_up_opponent = Opponent(env, difficulty=LESSON["warm_up_opponent"])
            memory = env.fill_replay_buffer(
                memory, warm_up_opponent
            )  # Fill replay buffer with transitions
            if LESSON["agent_warm_up"] > 0:
                print("Warming up agents ...")
                agent = pop[0]
                # Train on randomly collected samples
                for epoch in trange(LESSON["agent_warm_up"]):
                    experiences = memory.sample(agent.batch_size)
                    agent.learn(experiences)
                pop = [agent.clone() for _ in pop]
                elite = agent
                print("Agent population warmed up.")

        if max_episodes > 0:
            if wb:
                wandb.init(
                    # set the wandb project where this run will be logged
                    project="AgileRL",
                    name="{}-EvoHPO-{}-{}Opposition-CNN-{}".format(
                        "connect_four_v3",
                        INIT_HP["ALGO"],
                        LESSON["opponent"],
                        datetime.now().strftime("%m%d%Y%H%M%S"),
                    ),
                    # track hyperparameters and run metadata
                    config={
                        "algo": "Evo HPO Rainbow DQN",
                        "env": "connect_four_v3",
                        "INIT_HP": INIT_HP,
                        "lesson": LESSON,
                    },
                )

        total_steps = 0
        total_episodes = 0
        pbar = trange(int(max_episodes / episodes_per_epoch))

        # Training loop
        for idx_epi in pbar:
            turns_per_episode = []
            train_actions_hist = [0] * action_dim
            for agent in pop:  # Loop through population
                for episode in range(episodes_per_epoch):
                    env.reset()  # Reset environment at start of episode
                    observation, env_reward, done, truncation, _ = env.last()

                    (
                        p1_state,
                        p1_state_flipped,
                        p1_action,
                        p1_next_state,
                        p1_next_state_flipped,
                    ) = (None, None, None, None, None)

                    if LESSON["opponent"] == "self":
                        # Randomly choose opponent from opponent pool if using self-play
                        opponent = random.choice(opponent_pool)
                    else:
                        # Create opponent of desired difficulty
                        opponent = Opponent(env, difficulty=LESSON["opponent"])

                    # Randomly decide whether agent will go first or second
                    if random.random() > 0.5:
                        opponent_first = False
                    else:
                        opponent_first = True

                    score = 0
                    turns = 0  # Number of turns counter
```

--------------------------------

### raw_env.__init__(board_size, komi, render_mode)

Source: https://pettingzoo.farama.org/1.23.0/_modules/pettingzoo/classic/go/go

Constructor for the Go environment, defining the board configuration and agent spaces.

```APIDOC
## raw_env.__init__(board_size, komi, render_mode)

### Description
Initializes the Go environment with a specific board size and komi value.

### Parameters
- **board_size** (int) - Optional - The size of the square board (default 19).
- **komi** (float) - Optional - Points given to the second player (default 7.5).
- **render_mode** (str) - Optional - The rendering mode to use (e.g., 'human', 'rgb_array').
```

--------------------------------

### Parallel API Usage

Source: https://pettingzoo.farama.org/1.24.0/environments/atari/combat_plane

Example of how to interact with the environment using the Parallel API.

```APIDOC
from pettingzoo.atari import combat_plane_v2

env = combat_plane_v2.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### raw_env.__init__(board_size, komi, render_mode, screen_height)

Source: https://pettingzoo.farama.org/_modules/pettingzoo/classic/go/go

Constructor for the raw Go environment class.

```APIDOC
## raw_env.__init__(board_size, komi, render_mode, screen_height)

### Description
Initializes the raw Go environment with specific game parameters.

### Parameters
- **board_size** (int) - Optional - The size of the Go board (default: 19).
- **komi** (float) - Optional - Points given to the second player (default: 7.5).
- **render_mode** (str) - Optional - The mode for rendering the environment.
- **screen_height** (int) - Optional - The height of the rendering screen.
```

--------------------------------

### Initialize Simple Push Environment

Source: https://pettingzoo.farama.org/1.22.3/environments/mpe/simple_push

Configure the environment with custom max_cycles and action space type.

```python
simple_push_v2.env(max_cycles=25, continuous_actions=False)
```

--------------------------------

### simple_v3.env

Source: https://pettingzoo.farama.org/1.23.0/_modules/pettingzoo/mpe/simple/simple

Initializes the Simple environment instance.

```APIDOC
## simple_v3.env(max_cycles=25, continuous_actions=False)

### Description
Creates and returns an instance of the Simple environment.

### Parameters
- **max_cycles** (int) - Optional - Number of frames (a step for each agent) until the game terminates. Defaults to 25.
- **continuous_actions** (bool) - Optional - Whether agent action spaces are discrete (False) or continuous (True). Defaults to False.
```

--------------------------------

### Initialize the simple environment

Source: https://pettingzoo.farama.org/1.22.3/environments/mpe/simple

Configure the environment with custom cycle limits and action space types.

```python
simple_v2.env(max_cycles=25, continuous_actions=False)
```

--------------------------------

### Get Global State

Source: https://pettingzoo.farama.org/1.26.0/_modules/pettingzoo/butterfly/cooperative_pong/cooperative_pong

Returns a copy of the current screen as a global state observation.

```python
    def state(self) -> StateType:
        """Returns an observation of the global environment."""
        assert self.screen is not None
        state = pygame.surfarray.pixels3d(self.screen).copy()
        state = np.rot90(state, k=3)
        state = np.fliplr(state)
        return cast(StateType, state)
```

--------------------------------

### Get Ball Position

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/butterfly/pistonball/pistonball

Calculates the leftmost x-position of the ball, clipping it to the wall width.

```python
    def _get_ball_position(self) -> int:
        """Return the leftmost x-position of the ball.

        That leftmost x-position is generally referred to and treated as the
        balls' position in this class. If the ball extends beyond the leftmost
        wall, return the position of that wall-edge.
        """
        ball_position = int(self.ball.position[0] - self.ball_radius)
        # Check if the ball is touching/within the left-most wall.
        clipped_ball_position = max(self.wall_width, ball_position)
        return clipped_ball_position
```

--------------------------------

### Initialize and Configure PettingZoo Environment

Source: https://pettingzoo.farama.org/1.24.3/tutorials/cleanrl/advanced_PPO

Sets up the environment with Supersuit wrappers for observation processing and vectorization.

```python
# env setup
env = importlib.import_module(f"pettingzoo.atari.{args.env_id}").parallel_env()
env = ss.max_observation_v0(env, 2)
env = ss.frame_skip_v0(env, 4)
env = ss.clip_reward_v0(env, lower_bound=-1, upper_bound=1)
env = ss.color_reduction_v0(env, mode="B")
env = ss.resize_v1(env, x_size=84, y_size=84)
env = ss.frame_stack_v1(env, 4)
env = ss.agent_indicator_v0(env, type_only=False)
env = ss.pettingzoo_env_to_vec_env_v1(env)
envs = ss.concat_vec_envs_v1(
    env, args.num_envs // 2, num_cpus=0, base_class="gymnasium"
)
envs.single_observation_space = envs.observation_space
envs.single_action_space = envs.action_space
envs.is_vector_env = True
if args.capture_video:
    envs = gym.wrappers.RecordVideo(envs, f"videos/{run_name}")
assert isinstance(
    envs.single_action_space, gym.spaces.Discrete
), "only discrete action space is supported"

agent = Agent(envs).to(device)
optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)
```

--------------------------------

### Parallel API Usage

Source: https://pettingzoo.farama.org/1.26.0/environments/atari/volleyball_pong

Example of how to use the Parallel API for the Volleyball Pong environment.

```APIDOC
from pettingzoo.atari import volleyball_pong_v3

env = volleyball_pong_v3.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Parallel API Usage

Source: https://pettingzoo.farama.org/1.25.0/environments/mpe/simple

Example of running the simple_v3 environment using the Parallel API.

```python
from pettingzoo.mpe import simple_v3

env = simple_v3.parallel_env(render_mode="human")
observations, infos = env.reset()

while env.agents:
    # this is where you would insert your policy
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}

    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Initialize Go Environment Variables

Source: https://pettingzoo.farama.org/1.25.0/_modules/pettingzoo/classic/go/go

Configures board dimensions, coordinate systems, and neighbor/diagonal mappings for the Go board.

```python
    def _overwrite_go_global_variables(self, board_size: int):
        self._N = board_size
        go_base.N = self._N
        go_base.ALL_COORDS = [(i, j) for i in range(self._N) for j in range(self._N)]
        go_base.EMPTY_BOARD = np.zeros([self._N, self._N], dtype=np.int8)
        go_base.NEIGHBORS = {
            (x, y): list(
                filter(
                    self._check_bounds, [(x + 1, y), (x - 1, y), (x, y + 1), (x, y - 1)]
                )
            )
            for x, y in go_base.ALL_COORDS
        }
        go_base.DIAGONALS = {
            (x, y): list(
                filter(
                    self._check_bounds,
                    [(x + 1, y + 1), (x + 1, y - 1), (x - 1, y + 1), (x - 1, y - 1)],
                )
            )
            for x, y in go_base.ALL_COORDS
        }
        return

    def _check_bounds(self, c):
        return 0 <= c[0] < self._N and 0 <= c[1] < self._N
```