### Procedural API Example for DQN

Source: https://github.com/thu-ml/tianshou/blob/master/docs/01_user_guide/01_apis.md

This example demonstrates the procedural API for setting up a Deep Q-Network (DQN) agent. It involves explicit definition of hyperparameters, environment setup, network construction, policy and algorithm instantiation, and collector configuration.

```python
import gymnasium as gym
import tianshou as ts
from tianshou.algorithm.modelfree.dqn import DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.data import CollectStats
from tianshou.trainer import OffPolicyTrainerParams
from tianshou.utils.net.common import Net
from tianshou.utils.space_info import SpaceInfo
from torch.utils.tensorboard import SummaryWriter

# Define hyperparameters
task = "CartPole-v1"
lr, epoch, batch_size = 1e-3, 10, 64
num_training_envs, num_test_envs = 10, 100
gamma, n_step, target_freq = 0.9, 3, 320
buffer_size = 20000
olds_train, eps_test = 0.1, 0.05
epoch_num_steps, collection_step_num_env_steps = 10000, 10

# Set up logging
logger = ts.utils.TensorboardLogger(SummaryWriter("log/dqn"))

# Create environments
training_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_training_envs)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])

# Build the network
env = gym.make(task, render_mode="human")
space_info = SpaceInfo.from_env(env)
state_shape = space_info.observation_info.obs_shape
action_shape = space_info.action_info.action_shape
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128, 128])

# Create policy and algorithm
policy = DiscreteQLearningPolicy(
    model=net,
    action_space=env.action_space,
    eps_training=eps_train,
    eps_inference=eps_test,
)
algorithm = ts.algorithm.DQN(
    policy=policy,
    optim=AdamOptimizerFactory(lr=lr),
    gamma=gamma,
    n_step_return_horizon=n_step,
    target_update_freq=target_freq,
)

# Set up collectors
training_collector = ts.data.Collector[CollectStats](
    algorithm,
    training_envs,
    ts.data.VectorReplayBuffer(buffer_size, num_training_envs),
    exploration_noise=True,
)
test_collector = ts.data.Collector[CollectStats](
    algorithm,
    test_envs,
    exploration_noise=True,
)


# Define stop condition
def stop_fn(mean_rewards: float) -> bool:
    if env.spec and env.spec.reward_threshold:
        return mean_rewards >= env.spec.reward_threshold
    return False


# Train the algorithm
result = algorithm.run_training(
    OffPolicyTrainerParams(
        training_collector=training_collector,
        test_collector=test_collector,
        max_epochs=epoch,
        epoch_num_steps=epoch_num_steps,
        collection_step_num_env_steps=collection_step_num_env_steps,
        test_step_num_episodes=num_test_envs,
        batch_size=batch_size,
        update_step_num_gradient_steps_per_sample=1 / collection_step_num_env_steps,
        stop_fn=stop_fn,
        logger=logger,
        test_in_training=True,
    )
)
print(f"Finished training in {result.timing.total_time} seconds")

# Watch the trained agent
collector = ts.data.Collector[CollectStats](algorithm, env, exploration_noise=True)
collector.collect(n_episode=100, render=1 / 35)

```

--------------------------------

### Install Tianshou from Source

Source: https://context7.com/thu-ml/tianshou/llms.txt

Install Tianshou using Poetry for the latest version. Use extras for specific environments and acceleration.

```bash
git clone git@github.com:thu-ml/tianshou.git
cd tianshou
# Basic install
poetry install
# With extras for specific environments and acceleration
poetry install --extras "mujoco atari envpool argparse"
# Or install all extras
poetry install --all-extras
```

--------------------------------

### Install Tianshou with Poetry

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Clone the repository and use Poetry to install Tianshou and its development dependencies. Ensure Poetry is installed on your system first.

```bash
git clone git@github.com:thu-ml/tianshou.git
cd tianshou
poetry install
```

--------------------------------

### Install EnvPool for ViZDoom

Source: https://github.com/thu-ml/tianshou/blob/master/examples/vizdoom/README.md

Install the EnvPool library, which provides a faster backend for ViZDoom environments. This is recommended for improved performance.

```bash
pip install envpool
```

--------------------------------

### Standard Off-Policy RL Setup with VectorReplayBuffer

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L2_Buffer.ipynb

Illustrates a typical setup for off-policy reinforcement learning using a `VectorReplayBuffer` for collecting data from parallel environments and training a policy.

```python
# env = make_vectorized_env(num_envs=8)
# buffer = VectorReplayBuffer(total_size=100000, buffer_num=8)
# policy = SACPolicy(...)
# collector = Collector(policy, env, buffer)
# 
# # Collect and train
# collector.collect(n_step=1000)
# for _ in range(10):
#     batch, indices = buffer.sample(256)
#     policy.learn(batch)
```

--------------------------------

### Install Tianshou from GitHub

Source: https://github.com/thu-ml/tianshou/blob/master/docs/index.rst

Use this command to install the latest version of Tianshou directly from its master branch on GitHub. Ensure you have pip installed.

```bash
$ pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade
```

--------------------------------

### Verify Tianshou Installation

Source: https://github.com/thu-ml/tianshou/blob/master/docs/index.rst

After installation, run this Python code to import the tianshou library and print its version. A successful execution indicates the library is installed correctly.

```python
import tianshou
print(tianshou.__version__)
```

--------------------------------

### Launch Tensorboard for log visualization

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Command to start Tensorboard, pointing it to the directory where training logs are saved. This allows visualization of metrics like rewards and losses.

```bash
$ tensorboard --logdir log/dqn
```

--------------------------------

### Install Tianshou and Development Requirements

Source: https://github.com/thu-ml/tianshou/blob/master/docs/05_developer_guide/developer_guide.md

Installs Tianshou in editable mode along with all development dependencies using Poetry.

```bash
poetry install --with dev
```

--------------------------------

### Install Tianshou using Pip or Conda

Source: https://context7.com/thu-ml/tianshou/llms.txt

Install the stable PyPI release or the latest master from GitHub using pip. Conda installation is also supported.

```bash
# Stable PyPI release
pip install tianshou

# Conda
conda install tianshou -c conda-forge

# Latest master from GitHub
pip install git+https://github.com/thu-ml/tianshou.git@master --upgrade
```

--------------------------------

### High-Level API DQN Experiment Setup

Source: https://github.com/thu-ml/tianshou/blob/master/docs/01_user_guide/01_apis.md

Demonstrates setting up a DQN experiment using the high-level API with declarative configuration for environment, training, and algorithm parameters. Requires Tianshou and its dependencies.

```python
from tianshou.highlevel.config import OffPolicyTrainingConfig
from tianshou.highlevel.env import EnvFactoryRegistered, VectorEnvType
from tianshou.highlevel.experiment import DQNExperimentBuilder, ExperimentConfig
from tianshou.highlevel.params.algorithm_params import DQNParams
from tianshou.highlevel.trainer import EpochStopCallbackRewardThreshold

# Build the experiment through configuration
experiment = (
    DQNExperimentBuilder(
        # Environment configuration
        EnvFactoryRegistered(
            task="CartPole-v1",
            venv_type=VectorEnvType.DUMMY,
            training_seed=0,
            test_seed=10,
        ),
        # Experiment settings
        ExperimentConfig(
            persistence_enabled=False,
            watch=True,
            watch_render=1 / 35,
            watch_num_episodes=100,
        ),
        # Training configuration
        OffPolicyTrainingConfig(
            max_epochs=10,
            epoch_num_steps=10000,
            batch_size=64,
            num_training_envs=10,
            num_test_envs=100,
            buffer_size=20000,
            collection_step_num_env_steps=10,
            update_step_num_gradient_steps_per_sample=1 / 10,
        ),
    )
    # Algorithm-specific parameters
    .with_dqn_params(
        DQNParams(
            lr=1e-3,
            gamma=0.9,
            n_step_return_horizon=3,
            target_update_freq=320,
            eps_training=0.3,
            eps_inference=0.0,
        ),
    )
    # Network architecture
    .with_model_factory_default(hidden_sizes=(64, 64))
    # Stop condition
    .with_epoch_stop_callback(EpochStopCallbackRewardThreshold(195))
    .build()
)

```

--------------------------------

### DQN Training for Qbert

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the QbertNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "QbertNoFrameskip-v4" --num_test_envs 100
```

--------------------------------

### Train PPO Agent for D2_navigation

Source: https://github.com/thu-ml/tianshou/blob/master/examples/vizdoom/README.md

Example command to train a PPO agent specifically for the D2_navigation task. This demonstrates using a different algorithm for the same task.

```bash
python3 vizdoom_ppo.py --task "D2_navigation"
```

--------------------------------

### OptimizerFactory Configuration Examples

Source: https://context7.com/thu-ml/tianshou/llms.txt

Shows how to configure Adam and RMSprop optimizers with various settings, including weight decay and LR schedulers, using OptimizerFactory.

```python
from tianshou.algorithm.optim import (
    AdamOptimizerFactory,
    RMSpropOptimizerFactory,
    LRSchedulerFactoryLinear,
)

# Adam with default settings
adam = AdamOptimizerFactory(lr=3e-4)

# Adam with weight decay
adam_wd = AdamOptimizerFactory(lr=1e-3, weight_decay=1e-4)

# RMSprop
rmsprop = RMSpropOptimizerFactory(lr=7e-4, eps=1e-5, alpha=0.99)

# Attach a linear LR decay scheduler
linear_scheduler = LRSchedulerFactoryLinear(
    max_epochs=10,
    epoch_num_steps=50000,
    collection_step_num_env_steps=2048,
)
adam_with_decay = AdamOptimizerFactory(lr=3e-4).with_lr_scheduler_factory(linear_scheduler)

# The factory creates (optimizer, scheduler) when given a module
import torch.nn as nn
model = nn.Linear(4, 2)
optimizer, lr_scheduler = adam_with_decay.create_instances(model)
```

--------------------------------

### Install Tianshou with PyPI

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Install the latest release of Tianshou from the Python Package Index (PyPI). Note that this version may lag behind the master branch.

```bash
$ pip install tianshou
```

--------------------------------

### PPO Setup and Training in Tianshou

Source: https://context7.com/thu-ml/tianshou/llms.txt

Sets up and trains a Proximal Policy Optimization (PPO) agent for discrete or continuous action spaces. Requires Gymnasium and Tianshou libraries. Configure actor-critic networks, policy parameters, and trainer settings.

```python
import gymnasium as gym
import torch
from tianshou.algorithm.modelfree.ppo import PPO
from tianshou.algorithm.modelfree.reinforce import ProbabilisticActorPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.data import Collector, CollectStats, VectorReplayBuffer
from tianshou.env import DummyVectorEnv
from tianshou.trainer import OnPolicyTrainerParams
from tianshou.utils.net.common import ActorCritic, Net
from tianshou.utils.net.discrete import DiscreteActor, DiscreteCritic
from tianshou.utils.space_info import SpaceInfo

task = "CartPole-v1"
env = gym.make(task)
space_info = SpaceInfo.from_env(env)
obs_shape    = space_info.observation_info.obs_shape
action_shape = space_info.action_info.action_shape

net    = Net(state_shape=obs_shape, hidden_sizes=[64, 64])
actor  = DiscreteActor(preprocess_net=net, action_shape=action_shape, hidden_sizes=[64])
critic = DiscreteCritic(preprocess_net=net, hidden_sizes=[64])
actor_critic = ActorCritic(actor, critic)

policy = ProbabilisticActorPolicy(
    actor=actor,
    dist_fn=torch.distributions.Categorical,
    action_space=env.action_space,
)
algorithm = PPO(
    policy=policy,
    critic=critic,
    optim=AdamOptimizerFactory(lr=3e-4),
    eps_clip=0.2,
    vf_coef=0.5,
    ent_coef=0.01,
    gae_lambda=0.95,
    gamma=0.99,
    advantage_normalization=True,
    max_grad_norm=0.5,
)

n_envs = 8
train_envs = DummyVectorEnv([lambda: gym.make(task) for _ in range(n_envs)])
test_envs  = DummyVectorEnv([lambda: gym.make(task) for _ in range(10)])

train_collector = Collector[CollectStats](
    algorithm, train_envs,
    VectorReplayBuffer(total_size=2048 * n_envs, buffer_num=n_envs),
)
test_collector = Collector[CollectStats](algorithm, test_envs)

result = algorithm.run_training(
    OnPolicyTrainerParams(
        training_collector=train_collector,
        test_collector=test_collector,
        max_epochs=10,
        epoch_num_steps=50000,
        collection_step_num_env_steps=2048,
        test_step_num_episodes=10,
        batch_size=256,
        repeat_per_collect=10,    # gradient updates per rollout batch
        stop_fn=lambda r: r >= 475,
    )
)
print(f"Best reward: {result.best_reward}")

```

--------------------------------

### Train PPO Agent for D3_battle

Source: https://github.com/thu-ml/tianshou/blob/master/examples/vizdoom/README.md

Example command to train a PPO agent specifically for the D3_battle task. This demonstrates using a different algorithm for the same task.

```bash
python3 vizdoom_ppo.py --task "D3_battle"
```

--------------------------------

### DQN Training for Breakout

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the BreakoutNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "BreakoutNoFrameskip-v4" --num_test_envs 100
```

--------------------------------

### DQN Training for Pong

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the PongNoFrameskip-v4 Atari task. Adjust batch size for convergence speed.

```python
python3 atari_dqn.py --task "PongNoFrameskip-v4" --batch_size 64
```

--------------------------------

### DQN Setup and Training in Tianshou

Source: https://context7.com/thu-ml/tianshou/llms.txt

Sets up and trains a Deep Q-Network (DQN) agent for discrete action spaces. Requires Gymnasium and Tianshou libraries. Configure network architecture, policy parameters, and trainer settings.

```python
import gymnasium as gym
import tianshou as ts
from tianshou.algorithm.modelfree.dqn import DQN, DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.data import Collector, CollectStats, VectorReplayBuffer
from tianshou.trainer import OffPolicyTrainerParams
from tianshou.utils import TensorboardLogger
from tianshou.utils.net.common import Net
from tianshou.utils.space_info import SpaceInfo
from torch.utils.tensorboard import SummaryWriter
import gymnasium as gym

task = "CartPole-v1"
env = gym.make(task)
space_info = SpaceInfo.from_env(env)

net = Net(
    state_shape=space_info.observation_info.obs_shape,
    action_shape=space_info.action_info.action_shape,
    hidden_sizes=[128, 128, 128],
)
policy = DiscreteQLearningPolicy(
    model=net, action_space=env.action_space,
    eps_training=0.3, eps_inference=0.0,
)
algorithm = DQN(
    policy=policy,
    optim=AdamOptimizerFactory(lr=1e-3),
    gamma=0.9,
    n_step_return_horizon=3,
    target_update_freq=320,
)

num_train_envs, num_test_envs = 10, 100
train_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_train_envs)])
test_envs  = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])

train_collector = Collector[CollectStats](
    algorithm, train_envs,
    VectorReplayBuffer(20000, num_train_envs),
    exploration_noise=True,
)
test_collector = Collector[CollectStats](algorithm, test_envs, exploration_noise=True)

logger = TensorboardLogger(SummaryWriter("log/dqn"))

result = algorithm.run_training(
    OffPolicyTrainerParams(
        training_collector=train_collector,
        test_collector=test_collector,
        max_epochs=10,
        epoch_num_steps=10000,
        collection_step_num_env_steps=10,
        test_step_num_episodes=num_test_envs,
        batch_size=64,
        update_step_num_gradient_steps_per_sample=1/10,
        stop_fn=lambda mean_rew: mean_rew >= env.spec.reward_threshold,
        logger=logger,
        test_in_training=True,
    )
)
print(f"Finished in {result.timing.total_time:.1f}s | Best reward: {result.best_reward:.2f}")
# tensorboard --logdir log/dqn

```

--------------------------------

### Taking Random Actions in Vectorized Environments

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L3_Environments.ipynb

Demonstrates how to take random actions across all environments in a vectorized setup. It shows how to generate actions, step the environment, and interpret the returned rewards, termination flags, and info.

```python
# Take random actions in all environments
actions = np.random.choice(2, size=vector_env.env_num)
ob, rew, terminated, truncated, info = vector_env.step(actions)

print(f"Actions taken: {actions}")
print(f"Rewards received: {rew}")
print(f"Terminated flags: {terminated}")
print("Info", info)
```

--------------------------------

### DQN Training for MsPacman

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the MsPacmanNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "MsPacmanNoFrameskip-v4" --num_test_envs 100
```

--------------------------------

### Pre-filling ReplayBuffer Before Training

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L2_Buffer.ipynb

Shows how to pre-fill a `ReplayBuffer` with random exploration data before starting the main training loop. This ensures the buffer contains a diverse set of experiences.

```python
# Collect random exploration data
# collector.collect(n_step=10000)  # Fill buffer
# 
# # Then start training
# while not converged:
#     collector.collect(n_step=100)
#     for _ in range(10):
#         batch = buffer.sample(256)
#         policy.learn(batch)
```

--------------------------------

### Multi-Agent RL Policy Setup with PettingZoo

Source: https://context7.com/thu-ml/tianshou/llms.txt

Demonstrates setting up MultiAgentPolicy for cooperative/competitive MARL and MapPolicy for parameter sharing. Integrates with PettingZoo environments via PettingZooEnv.

```python
from tianshou.algorithm.multiagent.marl import MultiAgentPolicy, MapPolicy
from tianshou.env.pettingzoo_env import PettingZooEnv
from pettingzoo.classic import tictactoe_v3
import tianshou as ts

# Wrap PettingZoo env
env = PettingZooEnv(tictactoe_v3.env())

# Build one policy per agent (same or different architectures)
# ... (create policy_0 and policy_1 as usual Policy objects) ...

multi_policy = MultiAgentPolicy(
    policies={"player_0": policy_0, "player_1": policy_1},
)

# MapPolicy: all agents share the same policy (parameter sharing)
shared_policy = MapPolicy(
    policy=shared_single_policy,
    agent_ids=["player_0", "player_1"],
)

# Use with a MARL-capable algorithm
from tianshou.algorithm.multiagent.marl import MultiAgentOffPolicyAlgorithm

algorithm = MultiAgentOffPolicyAlgorithm(
    policy=multi_policy,
    # ... per-algorithm params ...
)

# Training: set multi_agent_return_reduction for monitoring
from tianshou.trainer import OffPolicyTrainerParams
import numpy as np

params = OffPolicyTrainerParams(
    # ... standard params ...
    multi_agent_return_reduction=lambda returns: np.mean(returns, axis=1),
)
```

--------------------------------

### Custom Network Factory Example

Source: https://github.com/thu-ml/tianshou/blob/master/docs/01_user_guide/01_apis.md

Demonstrates how to create a custom network factory by subclassing IntermediateModuleFactory. This allows integration of custom neural network architectures into the Tianshou high-level API, specifying hidden layer sizes and ensuring compatibility with the environment and device.

```python
from tianshou.highlevel.env import Environments
from tianshou.highlevel.module.core import TDevice
from tianshou.highlevel.module.intermediate import IntermediateModuleFactory, IntermediateModule

class CustomNetFactory(IntermediateModuleFactory):
    def __init__(self, hidden_sizes: tuple[int, ...] = (128, 128)):
        self.hidden_sizes = hidden_sizes
    
    def create_intermediate_module(self, envs: Environments, device: TDevice) -> IntermediateModule:
        obs_shape = envs.get_observation_shape()
        action_shape = envs.get_action_shape()
        
        # Your custom network creation logic
        net = CustomNetwork(
            obs_shape=obs_shape,
            action_shape=action_shape,
            hidden_sizes=self.hidden_sizes,
        ).to(device)
        
        return IntermediateModule(net, net.output_dim)

experiment = (
    DQNExperimentBuilder(...)
    .with_model_factory(CustomNetFactory(hidden_sizes=(256, 256)))
    .build()
)
```

--------------------------------

### Create a standard Gymnasium environment

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L3_Environments.ipynb

Demonstrates how to create a standard single environment instance using Gymnasium's make function.

```python
# Standard Gymnasium environment
gym_env = gym.make("CartPole-v1")
```

--------------------------------

### Install Tianshou via Pip

Source: https://github.com/thu-ml/tianshou/blob/master/docs/index.rst

Install the Tianshou library using pip. This command installs the latest stable release. Requires Python 3.11 or newer.

```bash
pip install tianshou
```

--------------------------------

### Install Tianshou with Conda

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Install Tianshou from the conda-forge channel using Anaconda or Miniconda.

```bash
$ conda install tianshou -c conda-forge
```

--------------------------------

### Build DQN Policy and Algorithm Components

Source: https://context7.com/thu-ml/tianshou/llms.txt

Illustrates creating a neural network, a DiscreteQLearningPolicy with exploration, and a DQN algorithm for learning. Shows forward pass and saving/loading policy weights.

```python
import gymnasium as gym
import torch
from tianshou.algorithm.modelfree.dqn import DQN, DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory
from tianshou.utils.net.common import Net
from tianshou.utils.space_info import SpaceInfo

env = gym.make("CartPole-v1")
space_info = SpaceInfo.from_env(env)
state_shape  = space_info.observation_info.obs_shape   # (4,)
action_shape = space_info.action_info.action_shape     # 2

# 1. Build the neural network
net = Net(state_shape=state_shape, action_shape=action_shape, hidden_sizes=[128, 128])

# 2. Build the Policy (action-selection logic + epsilon-greedy exploration)
policy = DiscreteQLearningPolicy(
    model=net,
    action_space=env.action_space,
    eps_training=0.1,   # 10 % random actions during training
    eps_inference=0.05, # 5 % random actions during evaluation
)

# 3. Build the Algorithm (DQN learning logic)
algorithm = DQN(
    policy=policy,
    optim=AdamOptimizerFactory(lr=1e-3),
    gamma=0.99,
    n_step_return_horizon=3,
    target_update_freq=320,
)

# Forward pass: compute action for a batch of observations
import numpy as np
from tianshou.data import Batch
obs_batch = Batch(obs=np.random.randn(4, 4).astype(np.float32), info={})
act_batch = policy(obs_batch)
print(act_batch.act)  # array of chosen actions

# Save / restore the policy weights
torch.save(policy.state_dict(), "dqn_policy.pth")
policy.load_state_dict(torch.load("dqn_policy.pth"))
```

--------------------------------

### Configure and Use Collector for Rollout Data

Source: https://context7.com/thu-ml/tianshou/llms.txt

Shows how to set up a Collector for training and testing, including buffer management and collecting data in steps or episodes. Demonstrates rendering during collection.

```python
import gymnasium as gym
import tianshou as ts
from tianshou.data import Collector, VectorReplayBuffer, CollectStats

# Assume `algorithm` is an instantiated Algorithm object (e.g., DQN)
train_envs = ts.env.DummyVectorEnv([lambda: gym.make("CartPole-v1") for _ in range(10)])
test_envs  = ts.env.DummyVectorEnv([lambda: gym.make("CartPole-v1") for _ in range(5)])

buffer = VectorReplayBuffer(total_size=20000, buffer_num=10)

# Training collector writes data into the replay buffer
train_collector = Collector[CollectStats](
    algorithm,
    train_envs,
    buffer,
    exploration_noise=True,  # add noise during training for exploration
)

# Test collector (no buffer needed — results are ephemeral)
test_collector = Collector[CollectStats](
    algorithm,
    test_envs,
    exploration_noise=False,
)

# Collect exactly 1000 environment steps
stats = train_collector.collect(n_step=1000)
print(f"Steps collected: {stats.n_collected_steps}")
print(f"Episodes done:   {stats.n_collected_episodes}")

# Or collect a fixed number of complete episodes
stats = test_collector.collect(n_episode=10)
print(f"Mean return: {stats.returns_stat.mean:.2f}")
print(f"Std  return: {stats.returns_stat.std:.2f}")
print(f"Mean length: {stats.lens_stat.mean:.1f}")

# Render at 35 FPS while collecting
env = gym.make("CartPole-v1", render_mode="human")
single_collector = ts.data.Collector(algorithm, env, exploration_noise=False)
single_collector.collect(n_episode=1, render=1/35)
```

--------------------------------

### Configure and Run PPO Experiment

Source: https://context7.com/thu-ml/tianshou/llms.txt

Sets up a PPO experiment for CartPole-v1 with specified environment and training configurations. It then runs the experiment and collects results.

```python
from tianshou.highlevel.experiment import ExperimentBuilder
from tianshou.env import VectorEnvType
from tianshou.highlevel.config import ExperimentConfig, OnPolicyTrainingConfig
from tianshou.highlevel.params import PPOParams

ppo_experiment = (
    PPOExperimentBuilder(
        EnvFactoryRegistered(task="CartPole-v1", venv_type=VectorEnvType.DUMMY,
                             training_seed=0, test_seed=10),
        ExperimentConfig(persistence_enabled=False, watch=False),
        OnPolicyTrainingConfig(
            max_epochs=10, epoch_num_steps=50000, batch_size=256,
            num_training_envs=8, num_test_envs=10, buffer_size=16384,
            collection_step_num_env_steps=2048,
        ),
    )
    .with_ppo_params(PPOParams(lr=3e-4, gamma=0.99, gae_lambda=0.95,
                               eps_clip=0.2, vf_coef=0.5, ent_coef=0.01,
                               repeat_per_collect=10))
    .with_model_factory_default(hidden_sizes=(64, 64))
    .build()
)
ppo_result = ppo_experiment.run()
```

--------------------------------

### Setup Environment and Agents for MARL

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L6_MARL.ipynb

Defines functions to create a PettingZoo environment and configure learning and opponent agents for multi-agent reinforcement learning. Handles observation and action space shapes, neural network creation, Q-learning policy, and DQN algorithm setup.

```python
def get_env(render_mode: str | None = None) -> PettingZooEnv:
    return PettingZooEnv(tictactoe_v3.env(render_mode=render_mode))

def get_agents(
    args,
    agent_learn: OffPolicyAlgorithm | None = None,
    agent_opponent: OffPolicyAlgorithm | None = None,
    optim: OptimizerFactory | None = None,
) -> tuple[MultiAgentOffPolicyAlgorithm, torch.optim.Optimizer | None, list]:
    """Create or load agents for training."""
    env = get_env()
    observation_space = (
        env.observation_space.spaces["observation"]
        if isinstance(env.observation_space, gymnasium.spaces.Dict)
        else env.observation_space
    )
    args.state_shape = observation_space.shape or int(observation_space.n)
    args.action_shape = env.action_space.shape or int(env.action_space.n)

    if agent_learn is None:
        # Create the neural network model
        net = Net(
            state_shape=args.state_shape,
            action_shape=args.action_shape,
            hidden_sizes=args.hidden_sizes,
        ).to(args.device)

        if optim is None:
            optim = AdamOptimizerFactory(lr=args.lr)

        # Create Q-learning policy for the learning agent
        algorithm = DiscreteQLearningPolicy(
            model=net,
            action_space=env.action_space,
            eps_training=args.eps_train,
            eps_inference=args.eps_test,
        )

        # Wrap in DQN algorithm
        agent_learn = DQN(
            policy=algorithm,
            optim=optim,
            n_step_return_horizon=args.n_step,
            gamma=args.gamma,
            target_update_freq=args.target_update_freq,
        )

        if args.resume_path:
            agent_learn.load_state_dict(torch.load(args.resume_path))

    if agent_opponent is None:
        if args.opponent_path:
            # Load a pre-trained opponent for self-play
            agent_opponent = deepcopy(agent_learn)
            agent_opponent.load_state_dict(torch.load(args.opponent_path))
        else:
            # Use a random opponent
            agent_opponent = MARLRandomDiscreteMaskedOffPolicyAlgorithm(
                action_space=env.action_space
            )

    # Arrange agents based on which player position the learning agent takes
    if args.agent_id == 1:
        agents = [agent_learn, agent_opponent]
    else:
        agents = [agent_opponent, agent_learn]

    ma_algorithm = MultiAgentOffPolicyAlgorithm(algorithms=agents, env=env)
    return ma_algorithm, optim, env.agents
```

--------------------------------

### Check Tianshou Version

Source: https://context7.com/thu-ml/tianshou/llms.txt

Verify the installed Tianshou version by importing the library and printing its version attribute.

```python
import tianshou
print(tianshou.__version__)  # e.g. "2.0.1"
```

--------------------------------

### Run CQL on PongNoFrameskip-v4 with small buffer size

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on PongNoFrameskip-v4 with a small expert buffer size (10,000 samples) and a min-q-weight parameter.

```bash
python3 atari_cql.py --task PongNoFrameskip-v4 --load-buffer-name log/PongNoFrameskip-v4/qrdqn/expert.size_1e4.hdf5 --epoch 5 --min-q-weight 1
```

--------------------------------

### Initialize Tensorboard logger

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Initializes a Tensorboard logger to track training metrics. Ensure Tensorboard is installed and accessible.

```python
logger = ts.utils.TensorboardLogger(SummaryWriter('log/dqn'))
```

--------------------------------

### Run CQL on BreakoutNoFrameskip-v4 with small buffer size and min-q-weight

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on BreakoutNoFrameskip-v4 with a small expert buffer size (10,000 samples) and a min-q-weight parameter.

```bash
python3 atari_cql.py --task BreakoutNoFrameskip-v4 --load-buffer-name log/BreakoutNoFrameskip-v4/qrdqn/expert.size_1e4.hdf5 --epoch 12 --min-q-weight 10
```

--------------------------------

### Create training and testing environments

Source: https://github.com/thu-ml/tianshou/blob/master/README.md

Sets up vectorized environments for training and testing. DummyVectorEnv is used here, but SubprocVectorEnv can be utilized for parallelization.

```python
# You can also try SubprocVectorEnv, which will use parallelization
training_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_training_envs)])
test_envs = ts.env.DummyVectorEnv([lambda: gym.make(task) for _ in range(num_test_envs)])
```

--------------------------------

### Tianshou ReplayBuffer Operations

Source: https://context7.com/thu-ml/tianshou/llms.txt

Illustrates creating, adding transitions to, sampling from, and saving/loading a Tianshou ReplayBuffer. Includes examples for PrioritizedReplayBuffer and VectorReplayBuffer.

```python
import numpy as np
from tianshou.data import ReplayBuffer, Batch

# Create a buffer with capacity 10000
buf = ReplayBuffer(size=10000)

# Add transitions (batch must include obs, act, rew, terminated, truncated, done, obs_next)
transition = Batch(
    obs=np.array([1.0, 0.5]),
    act=np.int64(1),
    rew=1.0,
    terminated=False,
    truncated=False,
    done=False,
    obs_next=np.array([0.9, 0.6]),
    info={},
)
buf.add(transition)
print(len(buf))   # 1

# Sample a mini-batch
batch, indices = buf.sample(batch_size=64)
print(batch.obs.shape)   # (64, 2)
print(batch.rew.shape)   # (64,)

# Prioritized Experience Replay
from tianshou.data import PrioritizedReplayBuffer
per_buf = PrioritizedReplayBuffer(size=10000, alpha=0.6, beta=0.4)
per_buf.add(transition)
batch, indices = per_buf.sample(64)
per_buf.update_weight(indices, new_weights=np.abs(batch.rew) + 1e-6)

# VectorReplayBuffer: one sub-buffer per parallel environment
from tianshou.data import VectorReplayBuffer
vec_buf = VectorReplayBuffer(total_size=20000, buffer_num=10)

# Save / Load (HDF5)
buf.save_hdf5("replay_buffer.hdf5")
buf2 = ReplayBuffer.load_hdf5("replay_buffer.hdf5")
```

--------------------------------

### Run CQL on BreakoutNoFrameskip-v4 with reduced buffer size and min-q-weight

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on BreakoutNoFrameskip-v4 with a reduced expert buffer size (100,000 samples) and a min-q-weight parameter.

```bash
python3 atari_cql.py --task BreakoutNoFrameskip-v4 --load-buffer-name log/BreakoutNoFrameskip-v4/qrdqn/expert.size_1e5.hdf5 --epoch 12 --min-q-weight 20
```

--------------------------------

### Import Necessary Modules for MARL

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L6_MARL.ipynb

Imports all required libraries and Tianshou modules for setting up and running MARL experiments. Ensure all dependencies are installed.

```python
import os
from copy import deepcopy
from functools import partial

import gymnasium
import torch
from pettingzoo.classic import tictactoe_v3
from torch.utils.tensorboard import SummaryWriter

from tianshou.algorithm import (
    DQN,
    Algorithm,
    MARLRandomDiscreteMaskedOffPolicyAlgorithm,
    MultiAgentOffPolicyAlgorithm,
)
from tianshou.algorithm.algorithm_base import OffPolicyAlgorithm
from tianshou.algorithm.modelfree.dqn import DiscreteQLearningPolicy
from tianshou.algorithm.optim import AdamOptimizerFactory, OptimizerFactory
from tianshou.data import Collector, CollectStats, VectorReplayBuffer
from tianshou.data.stats import InfoStats
from tianshou.env import DummyVectorEnv
from tianshou.env.pettingzoo_env import PettingZooEnv
from tianshou.trainer import OffPolicyTrainerParams
from tianshou.utils import TensorboardLogger
from tianshou.utils.net.common import Net
```

--------------------------------

### Run CQL on PongNoFrameskip-v4 with reduced buffer size

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on PongNoFrameskip-v4 with a reduced expert buffer size (100,000 samples).

```bash
python3 atari_cql.py --task PongNoFrameskip-v4 --load-buffer-name log/PongNoFrameskip-v4/qrdqn/expert.size_1e5.hdf5 --epoch 5
```

--------------------------------

### Train C51 Agent for D3_battle

Source: https://github.com/thu-ml/tianshou/blob/master/examples/vizdoom/README.md

Example command to train a C51 agent specifically for the D3_battle task. This is one of the tasks available for ViZDoom.

```bash
python3 vizdoom_c51.py --task "D3_battle"
```

--------------------------------

### Train C51 Agent for D2_navigation

Source: https://github.com/thu-ml/tianshou/blob/master/examples/vizdoom/README.md

Example command to train a C51 agent specifically for the D2_navigation task. This is one of the tasks available for ViZDoom.

```bash
python3 vizdoom_c51.py --task "D2_navigation"
```

--------------------------------

### Interact with Environment using env.step()

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L6_MARL.ipynb

Demonstrates taking an action in a multi-agent environment and observing the resulting state, reward, and game status. Follows the standard Gymnasium API.

```python
import numpy as np

# Take an action (place mark at position 0 - top-left corner)
action = 0  # action can be an integer or a numpy array with one element
obs, reward, done, truncated, info = env.step(action)  # follows the Gymnasium API

print("Observation after first move:")
print(obs)

# Examine the reward structure
# Reward has two items (one for each player): 1 for win, -1 for loss, 0 otherwise
print(f"\nReward: {reward}")

# Check if the game is over
print(f"Done: {done}")

# Info is typically an empty dict in Tic-Tac-Toe but may contain useful information in other environments
print(f"Info: {info}")
```

--------------------------------

### DQN Training for Seaquest

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the SeaquestNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "SeaquestNoFrameskip-v4" --num_test_envs 100
```

--------------------------------

### Run CQL on BreakoutNoFrameskip-v4 with min-q-weight

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on BreakoutNoFrameskip-v4, using a specific expert buffer and epoch, with a min-q-weight parameter.

```bash
python3 atari_cql.py --task BreakoutNoFrameskip-v4 --load-buffer-name log/BreakoutNoFrameskip-v4/qrdqn/expert.hdf5 --epoch 12 --min-q-weight 50
```

--------------------------------

### DQN Training for Enduro

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the EnduroNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "EnduroNoFrameskip-v4 " --num_test_envs 100
```

--------------------------------

### Initialize and Use VectorEnv for Parallel Environments

Source: https://context7.com/thu-ml/tianshou/llms.txt

Demonstrates initializing DummyVectorEnv and SubprocVectorEnv for sequential and parallel environment execution. Includes seeding, resetting, and stepping environments.

```python
import gymnasium as gym
from tianshou.env import DummyVectorEnv, SubprocVectorEnv

n_envs = 8
task = "CartPole-v1"

# Sequential (in-process) – simplest, good for debugging
train_envs = DummyVectorEnv([lambda: gym.make(task) for _ in range(n_envs)])

# Subprocess-based parallelism
test_envs = SubprocVectorEnv([lambda: gym.make(task) for _ in range(4)])

# Set different seeds per environment
train_envs.seed([i * 10 for i in range(n_envs)])

# Reset all or specific environments
obs, info = train_envs.reset()
obs_partial, info_partial = train_envs.reset(env_id=[0, 2, 5])

# Step synchronously
actions = [train_envs.action_space.sample() for _ in range(n_envs)]
obs, rew, terminated, truncated, info = train_envs.step(actions)

# Async stepping – return when wait_num envs finish
async_envs = DummyVectorEnv(
    [lambda: gym.make(task) for _ in range(n_envs)],
    wait_num=4,       # return when 4 envs are done
    timeout=0.1,      # or when 0.1s has elapsed
)

train_envs.close()
test_envs.close()
```

--------------------------------

### Batch with Scalar Values

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L1_Batch.ipynb

Shows how a Batch containing only scalar values is handled, noting that scalars do not have a length and attempting to get the length raises a TypeError.

```python
batch2 = Batch(a=5, b=10)
print("\nScalar batch:")
print(f"batch2.shape = {batch2.shape}")
try:
    print(f"len(batch2) = {len(batch2)}")
except TypeError as e:
    print(f"len(batch2) raises TypeError: {e}")
```

--------------------------------

### Loading ReplayBuffer from Raw Data with from_data()

Source: https://github.com/thu-ml/tianshou/blob/master/docs/02_deep_dives/L2_Buffer.ipynb

Illustrates creating a ReplayBuffer from pre-collected raw data stored in an HDF5 file, suitable for offline RL scenarios.

```python
# Simulate pre-collected offline dataset
import h5py

# Create temporary HDF5 file with raw data
with tempfile.NamedTemporaryFile(suffix=".hdf5", delete=False) as tmp:
    offline_path = tmp.name

with h5py.File(offline_path, "w") as f:
    # Create datasets
    n = 100
    f.create_dataset("obs", data=np.random.randn(n, 4))
    f.create_dataset("act", data=np.random.randint(0, 2, n))
    f.create_dataset("rew", data=np.random.randn(n))
    f.create_dataset("terminated", data=np.random.random(n) < 0.1)
    f.create_dataset("truncated", data=np.zeros(n, dtype=bool))
    f.create_dataset("done", data=np.random.random(n) < 0.1)
    f.create_dataset("obs_next", data=np.random.randn(n, 4))
```

--------------------------------

### Run Multiple Experiments with Collection

Source: https://context7.com/thu-ml/tianshou/llms.txt

Demonstrates how to collect multiple experiments and run them using a registered experiment launcher, such as sequentially.

```python
from tianshou.highlevel.experiment import ExperimentCollection
from tianshou.evaluation.launcher import RegisteredExpLauncher

experiments = ExperimentCollection([dqn_experiment])
experiments.run(RegisteredExpLauncher.sequential)
```

--------------------------------

### Monitor Logs with Tensorboard

Source: https://github.com/thu-ml/tianshou/blob/master/examples/mujoco/README.md

Command to launch Tensorboard for monitoring training logs saved in the './log/' directory.

```bash
tensorboard --logdir log
```

--------------------------------

### Run Mujoco SAC Experiment

Source: https://github.com/thu-ml/tianshou/blob/master/examples/mujoco/README.md

Example command to run a Soft Actor-Critic (SAC) experiment in the Ant-v3 Mujoco environment. Logs are saved in the './log/' directory.

```bash
python mujoco_sac.py --task Ant-v3
```

--------------------------------

### Run CQL on PongNoFrameskip-v4

Source: https://github.com/thu-ml/tianshou/blob/master/examples/offline/README.md

Execute CQL on the PongNoFrameskip-v4 task. Specify the expert buffer and the epoch for training.

```bash
python3 atari_cql.py --task PongNoFrameskip-v4 --load-buffer-name log/PongNoFrameskip-v4/qrdqn/expert.hdf5 --epoch 5
```

--------------------------------

### DQN Training for Space Invaders

Source: https://github.com/thu-ml/tianshou/blob/master/examples/atari/README.md

Example command to train a DQN agent on the SpaceInvadersNoFrameskip-v4 Atari task. Use a higher number of test environments for evaluation.

```python
python3 atari_dqn.py --task "SpaceInvadersNoFrameskip-v4" --num_test_envs 100
```