Stable Baselines3 Contrib

https://github.com/stable-baselines-team/stable-baselines3-contrib
Admin
Stable Baselines3 Contrib is an experimental reinforcement learning package that provides additional...
Tokens:10,770
Snippets:130
Trust Score:8.7
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

Install MicroRTS and other required packages. It is recommended to create a virtual environment before installation.

```bash
# Install MicroRTS:
rm -fR ~/microrts && mkdir ~/microrts && \
 wget -O ~/microrts/microrts.zip http://microrts.s3.amazonaws.com/microrts/artifacts/202004222224.microrts.zip && \
 unzip ~/microrts/microrts.zip -d ~/microrts/

# You may want to make a venv before installing packages
pip install -r requirements.txt
```

--------------------------------

### Install Development Version

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/install.md

Clone the repository and install the development version in editable mode for contributing.

```bash
git clone https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/ && cd stable-baselines3-contrib
pip install -e .
```

--------------------------------

### Install Sphinx and Theme

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/README.md

Installs the necessary Python packages for building documentation with Sphinx and the Read the Docs theme.

```bash
pip install sphinx sphinx-autobuild sphinx-rtd-theme
```

--------------------------------

### Install Bleeding-edge Version

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/install.md

Install the latest version directly from the GitHub repository using pip.

```bash
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/
```

--------------------------------

### TQC Training and Inference Example

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/tqc.md

Demonstrates how to train a TQC model on the Pendulum-v1 environment, save it, and then load it for inference. Ensure the environment is compatible with TQC policies.

```python
import gymnasium as gym
import numpy as np

from sb3_contrib import TQC

env = gym.make("Pendulum-v1", render_mode="human")

policy_kwargs = dict(n_critics=2, n_quantiles=25)
model = TQC("MlpPolicy", env, top_quantiles_to_drop_per_net=2, verbose=1, policy_kwargs=policy_kwargs)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("tqc_pendulum")

del model # remove to demonstrate saving and loading

model = TQC.load("tqc_pendulum")

obs, _ = env.reset()
while True:
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    env.render()
    if terminated or truncated:
        obs, _ = env.reset()
```

--------------------------------

### Install Stable Release

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/install.md

Use this command to install the latest stable version of sb3-contrib using pip.

```bash
pip install sb3-contrib
```

--------------------------------

### Install SB3-Contrib master version

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/README.md

Install the master version of SB3-Contrib directly from GitHub using pip. This provides the latest development version.

```bash
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
```

--------------------------------

### Documentation Template for RL Algorithms

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Use this template when documenting a new full training algorithm. It includes sections for name, description, references, example usage, results, replication code, and implementation comments.

```rst
[Feature/Algorithm name]
========================

- Non-abbreviated  name and/or one-sentence description of the method.
- Link and reference to the original publications the present the feature, or other established source(s).
- Links to any codebases that were used for reference (e.g. authors' implementations)

Example
-------

A minimal example on how to use the feature (full, runnable code).

Results
-------

A description and comparison of results (e.g. how the change improved results over the non-changed algorithm), if
applicable.
Please link the associated pull request, e.g., `Pull Request #4 <https://github.com/Stable-Baselines-Team/stable-baselines3-contrib/pull/4>`_.

Include the expected results from the work that originally proposed the method (e.g. original paper).

How to replicate the results?
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ "```

:param arg1: describe what is arg1
:param arg2: describe what is arg2
:return: describe what is returned
    """
    ...
    return my_variable
```
```

--------------------------------

### Install Stable Baselines3 master version

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/README.md

Install the master version of Stable Baselines3 using pip. This is recommended for compatibility with SB3-Contrib.

```bash
pip install git+https://github.com/DLR-RM/stable-baselines3
```

--------------------------------

### Train and Save QR-DQN Model

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/qrdqn.md

This example demonstrates how to initialize, train, and save a QR-DQN model for a CartPole environment. It also shows how to load a saved model and use it for inference.

```python
import gymnasium as gym

from sb3_contrib import QRDQN

env = gym.make("CartPole-v1", render_mode="human")

policy_kwargs = dict(n_quantiles=50)
model = QRDQN("MlpPolicy", env, policy_kwargs=policy_kwargs, verbose=1)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("qrdqn_cartpole")

del model # remove to demonstrate saving and loading

model = QRDQN.load("qrdqn_cartpole")

obs, _ = env.reset()
while True:
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    env.render()
    if terminated or truncated:
        obs, _ = env.reset()
```

--------------------------------

### Train TQC Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a Truncated Quantile Critics (TQC) agent on the Pendulum environment. This example shows basic TQC agent instantiation and training.

```python
from sb3_contrib import TQC

model = TQC("MlpPolicy", "Pendulum-v1", top_quantiles_to_drop_per_net=2, verbose=1)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("tqc_pendulum")
```

--------------------------------

### Train CrossQ Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a CrossQ agent on the Pendulum environment. This example shows how to configure the network architecture for the CrossQ policy.

```python
from sb3_contrib import CrossQ

model = CrossQ(
    "MlpPolicy",
    "Pendulum-v1",
    verbose=1,
    policy_kwargs=dict(
        net_arch=dict(
            pi=[256, 256],
            qf=[1024, 1024],
        )
    ),
)
model.learn(total_timesteps=5_000, log_interval=4)
model.save("crossq_pendulum")
```

--------------------------------

### Train QR-DQN Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a Quantile Regression DQN (QR-DQN) agent on the CartPole environment. This example demonstrates configuring the number of quantiles for the QR-DQN policy.

```python
from sb3_contrib import QRDQN

policy_kwargs = dict(n_quantiles=50)
model = QRDQN("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("qrdqn_cartpole")
```

--------------------------------

### Train ARS with Asynchronous Multi-processing

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ars.md

Demonstrates the use of experimental asynchronous multi-processing for ARS. This setup runs multiple environments in parallel and asynchronously, with callbacks being called after evaluation episodes.

```python
from sb3_contrib import ARS
from sb3_contrib.common.vec_env import AsyncEval

from stable_baselines3.common.env_util import make_vec_env

env_id = "CartPole-v1"
n_envs = 2

model = ARS("LinearPolicy", env_id, n_delta=2, n_top=1, verbose=1)
# Create env for asynchronous evaluation (run in different processes)
async_eval = AsyncEval([lambda: make_vec_env(env_id) for _ in range(n_envs)], model.policy)

model.learn(total_timesteps=200_000, log_interval=4, async_eval=async_eval)
```

--------------------------------

### Train ARS Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train an agent using Augmented Random Search (ARS) on the Pendulum environment. This example uses a LinearPolicy for the ARS agent.

```python
from sb3_contrib import ARS

model = ARS("LinearPolicy", "Pendulum-v1", verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("ars_pendulum")
```

--------------------------------

### Train TRPO Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a Trust Region Policy Optimization (TRPO) agent on the Pendulum environment. This example shows basic TRPO agent instantiation and training with a specified gamma value.

```python
from sb3_contrib import TRPO

model = TRPO("MlpPolicy", "Pendulum-v1", gamma=0.9, verbose=1)
model.learn(total_timesteps=100_000, log_interval=4)
model.save("trpo_pendulum")
```

--------------------------------

### Build Documentation

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Command to build the project's documentation. Ensure documentation is up-to-date with changes.

```bash
make doc
```

--------------------------------

### Build Documentation

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/README.md

Generates the HTML documentation from the Sphinx source files located in the 'docs/' folder.

```bash
make html
```

--------------------------------

### Update QRDQN learning_starts parameter

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/misc/changelog.md

Demonstrates the change in default learning_starts value for QRDQN in versions 2.3.0 and later.

```python
# SB3 < 2.3.0 default hyperparameters, 50_000 corresponded to Atari defaults hyperparameters
# model = QRDQN("MlpPolicy", env, learning_starts=50_000)
# SB3 >= 2.3.0:
model = QRDQN("MlpPolicy", env, learning_starts=100)
```

--------------------------------

### Clone Repository

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

Clone the repository for the PPO comparison experiment.

```bash
git clone git@github.com:kronion/microrts-ppo-comparison.git
cd microrts-ppo-comparison
```

--------------------------------

### Clone and Run Benchmark

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_recurrent.md

Instructions to clone the rl-baselines3-zoo repository and run a benchmark for PPO-LSTM on specified environments.

```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo

python train.py --algo ppo_lstm --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```

--------------------------------

### TQC Benchmark Replication - Plotting Results

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/tqc.md

Commands to plot the results of TQC benchmark experiments using scripts from the RL-Zoo.

```bash
python scripts/all_plots.py -a tqc -e HalfCheetah Ant Hopper Walker2D BipedalWalkerHardcore -f logs/ -o logs/tqc_results
python scripts/plot_from_file.py -i logs/tqc_results.pkl -latex -l TQC
```

--------------------------------

### TQC Benchmark Replication - Training

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/tqc.md

Command to train a TQC model for benchmarking using the RL-Zoo. Replace `$ENV_ID` with the target environment ID.

```bash
python train.py --algo tqc --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```

--------------------------------

### Run QR-DQN Benchmark

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/qrdqn.md

Command to run the QR-DQN benchmark using the RL-Zoo fork. Replace `$ENV_ID` with the desired environment ID.

```bash
python train.py --algo qrdqn --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```

--------------------------------

### Build Documentation Continuously

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/README.md

Builds the HTML documentation and automatically rebuilds it whenever a file changes. This is useful for development.

```bash
sphinx-autobuild . _build/html
```

--------------------------------

### Train and Save TRPO Model

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/trpo.md

Demonstrates how to instantiate, train, save, and load a TRPO model for a given environment. Ensure the environment and policy are compatible.

```python
import gymnasium as gym
import numpy as np

from sb3_contrib import TRPO

env = gym.make("Pendulum-v1", render_mode="human")

model = TRPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000, log_interval=4)
model.save("trpo_pendulum")

del model # remove to demonstrate saving and loading

model = TRPO.load("trpo_pendulum")

obs, _ = env.reset()
while True:
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    env.render()
    if terminated or truncated:
        obs, _ = env.reset()
```

--------------------------------

### Train PPO Models

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

Train PPO models with various seeds, both with and without action masking, for different MicroRTS map sizes.

```bash
# python sb/train_ppo.py [output dir] [MicroRTS map size] [--mask] [--seed int]

# 4x4 unmasked
python sb3/train_ppo.py zoo 4 --seed 42
python sb3/train_ppo.py zoo 4 --seed 43
python sb3/train_ppo.py zoo 4 --seed 44

# 4x4 masked
python sb3/train_ppo.py zoo 4 --mask --seed 42
python sb3/train_ppo.py zoo 4 --mask --seed 43
python sb3/train_ppo.py zoo 4 --mask --seed 44

# 10x10 unmasked
python sb3/train_ppo.py zoo 10 --seed 42
python sb3/train_ppo.py zoo 10 --seed 43
python sb3/train_ppo.py zoo 10 --seed 44

# 10x10 masked
python sb3/train_ppo.py zoo 10 --mask --seed 42
python sb3/train_ppo.py zoo 10 --mask --seed 43
python sb3/train_ppo.py zoo 10 --mask --seed 44
```

--------------------------------

### View TensorBoard Logs

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

View the TensorBoard log output to analyze training results for different environment sizes.

```bash
# For 4x4 environment
 tensorboard --logdir zoo/4x4/runs

 # For 10x10 environment
 tensorboard --logdir zoo/10x10/runs
```

--------------------------------

### Run All Code Checks

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Single command to run type checking, formatting, and linting. Recommended before committing.

```bash
make commit-checks
```

--------------------------------

### Train, Evaluate, Save, and Load Recurrent PPO Model

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_recurrent.md

Demonstrates the full lifecycle of a RecurrentPPO model, including training, evaluation, saving, loading, and prediction with LSTM state management. Ensure `lstm_states` and `episode_start` are passed to `predict` for correct state updates.

```python
import numpy as np

from sb3_contrib import RecurrentPPO
from stable_baselines3.common.evaluation import evaluate_policy

model = RecurrentPPO("MlpLstmPolicy", "CartPole-v1", verbose=1)
model.learn(5000)

vec_env = model.get_env()
mean_reward, std_reward = evaluate_policy(model, vec_env, n_eval_episodes=20, warn=False)
print(mean_reward)

model.save("ppo_recurrent")
del model # remove to demonstrate saving and loading

model = RecurrentPPO.load("ppo_recurrent")

obs = vec_env.reset()
# cell and hidden state of the LSTM
lstm_states = None
num_envs = 1
# Episode start signals are used to reset the lstm states
episode_starts = np.ones((num_envs,), dtype=bool)
while True:
    action, lstm_states = model.predict(obs, state=lstm_states, episode_start=episode_starts, deterministic=True)
    obs, rewards, dones, info = vec_env.step(action)
    episode_starts = dones
    vec_env.render("human")
```

--------------------------------

### Clone RL-Zoo and Checkout Branch

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/qrdqn.md

Instructions to clone the RL-Zoo repository and checkout the specific branch for QR-DQN experiments.

```bash
git clone https://github.com/ku2482/rl-baselines3-zoo/
cd rl-baselines3-zoo/
git checkout feat/qrdqn
```

--------------------------------

### Train CrossQ Model

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/crossq.md

Instantiate and train a CrossQ model with an MlpPolicy for a continuous control environment. The model is then saved to a file.

```python
from sb3_contrib import CrossQ

model = CrossQ("MlpPolicy", "Walker2d-v4")
model.learn(total_timesteps=1_000_000)
model.save("crossq_walker")
```

--------------------------------

### Replicate CrossQ Results with RL-Zoo

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/crossq.md

Clone the RL-Zoo repository and use its training script to benchmark CrossQ on specified environments. Replace `$ENV_ID` with the target environment ID.

```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
python train.py --algo crossq --env $ENV_ID --n-eval-envs 5 --eval-episodes 20 --eval-freq 25000
```

--------------------------------

### Plot QR-DQN Results

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/qrdqn.md

Commands to plot the results of QR-DQN benchmarks using scripts from the RL-Zoo repository.

```bash
python scripts/all_plots.py -a qrdqn -e Breakout Pong -f logs/ -o logs/qrdqn_results
python scripts/plot_from_file.py -i logs/qrdqn_results.pkl -latex -l QR-DQN
```

--------------------------------

### Run TRPO Benchmark

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/trpo.md

Executes the TRPO benchmark using the RL-Zoo training script. Replace $ENV_ID with the specific environment name.

```bash
python train.py --algo trpo --env $ENV_ID --n-eval-envs 10 --eval-episodes 20 --eval-freq 50000
```

--------------------------------

### Python Function Documentation Template

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Template for documenting Python functions, including type hints and a docstring with parameter and return descriptions. Use this for new functions and methods.

```python
def my_function(arg1: type1, arg2: type2) -> returntype:
    """
    Short description of the function.

    :param arg1: describe what is arg1
    :param arg2: describe what is arg2
    :return: describe what is returned
    """
    ...
    return my_variable
```

--------------------------------

### Train and Evaluate Maskable PPO Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

Train a Maskable PPO agent on an environment with discrete actions and invalid action masks. Includes evaluation and saving/loading the model. Use `MaskableEvalCallback` for evaluation.

```python
from sb3_contrib import MaskablePPO
from sb3_contrib.common.envs import InvalidActionEnvDiscrete
from sb3_contrib.common.maskable.evaluation import evaluate_policy
from sb3_contrib.common.maskable.utils import get_action_masks
# This is a drop-in replacement for EvalCallback
from sb3_contrib.common.maskable.callbacks import MaskableEvalCallback


env = InvalidActionEnvDiscrete(dim=80, n_invalid_actions=60)
model = MaskablePPO("MlpPolicy", env, gamma=0.4, seed=32, verbose=1)
model.learn(5_000)

evaluate_policy(model, env, n_eval_episodes=20, reward_threshold=90, warn=False)

model.save("ppo_mask")
del model # remove to demonstrate saving and loading

model = MaskablePPO.load("ppo_mask")

obs, _ = env.reset()
while True:
    # Retrieve current action mask
    action_masks = get_action_masks(env)
    action, _states = model.predict(obs, action_masks=action_masks)
    obs, reward, terminated, truncated, info = env.step(action)
```

--------------------------------

### Check Codestyle and Linting

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Commands to check code formatting with black and linting with ruff. Maintain consistent code style.

```bash
make check-codestyle
```

```bash
make lint
```

--------------------------------

### Plot CrossQ Benchmark Results

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/crossq.md

Use RL-Zoo scripts to plot the performance results of CrossQ benchmarks. The first command aggregates results, and the second plots them from a file.

```bash
python scripts/all_plots.py -a crossq -e HalfCheetah Ant Hopper Walker2D -f logs/ -o logs/crossq_results
python scripts/plot_from_file.py -i logs/crossq_results.pkl -latex -l CrossQ
```

--------------------------------

### Clone RL-Zoo Repository

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ars.md

Instructions to clone the RL-Zoo repository and checkout the specific branch for ARS experiments.

```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/
git checkout feat/ars
```

--------------------------------

### Train ARS with Multiple Seeds in Parallel

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ars.md

Shell script to train ARS with multiple seeds in parallel for specified environments. This is used for replicating benchmark results.

```bash
for ENV_ID in Swimmer-v3 HalfCheetah-v3 Hopper-v3
do
    for SEED_NUM in {1..8}
    do
        SEED=$RANDOM
        python train.py --algo ars --env $ENV_ID --eval-episodes 10 --eval-freq 10000 -n 20000000 --seed  $SEED &
        sleep 1
    done
    wait
done
```

--------------------------------

### Citing Stable-Baselines3

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/README.md

BibTeX entry for citing the Stable-Baselines3 project in academic publications. Please cite SB3 directly when referencing this work.

```bibtex
@article{stable-baselines3,
  author  = {Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann},
  title   = {Stable-Baselines3: Reliable Reinforcement Learning Implementations},
  journal = {Journal of Machine Learning Research},
  year    = {2021},
  volume  = {22},
  number  = {268},
  pages   = {1-8},
  url     = {http://jmlr.org/papers/v22/20-1364.html}
}
```

--------------------------------

### Wrap Environment with ActionMasker

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ppo_mask.md

Wrap a custom environment with `ActionMasker` to enable action masking if the environment does not natively support it or uses a different method name. The `mask_fn` should return the action mask for the current environment state.

```python
import gymnasium as gym
import numpy as np

from sb3_contrib.common.maskable.policies import MaskableActorCriticPolicy
from sb3_contrib.common.wrappers import ActionMasker
from sb3_contrib.ppo_mask import MaskablePPO


def mask_fn(env: gym.Env) -> np.ndarray:
    # Do whatever you'd like in this function to return the action mask
    # for the current env. In this example, we assume the env has a
    # helpful method we can rely on.
    return env.valid_action_mask()


env = ...  # Initialize env
env = ActionMasker(env, mask_fn)  # Wrap to enable masking

# MaskablePPO behaves the same as SB3's PPO unless the env is wrapped
# with ActionMasker. If the wrapper is detected, the masks are automatically
# retrieved and used when learning. Note that MaskablePPO does not accept
# a new action_mask_fn kwarg, as it did in an earlier draft.
model = MaskablePPO(MaskableActorCriticPolicy, env, verbose=1)
model.learn()

# Note that use of masks is manual and optional outside of learning,
# so masking can be "removed" at testing time
model.predict(observation, action_masks=valid_action_array)
```

--------------------------------

### Train MaskablePPO Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a PPO agent with invalid action masking on a toy environment. It is crucial to use `MaskableEvalCallback` and `evaluate_policy` from `sb3_contrib.common.maskable` for proper evaluation.

```python
from sb3_contrib import MaskablePPO
from sb3_contrib.common.envs import InvalidActionEnvDiscrete

env = InvalidActionEnvDiscrete(dim=80, n_invalid_actions=60)
model = MaskablePPO("MlpPolicy", env, verbose=1)
model.learn(5000)
model.save("maskable_toy_env")
```

--------------------------------

### Citing Stable Baselines3

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/index.md

BibTeX entry for citing the Stable Baselines3 project in academic publications.

```bibtex
@misc{stable-baselines3,
  author = {Raffin, Antonin and Hill, Ashley and Ernestus, Maximilian and Gleave, Adam and Kanervisto, Anssi and Dormann, Noah},
  title = {Stable Baselines3},
  year = {2019},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/DLR-RM/stable-baselines3}},
}
```

--------------------------------

### Clone RL-Zoo Repository

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/trpo.md

Clones the RL-Zoo repository, which is necessary for replicating TRPO benchmark results.

```bash
git clone https://github.com/cyprienc/rl-baselines3-zoo
cd rl-baselines3-zoo/
```

--------------------------------

### Train RecurrentPPO Agent

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/guide/examples.md

Train a PPO agent with a recurrent policy (LSTM) on the CartPole environment. When using `predict`, it's essential to pass `lstm_states` and `episode_start` to correctly manage LSTM states.

```python
import numpy as np

from sb3_contrib import RecurrentPPO

model = RecurrentPPO("MlpLstmPolicy", "CartPole-v1", verbose=1)
model.learn(5000)

vec_env = model.get_env()
obs = vec_env.reset()
# Cell and hidden state of the LSTM
lstm_states = None
num_envs = 1
# Episode start signals are used to reset the lstm states
episode_starts = np.ones((num_envs,), dtype=bool)
while True:
    action, lstm_states = model.predict(obs, state=lstm_states, episode_start=episode_starts, deterministic=True)
    # Note: vectorized environment resets automatically
    obs, rewards, dones, info = vec_env.step(action)
    episode_starts = dones
    vec_env.render("human")
```

--------------------------------

### Run Pytest Tests

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Command to execute all tests using pytest. Ensure all tests pass before submitting changes.

```bash
make pytest
```

--------------------------------

### Plot TRPO Benchmark Results

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/trpo.md

Generates plots for the TRPO benchmark results using scripts from the RL-Zoo. This command plots results from multiple environments.

```bash
python scripts/all_plots.py -a trpo -e HalfCheetah Ant Hopper Walker2d Swimmer -f logs/ -o logs/trpo_results
```

--------------------------------

### Run Mypy Type Checking

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/CONTRIBUTING.md

Command to perform static type checking using mypy. This helps catch type-related errors.

```bash
make type
```

--------------------------------

### Plot ARS Results

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ars.md

Commands to plot the results of ARS training experiments using scripts from the RL-Zoo repository.

```bash
python scripts/all_plots.py -a ars -e HalfCheetah Swimmer Hopper -f logs/ -o logs/ars_results -max 20000000
python scripts/plot_from_file.py -i logs/ars_results.pkl -l ARS
```

--------------------------------

### Train ARS with LinearPolicy

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/ars.md

Basic usage of the ARS algorithm with a LinearPolicy for a given environment. The model is trained for a specified number of timesteps and saved.

```python
from sb3_contrib import ARS

# Policy can be LinearPolicy or MlpPolicy
model = ARS("LinearPolicy", "Pendulum-v1", verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("ars_pendulum")
```

--------------------------------

### Plot TRPO Results from File

Source: https://github.com/stable-baselines-team/stable-baselines3-contrib/blob/master/docs/modules/trpo.md

Plots TRPO results specifically from a saved PKL file, with options for LaTeX formatting.

```bash
python scripts/plot_from_file.py -i logs/trpo_results.pkl -latex -l TRPO
```
Tokens:
Raw