### Atari Games Environment Setup and Training

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example demonstrating how to set up Atari environments using make_atari_env and train an A2C model.

```python
from stable_baselines3.common.env_util import make_atari_env

import ale_py

env = make_atari_env(
    "BreakoutNoFrameskip-v4",
    n_envs=1,
    wrapper_kwargs=dict(terminal_on_life_loss=False)
)
```

```python
from stable_baselines3.common.env_util import make_atari_env
from stable_baselines3.common.vec_env import VecFrameStack
from stable_baselines3 import A2C

import ale_py

# There already exists an environment generator
# that will make and wrap atari environments correctly.
# Here we are also multi-worker training (n_envs=4 => 4 environments)
vec_env = make_atari_env("PongNoFrameskip-v4", n_envs=4, seed=0)
# Frame-stacking with 4 frames
vec_env = VecFrameStack(vec_env, n_stack=4)

model = A2C("CnnPolicy", vec_env, verbose=1)
model.learn(total_timesteps=25_000)

obs = vec_env.reset()
while True:
    action, _states = model.predict(obs, deterministic=False)
    obs, rewards, dones, info = vec_env.step(action)
    vec_env.render("human")
```

--------------------------------

### Install Sphinx and Theme

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/README.md

Installs Sphinx and the theme required for building the documentation.

```bash
pip install -e ".[docs]"
```

--------------------------------

### One-liner training

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/quickstart.md

This code snippet shows a concise way to train a model using a one-liner command, assuming the environment is registered in Gymnasium and the policy is registered.

```python
from stable_baselines3 import A2C

model = A2C("MlpPolicy", "CartPole-v1").learn(10_000)

```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/README.md

Installs Stable-Baselines3 along with documentation and testing dependencies. Use this for development or testing purposes.

```sh
pip install -e '.[docs,tests,extra]'
```

--------------------------------

### Hugging Face Installation

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/integrations.md

Command to install the huggingface_sb3 library.

```bash
pip install huggingface_sb3
```

--------------------------------

### Train and run A2C on CartPole

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/quickstart.md

This code snippet demonstrates how to initialize a vectorized environment, create an A2C model with an MlpPolicy, train it for a specified number of timesteps, and then use the trained model to predict actions in the environment.

```python
import gymnasium as gym

from stable_baselines3 import A2C

env = gym.make("CartPole-v1", render_mode="rgb_array")

model = A2C("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)

vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    action, _state = model.predict(obs, deterministic=True)
    obs, reward, done, info = vec_env.step(action)
    vec_env.render("human")
    # VecEnv resets automatically
    # if done:
    #   obs = vec_env.reset()

```

--------------------------------

### Learning Rate Schedule

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example demonstrating how to implement and use a linear learning rate schedule with the PPO algorithm.

```python
from typing import Callable

from stable_baselines3 import PPO


def linear_schedule(initial_value: float) -> Callable[[float], float]:
    """
    Linear learning rate schedule.

    :param initial_value: Initial learning rate.
    :return: schedule that computes
      current learning rate depending on remaining progress
    """
    def func(progress_remaining: float) -> float:
        """
        Progress will decrease from 1 (beginning) to 0.

        :param progress_remaining:
        :return: current learning rate
        """
        return progress_remaining * initial_value

    return func

# Initial learning rate of 0.001
model = PPO("MlpPolicy", "CartPole-v1", learning_rate=linear_schedule(0.001), verbose=1)
model.learn(total_timesteps=20_000)
# By default, `reset_num_timesteps` is True, in which case the learning rate schedule resets.
# progress_remaining = 1.0 - (num_timesteps / total_timesteps)
model.learn(total_timesteps=10_000, reset_num_timesteps=True)

```

--------------------------------

### Bonus: Make a GIF of a Trained Agent

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example of creating a GIF animation from a trained agent's performance using imageio.

```python
import imageio
import numpy as np

from stable_baselines3 import A2C

model = A2C("MlpPolicy", "LunarLander-v3").learn(100_000)

images = []
obs = model.env.reset()
img = model.env.render(mode="rgb_array")
for i in range(350):
    images.append(img)
    action, _ = model.predict(obs)
    obs, _, _ ,_ = model.env.step(action)
    img = model.env.render(mode="rgb_array")

imageio.mimsave("lander_a2c.gif", [np.array(img) for i, img in enumerate(images) if i%2 == 0], fps=29)
```

--------------------------------

### Dict Observations

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

This example demonstrates how to use environments with dictionary observation spaces, which is useful when observations cannot be directly concatenated. It uses the `SimpleMultiObsEnv` as an example and trains a PPO model with a `MultiInputPolicy`.

```python
from stable_baselines3 import PPO
from stable_baselines3.common.envs import SimpleMultiObsEnv


# Stable Baselines provides SimpleMultiObsEnv as an example environment with Dict observations
env = SimpleMultiObsEnv(random_start=False)

model = PPO("MultiInputPolicy", env, verbose=1)
model.learn(total_timesteps=100_000)
```

--------------------------------

### Install Stable-Baselines3 with Extra Dependencies

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Install the stable release of Stable-Baselines3 including optional dependencies like Tensorboard, OpenCV, and ale-py for Atari games. Use quotation marks around brackets if your shell requires it.

```bash
pip install 'stable-baselines3[extra]'
```

--------------------------------

### Install Stable-Baselines3 Core

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Install the core Stable-Baselines3 package without optional dependencies. This is suitable if you do not need features like Tensorboard or Atari game support.

```bash
pip install stable-baselines3
```

--------------------------------

### Example: Train a Quantile Regression DQN (QR-DQN) agent

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/sb3_contrib.md

Example of training a Quantile Regression DQN (QR-DQN) agent on the CartPole environment.

```python
from sb3_contrib import QRDQN

policy_kwargs = dict(n_quantiles=50)
model = QRDQN("MlpPolicy", "CartPole-v1", policy_kwargs=policy_kwargs, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("qrdqn_cartpole")
```

--------------------------------

### Install Bleeding-edge Version with Extras

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Install the bleeding-edge version from GitHub, including extra dependencies for testing and documentation, using pip.

```bash
pip install "stable_baselines3[extra,tests,docs] @ git+https://github.com/DLR-RM/stable-baselines3"
```

--------------------------------

### PPO Training Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/algos.md

Example of training a PPO agent with a specified total_timesteps.

```python
from stable_baselines3 import PPO

model = PPO("MlpPolicy", "CartPole-v1").learn(total_timesteps=1_000)
```

--------------------------------

### Install Bleeding-edge Version from GitHub

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Install the latest bleeding-edge version of Stable-Baselines3 directly from its GitHub repository using pip.

```bash
pip install git+https://github.com/DLR-RM/stable-baselines3
```

--------------------------------

### Advanced Saving and Loading

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example showing how to save and load a policy independently from a model, and how to save and load a replay buffer separately.

```python
from stable_baselines3 import SAC
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.sac.policies import MlpPolicy

# Create the model and the training environment
model = SAC("MlpPolicy", "Pendulum-v1", verbose=1,
            learning_rate=1e-3)

# train the model
model.learn(total_timesteps=6000)

# save the model
model.save("sac_pendulum")

# the saved model does not contain the replay buffer
loaded_model = SAC.load("sac_pendulum")
print(f"The loaded_model has {loaded_model.replay_buffer.size()} transitions in its buffer")

# now save the replay buffer too
model.save_replay_buffer("sac_replay_buffer")

```

--------------------------------

### Record a Video

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example of recording an MP4 video of a random agent's performance using VecVideoRecorder.

```python
import gymnasium as gym
from stable_baselines3.common.vec_env import VecVideoRecorder, DummyVecEnv

env_id = "CartPole-v1"
video_folder = "logs/videos/"
video_length = 100

vec_env = DummyVecEnv([lambda: gym.make(env_id, render_mode="rgb_array")])

obs = vec_env.reset()

# Record the video starting at the first step
vec_env = VecVideoRecorder(vec_env, video_folder,
                       record_video_trigger=lambda x: x == 0, video_length=video_length,
                       name_prefix=f"random-agent-{env_id}")

vec_env.reset()
for _ in range(video_length + 1):
  action = [vec_env.action_space.sample()]
  obs, _, _, _ = vec_env.step(action)
# Save the video
vec_env.close()
```

--------------------------------

### Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/modules/td3.md

This example is only to demonstrate the use of the library and its functions, and the trained agents may not solve the environments. Optimized hyperparameters can be found in RL Zoo repository.

```python
import gymnasium as gym
import numpy as np

from stable_baselines3 import TD3
from stable_baselines3.common.noise import NormalActionNoise, OrnsteinUhlenbeckActionNoise

env = gym.make("Pendulum-v1", render_mode="rgb_array")

# The noise objects for TD3
n_actions = env.action_space.shape[-1]
action_noise = NormalActionNoise(mean=np.zeros(n_actions), sigma=0.1 * np.ones(n_actions))

model = TD3("MlpPolicy", env, action_noise=action_noise, verbose=1)
model.learn(total_timesteps=10000, log_interval=10)
model.save("td3_pendulum")
vec_env = model.get_env()

del model # remove to demonstrate saving and loading

model = TD3.load("td3_pendulum")

obs = vec_env.reset()
while True:
    action, _states = model.predict(obs)
    obs, rewards, dones, info = vec_env.step(action)
    vec_env.render("human")
```

--------------------------------

### HER Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/modules/her.md

This example demonstrates how to use the HER replay buffer with an off-policy algorithm like DQN, SAC, DDPG, or TD3. It initializes the model with `HerReplayBuffer` and trains it on the `BitFlippingEnv`.

```python
from stable_baselines3 import HerReplayBuffer, DDPG, DQN, SAC, TD3
from stable_baselines3.her.goal_selection_strategy import GoalSelectionStrategy
from stable_baselines3.common.envs import BitFlippingEnv

model_class = DQN  # works also with SAC, DDPG and TD3
N_BITS = 15

env = BitFlippingEnv(n_bits=N_BITS, continuous=model_class in [DDPG, SAC, TD3], max_steps=N_BITS)

# Available strategies (cf paper): future, final, episode
goal_selection_strategy = "future" # equivalent to GoalSelectionStrategy.FUTURE

# Initialize the model
model = model_class(
    "MultiInputPolicy",
    env,
    replay_buffer_class=HerReplayBuffer,
    # Parameters for HER
    replay_buffer_kwargs=dict(
        n_sampled_goal=4,
        goal_selection_strategy=goal_selection_strategy,
    ),
    verbose=1,
)

# Train the model
model.learn(1000)

model.save("./her_bit_env")
# Because it needs access to `env.compute_reward()`
# HER must be loaded with the env
model = model_class.load("./her_bit_env", env=env)

obs, info = env.reset()
for _ in range(100):
    action, _ = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, _ = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()

```

--------------------------------

### EvalCallback Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/callbacks.md

This example shows how to use the EvalCallback to periodically evaluate the agent's performance on a separate test environment and save the best model found.

```python
import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback

# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
# Use deterministic actions for evaluation
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/",
                             log_path="./logs/", eval_freq=500,
                             deterministic=True, render=False)

model = SAC("MlpPolicy", "Pendulum-v1")
model.learn(5000, callback=eval_callback)
```

--------------------------------

### DQN Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/modules/dqn.md

This example demonstrates the use of the DQN algorithm for training an agent on the CartPole-v1 environment. It includes model saving and loading.

```python
import gymnasium as gym

from stable_baselines3 import DQN

env = gym.make("CartPole-v1", render_mode="human")

model = DQN("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10000, log_interval=4)
model.save("dqn_cartpole")

del model # remove to demonstrate saving and loading

model = DQN.load("dqn_cartpole")

obs, info = env.reset()
while True:
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        obs, info = env.reset()
```

--------------------------------

### Install Development Version with Extras

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Set up the development version of Stable-Baselines3 from a local clone of the GitHub repository. This includes support for running tests and building documentation, along with extra dependencies.

```bash
git clone https://github.com/DLR-RM/stable-baselines3 && cd stable-baselines3
pip install -e '.[docs,tests,extra]'
```

--------------------------------

### Logging Videos to TensorBoard

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/tensorboard.md

This example shows how to log video data to TensorBoard periodically. It requires the `moviepy` library to be installed.

```python
from typing import Any, Dict

import gymnasium as gym
import torch as th
import numpy as np

from stable_baselines3 import A2C
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.evaluation import evaluate_policy
from stable_baselines3.common.logger import Video


class VideoRecorderCallback(BaseCallback):
    def __init__(self, eval_env: gym.Env, render_freq: int, n_eval_episodes: int = 1, deterministic: bool = True):
        """
        Records a video of an agent's trajectory traversing ``eval_env`` and logs it to TensorBoard

        :param eval_env: A gym environment from which the trajectory is recorded
        :param render_freq: Render the agent's trajectory every eval_freq call of the callback.
        :param n_eval_episodes: Number of episodes to render
        :param deterministic: Whether to use deterministic or stochastic policy
        """
        super().__init__()
        self._eval_env = eval_env
        self._render_freq = render_freq
        self._n_eval_episodes = n_eval_episodes
        self._deterministic = deterministic

    def _on_step(self) -> bool:
        if self.n_calls % self._render_freq == 0:
            screens = []

            def grab_screens(_locals: Dict[str, Any], _globals: Dict[str, Any]) -> None: 
                """
                Renders the environment in its current state, recording the screen in the captured `screens` list

                :param _locals: A dictionary containing all local variables of the callback's scope
                :param _globals: A dictionary containing all global variables of the callback's scope
                """
                # We expect `render()` to return a uint8 array with values in [0, 255] or a float array
                # with values in [0, 1], as described in
                # https://docs.pytorch.org/docs/stable/tensorboard.html#torch.utils.tensorboard.writer.SummaryWriter.add_video
                screen = self._eval_env.render(mode="rgb_array")
                # PyTorch uses CxHxW vs HxWxC gym (and tensorflow) image convention
                screens.append(screen.transpose(2, 0, 1))

            evaluate_policy(
                self.model,
                self._eval_env,
                callback=grab_screens,
                n_eval_episodes=self._n_eval_episodes,
                deterministic=self._deterministic,
            )
            self.logger.record(
                "trajectory/video",
                Video(th.from_numpy(np.asarray([screens])), fps=40),
                exclude=("stdout", "log", "json", "csv"),
            )
        return True


model = A2C("MlpPolicy", "CartPole-v1", tensorboard_log="runs/", verbose=1)
video_recorder = VideoRecorderCallback(gym.make("CartPole-v1"), render_freq=5000)
model.learn(total_timesteps=int(5e4), callback=video_recorder)
```

--------------------------------

### Loading with system info

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/save_format.md

Example of how to load a model and print system information for debugging loading issues.

```python
model = PPO.load("ppo_saved", print_system_info=True)
```

--------------------------------

### Logging Figures to TensorBoard

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/tensorboard.md

This example demonstrates how to log a matplotlib figure to TensorBoard at regular intervals. It requires the `matplotlib` library to be installed.

```python
import numpy as np
import matplotlib.pyplot as plt

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.logger import Figure

model = SAC("MlpPolicy", "Pendulum-v1", tensorboard_log="/tmp/sac/", verbose=1)


class FigureRecorderCallback(BaseCallback):
    def __init__(self, verbose=0):
        super().__init__(verbose)

    def _on_step(self):
        # Plot values (here a random variable)
        figure = plt.figure()
        figure.add_subplot().plot(np.random.random(3))
        # Close the figure after logging it
        self.logger.record("trajectory/figure", Figure(figure, close=True), exclude=("stdout", "log", "json", "csv"))
        plt.close()
        return True


model.learn(50000, callback=FigureRecorderCallback())
```

--------------------------------

### Install plotting dependencies

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/plotting.md

To use the plotting functionality, you need to install pandas and matplotlib. Alternatively, you can install the extra dependencies for Stable Baselines3.

```bash
pip install pandas matplotlib
```

```bash
pip install 'stable-baselines3[extra]'
```

--------------------------------

### How to replicate the results?

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/modules/td3.md

Clone the rl-zoo repo and run the benchmark.

```bash
git clone https://github.com/DLR-RM/rl-baselines3-zoo
cd rl-baselines3-zoo/

python train.py --algo td3 --env $ENV_ID --eval-episodes 10 --eval-freq 10000
```

--------------------------------

### Install SB3-Contrib with pip

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/sb3_contrib.md

Install Stable-Baselines3 contrib with pip.

```bash
pip install sb3-contrib
```

--------------------------------

### Build documentation

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/CONTRIBUTING.md

Command to build the project documentation.

```bash
make doc
```

--------------------------------

### Check Environment

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/rl_tips.md

Example of checking a custom environment for potential issues.

```python
check_env(env)
```

--------------------------------

### Training with EvalCallback

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example of training an SAC model with an EvalCallback for periodic evaluation.

```python
eval_callback = EvalCallback(eval_env, best_model_save_path=eval_log_dir,
                              log_path=eval_log_dir, eval_freq=max(500 // n_training_envs, 1),
                              n_eval_episodes=5, deterministic=True,
                              render=False)

model = SAC("MlpPolicy", train_env)
model.learn(5000, callback=eval_callback)
```

--------------------------------

### Custom Logger Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/common/logger.md

This example demonstrates how to configure and set a custom logger for an A2C model.

```python
from stable_baselines3 import A2C
from stable_baselines3.common.logger import configure

tmp_path = "/tmp/sb3_log/"
# set up logger
new_logger = configure(tmp_path, ["stdout", "csv", "tensorboard"])

model = A2C("MlpPolicy", "CartPole-v1", verbose=1)
# Set new logger
model.set_logger(new_logger)
model.learn(10000)
```

--------------------------------

### Install dependencies

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/rl_zoo.md

Installs necessary system dependencies and the RL Baselines3 Zoo package.

```bash
apt-get install swig cmake ffmpeg
# full dependencies
pip install -r requirements.txt
# minimal dependencies
pip install -e .
```

--------------------------------

### Loading and Saving Replay Buffer and Policy

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Demonstrates how to save and load a replay buffer and a policy independently from the model, and how to evaluate the loaded policy.

```python
from stable_baselines3.common.evaluation import evaluate_policy

# Load the replay buffer
loaded_model.load_replay_buffer("sac_replay_buffer")

# now the loaded replay is not empty anymore
print(f"The loaded_model has {loaded_model.replay_buffer.size()} transitions in its buffer")

# Save the policy independently from the model
# Note: if you don't save the complete model with `model.save()`
# you cannot continue training afterward
policy = model.policy
policy.save("sac_policy_pendulum")

# Retrieve the environment
env = model.get_env()

# Evaluate the policy
mean_reward, std_reward = evaluate_policy(policy, env, n_eval_episodes=10, deterministic=True)

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")

# Load the policy independently from the model
saved_policy = MlpPolicy.load("sac_policy_pendulum")

# Evaluate the loaded policy
mean_reward, std_reward = evaluate_policy(saved_policy, env, n_eval_episodes=10, deterministic=True)

print(f"mean_reward={mean_reward:.2f} +/- {std_reward}")
```

--------------------------------

### Install in develop mode

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/CONTRIBUTING.md

Installs Stable-Baselines3 in develop mode, enabling documentation building and testing.

```bash
pip install -e '.[docs,tests,extra]'
```

--------------------------------

### Launching Tensorboard

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/tensorboard.md

This bash command shows how to launch Tensorboard to visualize the logged data.

```bash
tensorboard --logdir ./a2c_cartpole_tensorboard/
```

--------------------------------

### Install Stable Baselines3 contrib master version

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/sb3_contrib.md

Install the master version of Stable Baselines3 contrib.

```bash
pip install git+https://github.com/Stable-Baselines-Team/stable-baselines3-contrib
```

--------------------------------

### Building the Docs

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/README.md

Builds the HTML documentation.

```bash
make html
```

--------------------------------

### CallbackList Example

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/callbacks.md

This example shows how to chain multiple callbacks, such as CheckpointCallback and EvalCallback, using CallbackList.

```python
import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import CallbackList, CheckpointCallback, EvalCallback

checkpoint_callback = CheckpointCallback(save_freq=1000, save_path="./logs/")
# Separate evaluation env
eval_env = gym.make("Pendulum-v1")
eval_callback = EvalCallback(eval_env, best_model_save_path="./logs/best_model",
                             log_path="./logs/results", eval_freq=500)
# Create the callback list
callback = CallbackList([checkpoint_callback, eval_callback])

model = SAC("MlpPolicy", "Pendulum-v1")
# Equivalent to:
```

--------------------------------

### Evaluate Agent Performance with EvalCallback

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

This example shows how to use `EvalCallback` to periodically evaluate an agent's performance on a separate test environment during training. It allows monitoring progress and saving the best performing model.

```python
import os
import gymnasium as gym

from stable_baselines3 import SAC
from stable_baselines3.common.callbacks import EvalCallback
from stable_baselines3.common.env_util import make_vec_env

env_id = "Pendulum-v1"
n_training_envs = 1
n_eval_envs = 5

# Create log dir where evaluation results will be saved
eval_log_dir = "./eval_logs/"
os.makedirs(eval_log_dir, exist_ok=True)

# Initialize a vectorized training environment with default parameters
train_env = make_vec_env(env_id, n_envs=n_training_envs, seed=0)

# Separate evaluation env, with different parameters passed via env_kwargs
# Eval environments can be vectorized to speed up evaluation.
eval_env = make_vec_env(env_id, n_envs=n_eval_envs, seed=0,
                        env_kwargs={'g':0.7})

# Create a callback that evaluates an agent for 5 episodes every 500 training environment steps.
# When using multiple training environments, agent will be evaluated every
# eval_freq calls to train_env.step(), thus it will be evaluated every
```

--------------------------------

### Install Ruff for Codestyle Check

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/README.md

Installs the ruff linter and runs a codestyle check on the project. This ensures adherence to coding standards.

```sh
pip install ruff
make lint
```

--------------------------------

### Install RL Baselines3 Zoo

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/plotting.md

To use the recommended RL Baselines3 Zoo plotting scripts, first install the zoo.

```bash
pip install 'rl_zoo3[plots]'
```

--------------------------------

### Build Stable-Baselines3 Docker Image (CPU)

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/install.md

Build the Docker image for Stable-Baselines3 with CPU-only support using make.

```bash
make docker-cpu
```

--------------------------------

### Basic Usage: Training, Saving, Loading

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

This code snippet demonstrates the fundamental workflow of training a DQN model on the Lunar Lander environment, saving the trained agent, and then loading it for evaluation and inference.

```python
import gymnasium as gym

from stable_baselines3 import DQN
from stable_baselines3.common.evaluation import evaluate_policy


# Create environment
env = gym.make("LunarLander-v3", render_mode="rgb_array")

# Instantiate the agent
model = DQN("MlpPolicy", env, verbose=1)
# Train the agent and display a progress bar
model.learn(total_timesteps=int(2e5), progress_bar=True)
# Save the agent
model.save("dqn_lunar")
del model  # delete trained model to demonstrate loading

# Load the trained agent
# NOTE: if you have loading issue, you can pass `print_system_info=True`
# to compare the system on which the model was trained vs the current one
# model = DQN.load("dqn_lunar", env=env, print_system_info=True)
model = DQN.load("dqn_lunar", env=env)

# Evaluate the agent
# NOTE: If you use wrappers with your environment that modify rewards,
#       this will be reflected here. To evaluate with original rewards,
#       wrap the environment in a "Monitor" wrapper before other wrappers.
mean_reward, std_reward = evaluate_policy(model, model.get_env(), n_eval_episodes=10)

# Enjoy trained agent
vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, rewards, dones, info = vec_env.step(action)
    vec_env.render("human")
```

--------------------------------

### Compiling models

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example of compiling a Stable Baselines3 model using torch.compile() for potential speed improvements.

```python
import torch as th
from stable_baselines3 import PPO

env_id = "CartPole-v1"
model = PPO("MlpPolicy", env_id, verbose=1)

model.policy = th.compile(model.policy, backend="inductor")
model.learn(total_timesteps=10_000)
```

--------------------------------

### Train and Run PPO on CartPole

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/README.md

This snippet shows how to train a PPO model on the CartPole-v1 environment and then run it. It includes environment setup, model initialization, training, and inference.

```python
import gymnasium as gym

from stable_baselines3 import PPO

env = gym.make("CartPole-v1", render_mode="human")

model = PPO("MlpPolicy", env, verbose=1)
model.learn(total_timesteps=10_000)

vec_env = model.get_env()
obs = vec_env.reset()
for i in range(1000):
    action, _states = model.predict(obs, deterministic=True)
    obs, reward, done, info = vec_env.step(action)
    vec_env.render()
    # VecEnv resets automatically
    # if done:
    #   obs = env.reset()

env.close()
```

--------------------------------

### PyBullet Environment Normalization

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/guide/examples.md

Example of using VecNormalize to normalize input features and rewards for PyBullet environments.

```python
from pathlib import Path

import pybullet_envs_gymnasium

from stable_baselines3.common.vec_env import VecNormalize
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3 import PPO

# Alternatively, you can use the MuJoCo equivalent "HalfCheetah-v4"
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
# Automatically normalize the input features and reward
vec_env = VecNormalize(vec_env, norm_obs=True, norm_reward=True, clip_obs=10.0)

model = PPO("MlpPolicy", vec_env)
model.learn(total_timesteps=2000)

# Don't forget to save the VecNormalize statistics when saving the agent
log_dir = Path("/tmp/")
model.save(log_dir / "ppo_halfcheetah")
stats_path = log_dir / "vec_normalize.pkl"
vec_env.save(stats_path)

# To demonstrate loading
del model, vec_env

# Load the saved statistics
vec_env = make_vec_env("HalfCheetahBulletEnv-v0", n_envs=1)
vec_env = VecNormalize.load(stats_path, vec_env)
#  do not update them at test time
vec_env.training = False
# reward normalization is not needed at test time
vec_env.norm_reward = False

# Load the agent
model = PPO.load(log_dir / "ppo_halfcheetah", env=vec_env)
```

--------------------------------

### Building the Docs (Live Reload)

Source: https://github.com/dlr-rm/stable-baselines3/blob/master/docs/README.md

Builds the documentation and rebuilds it automatically when files change.

```bash
sphinx-autobuild . _build/html
```