### Brax Training Example Setup

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

Imports necessary libraries for Brax training, including JAX, environment modules, and I/O utilities. Installs Brax if not already present.

```python
from datetime import datetime
import functools
import os

from IPython.display import HTML, clear_output

import jax
import jax.numpy as jnp
import matplotlib.pyplot as plt

try:
  import brax
except ImportError:
  !pip install git+https://github.com/google/brax.git@main
  clear_output()
  import brax

from brax import envs
from brax.io import html
from brax.io import model
```

--------------------------------

### Development Setup with uv

Source: https://github.com/denys88/rl_games/blob/master/README.md

Set up the development environment using uv, creating a virtual environment and installing dependencies for Atari and mujoco.

```bash
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[atari,mujoco]"
```

--------------------------------

### Train Ant Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/BRAX.md

Starts the training process for the Ant environment using a specified configuration file. This command assumes the setup is complete and Brax is installed.

```bash
poetry run python runner.py --train --file rl_games/configs/brax/ppo_ant.yaml
```

--------------------------------

### Install RL Games from Source

Source: https://github.com/denys88/rl_games/blob/master/README.md

Install the latest version of RL Games by cloning the repository and installing it from source.

```bash
pip install -e .
```

--------------------------------

### Manual Agent Instantiation with Custom Stop Callback

Source: https://github.com/denys88/rl_games/blob/master/README.md

Demonstrates how to manually instantiate an agent, assign a custom stop function to its 'stop_fn' attribute, and then start training. This method provides direct control over agent setup.

```python
import yaml
from rl_games.torch_runner import Runner

with open('rl_games/configs/atari/ppo_pong.yaml') as f:
    cfg = yaml.safe_load(f)

runner = Runner()
runner.load(cfg)
runner.load_config(runner.default_config)

agent = runner.algo_factory.create(
    runner.algo_name, base_name='run', params=runner.params
)
agent.stop_fn = lambda algo: algo.last_mean_rewards > 18.0 and algo.epoch_num >= 100
agent.train()
```

--------------------------------

### Install rl_games with uv

Source: https://github.com/denys88/rl_games/blob/master/README.md

Use uv to create a virtual environment and install rl_games with mujoco and envpool dependencies.

```bash
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[mujoco,envpool]"
```

--------------------------------

### Quick Start Atari Environments

Source: https://github.com/denys88/rl_games/blob/master/docs/ENVPOOL.md

Launch PPO training for Atari environments like Pong and Breakout using EnvPool configurations.

```bash
python runner.py --train --file rl_games/configs/atari/ppo_pong_envpool.yaml
python runner.py --train --file rl_games/configs/atari/ppo_breakout_envpool.yaml
```

--------------------------------

### Install Dependencies for RL Games and ONNX

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_discrete.ipynb

Installs necessary libraries including onnx, onnxruntime, rl_games, envpool, gym, and colabgymrender. Ensure these are installed before proceeding.

```python
!pip install onnx
!pip install onnxruntime
!pip install git+https://github.com/Denys88/rl_games
!pip install envpool
!pip install gym
!pip install -U colabgymrender
```

--------------------------------

### Quick Start MuJoCo Environments

Source: https://github.com/denys88/rl_games/blob/master/docs/ENVPOOL.md

Launch PPO training for MuJoCo environments including Ant, HalfCheetah, Hopper, Walker2d, and Humanoid using EnvPool configurations.

```bash
python runner.py --train --file rl_games/configs/mujoco/ant_envpool.yaml
python runner.py --train --file rl_games/configs/mujoco/halfcheetah_envpool.yaml
python runner.py --train --file rl_games/configs/mujoco/hopper_envpool.yaml
python runner.py --train --file rl_games/configs/mujoco/walker2d_envpool.yaml
python runner.py --train --file rl_games/configs/mujoco/humanoid_envpool.yaml
```

--------------------------------

### Install Dependencies for RL Games and ONNX

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_continuous.ipynb

Installs necessary libraries including onnx, onnxruntime, rl_games, and environment packages. Ensure these are installed before proceeding.

```python
!pip install onnx
!pip install onnxruntime
!pip install git+https://github.com/Denys88/rl_games
!pip install envpool
!pip install gym
!pip install pygame
!pip install -U colabgymrender
```

--------------------------------

### Start TensorBoard

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Launches TensorBoard to visualize training logs stored in the 'runs/' directory. This allows for monitoring metrics like rewards, losses, and episode lengths.

```python
%tensorboard --logdir 'runs/'
```

--------------------------------

### Initialize and Run RL Runner

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_discrete.ipynb

Creates an instance of the `Runner` class, loads the specified configuration, and starts the training process.

```python
runner = Runner()
runner.load(config)
runner.run({
    'train': True,
})
```

--------------------------------

### Install RL Games with Optional Extras

Source: https://github.com/denys88/rl_games/blob/master/README.md

Install RL Games with optional extras such as Atari, Mujoco, and EnvPool for enhanced functionality. Available extras include: atari, mujoco, envpool, brax, pufferlib.

```bash
pip install -e ".[atari,mujoco,envpool]"
```

--------------------------------

### Install EnvPool Extra

Source: https://github.com/denys88/rl_games/blob/master/docs/ENVPOOL.md

Install the rl_games library with the envpool extra dependency. Requires envpool version 1.2.5 or higher.

```bash
pip install -e ".[envpool]"
```

--------------------------------

### Multi-GPU Training with torchrun

Source: https://github.com/denys88/rl_games/blob/master/README.md

Orchestrate multi-GPU runs using torchrun. This example demonstrates training with two processes per node.

```bash
torchrun --standalone --nnodes=1 --nproc_per_node=2 runner.py --train --file rl_games/configs/ppo_cartpole.yaml
```

--------------------------------

### Install MuJoCo Dependencies and RL Games

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Installs necessary system libraries for MuJoCo and the RL Games package with MuJoCo support from GitHub. Redirects output to /dev/null to keep the output clean.

```bash
!apt-get install -y \n    libgl1-mesa-dev \n    libgl1-mesa-glx \n    libglew-dev \n    libosmesa6-dev \n    software-properties-common \n    patchelf > /dev/null 2>&1

!pip install "rl-games[mujoco] @ git+https://github.com/Denys88/rl_games"
```

--------------------------------

### Install RL Games with Pip

Source: https://github.com/denys88/rl_games/blob/master/README.md

Install the RL Games library using pip. For maximum training performance, PyTorch >= 2.2 with CUDA is recommended.

```bash
pip install rl-games
```

--------------------------------

### Install IsaacGymEnvs

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Installs the Isaac Gym Environments package in editable mode. This allows for immediate application of code changes.

```bash
cd /path/to/IsaacGym_Preview_4/IsaacGymEnvs
pip install -e .
```

--------------------------------

### Install MJLab Dependencies

Source: https://github.com/denys88/rl_games/blob/master/docs/MJLAB.md

Install MJLab with MuJoCo support and the MJLab package itself. Ensure you have the necessary build tools for MuJoCo.

```bash
pip install -e ".[mujoco]"
pip install mjlab
```

--------------------------------

### Train Humanoid Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/BRAX.md

Initiates training for the Humanoid environment with the provided configuration. This is used after all necessary dependencies are installed.

```bash
poetry run python runner.py --train --file rl_games/configs/brax/ppo_humanoid.yaml
```

--------------------------------

### Install IsaacGym

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Installs the Isaac Gym package in editable mode from its Python directory. This allows for immediate application of code changes.

```bash
cd /path/to/IsaacGym_Preview_4/isaacgym/python
pip install -e .
```

--------------------------------

### Install rl_games

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Installs the rl_games package in editable mode. This step ensures the correct PyTorch version and core dependencies are installed.

```bash
cd /path/to/rl_games
pip install -e .
```

--------------------------------

### Train PPO Agent

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Runs the training script for the PPO algorithm on the 'Ant' task in headless mode. This is a basic command to start training.

```bash
python train.py task=Ant headless=True
```

--------------------------------

### Install rl_games Library

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

Installs the rl_games library from its GitHub repository. Ensure you have a compatible Python version (>=3.7.1 and <3.11).

```python
!pip install git+https://github.com/Denys88/rl_games
```

--------------------------------

### Show RL-Games Package Information

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Displays information about the installed 'rl-games' package. Useful for verifying the installation and checking the version.

```bash
!pip show rl-games
```

--------------------------------

### Experiment Tracking with Weights and Biases

Source: https://github.com/denys88/rl_games/blob/master/README.md

Track experiments using Weights and Biases. Examples show basic tracking, API key usage, and custom project/entity names.

```bash
python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
```

```bash
WANDB_API_KEY=xxxx python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --track
```

```bash
python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test --track
```

```bash
python runner.py --train --file rl_games/configs/atari/ppo_breakout_torch.yaml --wandb-project-name rl-games-special-test -wandb-entity openrlbenchmark --track
```

--------------------------------

### Install Brax Dependencies

Source: https://github.com/denys88/rl_games/blob/master/docs/BRAX.md

Installs Brax and its dependencies, including specific versions of JAX and PyTorch with CUDA support. Ensure your environment meets the CUDA requirements.

```bash
poetry install -E brax
```

```bash
poetry run pip install --upgrade "jax[cuda]==0.3.13" -f https://storage.googleapis.com/jax-releases/jax_releases.html
```

```bash
poetry run pip install torch==1.10.2+cu113 torchvision==0.11.3+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html
```

--------------------------------

### Complete Custom Train.py Script with PushT Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/HOW_TO_RL_GAMES.md

A full example of a custom train.py script that integrates a new gym-like environment 'pushT' and a custom observer for logging metrics. Ensure the 'custom_envs' directory and its contents are correctly set up.

```python
import hydra

from omegaconf import DictConfig, OmegaConf
from omegaconf import DictConfig, OmegaConf


# Hydra decorator to pass in the config. Looks for a config file in the specified path. This file in turn has links to other configs 
@hydra.main(version_base="1.1", config_name="custom_config", config_path="./cfg")
def launch_rlg_hydra(cfg: DictConfig):

    import logging
    import os

    from hydra.utils import to_absolute_path
    import gym
    from isaacgymenvs.utils.reformat import omegaconf_to_dict, print_dict
    from rl_games.common import env_configurations, vecenv
    from rl_games.torch_runner import Runner


    # Naming the run
    time_str = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
    run_name = f"{cfg.run_name}_{time_str}"

    # ensure checkpoints can be specified as relative paths
    if cfg.checkpoint:
        cfg.checkpoint = to_absolute_path(cfg.checkpoint)


    # Creating a new function to return a pushT environment. This will then be added to rl_games env_configurations so that an env can be created from its name in the config
    from custom_envs.pusht_single_env import PushTEnv
    from custom_envs.customenv_utils import CustomRayVecEnv, PushTAlgoObserver

    def create_pusht_env(**kwargs):
        env =  PushTEnv()
        return env

    # env_configurations.register adds the env to the list of rl_games envs. 
    env_configurations.register('pushT', {
        'vecenv_type': 'CUSTOMRAY',
        'env_creator': lambda **kwargs: create_pusht_env(**kwargs),
    })

    # vecenv register calls the following lambda function which then returns an instance of CUSTOMRAY. 
    vecenv.register('CUSTOMRAY', lambda config_name, num_actors, **kwargs: CustomRayVecEnv(env_configurations.configurations, config_name, num_actors, **kwargs))

    # Convert to a big dictionary
    rlg_config_dict = omegaconf_to_dict(cfg.train)

    # Build an rl_games runner. You can add other algos and builders here
    def build_runner():
        runner = Runner(algo_observer=PushTAlgoObserver())
        return runner

    # create runner and set the settings
    runner = build_runner()
    runner.load(rlg_config_dict)
    runner.reset()

    # Run either training or playing via the rl_games runner
    runner.run({
        'train': not cfg.test,
        'play': cfg.test,
        # 'checkpoint': cfg.checkpoint,
        # 'sigma': cfg.sigma if cfg.sigma != '' else None
    })


if __name__ == "__main__":
    launch_rlg_hydra()

```

--------------------------------

### Initialize and Run RL Agent

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

Initializes the RL Games runner with the specified configuration and then starts the training process. The `run` method takes a dictionary specifying the mode, such as 'train'.

```python
runner = Runner()\nrunner.load(config)\nrunner.run({\n    'train': True,\n})
```

--------------------------------

### Train MiniGrid with Lava Environment and RNN

Source: https://github.com/denys88/rl_games/blob/master/docs/OTHER.md

Execute this command to train an agent on a MiniGrid environment with lava, utilizing an RNN. The configuration file path is crucial for this setup.

```bash
runner.py --train --file rl_games/configs/minigrid/lava_rnn.yaml
```

--------------------------------

### Run RL Games Training

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Initializes the RL Games Runner, loads a modified configuration for training (adjusting epochs, horizon, actors, and minibatch size), and starts the training process.

```python
import yaml
from rl_games.torch_runner import Runner

config = walker_config
config['params']['config']['full_experiment_name'] = 'Walker2d_mujoco'
config['params']['config']['max_epochs'] = 500
config['params']['config']['horizon_length'] = 512
config['params']['config']['num_actors'] = 8
config['params']['config']['minibatch_size'] = 1024

runner = Runner()
runner.load(config)
runner.run({
    'train': True,
})
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Creates a new conda environment named 'isaacgym' with Python 3.8 and activates it. This is the recommended setup to avoid dependency conflicts.

```bash
conda create -n isaacgym python=3.8 -y
conda activate isaacgym
```

--------------------------------

### Register Custom AMP Algorithm

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Demonstrates how to register a custom algorithm, AMPAgent, with the algo_factory. This is an example of extending RL Games with new training agents.

```python
_runner.algo_factory.register_builder(
    'amp_continuous',
    lambda **kwargs: amp_continuous.AMPAgent(**kwargs)
)
```

--------------------------------

### Run agent in environment and collect data

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_visualization.ipynb

Resets the environment, then enters a loop to collect observations, get actions from the agent, step the environment, and accumulate rewards and steps. A custom QP class is defined to store position and rotation.

```python
qps = []
obs = env.reset()
total_reward = 0
num_steps = 0

class QP:
    def __init__(self, qp):
        self.pos = jax.numpy.squeeze(qp.pos, axis=0)
        self.rot = jax.numpy.squeeze(qp.rot, axis=0)

is_done = False
while not is_done:
    qps.append(QP(env.env._state.qp))
    act = agent.get_action(obs)
    obs, reward, is_done, info = env.step(act.unsqueeze(0))
    total_reward += reward.item()
    num_steps += 1

print('Total Reward: ', total_reward)
print('Num steps: ', num_steps)
```

--------------------------------

### Advanced torch.compile Configuration with Dictionary

Source: https://github.com/denys88/rl_games/blob/master/docs/TORCH_COMPILE.md

Utilize a dictionary format for advanced torch.compile configuration, allowing separate modes for the actor and critic networks when using asymmetric actor-critic setups.

```yaml
torch_compile:
  mode: "reduce-overhead"
  critic_mode: "default"
```

--------------------------------

### Set Library Path and Verify Isaac Gym Import

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Sets the LD_LIBRARY_PATH environment variable to the conda environment's lib directory and then runs a Python command to import Isaac Gym and print a success message. This helps verify the installation.

```bash
export LD_LIBRARY_PATH=$HOME/anaconda3/envs/isaacgym/lib
python -c "import isaacgym; print('IsaacGym OK')"
```

--------------------------------

### Create and Run a DM Control Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/DEEPMIND_CONTROL.md

Demonstrates how to create a DeepMind Control Suite environment using Gymnasium, sample actions, and step through the environment.

```python
import gymnasium as gym

env = gym.make("dm_control/cartpole-balance-v0", render_mode="human")
observation, info = env.reset()
for _ in range(1000):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        observation, info = env.reset()
env.close()
```

--------------------------------

### Visualize Training Runs with TensorBoard

Source: https://github.com/denys88/rl_games/blob/master/docs/SMAC.md

Launch TensorBoard to visualize training metrics and results. Point it to the 'runs' directory where logs are stored.

```bash
tensorboard --logdir runs
```

--------------------------------

### Install Shimmy for DM Control Multi-Agent

Source: https://github.com/denys88/rl_games/blob/master/docs/DEEPMIND_CONTROL.md

Installs the Shimmy package with multi-agent support for DeepMind Control Suite.

```bash
pip install shimmy[dm-control-multi-agent]
```

--------------------------------

### Configure training parameters

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_visualization.ipynb

Defines configuration file names and paths to trained network weights for different Brax environments like Ant and Humanoid. Uncomment the desired configuration.

```python
# Training configs and path to the saved weights of the trained agent

# Ant
# config_name = 'rl_games/configs/brax/ppo_ant.yaml'
# trained_network = 'runs/Ant_brax/nn/Ant_brax.pth'

# Humanoid
config_name = 'rl_games/configs/brax/ppo_humanoid.yaml'
trained_network = 'runs/Humanoid_brax/nn/Humanoid_brax.pth'

# config_name = 'rl_games/configs/brax/sac_ant.yaml'
# trained_network = 'nn/Ant_brax_sac.pth'

# config_name = 'rl_games/configs/brax/sac_humanoid.yaml'
# trained_network = 'nn/humanoid_brax_sac.pth'

#config_name = 'rl_games/configs/brax/ppo_ur5e.yaml'
#trained_network = 'nn/Ur5e_brax.pth'

#config_name = 'rl_games/configs/brax/ppo_halfcheetah.yaml'
#trained_network = 'nn/Halfcheetah_brax.pth'

#config_name = 'rl_games/configs/brax/ppo_grasp.yaml'
#trained_network = 'nn/Grasp_brax.pth'

#config_name = 'rl_games/configs/brax/ppo_reacher.yaml'
#trained_network = 'nn/Reacher_brax.pth'

# config_name = 'rl_games/configs/brax/sac_humanoid.yaml'
# trained_network = './nn/humanoid_brax_sac.pth'
```

--------------------------------

### Install Shimmy for DM Control

Source: https://github.com/denys88/rl_games/blob/master/docs/DEEPMIND_CONTROL.md

Installs the necessary packages for DeepMind Control Suite integration, including dm_control and shimmy.

```bash
pip install shimmy[dm-control]
```

--------------------------------

### Train PPO Agent on ShadowHandOpenAI_FF

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Initiates training for the PPO algorithm on the 'ShadowHandOpenAI_FF' task in headless mode.

```bash
python train.py task=ShadowHandOpenAI_FF headless=True
```

--------------------------------

### Train PPO Agent (PyTorch)

Source: https://github.com/denys88/rl_games/blob/master/docs/SMAC.md

Use this command to train a PPO agent in Starcraft 2 environments with PyTorch. Ensure you have the correct configuration file.

```bash
runner.py --train --file rl_games/configs/smac/3m_torch.yaml
```

--------------------------------

### Train and Play PPO in Pong

Source: https://github.com/denys88/rl_games/blob/master/docs/HOW_TO_RL_GAMES.md

Execute training and then playing for PPO in Pong using the `runner.py` script with specified configuration files and checkpoints.

```bash
python runner.py --train --file rl_games/configs/atari/ppo_pong.yaml
```

```bash
python runner.py --play --file rl_games/configs/atari/ppo_pong.yaml --checkpoint nn/PongNoFrameskip.pth
```

--------------------------------

### Comment out Dependencies in IsaacGymEnvs setup.py

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Modifies the setup.py file in IsaacGymEnvs to comment out core dependencies like 'gym', 'torch', and 'rl-games'. This avoids version conflicts with rl_games.

```python
INSTALL_REQUIRES = [
    # Core deps managed by rl_games setup.py
    # "gym==0.23.1",
    # "torch",
    # "rl-games>=1.6.0",
    "omegaconf",
    "termcolor",
    ...
]
```

--------------------------------

### Simulate Environment Steps and Collect Data

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

Resets the environment, then enters a loop to take steps using the agent's actions. It collects joint positions and rotations (QP) for visualization, accumulates rewards, and counts steps until the episode is done.

```python
qps = []\nobs = env.reset()\ntotal_reward = 0\nnum_steps = 0\n\nclass QP:\n    def __init__(self, qp):\n        self.pos = jax.numpy.squeeze(qp.pos, axis=0)\n        self.rot = jax.numpy.squeeze(qp.rot, axis=0)\n\nis_done = False\nwhile not is_done:\n    qps.append(QP(env.env._state.qp))\n    act = agent.get_action(obs)\n    obs, reward, is_done, info = env.step(act.unsqueeze(0))\n    total_reward += reward.item()\n    num_steps += 1\n\nprint('Total Reward: ', total_reward)\nprint('Num steps: ', num_steps)
```

--------------------------------

### Downgrade NumPy Version

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Installs a NumPy version less than 1.24 to resolve compatibility issues with the 'urdfpy' package, which is a dependency of IsaacGymEnvs.

```bash
pip install "numpy<1.24"
```

--------------------------------

### Train Brax Ant

Source: https://github.com/denys88/rl_games/blob/master/README.md

Train and play the Ant task using the Brax environment. Requires installation of JAX with CUDA support and Brax.

```bash
pip install -U "jax[cuda12]"
pip install brax
python runner.py --train --file rl_games/configs/brax/ppo_ant.yaml
```

```bash
python runner.py --play --file rl_games/configs/brax/ppo_ant.yaml --checkpoint runs/Ant_brax/nn/Ant_brax.pth
```

--------------------------------

### Load Agent and Environment for Visualization

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

Loads a pre-trained agent and sets up the Brax environment for visualization. It restores the agent's weights and creates a Brax environment instance with specific configurations.

```python
from rl_games.envs.brax import BraxEnv\n\nfrom IPython.display import HTML, IFrame, display, clear_output\nimport os\n\nagent = runner.create_player()\nagent.restore(network_path)\n\nenv_config = runner.params['config']['env_config']\nnum_actors = 1\nenv = BraxEnv('', num_actors, **env_config)
```

--------------------------------

### Train with Custom Experiment Name

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Starts training for the 'Ant' task and assigns a custom name 'my_experiment_name' to the experiment. This helps in organizing training runs.

```bash
python train.py task=Ant experiment=my_experiment_name
```

--------------------------------

### Create and Run a DM Control Multi-Agent Environment

Source: https://github.com/denys88/rl_games/blob/master/docs/DEEPMIND_CONTROL.md

Initializes a multi-agent soccer environment from DeepMind Control Suite using Shimmy and PettingZoo, then steps through it with sampled actions.

```python
from shimmy import DmControlMultiAgentCompatibilityV0

env = DmControlMultiAgentCompatibilityV0(team_size=2, render_mode="human")
observations = env.reset()
while env.agents:
    actions = {agent: env.action_space(agent).sample() for agent in env.agents}
    observations, rewards, terminations, truncations, infos = env.step(actions)
env.close()
```

--------------------------------

### Play Trained PPO Agent (PyTorch)

Source: https://github.com/denys88/rl_games/blob/master/docs/SMAC.md

Use this command to play a trained PPO agent in Starcraft 2 environments with PyTorch. Specify the checkpoint path for the trained model.

```bash
runner.py --play --file rl_games/configs/smac/3m_torch.yaml --checkpoint 'nn/3m_cnn'
```

--------------------------------

### Comment out Dependencies in IsaacGym setup.py

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Modifies the setup.py file in IsaacGym to comment out conflicting dependencies, except for 'ninja'. This prevents version conflicts with rl_games.

```python
install_requires = [
    # Dependencies managed by rl_games setup.py
    # "torch>=1.8.0",
    # "torchvision>=0.9.0",
    # "numpy>=1.16.4",
    # "scipy>=1.5.0",
    # "pyyaml>=5.3.1",
    # "pillow",
    # "imageio",
    "ninja",
],
```

--------------------------------

### Enable torch.compile with Default Mode

Source: https://github.com/denys88/rl_games/blob/master/docs/TORCH_COMPILE.md

Enable torch.compile with default settings for general use. This is the recommended configuration for most scenarios.

```yaml
torch_compile: true
```

--------------------------------

### Train BipedalWalkerHardcore-v3

Source: https://github.com/denys88/rl_games/blob/master/docs/OTHER.md

Command to initiate training for the BipedalWalkerHardcore-v3 environment. The specified YAML configuration file dictates the training parameters.

```bash
runner.py --train --file rl_games/configs/ppo_walker_hardcode.yaml
```

--------------------------------

### Brax Environment Registration Error

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_training.ipynb

This error indicates a problem during the registration of the Brax environment, specifically a 'TypeError' where the constructor received multiple values for the 'env_name' argument. This often happens due to incorrect argument passing or environment setup.

```text
--> 280 register('BRAX', lambda config_name, num_actors, **kwargs: BraxEnv(config_name, num_actors, **kwargs))\n    282 from rl_games.envs.envpool import Envpool\n    283 register('ENVPOOL', lambda config_name, num_actors, **kwargs: Envpool(config_name, num_actors, **kwargs))\n\nTypeError: BraxEnv.__init__() got multiple values for argument 'env_name'
```

--------------------------------

### Import necessary libraries

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_visualization.ipynb

Imports required modules for YAML configuration, JAX, Brax I/O, rl_games, and environment interaction. Sets an environment variable for XLA preallocation.

```python
import yaml

import jax
from brax.io import html

from rl_games.torch_runner import Runner
from rl_games.envs.brax import BraxEnv

from IPython.display import HTML, IFrame, display, clear_output
import os
os.environ["XLA_PYTHON_CLIENT_PREALLOCATE"] = "false"
```

--------------------------------

### Inference and Rendering Loop with ONNX Runtime

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_discrete.ipynb

Initializes a Gym environment, resets it, and then enters a loop to perform inference using the ONNX Runtime model. The loop renders the environment, displays the frames, and updates the observation until the episode is done.

```python
is_done = False

# using regular openai gym to render
env = gym.make('CartPole-v1')
obs = env.reset()
prev_screen = env.render(mode='rgb_array')
plt.imshow(prev_screen)
total_reward = 0
num_steps = 0

while not is_done:
    outputs = ort_model.run(None, {"obs": np.expand_dims(obs, axis=0).astype(np.float32)},
    )

    action = np.argmax(outputs[0])
    obs, reward, done, info = env.step(action)
    total_reward += reward
    num_steps += 1
    is_done = done

    screen = env.render(mode='rgb_array')
    plt.imshow(screen)
    display.display(plt.gcf())    
    display.clear_output(wait=True)

print(total_reward, num_steps)
display.clear_output(wait=True)
```

--------------------------------

### Run ONNX Inference with ONNX Runtime

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_lstm_continuous.ipynb

This Python snippet demonstrates how to load an ONNX model using ONNX Runtime and perform inference. It shows how to get input names and run the model with specified inputs and retrieve the outputs.

```python
ort_model = ort.InferenceSession("pendulum.onnx")
print([o.name for o in ort_model.get_inputs()])
outputs = ort_model.run(
    None,
    {"obs": np.zeros((1, 3)).astype(np.float32), "out_state.1" : agent.states[0].cpu().numpy(), "hidden_state.1" : agent.states[1].cpu().numpy()},
)
print(outputs)
```

--------------------------------

### Enable WandB Logging

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Runs the training script for the 'Ant' task with Weights & Biases (WandB) logging activated. The 'wandb_project' argument specifies the project name in WandB.

```bash
python train.py task=Ant wandb_activate=True wandb_project=isaacgymenvs
```

--------------------------------

### Run Go1 Velocity Simulation

Source: https://github.com/denys88/rl_games/blob/master/docs/MJLAB.md

Execute the Go1 robot simulation on flat terrain using a PPO configuration. This command launches the training process.

```bash
python run_mjlab.py --config rl_games/configs/mjlab/ppo_go1_velocity.yaml
```

--------------------------------

### Export PyTorch Model to ONNX

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_lstm_continuous.ipynb

This snippet shows the complete process of exporting a trained PyTorch agent model to ONNX format. It involves creating a player, loading weights, initializing the RNN, tracing the model, and then exporting it. Ensure the 'rl_games' and 'torch' libraries are installed.

```python
agent = runner.create_player()
agent.restore('runs/pendulum_onnx/nn/pendulum.pth')
agent.init_rnn()
import rl_games.algos_torch.flatten as flatten
inputs = {
    'obs' : torch.zeros((1,) + agent.obs_shape).to(agent.device),
    'rnn_states' : agent.states,
}
with torch.no_grad():
    adapter = flatten.TracingAdapter(ModelWrapper(agent.model), inputs, allow_non_tensor=True)
    traced = torch.jit.trace(adapter, adapter.flattened_inputs, check_trace=False)
    flattened_outputs = traced(*adapter.flattened_inputs)

'''
we are using two states : ('out_state', 'hidden_state') because it is a lstm
'''
torch.onnx.export(traced, adapter.flattened_inputs, "pendulum.onnx", verbose=True, input_names=['obs', 'out_state', 'hidden_state'], output_names=['mu','log_std', 'value', 'out_state', 'hidden_state'])
```

--------------------------------

### Load Trained Agent for Playback

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Configures the player settings (disabling rendering, setting number of games) and loads the trained agent's weights from a specified path. This prepares the agent for evaluation or demonstration.

```python
config = player_walker_config
config['params']['config']['player']['render'] = False
config['params']['config']['player']['games_num'] = 2

runner.load(config)
agent = runner.create_player()
agent.restore('runs/Walker2d_mujoco/nn/Walker2d-v5.pth')
```

--------------------------------

### Train PPO Agent (TensorFlow)

Source: https://github.com/denys88/rl_games/blob/master/docs/SMAC.md

Use this command to train a PPO agent in Starcraft 2 environments with TensorFlow. The --tf flag indicates the TensorFlow backend.

```bash
runner.py --tf --train --file rl_games/configs/smac/3m_torch.yaml
```

--------------------------------

### Initialize Brax environment

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_visualization.ipynb

Creates an instance of the Brax environment using the loaded environment configuration. It specifies the number of actors to use for the environment.

```python
num_actors = 1
env = BraxEnv('', num_actors, **env_config)
```

--------------------------------

### Load configuration and trained agent

Source: https://github.com/denys88/rl_games/blob/master/notebooks/brax_visualization.ipynb

Loads the training configuration from a YAML file and initializes the RL agent. The agent's trained weights are then restored from the specified network path.

```python
with open(config_name, 'r') as stream:
    config = yaml.safe_load(stream)
    runner = Runner()
    runner.load(config)
    agent = runner.create_player()
    agent.restore(trained_network)

env_config = runner.params['config']['env_config']
```

--------------------------------

### Initialize RNN and Run RL Agent

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_lstm_continuous.ipynb

Initializes the agent's RNN, resets the environment, and runs the agent for a specified number of steps or until the episode is done. It renders the environment and updates the agent's states.

```python
agent.init_rnn()

is_done = False

env = gym.make('Pendulum-v1')
obs = env.reset()
prev_screen = env.render(mode='rgb_array')
plt.imshow(prev_screen)
total_reward = 0
num_steps = 0
out_state = agent.states[0].cpu().numpy()
hidden_state = agent.states[1].cpu().numpy()

while not is_done or num_steps < 100:
    outputs = ort_model.run(None, {"obs": np.expand_dims(obs, axis=0).astype(np.float32), "out_state.1" : out_state, "hidden_state.1" : hidden_state})
    mu = outputs[0].squeeze(1)
    sigma = np.exp(outputs[1].squeeze(1))
    action = np.random.normal(mu, sigma)
    obs, reward, done, info = env.step(action)
    total_reward += reward
    num_steps += 1
    is_done = done

    # Update out_state and hidden_state to next states   
    out_state = outputs[3]
    hidden_state = outputs[4]

    screen = env.render(mode='rgb_array')
    plt.imshow(screen)
    display.display(plt.gcf())
    display.clear_output(wait=True)

print(total_reward, num_steps)
display.clear_output(wait=True)
```

--------------------------------

### Run Inference Loop with ONNX Model

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_continuous.ipynb

Initializes a gym environment, resets it, and then enters a loop to perform inference using the ONNX model. It renders the environment, takes actions based on model outputs, and accumulates rewards until the episode is done.

```python
is_done = False

env = gym.make('Pendulum-v1')
obs = env.reset()
prev_screen = env.render(mode='rgb_array')
plt.imshow(prev_screen)
total_reward = 0
num_steps = 0

while not is_done:
    outputs = ort_model.run(None, {"obs": np.expand_dims(obs, axis=0).astype(np.float32)})
    mu = outputs[0].squeeze(1)
    sigma = np.exp(outputs[1].squeeze(1))
    action = np.random.normal(mu, sigma)
    obs, reward, done, info = env.step(action)
    total_reward += reward
    num_steps += 1
    is_done = done

    screen = env.render(mode='rgb_array')
    plt.imshow(screen)
    display.display(plt.gcf())    
    display.clear_output(wait=True)

print(total_reward, num_steps)
display.clear_output(wait=True)
```

--------------------------------

### Multi-GPU Training

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Launches a training script using torchrun for multi-GPU training with 2 processes per node. The 'multi_gpu=True' flag enables this functionality.

```bash
torchrun --standalone --nnodes=1 --nproc_per_node=2 train.py \
    multi_gpu=True task=ShadowHandOpenAI_FF headless=True
```

--------------------------------

### Play Trained PPO Agent (TensorFlow)

Source: https://github.com/denys88/rl_games/blob/master/docs/SMAC.md

Use this command to play a trained PPO agent in Starcraft 2 environments with TensorFlow. Specify the checkpoint path for the trained model.

```bash
runner.py --tf --play --file rl_games/configs/smac/3m_torch.yaml --checkpoint 'nn/3m_cnn'
```

--------------------------------

### Train PPO Agent on ShadowHandOpenAI_LSTM

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Initiates training for the PPO algorithm on the 'ShadowHandOpenAI_LSTM' task in headless mode.

```bash
python train.py task=ShadowHandOpenAI_LSTM headless=True
```

--------------------------------

### Run Training Procedure in RL Games

Source: https://github.com/denys88/rl_games/blob/master/docs/ISAAC_GYM.md

Initiates the training process for an agent. Ensure the agent and its parameters are correctly configured before calling.

```python
def run_train(self, args):
    """Run the training procedure from the algorithm passed in."""

    print('Started to train')
    agent = self.algo_factory.create(self.algo_name, base_name='run', params=self.params)
    _restore(agent, args)
    _override_sigma(agent, args)
    agent.train()
```

--------------------------------

### Record Agent's Environment Interaction

Source: https://github.com/denys88/rl_games/blob/master/notebooks/mujoco_training.ipynb

Wraps the agent's environment with a Recorder to save video of the agent's performance. The video will be saved in the './video' directory.

```python
from colabgymrender.recorder import Recorder
directory = './video'
agent.env = Recorder(agent.env, directory)
```

--------------------------------

### Train MiniGrid-MemoryS13Random-v0 with RNN

Source: https://github.com/denys88/rl_games/blob/master/docs/OTHER.md

Use this command to train an agent on the MiniGrid-MemoryS13Random-v0 environment with an RNN implementation. Ensure the configuration file is correctly specified.

```bash
runner.py --train --file rl_games/configs/minigrid/minigrid_rnn.yaml
```

--------------------------------

### Define Model Configuration

Source: https://github.com/denys88/rl_games/blob/master/notebooks/train_and_export_onnx_example_continuous.ipynb

Sets up the detailed configuration for the reinforcement learning model, including algorithm parameters, network architecture, and environment settings. This configuration is crucial for training and reproducibility.

```python
config = {'params': {'algo': {'name': 'a2c_continuous'},
  'config': {'bound_loss_type': 'regularisation',
   'bounds_loss_coef': 0.0,
   'clip_value': False,
   'critic_coef': 4,
   'e_clip': 0.2,
   'entropy_coef': 0.0,
   'env_config': {'env_name': 'Pendulum-v1', 'seed': 5},
   'env_name': 'envpool',
   'full_experiment_name' : 'pendulum_onnx',
   'save_best_after' : 20,
   'gamma': 0.99,
   'grad_norm': 1.0,
   'horizon_length': 32,
   'kl_threshold': 0.008,
   'learning_rate': '3e-4',
   'lr_schedule': 'adaptive',
   'max_epochs': 200,
   'mini_epochs': 5,
   'minibatch_size': 1024,
   'name': 'pendulum',
   'normalize_advantage': True,
   'normalize_input': True,
   'normalize_value': True,
   'num_actors': 64,
   'player': {'render': True},
   'ppo': True,
   'reward_shaper': {'scale_value': 0.1},
   'schedule_type': 'standard',
   'score_to_win': 20000,
   'tau': 0.95,
   'truncate_grads': True,
   'use_smooth_clamp': False,
   'value_bootstrap': True},
  'model': {'name': 'continuous_a2c_logstd'},
  'network': {'mlp': {'activation': 'elu',
    'initializer': {'name': 'default'},
    'units': [32, 32]},
   'name': 'actor_critic',
   'separate': False,
   'space': {'continuous': {'fixed_sigma': True,
     'mu_activation': 'None',
     'mu_init': {'name': 'default'},
     'sigma_activation': 'None',
     'sigma_init': {'name': 'const_initializer', 'val': 0}}}},
  'seed': 5}}
```