### Quick Start Installation with uv

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/installation.md

Clone the repository and install core dependencies using uv. Ensure you have Python >=3.7.1 and uv 0.7.19+ installed.

```bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .
```

--------------------------------

### Install and Run Procgen Examples

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/examples.md

Installs Procgen dependencies and runs PPO and PPG algorithms on the starpilot environment.

```bash
uv venv

uv pip install "[procgen]"
python cleanrl/ppo_procgen.py --env-id starpilot
python cleanrl/ppg_procgen.py --env-id starpilot
```

--------------------------------

### Install and Run Atari Examples

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/examples.md

Installs Atari dependencies and runs DQN, C51, PPO, and SAC algorithms on BreakoutNoFrameskip-v4. For Linux users, it shows how to install Envpool for faster Atari training.

```bash
uv venv

uv pip install "[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/ppo_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/sac_atari.py --env-id BreakoutNoFrameskip-v4

# NEW: 3-4x side-effects free speed up with envpool's atari (only available to linux)
uv pip install "[envpool]"
python cleanrl/ppo_atari_envpool.py --env-id BreakoutNoFrameskip-v4
# Learn Pong-v5 in ~5-10 mins
# Side effects such as lower sample efficiency might occur
uv run python ppo_atari_envpool.py --clip-coef=0.2 --num-envs=16 --num-minibatches=8 --num-steps=128 --update-epochs=3
```

--------------------------------

### Install and Run PPO with uv

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Installs the project using uv and runs the PPO script with help or a specific environment ID. Ensure the project is set up for local installation.

```bash
uv pip install .
uv run python cleanrl/ppo.py --help
uv run python cleanrl/ppo.py --env-id CartPole-v1
```

--------------------------------

### Install Dependencies with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ddpg.md

Install project dependencies for Mujoco and JAX using pip. This is an alternative to using Poetry for environment setup.

```bash
pip install -r requirements/requirements-mujoco.txt
pip install -r requirements/requirements-jax.txt
```

--------------------------------

### Install and Run PPO with PettingZoo (Poetry)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Use Poetry to install dependencies and run the PPO algorithm with PettingZoo and Atari environments. This includes installing extra packages and launching the training script with specific environment IDs.

```bash
uv pip install "[pettingzoo, atari]"
uv run AutoROM --accept-license
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --help
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
uv run  cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
```

--------------------------------

### Run PPO Algorithm Locally

Source: https://github.com/vwxyzjn/cleanrl/blob/master/README.md

This command clones the CleanRL repository, installs dependencies using uv, and runs the PPO algorithm on the CartPole-v0 environment with specified seed and timesteps. It also shows how to start Tensorboard for logging.

```bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# alternatively, you could use `uv venv` and do
# `python run cleanrl/ppo.py`
uv run python cleanrl/ppo.py \
    --seed 1 \
    --env-id CartPole-v0 \
    --total-timesteps 50000

# open another terminal and enter `cd cleanrl/cleanrl`
tensorboard --logdir runs
```

--------------------------------

### Install Dependencies and Run PQN Atari with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/pqn.md

Install the necessary dependencies for PQN with Envpool support using pip and execute the Atari environment experiment. This is a standard installation method.

```bash
pip install -r requirements/requirements-envpool.txt
python cleanrl/pqn_atari_envpool.py --help
python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
```

--------------------------------

### Install Dev Dependencies

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/contribution.md

Installs development dependencies, including those for documentation, using uv.

```bash
uv pip install ".[docs]"
```

--------------------------------

### Run PPO + LSTM Atari Example

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/examples.md

Installs Atari dependencies and runs the PPO algorithm with an LSTM network on BreakoutNoFrameskip-v4.

```bash
uv venv

uv pip install "[atari]"
python cleanrl/ppo_atari_lstm.py --env-id BreakoutNoFrameskip-v4
```

--------------------------------

### Install and Run SAC for ATARI (Poetry)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/sac.md

Install the necessary ATARI dependencies using Poetry and run the SAC algorithm for ATARI environments. Examples show default usage and manual alpha configuration.

```bash
uv pip install ".[atari]"
uv run python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4
```

```bash
uv run python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4 --autotune False --alpha 0.2
```

--------------------------------

### Install CleanRL and Dependencies

Source: https://context7.com/vwxyzjn/cleanrl/llms.txt

Clone the repository and install core dependencies using uv or pip. Optional environment groups can be installed for specific functionalities like Atari, MuJoCo, or JAX backends.

```bash
git clone https://github.com/vwxyzjn/cleanrl.git && cd cleanrl
uv pip install .

# Optional environment groups
uv pip install "[atari]"
uv pip install "[mujoco]"
uv pip install "[dm_control]"
uv pip install "[procgen]"
uv pip install "[envpool]"
uv pip install "[pettingzoo]"
uv pip install "[jax]"
uv pip install "[optuna]"
uv pip install "[cloud]"

# Alternative: pip + requirements files
pip install -r requirements/requirements.txt
pip install -r requirements/requirements-atari.txt
```

--------------------------------

### Install and Run PPO-TrXL with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-trxl.md

Installs dependencies using Poetry and runs the PPO-TrXL training script. Use this for environments like Endless-MortarMayhem-v0.

```bash
cd cleanrl/ppo_trxl
poetry install
uv run python ppo_trxl.py --help
uv run python ppo_trxl.py --env-id Endless-MortarMayhem-v0
```

--------------------------------

### Install and Run PPO with PettingZoo (pip)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Use pip to install dependencies and run the PPO algorithm with PettingZoo and Atari environments. This involves installing from requirements files and launching the training script with specific environment IDs.

```bash
pip install -r requirements/requirements-pettingzoo.txt
pip install -r requirements/requirements-atari.txt
AutoROM --accept-license
python cleanrl/ppo_pettingzoo_ma_atari.py --help
python cleanrl/ppo_pettingzoo_ma_atari.py --env-id pong_v3
python cleanrl/ppo_pettingzoo_ma_atari.py --env-id surround_v2
```

--------------------------------

### Install and Run DQN Atari

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/dqn.md

Installs the necessary packages and runs the DQN Atari implementation for specified environments. Ensure you have the correct requirements installed for Atari.

```bash
uv pip install "[atari]"
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

```bash
uv pip install "[atari]"
uv run python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/dqn_atari.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Install and Run QDagger DQN Atari

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/qdagger.md

Installs necessary packages and runs the QDagger DQN Atari implementation for specified environments. Ensure you have the correct dependencies installed.

```bash
uv pip install "[atari]"
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

```bash
uv pip install "[atari]"
uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id BreakoutNoFrameskip-v4
python cleanrl/qdagger_dqn_atari_impalacnn.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Install and Run JAX DQN Models

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/zoo.md

Install JAX and DQN dependencies, then run a JAX-based DQN model in the CartPole environment. This command installs packages for both DQN and JAX, followed by running the 'dqn_jax' experiment.

```bash
uv pip install ".[dqn, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_jax --env-id CartPole-v1
```

--------------------------------

### Install and Run C51 with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/c51.md

Installs dependencies and runs the C51 agent for general environments using pip. Ensure you have the correct JAX version installed, especially for CUDA support.

```bash
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/c51_jax.py --env-id CartPole-v1
```

--------------------------------

### Install and Run C51 with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/c51.md

Installs dependencies and runs the C51 agent for general environments using Poetry. Ensure you have the correct JAX version installed, especially for CUDA support.

```bash
uv pip install ".[jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/c51_jax.py --env-id CartPole-v1
```

--------------------------------

### Install and Run JAX Atari DQN Models

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/zoo.md

Install JAX and Atari dependencies, then run a JAX-based DQN model in an Atari environment. This command installs packages for both Atari and JAX, then executes the 'dqn_atari_jax' experiment.

```bash
uv pip install ".[atari, jax]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari_jax --env-id BreakoutNoFrameskip-v4
```

--------------------------------

### Install Dependencies with pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppg.md

Installs the required dependencies from a requirements file and then executes the PPG script. Usage includes displaying help or running with a specified environment ID.

```bash
pip install -r requirements/requirements-procgen.txt
python cleanrl/ppg_procgen.py --help
python cleanrl/ppg_procgen.py --env-id starpilot
```

--------------------------------

### Install All Extras and Run Tests

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/contribution.md

Installs all optional dependencies and runs pytest tests locally.

```bash
uv run install --all-extras
uv run pytest tests/.
```

--------------------------------

### Install and Run PPO-RND with EnvPool (Poetry)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-rnd.md

Installs the necessary packages using Poetry and runs the PPO-RND script with a specified environment ID. Use this for projects managed with Poetry.

```bash
uv pip install ".[envpool]"
uv run python cleanrl/ppo_rnd_envpool.py --help
uv run python cleanrl/ppo_rnd_envpool.py --env-id MontezumaRevenge-v5
```

--------------------------------

### Install and Run PPO-TrXL with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-trxl.md

Installs requirements using pip and runs the PPO-TrXL training script. This method is suitable for general use and specific environments.

```bash
pip install -r requirements/requirements-memory_gym.txt
python cleanrl/ppo_trxl/ppo_trxl.py --help
python cleanrl/ppo_trxl/ppo_trxl.py --env-id Endless-MortarMayhem-v0
```

--------------------------------

### Install Core Dependencies via pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/installation.md

Install the core dependencies using pip and a requirements file. This is an alternative to using uv for environment management.

```bash
# core dependencies
pip install -r requirements/requirements.txt
```

--------------------------------

### Install Dependencies for Mujoco and Jax

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/td3.md

Install project dependencies for Mujoco and Jax using pip. This is an alternative to Poetry for setting up the environment.

```bash
pip install -r requirements/requirements-mujoco.txt  
pip install -r requirements/requirements-jax.txt
```

--------------------------------

### Install and Run PPO Atari Envpool with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Installs the necessary packages using poetry and runs the PPO Atari Envpool script. Use this for projects managed with poetry.

```bash
uv pip install ".[envpool]"
uv run python cleanrl/ppo_atari_envpool.py --help
uv run python cleanrl/ppo_atari_envpool.py --env-id Breakout-v5
```

--------------------------------

### Install Dependencies with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Install project dependencies, including optional environments and JAX with CUDA support, using Poetry. This is a prerequisite for running the PPO JAX scan implementation.

```bash
uv pip install " B.[envpool, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --help
uv run python cleanrl/ppo_atari_envpool_xla_jax_scan.py --env-id Breakout-v5
```

--------------------------------

### Install Dependencies with Poetry for Jax

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/td3.md

Install project dependencies including Mujoco and Jax support using Poetry. This command is for setting up the environment for the Jax-based TD3 implementation.

```bash
uv pip install ".[mujoco, jax]"
```

--------------------------------

### Install Dependencies with uv

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppg.md

Installs the necessary packages for Procgen tasks using the 'uv' package manager. This is followed by running the PPG script with a help flag or a specific environment.

```bash
uv pip install "[procgen]"
uv run python cleanrl/ppg_procgen.py --help
uv run python cleanrl/ppg_procgen.py --env-id starpilot
```

--------------------------------

### Install Dependencies and Run SAC (Poetry)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/sac.md

Installs project dependencies using Poetry and then runs the SAC continuous action script. Use the `--help` flag to see all available arguments.

```bash
poetry install
uv pip install " B.[mujoco]"
uv run python cleanrl/sac_continuous_action.py --help
```

```bash
uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4
```

```bash
uv run python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
```

--------------------------------

### Install Dependencies and Run RPO with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/rpo.md

Demonstrates how to install dependencies using Poetry and run the RPO algorithm for various Mujoco and dm_control environments. Includes specific recommendations for the `--rpo-alpha` parameter.

```bash
# mujoco v4 environments
uv pip install "'.[mujoco]'"
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
# dm_control environments
uv pip install "'.[mujoco, dm_control]'"
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
# BipedalWalker-v3 experiment (hack)
uv pip install box2d-py==2.3.5
uv run python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
```

--------------------------------

### Install and Run SAC for ATARI (pip)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/sac.md

Install the necessary ATARI dependencies using pip and run the SAC algorithm for ATARI environments. Examples show default usage and manual alpha configuration.

```bash
pip install -r requirements/requirements-atari.txt
python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4
```

```bash
python cleanrl/sac_atari.py --env-id PongNoFrameskip-v4 --autotune False --alpha 0.2
```

--------------------------------

### Run Classic Control Examples

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/examples.md

Runs DQN, PPO, and C51 algorithms on the CartPole-v1 environment.

```bash
uv venv

python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1
```

--------------------------------

### Install Dependencies for JAX QDagger

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/qdagger.md

Install the necessary packages for running the JAX version of Q-Dagger, including specific JAX versions for CUDA support. Ensure you are using a compatible JAX version for your CUDA setup.

```bash
uv pip install " BLAH "
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
```

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/contribution.md

Builds and serves the project documentation locally using mkdocs.

```bash
uv run mkdocs serve
```

--------------------------------

### Docker Buildx Setup and Login

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/cloud/submit-experiments.md

Commands to set up Docker's buildx for multi-platform builds and log in to your preferred container registry.

```bash
docker buildx create --use
docker login
```

--------------------------------

### Install and Run DDPG with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ddpg.md

Installs the project using poetry and runs the DDPG algorithm with Hopper-v4 environment. Ensure to install mujoco dependencies.

```bash
uv pip install .
uv run python cleanrl/ddpg_continuous_action.py --help
uv pip install ".[mujoco]"
uv run python cleanrl/ddpg_continuous_action.py --env-id Hopper-v4
```

--------------------------------

### Configure and Run Tuner with Weights and Biases

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/advanced/hyperparameter-tuning.md

Instantiate the Tuner to define the script, metrics, parameter search space, and Weights and Biases project. Then, call tune() to start the hyperparameter search.

```python
import optuna

from cleanrl_utils.tuner import Tuner

tuner = Tuner(
    script="cleanrl/ppo.py",
    metric="charts/episodic_return",
    metric_last_n_average_window=50,
    direction="maximize",
    aggregation_type="average",
    target_scores={
        "CartPole-v1": None,
    },
    params_fn=lambda trial: {
        "learning-rate": trial.suggest_float("learning-rate", 0.0003, 0.003, log=True),
        "num-minibatches": trial.suggest_categorical("num-minibatches", [1, 2, 4]),
        "update-epochs": trial.suggest_categorical("update-epochs", [1, 2, 4, 8]),
        "num-steps": trial.suggest_categorical("num-steps", [5, 16, 32, 64, 128]),
        "vf-coef": trial.suggest_float("vf-coef", 0, 5),
        "max-grad-norm": trial.suggest_float("max-grad-norm", 0, 5),
        "total-timesteps": 100000,
        "num-envs": 16,
    },
    pruner=optuna.pruners.MedianPruner(n_startup_trials=5),
    sampler=optuna.samplers.TPESampler(),
    wandb_kwargs={"project": "cleanrl"},
)
tuner.tune(
    num_trials=100,
    num_seeds=3,
)
```

--------------------------------

### Install and Run DDPG with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ddpg.md

Installs project dependencies using pip and runs the DDPG algorithm with Hopper-v4 environment. Ensure to install mujoco dependencies.

```bash
python cleanrl/ddpg_continuous_action.py --help
pip install -r requirements/requirements-mujoco.txt
python cleanrl/ddpg_continuous_actions.py --env-id Hopper-v4
```

--------------------------------

### Install Dependencies and Run SAC (pip)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/sac.md

Installs project dependencies using pip and then runs the SAC continuous action script. Use the `--help` flag to see all available arguments.

```bash
pip install -r requirements/requirements-mujoco.txt
python cleanrl/sac_continuous_action.py --help
```

```bash
python cleanrl/sac_continuous_action.py --env-id Mujoco-v4
```

```bash
python cleanrl/sac_continuous_action.py --env-id Hopper-v4 --autotune False --alpha 0.2 ## Without Automatic entropy coef. tuning
```

--------------------------------

### Install Python with Shared Libraries using Pyenv

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-isaacgymenvs.md

When using pyenv, ensure Python is installed with shared library support by setting the PYTHON_CONFIGURE_OPTS environment variable before installation.

```bash
env PYTHON_CONFIGURE_OPTS="--enable-shared" pyenv install 3.7.8
```

--------------------------------

### Install Dependencies and Run RPO with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/rpo.md

Provides instructions for installing RPO dependencies using pip and running the algorithm with different environment configurations. Includes specific notes on recommended `--rpo-alpha` values.

```bash
pip install -r requirements/requirements-mujoco.txt
python cleanrl/rpo_continuous_action.py --help
python cleanrl/rpo_continuous_action.py --env-id Hopper-v4
# NOTE: we recommend using --rpo-alpha 0.01 for Ant Hopper InvertedDoublePendulum Reacher Pusher
python cleanrl/rpo_continuous_action.py --env-id Ant-v4 --rpo-alpha 0.01
pip install -r requirements/requirements-dm_control.txt
python cleanrl/rpo_continuous_action.py --env-id dm_control/cartpole-balance-v0
pip install box2d-py==2.3.5
python cleanrl/rpo_continuous_action.py --env-id BipedalWalker-v3
```

--------------------------------

### Install Dependencies and Run PQN Atari with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/pqn.md

Install the required packages for PQN with Envpool support using Poetry and run the Atari environment experiment. This is the recommended method for development environments.

```bash
uv pip install "'.[envpool]'"
uv run python cleanrl/pqn_atari_envpool.py --help
uv run python cleanrl/pqn_atari_envpool.py --env-id Breakout-v5
```

--------------------------------

### Run Classic Control Training Scripts

Source: https://github.com/vwxyzjn/cleanrl/blob/master/README.md

Execute training scripts for classic control environments using DQN, PPO, and C51 algorithms. Ensure the 'uv' tool is installed and the virtual environment is activated.

```bash
uv venv

# classic control
python cleanrl/dqn.py --env-id CartPole-v1
python cleanrl/ppo.py --env-id CartPole-v1
python cleanrl/c51.py --env-id CartPole-v1
```

--------------------------------

### Install Dependencies for PPO Continuous Action (Poetry)

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Install necessary dependencies using Poetry for running PPO with continuous action spaces, including support for Mujoco and dm_control environments.

```bash
uv pip install ".[mujoco]"
```

```bash
uv pip install ".[mujoco, dm_control]"
```

--------------------------------

### Remote Buildx Setup for Multi-Arch Images

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/cloud/submit-experiments.md

Commands to set up a remote buildx instance, append a remote builder, and inspect the bootstrap status. This can speed up multi-arch image builds by leveraging a native ARM server.

```bash
docker -H ssh://costa@gpu info
docker buildx create --name remote --use
docker buildx create --name remote --append ssh://costa@gpu
docker buildx inspect --bootstrap
python -m cleanrl_utils.submit_exp -b --archs linux/arm64,linux/amd64
```

--------------------------------

### Install and Run C51 Atari with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/c51.md

Installs dependencies and runs the C51 Atari agent using pip. Ensure you have the correct JAX version installed, especially for CUDA support.

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Install and Run C51 Atari with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/c51.md

Installs dependencies and runs the C51 Atari agent using Poetry. Ensure you have the correct JAX version installed, especially for CUDA support.

```bash
uv pip install ".[atari, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/c51_atari_jax.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/c51_atari_jax.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Get Help for PPO Script

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/basic-usage.md

Use the --help flag with the PPO script to view all available command-line arguments and their descriptions. This is useful for understanding configuration options.

```bash
python cleanrl/ppo.py --help
```

```text
usage: ppo.py [-h] [--exp-name EXP_NAME] [--env-id ENV_ID]
              [--learning-rate LEARNING_RATE] [--seed SEED]
              [--total-timesteps TOTAL_TIMESTEPS]
              [--torch-deterministic [TORCH_DETERMINISTIC]] [--cuda [CUDA]]
              [--track [TRACK]] [--wandb-project-name WANDB_PROJECT_NAME]
              [--wandb-entity WANDB_ENTITY] [--capture_video [CAPTURE_VIDEO]]
              [--num-envs NUM_ENVS] [--num-steps NUM_STEPS]
              [--anneal-lr [ANNEAL_LR]] [--gae [GAE]] [--gamma GAMMA]
              [--gae-lambda GAE_LAMBDA] [--num-minibatches NUM_MINIBATCHES]
              [--update-epochs UPDATE_EPOCHS] [--norm-adv [NORM_ADV]]
              [--clip-coef CLIP_COEF] [--clip-vloss [CLIP_VLOSS]]
              [--ent-coef ENT_COEF] [--vf-coef VF_COEF]
              [--max-grad-norm MAX_GRAD_NORM] [--target-kl TARGET_KL]

optional arguments:
  -h, --help            show this help message and exit
  --exp-name EXP_NAME   the name of this experiment
  --env-id ENV_ID       the id of the environment
  --learning-rate LEARNING_RATE
                        the learning rate of the optimizer
  --seed SEED           seed of the experiment
  --total-timesteps TOTAL_TIMESTEPS
                        total timesteps of the experiments
  --torch-deterministic [TORCH_DETERMINISTIC]
                        if toggled, `torch.backends.cudnn.deterministic=False`
  --cuda [CUDA]         if toggled, cuda will be enabled by default
  --track [TRACK]       if toggled, this experiment will be tracked with Weights
                        and Biases
  --wandb-project-name WANDB_PROJECT_NAME
                        the wandb's project name
  --wandb-entity WANDB_ENTITY
                        the entity (team) of wandb's project
  --capture_video [CAPTURE_VIDEO]
                        weather to capture videos of the agent performances (check
                        out `videos` folder)
  --num-envs NUM_ENVS   the number of parallel game environments
  --num-steps NUM_STEPS
                        the number of steps to run in each environment per policy
                        rollout
  --anneal-lr [ANNEAL_LR]
                        Toggle learning rate annealing for policy and value
                        networks
  --gae [GAE]           Use GAE for advantage computation
  --gamma GAMMA         the discount factor gamma
  --gae-lambda GAE_LAMBDA
                        the lambda for the general advantage estimation
  --num-minibatches NUM_MINIBATCHES
                        the number of mini-batches
  --update-epochs UPDATE_EPOCHS
                        the K epochs to update the policy
  --norm-adv [NORM_ADV]
                        Toggles advantages normalization
  --clip-coef CLIP_COEF
                        the surrogate clipping coefficient
  --clip-vloss [CLIP_VLOSS]
                        Toggles whether or not to use a clipped loss for the value
                        function, as per the paper.
  --ent-coef ENT_COEF   coefficient of the entropy
  --vf-coef VF_COEF     coefficient of the value function
  --max-grad-norm MAX_GRAD_NORM
                        the maximum norm for the gradient clipping
  --target-kl TARGET_KL
                        the target KL divergence threshold
```

--------------------------------

### Install Optional Dependencies for CleanRL

Source: https://github.com/vwxyzjn/cleanrl/blob/master/README.md

Install specific sets of optional dependencies for different environments like Atari, MuJoCo, Procgen, and more. Ensure you have the necessary base requirements installed first.

```bash
pip install -r requirements/requirements-atari.txt
```

```bash
pip install -r requirements/requirements-mujoco.txt
```

```bash
pip install -r requirements/requirements-mujoco_py.txt
```

```bash
pip install -r requirements/requirements-procgen.txt
```

```bash
pip install -r requirements/requirements-envpool.txt
```

```bash
pip install -r requirements/requirements-pettingzoo.txt
```

```bash
pip install -r requirements/requirements-jax.txt
```

```bash
pip install -r requirements/requirements-docs.txt
```

```bash
pip install -r requirements/requirements-cloud.txt
```

```bash
pip install -r requirements/requirements-memory_gym.txt
```

--------------------------------

### Install and Run Atari DQN Models

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/zoo.md

Install Atari dependencies and run a DQN model in an Atari environment. This command installs packages for Atari environments and then runs the 'dqn_atari' experiment on 'BreakoutNoFrameskip-v4'.

```bash
uv pip install ".[atari]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn_atari --env-id BreakoutNoFrameskip-v4
```

--------------------------------

### RL Algorithm Configuration Setup

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/CleanRL_Huggingface_Integration_Demo.ipynb

This code defines a flexible setup for different Reinforcement Learning algorithms (DQN, DQN-Atari, DQN-Jax, DQN-Atari-Jax) by mapping algorithm names to their respective Q-network, environment creation function, and evaluation utilities. It also includes argument parsing and environment/experiment name configurations.

```python
import argparse
from dataclasses import dataclass

from huggingface_hub import hf_hub_download

try:
    from pip import main as pipmain
except ImportError:
    from pip._internal import main as pipmain

@dataclass
class Args:
    exp_name: str = "dqn_atari_jax"
    seed: int = 1
    hf_entity: str = "cleanrl"
    hf_repository: str = ""
    env_id: str = "BreakoutNoFrameskip-v4"


def dqn():
    import cleanrl.dqn
    import cleanrl_utils.evals.dqn_eval
    return cleanrl.dqn.QNetwork, cleanrl.dqn.make_env, cleanrl_utils.evals.dqn_eval.evaluate

def dqn_atari():
    import cleanrl.dqn_atari
    import cleanrl_utils.evals.dqn_eval
    return cleanrl.dqn_atari.QNetwork, cleanrl.dqn_atari.make_env, cleanrl_utils.evals.dqn_eval.evaluate

def dqn_jax():
    import cleanrl.dqn_jax
    import cleanrl_utils.evals.dqn_jax_eval
    return cleanrl.dqn_jax.QNetwork, cleanrl.dqn_jax.make_env, cleanrl_utils.evals.dqn_jax_eval.evaluate

def dqn_atari_jax():
    import cleanrl.dqn_atari_jax
    import cleanrl_utils.evals.dqn_jax_eval
    return cleanrl.dqn_atari_jax.QNetwork, cleanrl.dqn_atari_jax.make_env, cleanrl_utils.evals.dqn_jax_eval.evaluate

MODELS = {
    "dqn": dqn,
    "dqn_atari": dqn_atari,
    "dqn_jax": dqn_jax,
    "dqn_atari_jax": dqn_atari_jax,
}



exp_names = ["dqn", "dqn_jax", "dqn_atari_jax", "dqn_atari"]
env_idss = [
    [
        "CartPole-v1",
        "Acrobot-v1",
        "MountainCar-v0",
    ],
    [
        "CartPole-v1",
        "Acrobot-v1",
        "MountainCar-v0",
    ],
    [
        "BreakoutNoFrameskip-v4",
        "PongNoFrameskip-v4",
        "BeamRiderNoFrameskip-v4"
    ],
    [
        "BreakoutNoFrameskip-v4",
        "PongNoFrameskip-v4",
        "BeamRiderNoFrameskip-v4"
    ]
  ]

```

--------------------------------

### Run PPO with EnvPool for Atari

Source: https://context7.com/vwxyzjn/cleanrl/llms.txt

Accelerates PPO training for Atari environments by using EnvPool. Requires Linux and specific installation.

```bash
uv pip install "[envpool]"
uv run python cleanrl/ppo_atari_envpool.py \
    --env-id BreakoutNoFrameskip-v4 \
    --clip-coef 0.2 \
    --num-envs 16 \
    --num-minibatches 8 \
    --num-steps 128 \
    --update-epochs 3
```

--------------------------------

### Run PQN with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/pqn.md

Install dependencies using Poetry and run the PQN script for CartPole-v1. Use `uv run` for environment activation.

```bash
poetry install
uv run python cleanrl/pqn.py --help
```

```bash
uv run python cleanrl/pqn.py --env-id CartPole-v1
```

--------------------------------

### Benchmark Utility Help Text

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/benchmark-utility.md

Displays the available command-line arguments for the benchmark utility. Use this to understand configuration options for running experiments.

```bash
$ python -m cleanrl_utils.benchmark --help
usage: benchmark.py [-h] --env-ids [STR
                    [STR ...]] --command STR [--num-seeds INT]
                    [--start-seed INT] [--workers INT]
                    [--auto-tag | --no-auto-tag]
                    [--slurm-template-path {None}|STR]
                    [--slurm-gpus-per-task {None}|INT]
                    [--slurm-total-cpus {None}|INT]
                    [--slurm-ntasks {None}|INT]
                    [--slurm-nodes {None}|INT]

╭─ arguments ──────────────────────────────────────────────────────────────╮
│ -h, --help                                                               │
│     show this help message and exit                                      │
│ --env-ids [STR [STR ...]]                                                │
│     the ids of the environment to compare (required)                     │
│ --command STR                                                            │
│     the command to run (required)                                        │
│ --num-seeds INT                                                          │
│     the number of random seeds (default: 3)                              │
│ --start-seed INT                                                         │
│     the number of the starting seed (default: 1)                         │
│ --workers INT                                                            │
│     the number of workers to run benchmark experimenets (default: 0)     │
│ --auto-tag, --no-auto-tag                                                │
│     if toggled, the runs will be tagged with git tags, commit, and pull  │
│     request number if possible (default: True)                           │
│ --slurm-template-path {None}|STR                                         │
│     the path to the slurm template file (see docs for more details)      │
│     (default: None)                                                      │
│ --slurm-gpus-per-task {None}|INT                                         │
│     the number of gpus per task to use for slurm jobs (default: None)    │
│ --slurm-total-cpus {None}|INT                                            │
│     the number of gpus per task to use for slurm jobs (default: None)    │
│ --slurm-ntasks {None}|INT                                                │
│     the number of tasks to use for slurm jobs (default: None)            │
│ --slurm-nodes {None}|INT                                                 │
│     the number of nodes to use for slurm jobs (default: None)            │
╰──────────────────────────────────────────────────────────────────────────╯
```

--------------------------------

### Install DQN Atari Jax with Pip

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/dqn.md

Installs the required packages for DQN Atari Jax using pip, including specific JAX versions for CUDA support. Verify your CUDA version matches the JAX installation.

```bash
pip install -r requirements/requirements-atari.txt
pip install -r requirements/requirements-jax.txt
pip install --upgrade "jax[cuda]==0.3.17" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Install DQN Atari Jax with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/dqn.md

Installs the necessary packages for DQN Atari Jax using Poetry, including specific JAX versions for CUDA support. Ensure you have the correct JAX version for your CUDA installation.

```bash
uv pip install "'.[atari, jax]'"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
uv run python cleanrl/dqn_atari_jax.py --env-id BreakoutNoFrameskip-v4
uv run python cleanrl/dqn_atari_jax.py --env-id PongNoFrameskip-v4
```

--------------------------------

### Install and Run DQN Models

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/zoo.md

Install DQN dependencies and run a model in the CartPole environment. This command installs the necessary packages for DQN and then uses `uv run` to execute the `enjoy.py` script for the 'dqn' experiment on 'CartPole-v1'.

```bash
uv pip install ".[dqn]"
uv run python -m cleanrl_utils.enjoy --exp-name dqn --env-id CartPole-v1
```

--------------------------------

### Run PPO Algorithm with Isaac Gym

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-isaacgymenvs.md

Execute the PPO algorithm for continuous action spaces using Isaac Gym environments. Use the --help flag for detailed options or specify an environment like 'Ant'.

```bash
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --help
```

```bash
uv run python cleanrl/ppo_continuous_action_isaacgym/ppo_continuous_action_isaacgym.py --env-id Ant
```

--------------------------------

### Install Dependencies with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/pqn.md

Use this command to install the necessary dependencies for Atari environments using Poetry.

```bash
uv pip install ".[atari]"
```

--------------------------------

### Visualize Training Metrics with Tensorboard

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/get-started/basic-usage.md

After training, use this command to launch Tensorboard and visualize recorded metrics. Metrics are saved in the 'runs' folder.

```bash
tensorboard --logdir runs
```

--------------------------------

### Install Dependencies with Poetry

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo.md

Installs the necessary dependencies for Envpool and Jax using Poetry. Ensure you have the correct CUDA version for Jax.

```bash
uv pip install "'.[envpool, jax]"
uv pip install --upgrade "jax[cuda11_cudnn82]==0.4.8" -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html
```

--------------------------------

### Enable Experiment Tracking and Video Recording

Source: https://context7.com/vwxyzjn/cleanrl/llms.txt

Demonstrates how to enable Weights & Biases experiment tracking and record gameplay videos using CLI flags. Requires `wandb login` for authentication.

```bash
wandb login   # one-time authentication

uv run python cleanrl/ppo.py \
    --env-id CartPole-v1 \
    --track \
    --wandb-project-name my-project \
    --wandb-entity my-team \
    --capture_video   # saves MP4s to videos/<run_name>/

# W&B run URL printed to console:
# wandb: View run at https://wandb.ai/my-team/my-project/runs/abc123

# Local Tensorboard (always active, no flag needed)
tensorboard --logdir runs
```

--------------------------------

### Install PyTorch with CUDA Support

Source: https://github.com/vwxyzjn/cleanrl/blob/master/docs/rl-algorithms/ppo-isaacgymenvs.md

Install PyTorch with CUDA 11.3 support using uv. This command is essential for GPU acceleration.

```bash
uv pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu113
```