### Setup Virtual Display and OpenGL

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit2/unit2.ipynb

Installs system dependencies including OpenGL and ffmpeg, and the pyvirtualdisplay library for creating a virtual screen. This is crucial for rendering environments in environments like Colab.

```bash
!sudo apt-get update
!sudo apt-get install -y python3-opengl
!apt install ffmpeg xvfb
!pip3 install pyvirtualdisplay
```

--------------------------------

### Setup Virtual Display

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit3/unit3.ipynb

Installs necessary libraries (python-opengl, xvfb, pyvirtualdisplay) and captures output to set up a virtual display environment, required for rendering in environments like Colab.

```python
%%capture
!apt install python-opengl
!apt install xvfb
!pip3 install pyvirtualdisplay
```

--------------------------------

### Install Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit8/hands-on-cleanrl.mdx

Installs necessary libraries for the exercise, including specific versions of `gym` and `imageio-ffmpeg` for compatibility.

```bash
pip install gym==0.22
pip install imageio-ffmpeg
pip install huggingface_hub
pip install gym[box2d]==0.22
```

--------------------------------

### Install Sample Factory

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part2.ipynb

Installs the Sample Factory library, a reinforcement learning framework that integrates with ViZDoom.

```python
!pip install sample-factory==2.1.1
```

--------------------------------

### Install setuptools

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part1.ipynb

Installs a specific version of setuptools. This might be required for compatibility with other packages.

```bash
!pip install setuptools==65.5.0
```

--------------------------------

### Install imitation library

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unitbonus5/train-our-robot.mdx

Install the imitation library using pip. This is a prerequisite for running the training script.

```bash
pip install imitation
```

--------------------------------

### Install Core Dependencies and Virtual Display

Source: https://context7.com/huggingface/deep-rl-class/llms.txt

Installs necessary packages for Unit 1 and sets up a virtual display for environment rendering in Colab. Ensure you have `swig` and `cmake` installed.

```bash
apt install swig cmake
pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit1/requirements-unit1.txt

sudo apt-get install -y python3-opengl ffmpeg xvfb
pip3 install pyvirtualdisplay
```

--------------------------------

### Install Sample Factory and ViZDoom

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit8/hands-on-sf.mdx

Installs the Sample Factory library and the ViZDoom package using pip. This step may take several minutes.

```bash
pip install sample-factory
pip install vizdoom
```

--------------------------------

### Install virtualenv and create environment

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/bonus-unit1/bonus_unit1.ipynb

Installs the virtualenv package and creates a new virtual environment named 'myenv'.

```bash
!pip install virtualenv
!virtualenv myenv
```

--------------------------------

### Setup Virtual Environment with Compatible Python

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unitbonus1/train.mdx

Install virtualenv and Miniconda, then create a virtual environment with Python 3.10.12, which is compatible with ML-Agents. This ensures that ML-Agents commands like 'mlagents-learn' can be found and executed.

```bash
# Install virtualenv and create a virtual environment
!pip install virtualenv
!virtualenv myenv

# Download and install Miniconda
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

# Activate Miniconda and install Python ver 3.10.12
!source /usr/local/bin/activate
!conda install -q -y --prefix /usr/local python=3.10.12 ujson  # Specify the version here
```

--------------------------------

### Install ML-Agents packages

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/bonus-unit1/bonus_unit1.ipynb

Navigates into the ml-agents repository and installs the ml-agents-envs and ml-agents packages in editable mode. This can take a few minutes.

```python
%%capture
# Go inside the repository and install the package (can take 3min)
%cd ml-agents
!pip3 install -e ./ml-agents-envs
!pip3 install -e ./ml-agents
```

--------------------------------

### Clone and Install ML-Agents

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit7/hands-on.mdx

Clone the ML-Agents repository from GitHub and install the package using pip. This involves navigating into the cloned directory and running pip install commands.

```bash
git clone https://github.com/Unity-Technologies/ml-agents
cd ml-agents
pip install -e ./ml-agents-envs
pip install -e ./ml-agents
```

--------------------------------

### Install Virtual Display Libraries

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit3/hands-on.mdx

Install necessary libraries for creating and managing a virtual display, which is required for rendering environments on headless systems.

```bash
apt install python-opengl
apt install ffmpeg
apt install xvfb
pip3 install pyvirtualdisplay
```

--------------------------------

### Install System Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit3/hands-on.mdx

Install essential system packages like swig, cmake, and ffmpeg required for certain functionalities.

```bash
apt-get install swig cmake ffmpeg
```

--------------------------------

### Install System Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit3/unit3.ipynb

Installs essential system libraries like swig, cmake, and ffmpeg, which are required for certain functionalities, including some Gymnasium environments.

```bash
!apt-get install swig cmake ffmpeg
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit4/unit4.ipynb

Installs project dependencies from a requirements file hosted on GitHub. This includes libraries like gym, gym-games, and huggingface_hub.

```bash
!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit4/requirements-unit4.txt
```

--------------------------------

### Install reinforcement learning dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part1.ipynb

Installs core reinforcement learning libraries including gym (with box2d support), imageio-ffmpeg for video processing, and huggingface_hub for model sharing.

```bash
!pip install gym==0.22
!pip install imageio-ffmpeg
!pip install huggingface_hub
!pip install gym[box2d]==0.22
```

--------------------------------

### Install Shimmy for Environment Compatibility

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Install the `shimmy` library to ensure compatibility when loading models trained with older Gym versions into Gymnasium environments.

```bash
!pip install shimmy
```

--------------------------------

### Install environment rendering dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part1.ipynb

Installs necessary libraries for rendering environments and creating virtual displays, including python-opengl, ffmpeg, xvfb, swig, cmake, pyglet, and pyvirtualdisplay.

```bash
%%capture
!apt install python-opengl
!apt install ffmpeg
!apt install xvfb
!apt install swig cmake
!pip install pyglet==1.5
!pip3 install pyvirtualdisplay
```

--------------------------------

### Start virtual display

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Initializes and starts a virtual display environment using pyvirtualdisplay. This is crucial for rendering environments when a physical display is not available, such as in Colab.

```python
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()
```

--------------------------------

### Initialize and Start Virtual Display

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit3/unit3.ipynb

Initializes and starts a virtual display with specified dimensions. This is crucial for environments that require a graphical interface for rendering, especially in headless environments.

```python
# Virtual display
from pyvirtualdisplay import Display

virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()
```

--------------------------------

### Install Libraries for Virtual Display

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit4/unit4.ipynb

Installs necessary libraries including python-opengl, ffmpeg, xvfb, pyvirtualdisplay, and a specific version of pyglet for creating a virtual screen to render environments.

```bash
%%capture
!apt install python-opengl
!apt install ffmpeg
!apt install xvfb
!pip install pyvirtualdisplay
!pip install pyglet==1.5.1
```

--------------------------------

### Start Virtual Display for Rendering

Source: https://context7.com/huggingface/deep-rl-class/llms.txt

Initializes and starts a virtual display environment, which is required for rendering environments when running in environments like Google Colab.

```python
from pyvirtualdisplay import Display
virtual_display = Display(visible=0, size=(1400, 900))
virtual_display.start()
```

--------------------------------

### Install virtual screen dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Installs necessary libraries for creating a virtual display, essential for rendering environments in environments like Colab. This includes Python OpenGL, ffmpeg, and pyvirtualdisplay.

```bash
!sudo apt-get update
!sudo apt-get install -y python3-opengl
!apt install ffmpeg
!apt install xvfb
!pip3 install pyvirtualdisplay
```

--------------------------------

### Activate Miniconda and install Python

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/bonus-unit1/bonus_unit1.ipynb

Activates the Miniconda environment and installs a specific Python version (3.10.12) along with the ujson library. Ensure the path to activate is correct for your setup.

```bash
!source /usr/local/bin/activate
!conda install -q -y --prefix /usr/local python=3.10.12 ujson  # Specify the version here
```

--------------------------------

### Train PPO on ViZDoom with Sample Factory

Source: https://context7.com/huggingface/deep-rl-class/llms.txt

Command-line example for training a PPO agent (APPO variant) on the ViZDoom environment using Sample Factory. This configuration is optimized for high throughput.

```bash
# --- PPO with Sample Factory on ViZDoom (high-throughput) ---
# pip install sample-factory[vizdoom]
# python -m sf_examples.vizdoom.train_vizdoom \
#   --env=doom_basic \
#   --train_for_env_steps=4000000 \
#   --algo=APPO \
#   --use_rnn=True \
#   --num_workers=8 \
#   --num_envs_per_worker=4 \
#   --experiment=doom_basic_example
```

--------------------------------

### Setup Virtual Environment with Conda

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit5/unit5.ipynb

Installs virtualenv and Miniconda, then creates a virtual environment with a specific Python version (3.10.12) compatible with ML-Agents. Sets necessary environment variables.

```python
# Install virtualenv and create a virtual environment
!pip install virtualenv
!virtualenv myenv

# Download and install Miniconda
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local

# Activate Miniconda and install Python ver 3.10.12
!source /usr/local/bin/activate
!conda install -q -y --prefix /usr/local python=3.10.12 ujson  # Specify the version here

# Set environment variables for Python and conda paths
!export PYTHONPATH=/usr/local/lib/python3.10/site-packages/
!export CONDA_PREFIX=/usr/local/envs/myenv
```

--------------------------------

### Environment and Agent Setup for PPO

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part1.ipynb

Initializes the environment, agent, and optimizer. Ensures discrete action space and sets up data storage for observations, actions, log probabilities, rewards, dones, and values.

```python
torch.backends.cudnn.deterministic = args.torch_deterministic

device = torch.device("cuda" if torch.cuda.is_available() and args.cuda else "cpu")

# env setup
envs = gym.vector.SyncVectorEnv(
    [make_env(args.env_id, args.seed + i, i, args.capture_video, run_name) for i in range(args.num_envs)]
)
assert isinstance(envs.single_action_space, gym.spaces.Discrete), "only discrete action space is supported"

agent = Agent(envs).to(device)
optimizer = optim.Adam(agent.parameters(), lr=args.learning_rate, eps=1e-5)

# ALGO Logic: Storage setup
obs = torch.zeros((args.num_steps, args.num_envs) + envs.single_observation_space.shape).to(device)
actions = torch.zeros((args.num_steps, args.num_envs) + envs.single_action_space.shape).to(device)
logprobs = torch.zeros((args.num_steps, args.num_envs)).to(device)
rewards = torch.zeros((args.num_steps, args.num_envs)).to(device)
dones = torch.zeros((args.num_steps, args.num_envs)).to(device)
values = torch.zeros((args.num_steps, args.num_envs)).to(device)
```

--------------------------------

### Install RL-Baselines3 Zoo

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit3/hands-on.mdx

Install the RL-Baselines3 Zoo library using pip. This command installs the latest version from GitHub.

```python
# For now we install this update of RL-Baselines3 Zoo
pip install git+https://github.com/DLR-RM/rl-baselines3-zoo
```

--------------------------------

### Install grpcio for Mac (Apple Silicon)

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit7/hands-on.mdx

If you are on a Mac with Apple Silicon and encounter issues with gRPC installation, try installing it using conda.

```bash
conda install grpcio
```

--------------------------------

### Initialize Policy and Device for Training

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit4/unit4.ipynb

Prepare for training by creating the policy network instance and moving it to the appropriate computation device (e.g., GPU).

```python
# Create policy and place it to the device

```

--------------------------------

### Launch Training for SoccerTwos Environment (Windows)

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit7/hands-on.mdx

Use this command to start training your agent on Windows. Ensure the paths to the configuration file and the environment executable are correct.

```bash
mlagents-learn ./config/poca/SoccerTwos.yaml --env=./training-envs-executables/SoccerTwos.exe --run-id="SoccerTwos" --no-graphics
```

--------------------------------

### Install RL-Baselines3 Zoo

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit3/unit3.ipynb

Installs the rl-baselines3-zoo library from GitHub. If you see a dependency resolver error, it's normal due to version conflicts but the required packages are installed.

```python
!pip install git+https://github.com/DLR-RM/rl-baselines3-zoo
```

--------------------------------

### Create and Inspect a Gym Environment

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit1/hands-on.mdx

Demonstrates how to create a Gym environment, reset it, and inspect its observation space. Use this to understand the input your agent will receive.

```python
import gymnasium as gym

env = gym.make("LunarLander-v2")
env.reset()
print("_____OBSERVATION SPACE_____ \n")
print("Observation Space Shape", env.observation_space.shape)
print("Sample observation", env.observation_space.sample())  # Get a random observation
```

--------------------------------

### Install ML-Agents Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unitbonus1/train.mdx

Navigate to the ml-agents repository and install the ml-agents-envs and ml-agents packages using pip.

```bash
# Go inside the repository and install the package (can take 3min)
%cd ml-agents
pip3 install -e ./ml-agents-envs
pip3 install -e ./ml-agents
```

--------------------------------

### Initialize WandB and SummaryWriter

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit8/hands-on-cleanrl.mdx

Initializes Weights & Biases for experiment tracking and TensorBoard for logging, based on parsed arguments.

```python
if __name__ == "__main__":
    args = parse_args()
    run_name = f"{args.env_id}__{args.exp_name}__{args.seed}__{int(time.time())}"
    if args.track:
        import wandb

        wandb.init(
            project=args.wandb_project_name,
            entity=args.wandb_entity,
            sync_tensorboard=True,
            config=vars(args),
            name=run_name,
            monitor_gym=True,
            save_code=True,
        )
    writer = SummaryWriter(f"runs/{run_name}")
    writer.add_text(
        "hyperparameters",

```

--------------------------------

### Launch Training for SoccerTwos Environment (Mac)

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit7/hands-on.mdx

Use this command to start training your agent on Mac. Ensure the paths to the configuration file and the environment executable are correct.

```bash
mlagents-learn ./config/poca/SoccerTwos.yaml --env=./training-envs-executables/SoccerTwos/SoccerTwos.app --run-id="SoccerTwos" --no-graphics
```

--------------------------------

### Install System Dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit1/hands-on.mdx

Installs system-level dependencies required for certain environments and tools. This includes swig and cmake.

```bash
apt install swig cmake
```

--------------------------------

### Install Python Libraries for ViZDoom

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part2.ipynb

Installs the `faster-fifo` and `vizdoom` Python packages. `faster-fifo` is a dependency for efficient communication.

```python
# install python libraries
# thanks toinsson
!pip install faster-fifo==1.4.2
!pip install vizdoom
```

--------------------------------

### Install Unit 2 Requirements

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit2/unit2.ipynb

Installs all necessary Python packages for Unit 2 from a requirements file hosted on GitHub.

```python
!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit2/requirements-unit2.txt
```

--------------------------------

### Configure and Run Doom Deathmatch Agent

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit8/hands-on-sf.mdx

Sets up and runs the Doom Deathmatch agent for evaluation. This configuration saves a video of the agent playing a single episode and requires registering the ViZDoom components.

```python
from sample_factory.enjoy import enjoy

register_vizdoom_components()
env = "doom_deathmatch_bots"
cfg = parse_vizdoom_cfg(
    argv=[
        f"--env={env}",
        "--num_workers=1",
        "--save_video",
        "--no_render",
        "--max_num_episodes=1",
        "--experiment=doom_deathmatch_bots_2222",
        "--train_dir=train_dir",
    ],
    evaluation=True,
)
status = enjoy(cfg)
mp4 = open("/content/train_dir/doom_deathmatch_bots_2222/replay.mp4", "rb").read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML(
    """
<video width=640 controls>
      <source src="%s" type="video/mp4">
</video>
"""
    % data_url
)
```

--------------------------------

### Configure and Instantiate PPO Agent

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Instantiates a PPO agent with specific hyperparameters for policy, environment, steps, batch size, epochs, gamma, GAE lambda, and entropy coefficient. Use this for fine-tuned training.

```python
from stable_baselines3 import PPO

# SOLUTION
# We added some parameters to accelerate the training
model = PPO(
    policy = 'MlpPolicy',
    env = env,
    n_steps = 1024,
    batch_size = 64,
    n_epochs = 4,
    gamma = 0.999,
    gae_lambda = 0.98,
    ent_coef = 0.01,
    verbose=1)

```

--------------------------------

### Register ViZDoom Environments and Models in Sample Factory

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit8/unit8_part2.ipynb

Sets up Sample Factory to recognize and utilize ViZDoom environments and custom neural network architectures. This involves registering environment creators and model encoders.

```python
import functools

from sample_factory.algo.utils.context import global_model_factory
from sample_factory.cfg.arguments import parse_full_cfg, parse_sf_args
from sample_factory.envs.env_utils import register_env
from sample_factory.train import run_rl

from sf_examples.vizdoom.doom.doom_model import make_vizdoom_encoder
from sf_examples.vizdoom.doom.doom_params import add_doom_env_args, doom_override_defaults
from sf_examples.vizdoom.doom.doom_utils import DOOM_ENVS, make_doom_env_from_spec


# Registers all the ViZDoom environments
def register_vizdoom_envs():
    for env_spec in DOOM_ENVS:
        make_env_func = functools.partial(make_doom_env_from_spec, env_spec)
        register_env(env_spec.name, make_env_func)

# Sample Factory allows the registration of a custom Neural Network architecture
# See https://github.com/alex-petrenko/sample-factory/blob/master/sf_examples/vizdoom/doom/doom_model.py for more details
def register_vizdoom_models():
    global_model_factory().register_encoder_factory(make_vizdoom_encoder)


def register_vizdoom_components():
    register_vizdoom_envs()
    register_vizdoom_models()

# parse the command line args and create a config
def parse_vizdoom_cfg(argv=None, evaluation=False):
    parser, _ = parse_sf_args(argv=argv, evaluation=evaluation)
    # parameters specific to Doom envs
    add_doom_env_args(parser)
    # override Doom default values for algo parameters
    doom_override_defaults(parser)
    # second parsing pass yields the final configuration
    final_cfg = parse_full_cfg(parser, argv)
    return final_cfg
```

--------------------------------

### Download and install Miniconda

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/bonus-unit1/bonus_unit1.ipynb

Downloads, makes executable, and installs Miniconda for Linux. This is a prerequisite for managing Python environments and packages.

```bash
!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
!chmod +x Miniconda3-latest-Linux-x86_64.sh
!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/local
```

--------------------------------

### Install Deep RL Class Requirements

Source: https://github.com/huggingface/deep-rl-class/blob/main/units/en/unit2/hands-on.mdx

Installs the necessary Python packages for the Deep RL class, including gymnasium, pygame, and numpy.

```bash
pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit2/requirements-unit2.txt
```

--------------------------------

### Initialize Q-Table for Q-Learning

Source: https://context7.com/huggingface/deep-rl-class/llms.txt

Initializes a Q-table with zeros for a given state and action space. This is a fundamental step before training a Q-Learning agent.

```python
import numpy as np
import gymnasium as gym

env = gym.make("FrozenLake-v1", map_name="4x4", is_slippery=False, render_mode="rgb_array")
state_space  = env.observation_space.n   # 16
action_space = env.action_space.n        # 4 (LEFT, DOWN, RIGHT, UP)

def initialize_q_table(state_space, action_space):
    return np.zeros((state_space, action_space))

Qtable = initialize_q_table(state_space, action_space)
```

--------------------------------

### Initialize Q-Learning Training Function

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit2/unit2.ipynb

Initializes the Q-Learning training function with parameters for episodes, epsilon decay, environment, max steps, and the Q-table.

```python
def train(n_training_episodes, min_epsilon, max_epsilon, decay_rate, env, max_steps, Qtable):
  for episode in tqdm(range(n_training_episodes)):
    # Reduce epsilon (because we need less and less exploration)
    epsilon = min_epsilon + (max_epsilon - min_epsilon)*np.exp(-decay_rate*episode)
    # Reset the environment
    state, info = env.reset()
    step = 0
    terminated = False
    truncated = False

    # repeat
    for step in range(max_steps):
      # Choose the action At using epsilon greedy policy
      action =

      # Take action At and observe Rt+1 and St+1
      # Take the action (a) and observe the outcome state(s') and reward (r)
      new_state, reward, terminated, truncated, info =

      # Update Q(s,a):= Q(s,a) + lr [R(s,a) + gamma * max Q(s',a') - Q(s,a)]
      Qtable[state][action] =

      # If terminated or truncated finish the episode
      if terminated or truncated:
        break

      # Our next state is the new state
      state = new_state
  return Qtable
```

--------------------------------

### Install Python dependencies from requirements file

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Installs Python packages listed in a requirements file. Ensure the file path is correct for your environment.

```bash
!pip install -r https://raw.githubusercontent.com/huggingface/deep-rl-class/main/notebooks/unit1/requirements-unit1.txt
```

--------------------------------

### Install apt dependencies

Source: https://github.com/huggingface/deep-rl-class/blob/main/notebooks/unit1/unit1.ipynb

Installs system-level dependencies required for certain environments or tools. Use when needing packages like swig or cmake.

```bash
!apt install swig cmake
```