### RL Coach Command Line Usage

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Demonstrates how to use the RL Coach command-line interface to run experiments, list available presets, and access help. It shows examples of running a specific preset, listing all presets, and specifying custom preset paths.

```bash
coach -p CartPole_DQN
coach -l
coach -p /home/my_user/my_agent_dir/my_preset.py:graph_manager
coach -p Atari_DQN -lvl breakout
coach -h
```

--------------------------------

### Training a Preset with CoachInterface

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Demonstrates how to instantiate and run a reinforcement learning experiment using the CoachInterface in Python. This example shows how to specify a preset, override parameters, set worker count, and configure checkpoint saving.

```python
coach = CoachInterface(preset='CartPole_ClippedPPO',
                       # The optional custom_parameter enables overriding preset settings
                       custom_parameter='heatup_steps=EnvironmentSteps(5);improve_steps=TrainingSteps(3)',
                       # Other optional parameters enable easy access to advanced functionalities
                       num_workers=1, checkpoint_save_secs=10)
coach.run()
```

--------------------------------

### Setup Virtual Environment and Install Coach

Source: https://github.com/intellabs/coach/blob/master/README.md

Guides users through creating a Python virtual environment and installing the Coach library using pip. It also covers installing from a cloned repository for development.

```bash
sudo -E pip3 install virtualenv
virtualenv -p python3 coach_env
. coach_env/bin/activate
pip3 install rl_coach
```

```bash
cd coach
pip3 install -e .
```

--------------------------------

### RL Coach Python Library Setup

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Initializes the RL Coach environment by adding necessary module paths to sys.path and importing the CoachInterface. This setup is crucial for using Coach as a Python library.

```python
import os
import sys
import tensorflow as tf
module_path = os.path.abspath(os.path.join('..'))
resources_path = os.path.abspath(os.path.join('Resources'))
if module_path not in sys.path:
    sys.path.append(module_path)
if resources_path not in sys.path:
    sys.path.append(resources_path)
    
from rl_coach.coach import CoachInterface
```

--------------------------------

### Manual Inference with CoachInterface and GymEnvironment

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Illustrates how to perform inference manually using the CoachInterface and a GymEnvironment. It shows how to reset the environment, choose actions using the agent, and step through the environment to get responses.

```python
# inference
env_params = GymVectorEnvironment(level='CartPole-v0')
env = GymEnvironment(**env_params.__dict__, visualization_parameters=VisualizationParameters())

response = env.reset_internal_state()
for _ in range(10):
    action_info = coach.graph_manager.get_agent().choose_action(response.next_state)
    print("State:{}, Action:{}".format(response.next_state,action_info.action))
    response = env.step(action_info.action)
    print("Reward:{}".format(response.reward))
```

--------------------------------

### Basic Coach CLI Execution

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/usage.rst.txt

This example demonstrates a basic command to run a Coach experiment. It specifies the preset ('Atari_DQN'), the level ('breakout'), and enables rendering.

```bash
coach -p Atari_DQN -lvl breakout -r
```

--------------------------------

### Basic Coach CLI Execution

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/usage.rst

This example demonstrates a basic command to run a Coach experiment. It specifies the preset ('Atari_DQN'), the level ('breakout'), and enables rendering.

```bash
coach -p Atari_DQN -lvl breakout -r
```

--------------------------------

### Mujoco Dockerfile Setup

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/dist_usage.rst.txt

This snippet shows the Dockerfile instructions for setting up the Mujoco environment. It includes installing Mujoco, setting environment variables, and installing necessary Python packages.

```docker
FROM coach-base:master as builder

ARG MUJOCO_KEY
ENV MUJOCO_KEY=$MUJOCO_KEY
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
RUN echo $MUJOCO_KEY | base64 --decode > /root/.mujoco/mjkey.txt
RUN pip3 install mujoco_py

# add coach source starting with files that could trigger
# re-build if dependencies change.
RUN mkdir /root/src
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
RUN pip3 install -r /root/src/requirements.txt

FROM coach-base:master
WORKDIR /root/src
COPY --from=builder /root/.mujoco /root/.mujoco
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
COPY --from=builder /root/.cache /root/.cache
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
COPY README.md /root/src/.
RUN pip3 install mujoco_py && pip3 install -e .[all] && rm -rf /root/.cache
COPY . /root/src
```

--------------------------------

### CARLA Environment Setup

Source: https://github.com/intellabs/coach/blob/master/README.md

Instructions for setting up the CARLA environment, including installing the Python client and dependencies, and setting the CARLA_ROOT environment variable. It also mentions the inclusion of a CarlaSettings.ini file.

```bash
pip3 install -r PythonClient/requirements.txt
pip3 install PythonClient
```

--------------------------------

### Training Custom Gym Environment with GraphManager

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Demonstrates training an agent with a custom Gym environment using the GraphManager directly. This approach bypasses presets and allows for custom environment configurations, including additional simulator parameters.

```python
from rl_coach.agents.clipped_ppo_agent import ClippedPPOAgentParameters
from rl_coach.environments.gym_environment import GymVectorEnvironment
from rl_coach.graph_managers.basic_rl_graph_manager import BasicRLGraphManager
from rl_coach.graph_managers.graph_manager import SimpleSchedule
from rl_coach.architectures.embedder_parameters import InputEmbedderParameters

# Resetting tensorflow graph as the network has changed.
tf.reset_default_graph()

# define the environment parameters
bit_length = 10
env_params = GymVectorEnvironment(level='rl_coach.environments.toy_problems.bit_flip:BitFlip')
env_params.additional_simulator_parameters = {'bit_length': bit_length, 'mean_zero': True}

# Clipped PPO
agent_params = ClippedPPOAgentParameters()
agent_params.network_wrappers['main'].input_embedders_parameters = {
    'state': InputEmbedderParameters(scheme=[]),
    'desired_goal': InputEmbedderParameters(scheme=[])
}

graph_manager = BasicRLGraphManager(
    agent_params=agent_params,
    env_params=env_params,
    schedule_params=SimpleSchedule()
)
```

```python
graph_manager.improve()
```

--------------------------------

### Mujoco Dockerfile Setup

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/dist_usage.rst

This snippet shows the Dockerfile instructions for setting up the Mujoco environment. It includes installing Mujoco, setting environment variables, and installing necessary Python packages.

```docker
FROM coach-base:master as builder

ARG MUJOCO_KEY
ENV MUJOCO_KEY=$MUJOCO_KEY
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
RUN echo $MUJOCO_KEY | base64 --decode > /root/.mujoco/mjkey.txt
RUN pip3 install mujoco_py

# add coach source starting with files that could trigger
# re-build if dependencies change.
RUN mkdir /root/src
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
RUN pip3 install -r /root/src/requirements.txt

FROM coach-base:master
WORKDIR /root/src
COPY --from=builder /root/.mujoco /root/.mujoco
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
COPY --from=builder /root/.cache /root/.cache
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
COPY README.md /root/src/.
RUN pip3 install mujoco_py && pip3 install -e .[all] && rm -rf /root/.cache
COPY . /root/src
```

--------------------------------

### Manual Training Iteration with CoachInterface

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Demonstrates how to manually control training iterations using the CoachInterface. It shows how to access the graph manager, log signals, perform heatup, and execute training steps with action selection and reward retrieval.

```python
from rl_coach.environments.gym_environment import GymEnvironment, GymVectorEnvironment
from rl_coach.base_parameters import VisualizationParameters
from rl_coach.core_types import EnvironmentSteps

tf.reset_default_graph()
coach = CoachInterface(preset='CartPole_ClippedPPO')

# registering an iteration signal before starting to run
coach.graph_manager.log_signal('iteration', -1)

coach.graph_manager.heatup(EnvironmentSteps(100))

# training
for it in range(10):
    # logging the iteration signal during training
    coach.graph_manager.log_signal('iteration', it)
    # using the graph manager to train and act a given number of steps
    coach.graph_manager.train_and_act(EnvironmentSteps(100))
    # reading signals during training
    training_reward = coach.graph_manager.get_signal_value('Training Reward')
```

--------------------------------

### ViZDoom Dockerfile Setup

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/dist_usage.rst.txt

This snippet outlines the Dockerfile configuration for the ViZDoom environment. It details the installation of ViZDoom and related dependencies.

```docker
FROM coach-base:master as builder

# prep vizdoom and any of its related requirements.
RUN pip3 install vizdoom

# add coach source starting with files that could trigger
# re-build if dependencies change.
RUN mkdir /root/src
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
RUN pip3 install -r /root/src/requirements.txt

FROM coach-base:master
WORKDIR /root/src
COPY --from=builder /root/.cache /root/.cache
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
COPY README.md /root/src/.
RUN pip3 install vizdoom && pip3 install -e .[all] && rm -rf /root/.cache
COPY . /root/src
```

--------------------------------

### Switching Deep Learning Frameworks

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/usage.rst.txt

This example demonstrates how to switch the deep learning backend framework from the default TensorFlow to MXNet using the '-f' flag. This requires MXNet to be installed.

```bash
coach -p Doom_Basic_DQN -f mxnet
```

--------------------------------

### Install Gym Dependencies

Source: https://github.com/intellabs/coach/blob/master/README.md

Installs additional dependencies required for OpenAI Gym, including multimedia libraries and build tools.

```bash
sudo -E apt-get install libav-tools libsdl2-dev swig cmake -y
```

--------------------------------

### Coach Installation Prerequisites

Source: https://github.com/intellabs/coach/blob/master/README.md

Provides the necessary commands to install the prerequisites for running Coach on Ubuntu. This includes general development tools, Boost libraries, SciPy requirements, and PyGame dependencies.

```bash
# General
sudo -E apt-get install python3-pip cmake zlib1g-dev python3-tk python-opencv -y

# Boost libraries
sudo -E apt-get install libboost-all-dev -y

# Scipy requirements
sudo -E apt-get install libblas-dev liblapack-dev libatlas-base-base-dev gfortran -y

# PyGame
sudo -E apt-get install libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev
libsmpeg-dev libportmidi-dev libavformat-dev libswscale-dev -y
```

--------------------------------

### Install Dependencies

Source: https://github.com/intellabs/coach/blob/master/README.md

Installs necessary development tools and libraries for building and running Coach, including Python 3.5 development headers and image manipulation libraries.

```bash
sudo -E apt-get install dpkg-dev build-essential python3.5-dev libjpeg-dev  libtiff-dev libsdl1.2-dev libnotify-dev 
freeglut3 freeglut3-dev libsm-dev libgtk2.0-dev libgtk-3-dev libwebkitgtk-dev libgtk-3-dev libwebkitgtk-3.0-dev
libgstreamer-plugins-base1.0-dev -y
```

--------------------------------

### ViZDoom Dockerfile Setup

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/dist_usage.rst

This snippet outlines the Dockerfile configuration for the ViZDoom environment. It details the installation of ViZDoom and related dependencies.

```docker
FROM coach-base:master as builder

# prep vizdoom and any of its related requirements.
RUN pip3 install vizdoom

# add coach source starting with files that could trigger
# re-build if dependencies change.
RUN mkdir /root/src
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
RUN pip3 install -r /root/src/requirements.txt

FROM coach-base:master
WORKDIR /root/src
COPY --from=builder /root/.cache /root/.cache
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
COPY README.md /root/src/.
RUN pip3 install vizdoom && pip3 install -e .[all] && rm -rf /root/.cache
COPY . /root/src
```

--------------------------------

### Switching Deep Learning Frameworks

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/usage.rst

This example demonstrates how to switch the deep learning backend framework from the default TensorFlow to MXNet using the '-f' flag. This requires MXNet to be installed.

```bash
coach -p Doom_Basic_DQN -f mxnet
```

--------------------------------

### DQN Agent Configuration for CartPole

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Configures a DQN agent to solve the CartPole environment. Sets up training schedules, agent parameters like learning rate and discount factor, and memory size. It also defines the environment as a GymVectorEnvironment.

```python
from rl_coach.agents.dqn_agent import DQNAgentParameters
from rl_coach.base_parameters import VisualizationParameters, TaskParameters
from rl_coach.core_types import TrainingSteps, EnvironmentEpisodes, EnvironmentSteps
from rl_coach.environments.gym_environment import GymVectorEnvironment
from rl_coach.graph_managers.basic_rl_graph_manager import BasicRLGraphManager
from rl_coach.graph_managers.graph_manager import ScheduleParameters
from rl_coach.memories.memory import MemoryGranularity


####################
# Graph Scheduling #
####################

# Resetting tensorflow graph as the network has changed.
tf.reset_default_graph()

schedule_params = ScheduleParameters()
schedule_params.improve_steps = TrainingSteps(4000)
schedule_params.steps_between_evaluation_periods = EnvironmentEpisodes(10)
schedule_params.evaluation_steps = EnvironmentEpisodes(1)
schedule_params.heatup_steps = EnvironmentSteps(1000)

#########
# Agent #
#########
agent_params = DQNAgentParameters()

# DQN params
agent_params.algorithm.num_steps_between_copying_online_weights_to_target = EnvironmentSteps(100)
agent_params.algorithm.discount = 0.99
agent_params.algorithm.num_consecutive_playing_steps = EnvironmentSteps(1)

# NN configuration
agent_params.network_wrappers['main'].learning_rate = 0.00025
agent_params.network_wrappers['main'].replace_mse_with_huber_loss = False

# ER size
agent_params.memory.max_size = (MemoryGranularity.Transitions, 40000)

################
#  Environment #
################
env_params = GymVectorEnvironment(level='CartPole-v0')
```

--------------------------------

### Install Pip Requirements and Patchelf

Source: https://github.com/intellabs/coach/blob/master/docs/dist_usage.html

Upgrades pip, installs specific versions of setuptools and pytest, and downloads and makes the patchelf executable for environment setup.

```bash
RUN pip3 install --upgrade pip
RUN pip3 install setuptools==39.1.0 && pip3 install pytest && pip3 install pytest-xdist

RUN curl -o /usr/local/bin/patchelf https://s3-us-west-2.amazonaws.com/openai-sci-artifacts/manual-builds/patchelf_0.9_amd64.elf \
    && chmod +x /usr/local/bin/patchelf
```

--------------------------------

### ViZDoom Environment Setup and Filters

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/environments/doom_environment.html

This snippet demonstrates the setup of the ViZDoom environment within RL Coach. It includes the definition of available ViZDoom levels using an Enum, a mapping of keyboard actions to their corresponding key codes, and the configuration of input and output filters for processing observations and actions.

```python
try:
    import vizdoom
except ImportError:
    from rl_coach.logger import failed_imports
    failed_imports.append("ViZDoom")

import os
from enum import Enum
from os import path, environ
from typing import Union, List

import numpy as np

from rl_coach.base_parameters import VisualizationParameters
from rl_coach.environments.environment import Environment, EnvironmentParameters, LevelSelection
from rl_coach.filters.action.full_discrete_action_space_map import FullDiscreteActionSpaceMap
from rl_coach.filters.filter import InputFilter, OutputFilter
from rl_coach.filters.observation.observation_rescale_to_size_filter import ObservationRescaleToSizeFilter
from rl_coach.filters.observation.observation_rgb_to_y_filter import ObservationRGBToYFilter
from rl_coach.filters.observation.observation_stacking_filter import ObservationStackingFilter
from rl_coach.filters.observation.observation_to_uint8_filter import ObservationToUInt8Filter
from rl_coach.spaces import MultiSelectActionSpace, ImageObservationSpace, \
    VectorObservationSpace, StateSpace

# enum of the available levels and their path
class DoomLevel(Enum):
    BASIC = "basic.cfg"
    DEFEND = "defend_the_center.cfg"
    DEATHMATCH = "deathmatch.cfg"
    MY_WAY_HOME = "my_way_home.cfg"
    TAKE_COVER = "take_cover.cfg"
    HEALTH_GATHERING = "health_gathering.cfg"
    HEALTH_GATHERING_SUPREME_COACH_LOCAL = "D2_navigation.cfg"  # from https://github.com/IntelVCL/DirectFuturePrediction/tree/master/maps
    DEFEND_THE_LINE = "defend_the_line.cfg"
    DEADLY_CORRIDOR = "deadly_corridor.cfg"
    BATTLE_COACH_LOCAL = "D3_battle.cfg"  # from https://github.com/IntelVCL/DirectFuturePrediction/tree/master/maps

key_map = {
    'NO-OP': 96,  # `
    'ATTACK': 13,  # enter
    'CROUCH': 306,  # ctrl
    'DROP_SELECTED_ITEM': ord("t"),
    'DROP_SELECTED_WEAPON': ord("t"),
    'JUMP': 32,  # spacebar
    'LAND': ord("l"),
    'LOOK_DOWN': 274,  # down arrow
    'LOOK_UP': 273,  # up arrow
    'MOVE_BACKWARD': ord("s"),
    'MOVE_DOWN': ord("s"),
    'MOVE_FORWARD': ord("w"),
    'MOVE_LEFT': 276,
    'MOVE_RIGHT': 275,
    'MOVE_UP': ord("w"),
    'RELOAD': ord("r"),
    'SELECT_NEXT_WEAPON': ord("q"),
    'SELECT_PREV_WEAPON': ord("e"),
    'SELECT_WEAPON0': ord("0"),
    'SELECT_WEAPON1': ord("1"),
    'SELECT_WEAPON2': ord("2"),
    'SELECT_WEAPON3': ord("3"),
    'SELECT_WEAPON4': ord("4"),
    'SELECT_WEAPON5': ord("5"),
    'SELECT_WEAPON6': ord("6"),
    'SELECT_WEAPON7': ord("7"),
    'SELECT_WEAPON8': ord("8"),
    'SELECT_WEAPON9': ord("9"),
    'SPEED': 304,  # shift
    'STRAFE': 9,  # tab
    'TURN180': ord("u"),
    'TURN_LEFT': ord("a"),  # left arrow
    'TURN_RIGHT': ord("d"),  # right arrow
    'USE': ord("f"),
}

DoomInputFilter = InputFilter(is_a_reference_filter=True)
DoomInputFilter.add_observation_filter('observation', 'rescaling',
                                       ObservationRescaleToSizeFilter(ImageObservationSpace(np.array([60, 76, 3]),
                                                                                            high=255)))
DoomInputFilter.add_observation_filter('observation', 'to_grayscale', ObservationRGBToYFilter())
DoomInputFilter.add_observation_filter('observation', 'to_uint8', ObservationToUInt8Filter(0, 255))
DoomInputFilter.add_observation_filter('observation', 'stacking', ObservationStackingFilter(3))

DoomOutputFilter = OutputFilter(is_a_reference_filter=True)
DoomOutputFilter.add_action_filter('to_discrete', FullDiscreteActionSpaceMap())

```

--------------------------------

### Build and Run Docker Image

Source: https://github.com/intellabs/coach/blob/master/rl_coach/tests/README.md

Commands to build the Docker image for Coach and run it. This verifies the installation process and component correctness.

```bash
cd docker
make build_base && make build
make run
```

--------------------------------

### Install Sphinx Documentation Dependencies

Source: https://github.com/intellabs/coach/blob/master/docs_raw/README.md

Installs the necessary Python packages required for building the Sphinx documentation website. This includes Sphinx itself, extensions for Markdown support, themes, and live reloading.

```python
pip install Sphinx
pip install recommonmark
pip install sphinx_rtd_theme
pip install sphinx-autobuild
pip install sphinx-argparse
```

--------------------------------

### Install System Dependencies for Environments

Source: https://github.com/intellabs/coach/blob/master/docs/dist_usage.html

Installs essential system packages required for Gym, Mujoco, and ViZDoom environments, including libraries for graphics, audio, and build tools.

```bash
# Install dependencies for Gym
RUN apt-get update && apt-get install -y --no-install-recommends \
    freeglut3 freeglut3-dev libsm-dev libgtk2.0-dev libgtk-3-dev libwebkitgtk-dev libgtk-3-dev \
    libwebkitgtk-3.0-dev libgstreamer-plugins-base1.0-dev \
    # Gym
    libav-tools libsdl2-dev swig cmake \
    # Mujoco_py
    curl libgl1-mesa-dev libgl1-mesa-glx libglew-dev libosmesa6-dev software-properties-common \
    # ViZDoom
    build-essential zlib1g-dev libsdl2-dev libjpeg-dev \
    nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \
    libopenal-dev timidity libwildmidi-dev unzip wget && \
    apt-get clean autoclean && \
    apt-get autoremove -y
```

--------------------------------

### Custom Exploration Policy and Checkpoint Saving

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Overrides the default DQN exploration policy with a custom one and configures checkpoint saving. It initializes a graph manager, sets the checkpoint directory and save interval, and starts the training process.

```python
from exploration import MyExplorationParameters

# Overriding the default DQN Agent exploration policy with my exploration policy
agent_params.exploration = MyExplorationParameters()

# Creating a graph manager to train a DQN agent to solve CartPole
graph_manager = BasicRLGraphManager(agent_params=agent_params, env_params=env_params,
                                    schedule_params=schedule_params, vis_params=VisualizationParameters())

# Resources path was defined at the top of this notebook
my_checkpoint_dir = resources_path + '/checkpoints'

# Checkpoints will be stored every 5 seconds to the given directory
task_parameters1 = TaskParameters()
task_parameters1.checkpoint_save_dir = my_checkpoint_dir
task_parameters1.checkpoint_save_secs = 5

graph_manager.create_graph(task_parameters1)
graph_manager.improve()
```

--------------------------------

### Loading Checkpoint and Evaluating Agent

Source: https://github.com/intellabs/coach/blob/master/tutorials/0. Quick Start Guide.ipynb

Loads the latest checkpoint from a specified directory and evaluates the agent's performance. It resets the TensorFlow graph, sets the restore path for checkpoints, creates the graph, and runs the evaluation.

```python
import tensorflow as tf
import shutil

# Clearing the previous graph before creating the new one to avoid name conflicts
tf.reset_default_graph()

# Updating the graph manager's task parameters to restore the latest stored checkpoint from the checkpoints directory
task_parameters2 = TaskParameters()
task_parameters2.checkpoint_restore_path = my_checkpoint_dir

graph_manager.create_graph(task_parameters2)
graph_manager.evaluate(EnvironmentSteps(5))

# Clearning up
shutil.rmtree(my_checkpoint_dir)
```

--------------------------------

### PPO Agent Parameters Initialization

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/agents/ppo_agent.html

Initializes the PPOAgentParameters, setting up the algorithm, exploration strategies, memory, and network configurations.

```python
class PPOAgentParameters(AgentParameters):
    def __init__(self):
        super().__init__(algorithm=PPOAlgorithmParameters(),
                         exploration={DiscreteActionSpace: CategoricalParameters(),
                                      BoxActionSpace: AdditiveNoiseParameters()},
                         memory=EpisodicExperienceReplayParameters(),
                         networks={"critic": PPOCriticNetworkParameters(), "actor": PPOActorNetworkParameters()})
```

--------------------------------

### Build Documentation with Live Reload

Source: https://github.com/intellabs/coach/blob/master/docs_raw/README.md

Starts a local development server that automatically rebuilds and reloads the documentation website whenever changes are detected in the source files.

```bash
sphinx-autobuild source build/html
```

--------------------------------

### Coach Preliminaries and Acrobot Setup

Source: https://github.com/intellabs/coach/blob/master/tutorials/4. Batch Reinforcement Learning.ipynb

Imports necessary libraries from Coach and TensorFlow, sets up task parameters, and configures graph scheduling for Batch RL. It defines environment parameters for Acrobot-v1 and specifies dataset size and training steps.

```python
from copy import deepcopy
import tensorflow as tf
import os

from rl_coach.agents.dqn_agent import DQNAgentParameters
from rl_coach.agents.ddqn_bcq_agent import DDQNBCQAgentParameters, KNNParameters
from rl_coach.base_parameters import VisualizationParameters
from rl_coach.core_types import TrainingSteps, EnvironmentEpisodes, EnvironmentSteps, CsvDataset
from rl_coach.environments.gym_environment import GymVectorEnvironment
from rl_coach.graph_managers.batch_rl_graph_manager import BatchRLGraphManager
from rl_coach.graph_managers.graph_manager import ScheduleParameters
from rl_coach.memories.memory import MemoryGranularity
from rl_coach.schedules import LinearSchedule
from rl_coach.memories.episodic import EpisodicExperienceReplayParameters
from rl_coach.architectures.head_parameters import QHeadParameters
from rl_coach.agents.ddqn_agent import DDQNAgentParameters
from rl_coach.base_parameters import TaskParameters
from rl_coach.spaces import SpacesDefinition, DiscreteActionSpace, VectorObservationSpace, StateSpace, RewardSpace

# Get all the outputs of this tutorial out of the 'Resources' folder
os.chdir('Resources')

# the dataset size to collect 
DATASET_SIZE = 50000

task_parameters = TaskParameters(experiment_path='.')

####################
# Graph Scheduling #
####################

schedule_params = ScheduleParameters()

# 100 epochs (we run train over all the dataset, every epoch) of training
schedule_params.improve_steps = TrainingSteps(100)

# we evaluate the model every epoch
schedule_params.steps_between_evaluation_periods = TrainingSteps(1)

# only for when we have an enviroment
schedule_params.evaluation_steps = EnvironmentEpisodes(10)
schedule_params.heatup_steps = EnvironmentSteps(DATASET_SIZE)

################
#  Environment #
################
env_params = GymVectorEnvironment(level='Acrobot-v1')

```

--------------------------------

### Run Coach Preset

Source: https://github.com/intellabs/coach/blob/master/README.md

Demonstrates how to run Coach with a specified preset for training an RL agent. Includes examples for CartPole and Doom environments, and how to use the -lvl flag for specific levels within suites like Atari.

```bash
coach -r -p <preset_name>
```

```bash
coach -r -p CartPole_PG
```

```bash
coach -r -p Doom_Basic_Dueling_DDQN
```

```bash
coach -r -p Atari_NEC -lvl pong
```

--------------------------------

### Dockerfile Base Example

Source: https://github.com/intellabs/coach/blob/master/docs/dist_usage.html

A sample Dockerfile for building a base image for distributed Coach. It installs essential system packages and libraries required for reinforcement learning tasks.

```dockerfile
FROM nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04

################################
# Install apt-get Requirements #
################################

# General
RUN apt-get update && \
    apt-get install -y python3-pip cmake zlib1g-dev python3-tk python-opencv \
    # Boost libraries
    libboost-all-dev \
    # Scipy requirements
    libblas-dev liblapack-dev libatlas-base-dev gfortran \
    # Pygame requirements
    libsdl-dev libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev \
    libsmpeg-dev libportmidi-dev libavformat-dev libswscale-dev \
    # Dashboard
    dpkg-dev build-essential python3.5-dev libjpeg-dev  libtiff-dev libsdl1.2-dev libnotify-dev \
```

--------------------------------

### Environment Wrapper Initialization

Source: https://github.com/intellabs/coach/blob/master/tutorials/2. Adding an Environment.ipynb

Demonstrates the initialization of an environment wrapper for Coach, including loading the simulator, defining state and action spaces, and handling observations.

```python
import numpy as np
import random
from typing import Union

from dm_control import suite
from dm_control.suite.wrappers import pixels

from rl_coach.base_parameters import VisualizationParameters
from rl_coach.spaces import BoxActionSpace, ImageObservationSpace, VectorObservationSpace, StateSpace
from rl_coach.environments.environment import Environment, LevelSelection


class DeepMindControlSuiteEnvWrapper(Environment):
    def __init__(self, level: LevelSelection = None, visualization_params: VisualizationParameters = None):
        super().__init__(visualization_params=visualization_params)

        self.env_id = level.level_id if level is not None else 'cartpole.swingup'
        self.domain_name, self.task_name = self.env_id.split('.')

        self.env = suite.load(domain_name=self.domain_name, task_name=self.task_name, task_kwargs={'time_limit': 100})
        self.env = pixels.Wrapper(self.env)

        self.state_space = StateSpace({
            'pixels': ImageObservationSpace(shape=self.env.observation_spec()['pixels'].shape, low=0, high=255, dtype=np.uint8),
            'measurements': VectorObservationSpace(shape=self.env.observation_spec()['measurements'].shape, low=-np.inf, high=np.inf, dtype=np.float32)
        })

        self.action_space = BoxActionSpace(shape=self.env.action_spec().shape, low=self.env.action_spec().min, high=self.env.action_spec().max, dtype=np.float32)

```

--------------------------------

### Dumping GIFs of Agent Gameplay

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/usage.rst

This example shows how to use the '-dg' flag to dump GIFs of the agent's gameplay after each evaluation episode. These GIFs are saved in a 'gifs' subdirectory within the experiment directory.

```bash
coach -p Atari_A3C -lvl breakout -n 4 -dg
```

--------------------------------

### Dumping GIFs of Agent Gameplay

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/usage.rst.txt

This example shows how to use the '-dg' flag to dump GIFs of the agent's gameplay after each evaluation episode. These GIFs are saved in a 'gifs' subdirectory within the experiment directory.

```bash
coach -p Atari_A3C -lvl breakout -n 4 -dg
```

--------------------------------

### Get Variable Value

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/architectures/architecture.html

Retrieves the value of a specified variable. The type of the variable depends on the framework being used. For example, a variable could be a symbolic representation like `head.kl_coefficient` or a string.

```APIDOC
get_variable_value(self, variable: Any) -> np.ndarray:
    """
    Gets value of a specified variable. Type of variable is dependant on the framework.
    Example of a variable is head.kl_coefficient, which could be a symbol for evaluation
    or could be a string representing the value.

    :param variable: variable of interest
    :return: value of the specified variable
    """
    raise NotImplementedError
```

--------------------------------

### Build Documentation using Makefile

Source: https://github.com/intellabs/coach/blob/master/docs_raw/README.md

Builds the Sphinx documentation website using the provided Makefile. This process generates HTML output and copies it to the deployment directory, including custom CSS.

```bash
make html
cp source/_static/css/custom.css build/html/_static/css/
rm -rf ../docs/
mkdir ../docs
touch ../docs/.nojekyll
cp -R build/html/* ../docs/
```

--------------------------------

### Dockerfile for Mujoco Environment

Source: https://github.com/intellabs/coach/blob/master/docs/dist_usage.html

Builds a Docker image for the Mujoco environment, downloading and installing Mujoco, setting environment variables, and installing Coach dependencies.

```dockerfile
FROM coach-base:master as builder

# prep mujoco and any of its related requirements.
# Mujoco
RUN mkdir -p ~/.mujoco \
    && wget https://www.roboti.us/download/mjpro150_linux.zip -O mujoco.zip \
    && unzip -n mujoco.zip -d ~/.mujoco \
    && rm mujoco.zip
ARG MUJOCO_KEY
ENV MUJOCO_KEY=$MUJOCO_KEY
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
RUN echo $MUJOCO_KEY | base64 --decode > /root/.mujoco/mjkey.txt
RUN pip3 install mujoco_py

# add coach source starting with files that could trigger
# re-build if dependencies change.
RUN mkdir /root/src
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
RUN pip3 install -r /root/src/requirements.txt

FROM coach-base:master
WORKDIR /root/src
COPY --from=builder /root/.mujoco /root/.mujoco
ENV LD_LIBRARY_PATH /root/.mujoco/mjpro150/bin:$LD_LIBRARY_PATH
COPY --from=builder /root/.cache /root/.cache
COPY setup.py /root/src/.
COPY requirements.txt /root/src/.
COPY README.md /root/src/.
RUN pip3 install mujoco_py && pip3 install -e .[all] && rm -rf /root/.cache
COPY . /root/src
```

--------------------------------

### Run Doom Basic DFP

Source: https://github.com/intellabs/coach/blob/master/benchmarks/dfp/README.md

Executes the Doom Basic DFP experiment with 8 workers. This command initiates a training or evaluation process for the specified DFP configuration.

```bash
coach -p Doom_Basic_DFP -n 8
```

--------------------------------

### Run Coach Dashboard

Source: https://github.com/intellabs/coach/blob/master/README.md

Command to launch the Coach dashboard, which visualizes training signals and algorithmic behavior from CSV files generated during training.

```bash
dashboard
```

--------------------------------

### Load Imitation Learning Dataset

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/usage.rst

Loads a previously collected dataset (replay buffer) for imitation learning. The path to the pickle file can be specified via command line or preset.

```python
from rl_coach.core_types import PickledReplayBuffer
coach -p Doom_Basic_BC -cp='agent.load_memory_from_file_path=PickledReplayBuffer("<experiment dir>/replay_buffer.p")'
```

--------------------------------

### Environment Initialization and State Setup

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/environments/gym_environment.html

Initializes the environment, sets up the state space, and defines reward thresholds. It handles custom reward thresholds and success rates, and prepares the environment for training or testing.

```python
self.renderer.create_screen(image.shape[1]*scale, image.shape[0]*scale)

        # the info is only updated after the first step
        self.state_space['measurements'] = VectorObservationSpace(shape=len(self.info.keys()))

        if self.env.spec and custom_reward_threshold is None:
                self.reward_success_threshold = self.env.spec.reward_threshold
                self.reward_space = RewardSpace(1, reward_success_threshold=self.reward_success_threshold)

        self.target_success_rate = target_success_rate
```

--------------------------------

### API Compatibility Get Transition

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/memories/non_episodic/experience_replay.html

Provides an alternative method name 'get' for retrieving a transition by its index, maintaining API compatibility. It calls the primary 'get_transition' method.

```Python
def get(self, transition_index: int, lock: bool=True) -> Union[None, Transition]:
    """
    Returns the transition in the given index. If the transition does not exist, returns None instead.
    :param transition_index: the index of the transition to return
    :return: the corresponding transition
    """
    return self.get_transition(transition_index, lock)
```

--------------------------------

### ACER Agent Initialization

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/agents/acer_agent.html

Sets up the ACER agent by defining its algorithm, exploration policy, memory, and network configurations.

```python
class ACERAgentParameters(AgentParameters):
    def __init__(self):
        super().__init__(algorithm=ACERAlgorithmParameters(),
                         exploration={DiscreteActionSpace: CategoricalParameters()},
                         memory=EpisodicExperienceReplayParameters(),
                         networks={"main": ACERNetworkParameters()})
    @property
    def path(self):
        return 'rl_coach.agents.acer_agent:ACERAgent'
```

--------------------------------

### Get State Embedding

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/agents/agent.html

Retrieves the state embedding from the main network for a given state. This method is used to get a vectorized representation of the state, which is often used as input for the agent's decision-making process. It prepares the state for inference.

```python
def get_state_embedding(self, state: dict) -> np.ndarray:
        """
        Given a state, get the corresponding state embedding  from the main network

        :param state: a state dict
        :return: a numpy embedding vector
        """
        # TODO: this won't work anymore
        # TODO: instead of the state embedding (which contains the goal) we should use the observation embedding
        embedding = self.networks['main'].online_network.predict(
            self.prepare_batch_for_inference(state, "main"),
            outputs=self.networks['main'].online_network.state_embedding)
        return embedding
```

--------------------------------

### Load Imitation Learning Dataset

Source: https://github.com/intellabs/coach/blob/master/docs/_sources/usage.rst.txt

Loads a previously collected dataset (replay buffer) for imitation learning. The path to the pickle file can be specified via command line or preset.

```python
from rl_coach.core_types import PickledReplayBuffer
coach -p Doom_Basic_BC -cp='agent.load_memory_from_file_path=PickledReplayBuffer("<experiment dir>/replay_buffer.p")'
```

--------------------------------

### ControlSuiteEnvironment Initialization and Setup

Source: https://github.com/intellabs/coach/blob/master/docs/_modules/rl_coach/environments/control_suite_environment.html

Initializes the ControlSuiteEnvironment, loads the specified dm_control task, and optionally wraps it with a pixel wrapper for image observations. It also sets up the state space based on the chosen observation type.

```Python
import random
from enum import Enum
from typing import Union

import numpy as np

try:
    from dm_control import suite
    from dm_control.suite.wrappers import pixels
except ImportError:
    from rl_coach.logger import failed_imports
    failed_imports.append("DeepMind Control Suite")

from rl_coach.base_parameters import VisualizationParameters
from rl_coach.environments.environment import Environment, EnvironmentParameters, LevelSelection
from rl_coach.filters.filter import NoInputFilter, NoOutputFilter
from rl_coach.spaces import BoxActionSpace, ImageObservationSpace, VectorObservationSpace, StateSpace

class ObservationType(Enum):
    Measurements = 1
    Image = 2
    Image_and_Measurements = 3

class ControlSuiteEnvironmentParameters(EnvironmentParameters):
    def __init__(self, level=None):
        super().__init__(level=level)
        self.observation_type = ObservationType.Measurements
        self.default_input_filter = ControlSuiteInputFilter
        self.default_output_filter = ControlSuiteOutputFilter

    @property
    def path(self):
        return 'rl_coach.environments.control_suite_environment:ControlSuiteEnvironment'

ControlSuiteInputFilter = NoInputFilter()
ControlSuiteOutputFilter = NoOutputFilter()

control_suite_envs = {':'.join(env): ':'.join(env) for env in suite.BENCHMARKING}

class ControlSuiteEnvironment(Environment):
    def __init__(self, level: LevelSelection, frame_skip: int, visualization_parameters: VisualizationParameters,
                 target_success_rate: float=1.0, seed: Union[None, int]=None, human_control: bool=False,
                 observation_type: ObservationType=ObservationType.Measurements,
                 custom_reward_threshold: Union[int, float]=None, **kwargs):
        """
        :param level: (str)
            A string representing the control suite level to run. This can also be a LevelSelection object.
            For example, cartpole:swingup.

        :param frame_skip: (int)
            The number of frames to skip between any two actions given by the agent. The action will be repeated
            for all the skipped frames.

        :param visualization_parameters: (VisualizationParameters)
            The parameters used for visualizing the environment, such as the render flag, storing videos etc.

        :param target_success_rate: (float)
            Stop experiment if given target success rate was achieved.

        :param seed: (int)
            A seed to use for the random number generator when running the environment.

        :param human_control: (bool)
            A flag that allows controlling the environment using the keyboard keys.

        :param observation_type: (ObservationType)
            An enum which defines which observation to use. The current options are to use:
            * Measurements only - a vector of joint torques and similar measurements
            * Image only - an image of the environment as seen by a camera attached to the simulator
            * Measurements & Image - both type of observations will be returned in the state using the keys
            'measurements' and 'pixels' respectively.

        :param custom_reward_threshold: (float)
            Allows defining a custom reward that will be used to decide when the agent succeeded in passing the environment.

        """
        super().__init__(level, seed, frame_skip, human_control, custom_reward_threshold, visualization_parameters, target_success_rate)

        self.observation_type = observation_type

        # load and initialize environment
        domain_name, task_name = self.env_id.split(":")
        self.env = suite.load(domain_name=domain_name, task_name=task_name, task_kwargs={'random': seed})

        if observation_type != ObservationType.Measurements:
            self.env = pixels.Wrapper(self.env, pixels_only=observation_type == ObservationType.Image)

        # seed
        if self.seed is not None:
            np.random.seed(self.seed)
            random.seed(self.seed)

        self.state_space = StateSpace({})

        # image observations
        if observation_type != ObservationType.Measurements:
            self.state_space['pixels'] = ImageObservationSpace(shape=self.env.observation_spec()['pixels'].shape,
                                                               high=255)

        # measurements observations
        if observation_type != ObservationType.Image:
            measurements_space_size = 0
            measurements_names = []
            for observation_space_name, observation_space in self.env.observation_spec().items():
                if len(observation_space.shape) == 0:

```

--------------------------------

### Render Environment

Source: https://github.com/intellabs/coach/blob/master/docs_raw/source/usage.rst

Renders the environment during the experiment, allowing visualization of the agent's interaction.

```python
coach -r
```