### Test SpinningUp Installation Policy (Shell)

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

Executes a test policy script provided by SpinningUp using the data from the installation test run. This verifies that the basic policy execution is working correctly after installation.

```shell
python -m spinup.run test_policy data/installtest/installtest_s0
```

--------------------------------

### Cloning and Installing Spinning Up

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This sequence of commands first clones the Spinning Up repository from GitHub, then changes the current directory into the newly cloned repository, and finally installs the library in editable mode using pip. The editable install allows for local modifications to the source code.

```Shell
git clone https://github.com/openai/spinningup.git
cd spinningup
pip install -e .
```

--------------------------------

### Install Gym MuJoCo/Robotics Environments (Shell)

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

Installs the `gym` package along with the necessary dependencies for MuJoCo and Robotics environments using pip. This step is required to use MuJoCo-based environments within Gym after installing MuJoCo itself.

```shell
pip install gym[mujoco,robotics]
```

--------------------------------

### Running Spinning Up Installation Test

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This command executes a test run of the PPO algorithm on the LunarLander-v2 environment using the installed Spinning Up library. It specifies network architecture, environment, experiment name, and discount factor. This helps verify that the installation is functional.

```Shell
python -m spinup.run ppo --hid "[32,32]" --env LunarLander-v2 --exp_name installtest --gamma 0.999
```

--------------------------------

### Plot SpinningUp Installation Test Results (Shell)

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

Runs the plotting script from SpinningUp on the data generated during the installation test. This helps visualize the training progress and confirm the data logging and plotting functionality.

```shell
python -m spinup.run plot data/installtest/installtest_s0
```

--------------------------------

### Installing OpenMPI on macOS with Homebrew

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This command installs the OpenMPI library on macOS using the Homebrew package manager. Homebrew is a prerequisite for this installation method. OpenMPI provides parallel processing support.

```Shell
brew install openmpi
```

--------------------------------

### Run PPO on MuJoCo Environment (Shell)

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

Executes a PPO training script from SpinningUp on the `Walker2d-v2` environment, which requires MuJoCo. This command tests the successful integration of MuJoCo, Gym, and SpinningUp by running a simple training job.

```shell
python -m spinup.run ppo --hid "[32,32]" --env Walker2d-v2 --exp_name mujocotest
```

--------------------------------

### Installing OpenMPI on Ubuntu

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This command updates the package list and then installs the OpenMPI development libraries on Ubuntu systems using the apt-get package manager. OpenMPI is required for parallel processing capabilities used by some parts of the library.

```Shell
sudo apt-get update && sudo apt-get install libopenmpi-dev
```

--------------------------------

### Example: Running PPO on Walker2d-v2

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

A specific example demonstrating how to run the PPO algorithm on the Walker2d-v2 environment with a custom experiment name 'walker'.

```Python (Command Line)
python -m spinup.run ppo --env Walker2d-v2 --exp_name walker
```

--------------------------------

### Creating Conda Environment for Spinning Up

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This command uses Conda to create a new virtual environment named 'spinningup' specifically for the project. It specifies Python version 3.6 to ensure compatibility with the library's requirements.

```Shell
conda create -n spinningup python=3.6
```

--------------------------------

### Detailed Example: Running PPO with Multiple Hyperparameters

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

A comprehensive example showing how to run PPO on Ant-v2, specifying multiple values for hyperparameters like clip_ratio, hidden layers (hid), activation function (act), and random seeds. Includes flags for timestamping (dt) and specifying data directory (data_dir).

```Python (Command Line)
python -m spinup.run ppo --exp_name ppo_ant --env Ant-v2 --clip_ratio 0.1 0.2 
    --hid[h] [32,32] [64,32] --act torch.nn.Tanh --seed 0 10 20 --dt
    --data_dir path/to/data
```

--------------------------------

### Activating Conda Environment

Source: https://github.com/openai/spinningup/blob/master/docs/user/installation.rst

This command activates the previously created Conda environment named 'spinningup'. Activating the environment ensures that subsequent commands use the Python interpreter and packages installed within this specific environment.

```Shell
conda activate spinningup
```

--------------------------------

### Getting Algorithm Hyperparameter Help

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Command to display the available keyword arguments (hyperparameters) for a specific Spinning Up algorithm by showing its docstring.

```Python (Command Line)
python -m spinup.run [algo name] --help
```

--------------------------------

### Get Actions from Saved PyTorch PPO Model

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/ppo.rst

Loads a saved PyTorch actor-critic model and demonstrates how to use its `act` method to get actions for a given observation. Requires the model to be loaded first using `torch.load`.

```python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### Defining Hyperparameter Sweeps with Spinup ExperimentGrid (PyTorch)

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

This example shows how to use the `ExperimentGrid` utility from Spinning Up to manage multiple training runs with different hyperparameter configurations for a PyTorch-based PPO algorithm. It covers initializing the grid, adding parameters with lists of values for sweeping, assigning shorthands, and executing the grid which automatically launches individual experiments for each parameter combination.

```python
from spinup.utils.run_utils import ExperimentGrid
from spinup import ppo_pytorch
import torch

if __name__ == '__main__':
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--cpu', type=int, default=4)
    parser.add_argument('--num_runs', type=int, default=3)
    args = parser.parse_args()

    eg = ExperimentGrid(name='ppo-pyt-bench')
    eg.add('env_name', 'CartPole-v0', '', True)
    eg.add('seed', [10*i for i in range(args.num_runs)])
    eg.add('epochs', 10)
    eg.add('steps_per_epoch', 4000)
    eg.add('ac_kwargs:hidden_sizes', [(32,), (64,64)], 'hid')
    eg.add('ac_kwargs:activation', [torch.nn.Tanh, torch.nn.ReLU], '')
    eg.run(ppo_pytorch, num_cpu=args.cpu)
```

--------------------------------

### Launching Multiple Experiments with Different Seeds

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Shows how to launch multiple experiments sequentially by providing multiple values for a single command-line argument. In this example, three separate runs of the PPO algorithm on Walker2d-v2 are launched, each with a different random seed (0, 10, and 20).

```Shell
python -m spinup.run ppo --env Walker2d-v2 --exp_name walker --seed 0 10 20
```

--------------------------------

### Initializing Data Structures for Tensorflow Policy Training Epoch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This code defines the start of a function `train_one_epoch` and initializes empty lists to store data collected during one epoch of policy training. These lists are intended to hold observations, actions, and corresponding episode weights (returns) for batch processing.

```python
# for training policy
def train_one_epoch():
    # make some empty lists for logging.
    batch_obs = []          # for observations
    batch_acts = []         # for actions
    batch_weights = []      # for R(tau) weighting in policy gradient
```

--------------------------------

### Creating Tensorflow Policy Network

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This code block defines the core components of a feedforward neural network policy in Tensorflow. It sets up a placeholder for observations, passes them through an MLP to get action logits, and defines an operation to sample actions from the categorical distribution implied by the logits.

```python
# make core of policy network
obs_ph = tf.placeholder(shape=(None, obs_dim), dtype=tf.float32)
logits = mlp(obs_ph, sizes=hidden_sizes+[n_acts])

# make action selection op (outputs int actions, sampled from policy)
actions = tf.squeeze(tf.multinomial(logits=logits,num_samples=1), axis=1)
```

--------------------------------

### Getting Actions from PyTorch VPG Model

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/vpg.rst

Demonstrates how to obtain actions from a loaded PyTorch actor-critic model trained with the VPG algorithm. It uses the model's `act` method with observations converted to a PyTorch tensor.

```python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### Using Deterministic Policy Network PyTorch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

This snippet demonstrates how to use the previously defined `pi_net` to compute actions from observations. It converts a batch of observations (assumed to be a Numpy array `obs`) into a PyTorch tensor and then passes it through the policy network to get the corresponding actions.

```python
obs_tensor = torch.as_tensor(obs, dtype=torch.float32)
actions = pi_net(obs_tensor)
```

--------------------------------

### Run One Epoch of Simple Policy Gradient - Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This code snippet represents the core logic of the `train_one_epoch` function in a simple policy gradient implementation. It collects experience by simulating agent interaction with the environment for multiple episodes until a batch size is reached, and then performs a single policy gradient update step using the collected data. This version uses the total episode return as the weight for each step's log probability.

```Python
batch_rets = []         # for measuring episode returns
batch_lens = []         # for measuring episode lengths

# reset episode-specific variables
obs = env.reset()       # first obs comes from starting distribution
done = False            # signal from environment that episode is over
ep_rews = []            # list for rewards accrued throughout ep

# render first episode of each epoch
finished_rendering_this_epoch = False

# collect experience by acting in the environment with current policy
while True:

    # rendering
    if not(finished_rendering_this_epoch):
        env.render()

    # save obs
    batch_obs.append(obs.copy())

    # act in the environment
    act = sess.run(actions, {obs_ph: obs.reshape(1,-1)})[0]
    obs, rew, done, _ = env.step(act)

    # save action, reward
    batch_acts.append(act)
ep_rews.append(rew)

    if done:
        # if episode is over, record info about episode
        ep_ret, ep_len = sum(ep_rews), len(ep_rews)
        batch_rets.append(ep_ret)
        batch_lens.append(ep_len)

        # the weight for each logprob(a|s) is R(tau)
        batch_weights += [ep_ret] * ep_len

        # reset episode-specific variables
        obs, done, ep_rews = env.reset(), False, []

        # won't render again this epoch
        finished_rendering_this_epoch = True

        # end experience loop if we have enough of it
        if len(batch_obs) > batch_size:
            break

# take a single policy gradient update step
batch_loss, _ = sess.run([loss, train_op],
                         feed_dict={
                            obs_ph: np.array(batch_obs),
                            act_ph: np.array(batch_acts),
                            weights_ph: np.array(batch_weights)
                         })
return batch_loss, batch_rets, batch_lens
```

--------------------------------

### Calculate Reward-to-Go - Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This function calculates the reward-to-go for a given list of rewards from an episode. It iterates backward through the rewards, summing up all future rewards at each time step `t` to get the return from that point onwards. This is used as the weight for the log probability of the action taken at time `t` in the reward-to-go policy gradient algorithm.

```Python
def reward_to_go(rews):
    n = len(rews)
    rtgs = np.zeros_like(rews)
    for i in reversed(range(n)):
        rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0)
    return rtgs
```

--------------------------------

### Plotting Results using plot.py with Autocomplete

Source: https://github.com/openai/spinningup/blob/master/docs/user/plotting.rst

An alternative way to run the plotter is directly using the `plot.py` script. This example demonstrates using a directory prefix (`data/bench_algo`) which the plotter will autocomplete internally to find matching log directories recursively, such as `data/bench_algo1` and `data/bench_algo2`.

```bash
python spinup/utils/plot.py data/bench_algo
```

--------------------------------

### Running Policy After Environment Not Found Error - Python

Source: https://github.com/openai/spinningup/blob/master/docs/user/saving_and_loading.rst

This snippet shows how to manually load a saved policy and run it in a recreated environment when the original environment was not saved with the model. It uses `load_policy_and_env` to get the policy's action function and then runs the policy using `run_policy` with a newly created environment instance.

```Python
from spinup.utils.test_policy import load_policy_and_env, run_policy
import your_env
_, get_action = load_policy_and_env('/path/to/output_directory')
env = your_env.make()
run_policy(env, get_action)
```

--------------------------------

### Getting Actions from PyTorch DDPG Model (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/ddpg.rst

This snippet demonstrates how to obtain actions from a loaded PyTorch actor-critic model object. It requires the input observation 'obs' to be converted to a PyTorch tensor with the correct data type.

```python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### Calculating Log Probability for Gaussian Policy (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

Demonstrates how to calculate the total log probability for a vector-valued action when using a diagonal Gaussian policy in PyTorch. It sums the log probabilities of individual action components to get a single log probability per action.

```python
logp = get_policy(obs).log_prob(act).sum(axis=-1)
```

--------------------------------

### Getting Actions from Loaded PyTorch SAC Model

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/sac.rst

This snippet demonstrates how to obtain actions from a loaded PyTorch SAC actor-critic model. It shows how to convert observations into a PyTorch tensor and use the model's 'act' method.

```python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### Full Training Loop Integration with EpochLogger (TensorFlow/Python)

Source: https://github.com/openai/spinningup/blob/master/docs/utils/logger.rst

Provides a complete example of integrating `EpochLogger` into a TensorFlow training script for an MLP on MNIST. It illustrates how to save hyperparameters (`save_config`), set up model saving (`setup_tf_saver`), log metrics per step (`store`), save the model state (`save_state`), and log epoch-level statistics (`log_tabular`, `dump_tabular`). Requires TensorFlow and NumPy.

```python
import numpy as np
import tensorflow as tf
import time
from spinup.utils.logx import EpochLogger


def mlp(x, hidden_sizes=(32,), activation=tf.tanh, output_activation=None):
    for h in hidden_sizes[:-1]:
        x = tf.layers.dense(x, units=h, activation=activation)
    return tf.layers.dense(x, units=hidden_sizes[-1], activation=output_activation)


# Simple script for training an MLP on MNIST.
def train_mnist(steps_per_epoch=100, epochs=5,
                lr=1e-3, layers=2, hidden_size=64,
                logger_kwargs=dict(), save_freq=1):

    logger = EpochLogger(**logger_kwargs)
    logger.save_config(locals())

    # Load and preprocess MNIST data
    (x_train, y_train), _ = tf.keras.datasets.mnist.load_data()
    x_train = x_train.reshape(-1, 28*28) / 255.0

    # Define inputs & main outputs from computation graph
    x_ph = tf.placeholder(tf.float32, shape=(None, 28*28))
    y_ph = tf.placeholder(tf.int32, shape=(None,))
    logits = mlp(x_ph, hidden_sizes=[hidden_size]*layers + [10], activation=tf.nn.relu)
    predict = tf.argmax(logits, axis=1, output_type=tf.int32)

    # Define loss function, accuracy, and training op
    y = tf.one_hot(y_ph, 10)
    loss = tf.losses.softmax_cross_entropy(y, logits)
    acc = tf.reduce_mean(tf.cast(tf.equal(y_ph, predict), tf.float32))
    train_op = tf.train.AdamOptimizer().minimize(loss)

    # Prepare session
    sess = tf.Session()
    sess.run(tf.global_variables_initializer())

    # Setup model saving
    logger.setup_tf_saver(sess, inputs={'x': x_ph},
                                outputs={'logits': logits, 'predict': predict})

    start_time = time.time()

    # Run main training loop
    for epoch in range(epochs):
        for t in range(steps_per_epoch):
            idxs = np.random.randint(0, len(x_train), 32)
            feed_dict = {x_ph: x_train[idxs],
                         y_ph: y_train[idxs]}
            outs = sess.run([loss, acc, train_op], feed_dict=feed_dict)
            logger.store(Loss=outs[0], Acc=outs[1])

        # Save model
        if (epoch % save_freq == 0) or (epoch == epochs-1):
            logger.save_state(state_dict=dict(), itr=None)

        # Log info about epoch
        logger.log_tabular('Epoch', epoch)
        logger.log_tabular('Acc', with_min_and_max=True)
        logger.log_tabular('Loss', average_only=True)
        logger.log_tabular('TotalGradientSteps', (epoch+1)*steps_per_epoch)
        logger.log_tabular('Time', time.time()-start_time)
        logger.dump_tabular()

    if __name__ == '__main__':
        train_mnist()
```

--------------------------------

### Get Actions from PyTorch TD3 Model

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/td3.rst

Demonstrates how to use a loaded PyTorch actor-critic model object (`ac`) to compute actions for a given observation (`obs`). The observation must be converted to a PyTorch tensor with `float32` dtype.

```Python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### Defining Tensorflow Policy Gradient Loss

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This code block constructs a 'loss' function for the policy gradient algorithm in Tensorflow. It uses placeholders for episode weights and actions, creates action masks, calculates the log probabilities of chosen actions, and computes the weighted negative mean of log probabilities, which serves as the objective for gradient ascent.

```python
# make loss function whose gradient, for the right data, is policy gradient
weights_ph = tf.placeholder(shape=(None,), dtype=tf.float32)
act_ph = tf.placeholder(shape=(None,), dtype=tf.int32)
action_masks = tf.one_hot(act_ph, n_acts)
log_probs = tf.reduce_sum(action_masks * tf.nn.log_softmax(logits), axis=1)
loss = -tf.reduce_mean(weights_ph * log_probs)
```

--------------------------------

### Evaluating TensorFlow Tensor z3 (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Runs a TensorFlow session to evaluate the tensor `z3`. The resulting 5x5 array provides a third example of tensor operation output, further illustrating the shapes and values produced by operations discussed in the context of broadcasting issues.

```Python
>>> sess.run(z3)
array([[ 0,  1,  2,  3,  4],
       [ 0,  2,  4,  6,  8],
       [ 0,  3,  6,  9, 12],
       [ 0,  4,  8, 12, 16],
       [ 0,  5, 10, 15, 20]])
```

--------------------------------

### Tensorflow Broadcasting Example

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Demonstrates Tensorflow's broadcasting behavior with tensors of shapes `[batch size]` and `[batch size, 1]`. It shows that operations like multiplication and addition between these shapes result in a tensor of shape `[batch size, batch size]`, which is often not the intended behavior in reinforcement learning value calculations.

```python
>>> import tensorflow as tf
>>> import numpy as np
>>> x = tf.constant(np.arange(5))
>>> y = tf.constant(np.arange(5).reshape(-1,1))
>>> z1 = x * y
>>> z2 = x + y
>>> z3 = x + z1
>>> x.shape
TensorShape([Dimension(5)])
>>> y.shape
TensorShape([Dimension(5), Dimension(1)])
>>> z1.shape
TensorShape([Dimension(5), Dimension(5)])
>>> z2.shape
TensorShape([Dimension(5), Dimension(5)])
```

--------------------------------

### Evaluating TensorFlow Tensor z2 (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Executes a TensorFlow session to evaluate the tensor `z2`. The output is a 5x5 array, illustrating another example of tensor operation results, potentially demonstrating different broadcasting behavior or intermediate values compared to `z1`.

```Python
>>> sess.run(z2)
array([[0, 1, 2, 3, 4],
       [1, 2, 3, 4, 5],
       [2, 3, 4, 5, 6],
       [3, 4, 5, 6, 7],
       [4, 5, 6, 7, 8]])
```

--------------------------------

### Update Batch Weights with Reward-to-Go - Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/extra_tf_pg_implementation.rst

This code snippet shows the modification needed in the experience collection loop to implement the reward-to-go policy gradient. Instead of adding the total episode return (`ep_ret`) for every step in the episode, it calculates the reward-to-go for the episode's rewards (`ep_rews`) using the `reward_to_go` function and adds these values as weights to `batch_weights`. This aligns the weight for each action's log probability with the sum of future rewards from that point.

```Python
# the weight for each logprob(a_t|s_t) is reward-to-go from t
batch_weights += list(reward_to_go(ep_rews))
```

--------------------------------

### Running Spinning Up Algorithm from Command Line

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

The standard command format for launching any Spinning Up algorithm with specified experiment flags.

```Python (Command Line)
python -m spinup.run [algo name] [experiment flags]
```

--------------------------------

### Running Spinup PPO Algorithm from Python Script (TensorFlow)

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

This snippet illustrates how to programmatically launch a reinforcement learning experiment using the Spinning Up library's PPO algorithm implemented in TensorFlow 1. It demonstrates setting up the environment function, configuring the actor-critic network architecture and activation, defining logging parameters, and executing the training process by calling the `ppo` function directly.

```python
from spinup import ppo_tf1 as ppo
import tensorflow as tf
import gym

env_fn = lambda : gym.make('LunarLander-v2')

ac_kwargs = dict(hidden_sizes=[64,64], activation=tf.nn.relu)

logger_kwargs = dict(output_dir='path/to/output_dir', exp_name='experiment_name')

ppo(env_fn=env_fn, ac_kwargs=ac_kwargs, steps_per_epoch=5000, epochs=250, logger_kwargs=logger_kwargs)
```

--------------------------------

### Passing Dictionary Arguments on Command Line

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Demonstrates two equivalent methods for providing dictionary values to command-line arguments. The first uses standard Python dictionary syntax within the argument value, while the second uses a colon-separated key-value syntax (`--key:subkey value`).

```Shell
--key dict(v1=value_1, v2=value_2)
```

```Shell
--key:v1 value_1 --key:v2 value_2
```

--------------------------------

### Running PyTorch Version of Algorithm

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Command format to explicitly select the PyTorch implementation of a Spinning Up algorithm.

```Python (Command Line)
python -m spinup.run [algo]_pytorch
```

--------------------------------

### Running Trained Policy via Command Line

Source: https://github.com/openai/spinningup/blob/master/docs/user/saving_and_loading.rst

Execute the `test_policy` script from the command line to load a trained agent from a specified output directory and run it in its environment. This is the standard way to evaluate a saved policy.

```shell
python -m spinup.run test_policy path/to/output_directory
```

--------------------------------

### Run PPO with Specific Activation Function

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Launches a PPO experiment using `spinup.run` for the Walker2d-v2 environment, naming it 'walker', and setting the activation function for the neural networks to `torch.nn.ELU` via the `--act` flag. The text also mentions the TensorFlow equivalent `tf.nn.elu` for `ppo_tf1`.

```Shell
python -m spinup.run ppo --env Walker2d-v2 --exp_name walker --act torch.nn.ELU
```

--------------------------------

### Soft Actor-Critic Algorithm Pseudocode

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/sac.rst

This pseudocode outlines the main steps of the Soft Actor-Critic (SAC) algorithm, including initialization, interaction with the environment, storing transitions in a replay buffer, and the update steps for the Q-functions and the policy using sampled batches.

```Pseudocode
Input: initial policy parameters $\theta$, Q-function parameters $\phi_1$, $\phi_2$, empty replay buffer $\mathcal{D}$
Set target parameters equal to main parameters $\phi_{\text{targ},1} \leftarrow \phi_1$, $\phi_{\text{targ},2} \leftarrow \phi_2$
REPEAT
    Observe state $s$ and select action $a \sim \pi_{\theta}(\cdot|s)$
    Execute $a$ in the environment
    Observe next state $s'$, reward $r$, and done signal $d$ to indicate whether $s'$ is terminal
    Store $(s,a,r,s',d)$ in replay buffer $\mathcal{D}$
    If $s'$ is terminal, reset environment state.
    IF{it's time to update}
        FOR{$j$ in range(however many updates)}
            Randomly sample a batch of transitions, $B = { (s,a,r,s',d) }$ from $\mathcal{D}$
            Compute targets for the Q functions:
            $y (r,s',d) = r + \gamma (1-d) \left(\min_{i=1,2} Q_{\phi_{\text{targ}, i}} (s', \tilde{a}') - \alpha \log \pi_{\theta}(\tilde{a}'|s')\right), \tilde{a}' \sim \pi_{\theta}(\cdot|s')$
            Update Q-functions by one step of gradient descent using
            $\nabla_{\phi_i} \frac{1}{|B|}\sum_{(s,a,r,s',d) \in B} \left( Q_{\phi_i}(s,a) - y(r,s',d) \right)^2 \text{for } i=1,2$
            Update policy by one step of gradient ascent using
            $\nabla_{\theta} \frac{1}{|B|}\sum_{s \in B} \Big(\min_{i=1,2} Q_{\phi_i}(s, \tilde{a}_{\theta}(s)) - \alpha \log \pi_{\theta} \left(\left. \tilde{a}_{\theta}(s) \right| s
ight) \Big),$
            where $\tilde{a}_{\theta}(s)$ is a sample from $\pi_{\theta}(\cdot|s)$ which is differentiable wrt $\theta$ via the reparametrization trick.
```

--------------------------------

### Collecting Experience in Training Epoch (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

Initializes lists to store batch data (observations, actions, weights, returns, lengths) and episode data (rewards). Resets the environment and episode variables, then enters a loop to collect experience by acting in the environment using the current policy, saving observations, actions, and rewards until an episode finishes.

```python
def train_one_epoch():
    # make some empty lists for logging.
    batch_obs = []          # for observations
    batch_acts = []         # for actions
    batch_weights = []      # for R(tau) weighting in policy gradient
    batch_rets = []         # for measuring episode returns
    batch_lens = []         # for measuring episode lengths

    # reset episode-specific variables
    obs = env.reset()       # first obs comes from starting distribution
    done = False            # signal from environment that episode is over
    ep_rews = []            # list for rewards accrued throughout ep

    # render first episode of each epoch
    finished_rendering_this_epoch = False

    # collect experience by acting in the environment with current policy
    while True:

        # rendering
        if (not finished_rendering_this_epoch) and render:
            env.render()

        # save obs
        batch_obs.append(obs.copy())

        # act in the environment
        act = get_action(torch.as_tensor(obs, dtype=torch.float32))
        obs, rew, done, _ = env.step(act)

        # save action, reward
        batch_acts.append(act)
        ep_rews.append(rew)

        if done:
            # if episode is over, record info about episode
            ep_ret, ep_len = sum(ep_rews), len(ep_rews)
            batch_rets.append(ep_ret)
            batch_lens.append(ep_len)

```

--------------------------------

### Running TRPO Experiments with Varying Value Iterations (Shell)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercises.rst

Executes the Spinning Up TRPO implementation via the command line, comparing runs with `train_v_iters` set to 0 and 80. It uses the Hopper-v2 environment, runs for 250 epochs with 4000 steps per epoch, and uses seeds 0, 10, and 20. The `--dt` flag is also included. This command launches six experiments in total.

```Shell
python -m spinup.run trpo --env Hopper-v2 --train_v_iters[v] 0 80 --exp_name ex2-1 --epochs 250 --steps_per_epoch 4000 --seed 0 10 20 --dt
```

--------------------------------

### Basic Diagnostic Logging with EpochLogger (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/utils/logger.rst

Demonstrates the basic usage of `EpochLogger` to track a diagnostic value. It shows how to use `store` to accumulate values, `log_tabular` to compute statistics (average, std dev, min, max) over accumulated values and clear the internal state, and `dump_tabular` to output the results.

```python
from spinup.utils.logx import EpochLogger
epoch_logger = EpochLogger()
for i in range(10):
    epoch_logger.store(Test=i)
epoch_logger.log_tabular('Test', with_min_and_max=True)
epoch_logger.dump_tabular()
```

--------------------------------

### Optimal Action from Optimal Q-Function (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

This equation shows how to derive the optimal action a*(s) in a given state s directly from the optimal action-value function Q*(s,a) by selecting the action that maximizes Q*.

```math
a^*(s) = \arg \max_a Q^* (s,a).
```

--------------------------------

### BibTeX Citation for Spinning Up

Source: https://github.com/openai/spinningup/blob/master/readme.md

This snippet provides the standard BibTeX entry for citing the Spinning Up in Deep Reinforcement Learning resource in academic papers or research.

```BibTeX
@article{SpinningUp2018,
    author = {Achiam, Joshua},
    title = {{Spinning Up in Deep Reinforcement Learning}},
    year = {2018}
}
```

--------------------------------

### Running Tensorflow Version of Algorithm

Source: https://github.com/openai/spinningup/blob/master/docs/user/running.rst

Command format to explicitly select the Tensorflow (TF1) implementation of a Spinning Up algorithm.

```Python (Command Line)
python -m spinup.run [algo]_tf1
```

--------------------------------

### Plotting Results with spinup.run

Source: https://github.com/openai/spinningup/blob/master/docs/user/plotting.rst

Run the main plotting utility via the `spinup.run` module. Provide one or more paths to output directories. Optional arguments control the legend, axes, smoothing, and which data to include or exclude.

```bash
python -m spinup.run plot [path/to/output_directory ...] [--legend [LEGEND ...]] \
    [--xaxis XAXIS] [--value [VALUE ...]] [--count] [--smooth S] \
    [--select [SEL ...]] [--exclude [EXC ...]]
```

--------------------------------

### Loading PyTorch Model

Source: https://github.com/openai/spinningup/blob/master/docs/utils/logger.rst

Loads a saved PyTorch actor-critic model from a specified file path using `torch.load`. The loaded model is expected to have an `act` method for sampling actions.

```python
ac = torch.load('path/to/model.pt')
```

--------------------------------

### Bellman Equations for On-Policy Value Functions (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

These equations define the recursive relationships for the state-value function V^pi and the action-value function Q^pi under a fixed policy pi, expressing values in terms of expected immediate rewards and discounted future values.

```math
\begin{align*}\nV^{\pi}(s) &= \underE{a \sim \pi \\ s'\sim P}{r(s,a) + \gamma V^{\pi}(s')}, \\nQ^{\pi}(s,a) &= \underE{s'\sim P}{r(s,a) + \gamma \underE{a'\sim \pi}{Q^{\pi}(s',a')}},\n\end{align*}
```

--------------------------------

### Bellman Equations for Optimal Value Functions (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

These equations define the recursive relationships for the optimal state-value function V* and the optimal action-value function Q*, incorporating a maximization over actions to reflect optimal decision-making.

```math
\begin{align*}\nV^*(s) &= \max_a \underE{s'\sim P}{r(s,a) + \gamma V^*(s')}, \\nQ^*(s,a) &= \underE{s'\sim P}{r(s,a) + \gamma \max_{a'} Q^*(s',a')}.\n\end{align*}
```

--------------------------------

### Markov Decision Process (MDP) Formal Definition (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

This defines a Markov Decision Process (MDP) as a 5-tuple, providing the standard mathematical formalism for sequential decision-making problems in reinforcement learning.

```math
\langle S, A, R, P, \rho_0 \rangle
```

--------------------------------

### Building Deterministic Policy Network PyTorch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

This snippet defines a simple multi-layer perceptron (MLP) using PyTorch's `nn.Sequential` module. It creates a deterministic policy network suitable for continuous action spaces, consisting of two hidden layers with Tanh activation and input/output layers matching observation and action dimensions.

```python
pi_net = nn.Sequential(
                  nn.Linear(obs_dim, 64),
                  nn.Tanh(),
                  nn.Linear(64, 64),
                  nn.Tanh(),
                  nn.Linear(64, act_dim)
                )
```

--------------------------------

### Performing a Policy Gradient Update Step in PyTorch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

This snippet demonstrates the standard pattern for performing a single gradient descent step in PyTorch for a policy gradient update. It clears previous gradients, computes the loss using batched observations, actions, and weights, performs a backward pass to compute gradients, and finally updates the model parameters using the optimizer. It requires an optimizer and a 'compute_loss' function.

```Python
optimizer.zero_grad()
batch_loss = compute_loss(obs=torch.as_tensor(batch_obs, dtype=torch.float32),
 act=torch.as_tensor(batch_acts, dtype=torch.int32),
 weights=torch.as_tensor(batch_weights, dtype=torch.float32)
 )
batch_loss.backward()
optimizer.step()
return batch_loss, batch_rets, batch_lens
```

--------------------------------

### Creating Policy Network and Action Selection in PyTorch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

This snippet defines the neural network (`logits_net`) that outputs action logits, a function (`get_policy`) to convert logits into a categorical distribution using PyTorch's `Categorical`, and a function (`get_action`) to sample an integer action from the distribution for a single observation. It relies on an `mlp` function (presumably a Multi-Layer Perceptron builder) and PyTorch's `Categorical` distribution. `get_action` uses `.item()` assuming a single observation input.

```python
# make core of policy network
logits_net = mlp(sizes=[obs_dim]+hidden_sizes+[n_acts])

# make function to compute action distribution
def get_policy(obs):
    logits = logits_net(obs)
    return Categorical(logits=logits)

# make action selection function (outputs int actions, sampled from policy)
def get_action(obs):
    return get_policy(obs).sample().item()
```

--------------------------------

### Reward-to-Go Policy Gradient Batch Weights (Modified) - Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

Shows the modified method for calculating batch weights using the reward-to-go concept. Each action's log probability at time t is weighted by the sum of rewards from time t onwards, calculated using the reward_to_go function.

```python
# the weight for each logprob(a_t|s_t) is reward-to-go from t
batch_weights += list(reward_to_go(ep_rews))
```

--------------------------------

### Average PyTorch Gradients with MPI

Source: https://github.com/openai/spinningup/blob/master/docs/utils/mpi.rst

This snippet illustrates the standard pattern for integrating `mpi_avg_grads` into a PyTorch training loop. It shows how to average the gradient buffers across all MPI processes immediately after the backward pass and before the optimizer takes a step, enabling data-parallel training.

```python
optimizer.zero_grad()
loss = compute_loss(module)
loss.backward()
mpi_avg_grads(module)   # averages gradient buffers across MPI processes!
optimizer.step()
```

--------------------------------

### Simple Policy Gradient Batch Weights (Original) - Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

Shows the original method for calculating batch weights in a simple policy gradient implementation. Each action's log probability is weighted by the total return (ep_ret) of the entire episode (tau).

```python
# the weight for each logprob(a|s) is R(tau)
batch_weights += [ep_ret] * ep_len
```

--------------------------------

### Saving Logger State (Default) - Python

Source: https://github.com/openai/spinningup/blob/master/docs/user/saving_and_loading.rst

This is the default line of code found in Spinning Up algorithms for saving the logger state. The `None` argument typically indicates that only the latest state snapshot is saved, overwriting previous ones.

```python
logger.save_state({'env': env}, None)
```

--------------------------------

### Sampling Actions from Loaded PyTorch Model

Source: https://github.com/openai/spinningup/blob/master/docs/utils/logger.rst

Uses the `act` method of a loaded PyTorch actor-critic model (`ac`) to sample actions given an observation tensor (`obs`). The observation is cast to `torch.float32` before being passed to the model.

```python
actions = ac.act(torch.as_tensor(obs, dtype=torch.float32))
```

--------------------------------

### DDPG Loss Calculation (TensorFlow)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercises.rst

Shows the TensorFlow implementation for calculating the Bellman backup target for the Q function and the policy (`pi_loss`) and Q function (`q_loss`) losses in a Deep Deterministic Policy Gradient (DDPG) algorithm. The backup target uses `tf.stop_gradient` to prevent gradients from flowing through the target network. The policy loss maximizes the Q value of the chosen action, and the Q loss minimizes the squared difference between the predicted Q value and the Bellman backup target.

```Python
# Bellman backup for Q function
backup = tf.stop_gradient(r_ph + gamma*(1-d_ph)*q_pi_targ)

# DDPG losses
pi_loss = -tf.reduce_mean(q_pi)
q_loss = tf.reduce_mean((q-backup)**2)
```

--------------------------------

### Advantage Function Definition (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro.rst

This equation defines the advantage function A^pi(s,a) as the difference between the action-value Q^pi(s,a) and state-value V^pi(s) functions, quantifying the relative benefit of taking action 'a' compared to the policy's average.

```math
A^{\pi}(s,a) = Q^{\pi}(s,a) - V^{\pi}(s).
```

--------------------------------

### Saving Logger State (with Epoch) - Python

Source: https://github.com/openai/spinningup/blob/master/docs/user/saving_and_loading.rst

Modify the default logger state saving call by passing the current `epoch` instead of `None`. This enables saving multiple policy snapshots at different training iterations, allowing evaluation of the agent's performance throughout training. Requires adjusting the `save_freq` setting.

```python
logger.save_state({'env': env}, epoch)
```

--------------------------------

### Calculate Reward-to-Go in Python

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

Implements the reward-to-go calculation for a list of rewards. It iterates backward through the rewards, accumulating the sum of future rewards at each step. This is used as the weight for the log probability of the action taken at that step.

```python
def reward_to_go(rews):
    n = len(rews)
    rtgs = np.zeros_like(rews)
    for i in reversed(range(n)):
        rtgs[i] = rews[i] + (rtgs[i+1] if i+1 < n else 0)
    return rtgs
```

--------------------------------

### SAC Entropy-Regularized Bellman Equation for Q

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/sac.rst

This equation shows the recursive definition of the entropy-regularized Q-function, expanding the entropy term into the expected negative log-probability of the action. It forms the basis for the Q-learning target.

```LaTeX
Q^{\pi}(s,a) &= \underE{s' \sim P \\ a' \sim \pi}{R(s,a,s') + \gamma\left(Q^{\pi}(s',a') + \alpha H\left(\pi(\cdot|s')\right) \right)} \\
&= \underE{s' \sim P \\ a' \sim \pi}{R(s,a,s') + \gamma\left(Q^{\pi}(s',a') - \alpha \log \pi(a'|s') \right)}
```

--------------------------------

### Project Dependencies List

Source: https://github.com/openai/spinningup/blob/master/docs/docs_requirements.txt

Specifies the necessary Python libraries and their version constraints for the project. This list is commonly used with package managers like pip to ensure the correct environment is set up.

```Python
cloudpickle~=1.2.1
gym~=0.15.3
ipython
joblib
matplotlib
numpy
pandas
pytest
psutil
scipy
seaborn==0.8.1
sphinx==1.5.6
sphinx-autobuild==0.7.1
sphinx-rtd-theme==0.4.1
tensorflow>=1.8.0,<2.0
tqdm
```

--------------------------------

### SAC Sampled Bellman Target Approximation

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/sac.rst

This equation provides a sample-based approximation of the entropy-regularized Bellman target for the Q-function. It shows how the target is computed using a sample reward, next state from the replay buffer, and a next action sampled from the current policy.

```LaTeX
Q^{\pi}(s,a) &\approx r + \gamma\left(Q^{\pi}(s',\tilde{a}') - \alpha \log \pi(\tilde{a}'|s') \right), \;;;;;  \tilde{a}' \sim \pi(\cdot|s').
```

--------------------------------

### Polyak Averaging Update for Target Networks (Math)

Source: https://github.com/openai/spinningup/blob/master/docs/algorithms/ddpg.rst

Describes the polyak averaging method used to update the parameters of target networks in DDPG-style algorithms. The target parameters are updated as a weighted average of the old target parameters and the current main network parameters.

```Math
\phi_{\text{targ}} \leftarrow \rho \phi_{\text{targ}} + (1 - \rho) \phi
```

--------------------------------

### Correct PyTorch DDPG Q-Function Class

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Defines the correct multi-layer perceptron (MLP) based Q-function class for DDPG in PyTorch. The `forward` method computes the Q-value and explicitly squeezes the output tensor along the last dimension to ensure it has shape `[batch size]`.

```python
"""
Correct Q-Function
"""
class MLPQFunction(nn.Module):

    def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
        super().__init__()
        self.q = mlp([obs_dim + act_dim] + list(hidden_sizes) + [1], activation)

    def forward(self, obs, act):
        q = self.q(torch.cat([obs, act], dim=-1))
        return torch.squeeze(q, -1) # Critical to ensure q has right shape.
```

--------------------------------

### Defining Policy Gradient Loss Function in PyTorch

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/rl_intro3.rst

This snippet defines the loss function (`compute_loss`) used for the policy gradient update. It calculates the log probability of the taken action (`act`) given the observation (`obs`) using the previously defined `get_policy` function, multiplies it by corresponding `weights` (presumably returns or advantages), and takes the negative mean to form the loss. Minimizing this loss function with respect to policy parameters performs the policy gradient update.

```python
# make loss function whose gradient, for the right data, is policy gradient
def compute_loss(obs, act, weights):
    logp = get_policy(obs).log_prob(act)
    return -(logp * weights).mean()
```

--------------------------------

### Evaluating TensorFlow Tensor z1 (Python)

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Runs a TensorFlow session to evaluate the tensor `z1`. The resulting 5x5 array demonstrates the output shape, likely related to tensor operations discussed in the text, such as broadcasting between tensors of shapes [5] and [5,1].

```Python
>>> sess = tf.InteractiveSession()
>>> sess.run(z1)
array([[ 0,  0,  0,  0,  0],
       [ 0,  1,  2,  3,  4],
       [ 0,  2,  4,  6,  8],
       [ 0,  3,  6,  9, 12],
       [ 0,  4,  8, 12, 16]])
```

--------------------------------

### Bugged PyTorch DDPG Q-Function Class

Source: https://github.com/openai/spinningup/blob/master/docs/spinningup/exercise2_2_soln.rst

Defines the bugged MLP-based Q-function class for DDPG in PyTorch. The `forward` method computes the Q-value but *does not* squeeze the output tensor, leaving it with shape `[batch size, 1]`. This mirrors the bug in the Tensorflow version.

```python
"""
Bugged Q-Function
"""
class BuggedMLPQFunction(nn.Module):

    def __init__(self, obs_dim, act_dim, hidden_sizes, activation):
        super().__init__()
        self.q = mlp([obs_dim + act_dim] + list(hidden_sizes) + [1], activation)

    def forward(self, obs, act):
        return self.q(torch.cat([obs, act], dim=-1))
```