### Install gym-retro

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_retro.html

Install the gym-retro package, which is required to use the OpenAI Retro environment adapter.

```bash
pip3 install gym-retro
```

--------------------------------

### Install OpenAI Gym dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_gym.html

Commands to install the OpenAI Gym library and its full suite of environments.

```bash
pip3 install gym
pip3 install gym[all]
```

--------------------------------

### Initializing a PPO agent

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Example of initializing a Proximal Policy Optimization (PPO) agent.

```python
agent = Agent.create(
    agent='ppo', environment=environment, batch_size=10, learning_rate=1e-3
)
```

--------------------------------

### Install ALE dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/ale.html

Install the system dependencies required by the Arcade Learning Environment adapter before using it.

```bash
sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake

```

--------------------------------

### Install stable Tensorforce from PyPI

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Installs the latest stable version of Tensorforce from PyPI.

```bash
pip3 install tensorforce

```

--------------------------------

### Install PyGame Learning Environment dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/ple.html

Install the system dependencies and Python packages required to use the PyGame Learning Environment adapter. Run the apt-get command first, then the two pip3 install commands in order.

```bash
sudo apt-get install git python3-dev python3-setuptools python3-numpy python3-opengl     libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsmpeg-dev libsdl1.2-dev     libportmidi-dev libswscale-dev libavformat-dev libavcodec-dev libtiff5-dev libx11-6     libx11-dev fluid-soundfont-gm timgm6mb-soundfont xfonts-base xfonts-100dpi xfonts-75dpi     xfonts-cyrillic fontconfig fonts-freefont-ttf libfreetype6-dev
```

```bash
pip3 install pygame
```

```bash
pip3 install git+https://github.com/ntasfi/PyGame-Learning-Environment.git
```

--------------------------------

### Install ViZDoom dependencies and package

Source: https://tensorforce.readthedocs.io/en/latest/environments/vizdoom.html

System dependencies and the vizdoom Python package required to use the ViZDoom environment adapter.

```bash
sudo apt-get install g++ build-essential libsdl2-dev zlib1g-dev libmpg123-dev libjpeg-dev     libsndfile1-dev nasm tar libbz2-dev libgtk2.0-dev make cmake git chrpath timidity     libfluidsynth-dev libgme-dev libopenal-dev timidity libwildmidi-dev unzip libboost-all-dev     liblua5.1-dev

pip3 install vizdoom
```

--------------------------------

### Specify state and reward preprocessing in Agent.create

Source: https://tensorforce.readthedocs.io/en/latest/modules/preprocessing.html

Example of how to define reward_estimation and state_preprocessing configurations within the Agent.create method.

```python
Agent.create(
    ...
    reward_estimation=dict(
        ...
        reward_processing=dict(type='clipping', lower=-1.0, upper=1.0)
    ),
    state_preprocessing=[
        dict(type='image', height=4, width=4, grayscale=True),
        dict(type='exponential_normalization', decay=0.999)
    ],
    ...
)
```

--------------------------------

### Install latest Tensorforce from GitHub

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Clones the Tensorforce repository and installs the latest version in editable mode.

```bash
git clone https://github.com/tensorforce/tensorforce.git
cd tensorforce
pip3 install -e .

```

--------------------------------

### Initializing a generic Tensorforce agent

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Example of initializing a generic agent with specific optimizer, objective, and reward estimation settings.

```python
agent = Agent.create(
    agent='tensorforce', environment=environment, update=64,
    optimizer=dict(optimizer='adam', learning_rate=1e-3),
    objective='policy_gradient', reward_estimation=dict(horizon=20)
)
```

--------------------------------

### Environment.reset

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Resets the environment to start a new episode.

```APIDOC
## Environment.reset

### Description
Resets the environment to start a new episode.

### Method
instance

### Signature
`Environment.reset(num_parallel=None)`

### Parameters
- **num_parallel** (int >= 1) - Optional - Number of environment instances executed in parallel, only valid if environment is vectorizable (no vectorization).

### Returns
- **(parallel,) dict[state]** - Dictionary containing initial state(s) and auxiliary information, and parallel index vector in case of vectorized execution.
```

--------------------------------

### Configure Agent memory with capacity

Source: https://tensorforce.readthedocs.io/en/latest/modules/memories.html

Example of specifying a Replay memory with a capacity of 10000 using an integer shorthand.

```python
Agent.create(
    ...
    memory=10000,
    ...
)
```

--------------------------------

### Configure Adam optimizer with OptimizerWrapper

Source: https://tensorforce.readthedocs.io/en/latest/modules/optimizers.html

Shows how to specify a customized Adam optimizer via the OptimizerWrapper by passing a dictionary to the optimizer argument of Agent.create. The example sets learning_rate, clipping_threshold, multi_step, subsampling_fraction, linesearch_iterations, and doublecheck_update.

```python
Agent.create(
    ...
    optimizer=dict(
        optimizer='adam', learning_rate=1e-3, clipping_threshold=1e-2,
        multi_step=10, subsampling_fraction=64, linesearch_iterations=5,
        doublecheck_update=True
    ),
    ...
)
```

--------------------------------

### Install Tensorforce on M1 Macs without dependencies

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Installs Tensorforce on M1 Macs after manually installing its dependencies, ignoring the tensorflow dependency.

```bash
pip3 install tensorforce --no-deps

```

--------------------------------

### Customize policy distributions with Agent.create

Source: https://tensorforce.readthedocs.io/en/latest/modules/distributions.html

Configure a policy with custom distributions via the `distributions` argument. The example sets a Gaussian distribution for continuous actions with global standard deviation mode and a Beta distribution for bounded actions.

```python
Agent.create(
    ...
    policy=dict(distributions=dict(
        float=dict(type='gaussian', stddev_mode='global'),
        bounded_action=dict(type='beta')
    ))
    ...
)

```

--------------------------------

### Dockerfile for latest Tensorforce from GitHub

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Dockerfile to run the latest Tensorforce version by cloning the GitHub repository and installing it in editable mode.

```dockerfile
FROM python:3.8
RUN \
  git clone https://github.com/tensorforce/tensorforce.git && \
  pip3 install -e tensorforce

```

--------------------------------

### Specify multi-input network with Register and Retrieve layers

Source: https://tensorforce.readthedocs.io/en/latest/modules/networks.html

Specifies a multi-input, non-sequential network using nested lists of layers. Each inner list forms a sequential component, and the special layers 'retrieve' and 'register' combine these components. The example uses two input states (observation and attributes) and concatenates their embeddings before a final dense layer.

```python
Agent.create(
    states=dict(
        observation=dict(type='float', shape=(16, 16, 3), min_value=-1.0, max_value=1.0),
        attributes=dict(type='int', shape=(4, 2), num_values=5)
    ),
    ...
    policy=[
        [
            dict(type='retrieve', tensors=['observation']),
            dict(type='conv2d', size=32),
            dict(type='flatten'),
            dict(type='register', tensor='obs-embedding')
        ],
        [
            dict(type='retrieve', tensors=['attributes']),
            dict(type='embedding', size=32),
            dict(type='flatten'),
            dict(type='register', tensor='attr-embedding')
        ],
        [
            dict(
                type='retrieve', aggregation='concat',
                tensors=['obs-embedding', 'attr-embedding']
            ),
            dict(type='dense', size=64)
        ]
    ],
    ...
)

```

--------------------------------

### TensorforceAgent.observe

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Observes reward and whether a terminal state is reached, needs to be preceded by act(). See the act-observe script for an example application as part of the act-observe interface.

```APIDOC
## TensorforceAgent.observe

### Description
Observes reward and whether a terminal state is reached, needs to be preceded by act(). See the act-observe script for an example application as part of the act-observe interface.

### Method
Instance method

### Signature
`TensorforceAgent.observe(reward=0.0, terminal=False, parallel=0)`

### Parameters
- **reward** (float | iter[float]) - Optional - Reward. Default: 0.0.
- **terminal** (bool | 0 | 1 | 2 | iter[..]) - Optional - Whether a terminal state is reached, or 2 if the episode was aborted. Default: false.
- **parallel** (int, iter[int]) - Optional - Parallel execution index. Default: 0.

### Returns
- **int** - Number of performed updates.
```

--------------------------------

### Loading agent configuration from a file

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

How to initialize an agent by passing the path to a JSON configuration file.

```python
agent = Agent.create(agent='agent.json', environment=environment)
```

--------------------------------

### Initialize OpenAI CartPole-v1 environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Access pre-defined Gym versions directly.

```python
environment = Environment.create(environment='gym', level='CartPole-v1')
```

--------------------------------

### Load environment from config file path

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Load an environment by passing the path to a configuration file.

```python
environment = Environment.create(
    environment='environment.json', max_episode_timesteps=500
)
```

--------------------------------

### ViZDoom

Source: https://tensorforce.readthedocs.io/en/latest/environments/vizdoom.html

Constructor for the ViZDoom environment adapter, which initializes a ViZDoom environment with configurable parameters.

```APIDOC
## ViZDoom

### Description
ViZDoom environment adapter (specification key: `vizdoom`).

### Constructor
`tensorforce.environments.ViZDoom(level, visualize=False, include_variables=False, factored_action=False, frame_skip=12, seed=None)`

### Parameters
- **level** (string) - Required - ViZDoom configuration file.
- **include_variables** (bool) - Optional - Whether to include game variables to state (default: false).
- **factored_action** (bool) - Optional - Whether to use factored action representation (default: false).
- **visualize** (bool) - Optional - Whether to visualize interaction (default: false).
- **frame_skip** (int > 0) - Optional - Number of times to repeat an action without observing (default: 12).
- **seed** (int) - Optional - Random seed (default: none).

### Installation Requirements
```
sudo apt-get install g++ build-essential libsdl2-dev zlib1g-dev libmpg123-dev libjpeg-dev libsndfile1-dev nasm tar libbz2-dev libgtk2.0-dev make cmake git chrpath timidity libfluidsynth-dev libgme-dev libopenal-dev timidity libwildmidi-dev unzip libboost-all-dev liblua5.1-dev

pip3 install vizdoom
```
```

--------------------------------

### OpenAIGym constructor

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_gym.html

Creates an OpenAI Gym environment adapter. The level parameter is required and can be a Gym id string or a Gym environment instance. Optional parameters control visualization, value clipping, terminal rewards, reward thresholds, state dropping, and additional Gym arguments.

```APIDOC
## OpenAIGym(level, visualize=False, import_modules=None, min_value=None, max_value=None, terminal_reward=0.0, reward_threshold=None, drop_states_indices=None, visualize_directory=None, **kwargs)

### Description
Creates an OpenAI Gym environment adapter for Tensorforce. The adapter allows Tensorforce agents to interact with OpenAI Gym environments.

### Method
Constructor

### Parameters
- **level** (string | gym.Env) - Required - Gym id or instance.
- **visualize** (bool) - Optional - Whether to visualize interaction (default: false).
- **import_modules** (list) - Optional - Modules to import (default: none).
- **min_value** (float) - Optional - Lower bound clipping for otherwise unbounded state values (default: no clipping).
- **max_value** (float) - Optional - Upper bound clipping for otherwise unbounded state values (default: no clipping).
- **terminal_reward** (float) - Optional - Additional reward for early termination (default: Gym default).
- **reward_threshold** (float) - Optional - Gym environment argument, the reward threshold before the task is considered solved (default: Gym default).
- **drop_states_indices** (list[int]) - Optional - Drop states indices (default: none).
- **visualize_directory** (string) - Optional - Visualization output directory (default: none).
- **kwargs** - Optional - Additional Gym environment arguments.

### Example
```python
from tensorforce.environments import OpenAIGym

environment = OpenAIGym(
    level='CartPole-v1',
    visualize=True,
    max_value=10.0
)
```
```

--------------------------------

### Run basic experiment with Runner

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Configure and run a basic training and evaluation experiment using the Runner utility.

```python
runner = Runner(
    agent='agent.json',
    environment=dict(environment='gym', level='CartPole'),
    max_episode_timesteps=500
)

runner.run(num_episodes=200)

runner.run(num_episodes=100, evaluation=True)

runner.close()
```

--------------------------------

### TensorforceAgent.act

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Returns action(s) for the given state(s), needs to be followed by observe() unless independent mode. See the act-observe script for an example application as part of the act-observe interface.

```APIDOC
## TensorforceAgent.act

### Description
Returns action(s) for the given state(s), needs to be followed by observe() unless independent mode. See the act-observe script for an example application as part of the act-observe interface.

### Method
Instance method

### Signature
`TensorforceAgent.act(states, internals=None, parallel=0, independent=False, deterministic=True, evaluation=None)`

### Parameters
- **states** (dict[state] | iter[dict[state]]) - Required - Dictionary containing state(s) to be acted on.
- **internals** (dict[internal] | iter[dict[internal]]) - Optional - Dictionary containing current internal agent state(s), either given by initial_internals() at the beginning of an episode or as return value of the preceding act() call. Required if independent mode and agent has internal states.
- **parallel** (int | iter[int]) - Optional - Parallel execution index. Default: 0.
- **independent** (bool) - Optional - Whether this act() call is not part of the training agent-environment interaction and thus not followed by observe(), meaning its inputs/outputs/internals are not stored in memory and not used in updates, e.g. for independent evaluation episodes which should not be learned from. Default: false.
- **deterministic** (bool) - Optional - Whether action should be chosen deterministically, so no action distribution sampling and no exploration, only valid in independent mode. Default: true.
- **evaluation** - Optional - Not described in source.

### Returns
- If internals argument given: dict[action] | iter[dict[action]], dict[internal] | iter[dict[internal]] - Dictionary containing action(s), dictionary containing next internal agent state(s) if independent mode.
- Otherwise: dict[action] | iter[dict[action]] - Dictionary containing action(s).
```

--------------------------------

### PyGameLearningEnvironment.__init__

Source: https://tensorforce.readthedocs.io/en/latest/environments/ple.html

Initializes a PyGame Learning Environment adapter with the specified game level and configuration options. This is the constructor for the environment class.

```APIDOC
## PyGameLearningEnvironment.__init__

### Description
Initializes a PyGame Learning Environment adapter with the specified game level and configuration options. This is the constructor for the environment class.

### Method
Constructor

### Signature
`PyGameLearningEnvironment(level, visualize=False, frame_skip=1, fps=30)`

### Parameters
- **level** (string | subclass of `ple.games.base`) - Required - Game instance or name of class in `ple.games`, like "Catcher", "Doom", "FlappyBird", "MonsterKong", "Pixelcopter", "Pong", "PuckWorld", "RaycastMaze", "Snake", "WaterWorld".
- **visualize** (bool) - Optional - Whether to visualize interaction. Default: false.
- **frame_skip** (int > 0) - Optional - Number of times to repeat an action without observing. Default: 1.
- **fps** (int > 0) - Optional - The desired frames per second we want to run our game at. Default: 30.

### Usage Example
```python
from tensorforce.environments import PyGameLearningEnvironment

# Create a PyGame Learning Environment adapter
env = PyGameLearningEnvironment(
    level="Catcher",
    visualize=False,
    frame_skip=1,
    fps=30
)
```

### Notes
- The environment adapter uses the specification key `ple` or `pygame_learning_environment`.
- May require additional system packages and Python packages as listed in the documentation.
```

--------------------------------

### Initialize OpenAI CartPole environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Initialize the OpenAI CartPole environment using the Environment.create interface.

```python
environment = Environment.create(
    environment='gym', level='CartPole', max_episode_timesteps=500
)
```

--------------------------------

### Load custom environment by module path

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Load a custom environment by specifying its module path.

```python
environment = Environment.create(
    environment='envs.custom_env', max_episode_timesteps=100
)
```

--------------------------------

### Environment configuration file

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Define an environment using a JSON configuration file.

```json
{
    "environment": "gym",
    "level": "CartPole"
}
```

--------------------------------

### Import Environment class

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Import the Environment class to begin creating environments.

```python
from tensorforce.environments import Environment
```

--------------------------------

### Implement manual act-experience-update training loop

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Use the act-experience-update interface for more flexibility, noting that some stateful network layers may not update correctly in independent-mode.

```python
# Train for 100 episodes
for _ in range(100):
    episode_states = list()
    episode_internals = list()
    episode_actions = list()
    episode_terminal = list()
    episode_reward = list()

    states = environment.reset()
    internals = agent.initial_internals()
    terminal = False
    while not terminal:
        episode_states.append(states)
        episode_internals.append(internals)
        actions, internals = agent.act(
            states=states, internals=internals, independent=True
        )
        episode_actions.append(actions)
        states, terminal, reward = environment.execute(actions=actions)
        episode_terminal.append(terminal)
        episode_reward.append(reward)

    agent.experience(
        states=episode_states, internals=episode_internals,
        actions=episode_actions, terminal=episode_terminal,
        reward=episode_reward
    )
    agent.update()
```

--------------------------------

### Configure TensorBoard Summarizer

Source: https://tensorforce.readthedocs.io/en/latest/basics/features.html

Enable TensorBoard logging by providing a summarizer configuration with specific labels to track during training.

```python
Agent.create(...
    summarizer=dict(
        directory='data/summaries',
        # list of labels, or 'all'
        labels=['entropy', 'kl-divergence', 'loss', 'reward', 'update-norm']
    ), ...
)
```

--------------------------------

### Implement a custom environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Create a custom environment by subclassing the Tensorforce Environment interface.

```python
class CustomEnvironment(Environment):

    def __init__(self):
        super().__init__()

    def states(self):
        return dict(type='float', shape=(8,))

    def actions(self):
        return dict(type='int', num_values=4)

    # Optional: should only be defined if environment has a natural fixed
    # maximum episode length; otherwise specify maximum number of training
    # timesteps via Environment.create(..., max_episode_timesteps=???)
    def max_episode_timesteps(self):
        return super().max_episode_timesteps()

    # Optional additional steps to close environment
    def close(self):
        super().close()

    def reset(self):
        state = np.random.random(size=(8,))
        return state

    def execute(self, actions):
        next_state = np.random.random(size=(8,))
        terminal = False  # Always False if no "natural" terminal state
        reward = np.random.random()
        return next_state, terminal, reward
```

--------------------------------

### TensorforceAgent.pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Simple pretraining approach as a combination of experience() and update, akin to behavioral cloning, using experience traces obtained e.g. via recording agent interactions.

```APIDOC
## TensorforceAgent.pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

### Description
Simple pretraining approach as a combination of experience() and update, akin to behavioral cloning, using experience traces obtained e.g. via recording agent interactions.

### Method
pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

### Parameters
- **directory** (path) - Required - Directory with experience traces, e.g. obtained via recorder; episode length has to be consistent with agent configuration.
- **num_iterations** (int > 0) - Required - Number of iterations consisting of loading new traces and performing multiple updates.
- **num_traces** (int > 0) - Optional (default: 1) - Number of traces to load per iteration; has to at least satisfy the update batch size.
- **num_updates** (int > 0) - Optional (default: 1) - Number of updates per iteration.
- **extension** (str) - Optional (default: ".npz") - Traces file extension to filter the given directory for.
```

--------------------------------

### OpenSim Constructor

Source: https://tensorforce.readthedocs.io/en/latest/environments/open_sim.html

Creates an OpenSim environment adapter instance. The level parameter is required and specifies the environment id, while visualize and integrator_accuracy are optional.

```APIDOC
## OpenSim(level, visualize=False, **kwargs)

### Description
Creates an OpenSim environment adapter instance. The level parameter is required and specifies the environment id, while visualize and integrator_accuracy are optional.

### Method
Constructor

### Signature
`tensorforce.environments.OpenSim(level, visualize=False, **kwargs)`

### Parameters
- **level** (_'Arm2D' | 'L2Run' | 'Prosthetics'_) - Required - Environment id.
- **visualize** (_bool_) - Optional - Whether to visualize interaction (default: false).
- **integrator_accuracy** (_float_) - Optional - Integrator accuracy (default: 5e-5).

### Specification Key
`osim`, `open_sim`
```

--------------------------------

### Environment.actions

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Returns the action space specification.

```APIDOC
## Environment.actions

### Description
Returns the action space specification.

### Method
instance

### Signature
`Environment.actions()`

### Parameters
None

### Returns
- **specification** - Arbitrarily nested dictionary of action descriptions with the following attributes:
  - **type** ("bool" | "int" | "float") - action data type (required).
  - **shape** (int > 0 | iter[int > 0]) - action shape (default: scalar).
  - **num_actions** (int > 0) - number of discrete action values (required for type "int").
  - **min_value/max_value** (float) - minimum/maximum action value (optional for type "float").
```

--------------------------------

### Environment.execute

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Executes the given action(s) and advances the environment by one step.

```APIDOC
## Environment.execute

### Description
Executes the given action(s) and advances the environment by one step.

### Method
instance

### Signature
`Environment.execute(actions)`

### Parameters
- **actions** (dict[action]) - Required - Dictionary containing action(s) to be executed.

### Returns
- **(parallel,) dict[state], bool | 0 | 1 | 2, float** - Dictionary containing next state(s) and auxiliary information, whether a terminal state is reached or 2 if the episode was aborted, observed reward, and parallel index vector in case of vectorized execution.
```

--------------------------------

### ProximalPolicyOptimization constructor

Source: https://tensorforce.readthedocs.io/en/latest/agents/ppo.html

Creates a Proximal Policy Optimization agent with the given configuration. The constructor accepts states and actions specifications, environment-related parameters, and various hyperparameters for the PPO algorithm.

```APIDOC
## ProximalPolicyOptimization(states, actions, max_episode_timesteps, batch_size, network='auto', use_beta_distribution=False, memory='minimum', update_frequency=1.0, learning_rate=0.001, multi_step=10, subsampling_fraction=0.33, likelihood_ratio_clipping=0.25, discount=0.99, reward_processing=None, return_processing=None, advantage_processing=None, predict_terminal_values=False, baseline=None, baseline_optimizer=None, state_preprocessing='linear_normalization', exploration=0.0, variable_noise=0.0, l2_regularization=0.0, entropy_regularization=0.0, parallel_interactions=1, config=None, saver=None, summarizer=None, tracking=None, recorder=None, **kwargs)

### Description
Creates a Proximal Policy Optimization agent. This constructor initializes the agent with the given states and actions specifications, environment parameters, and PPO-specific hyperparameters.

### Parameters
- **states** (specification) - Required - States specification, arbitrarily nested dictionary of state descriptions with attributes: type ("bool" | "int" | "float"), shape (int | iter[int]), num_values (int > 0, required for type "int"), min_value/max_value (float, optional for type "float").
- **actions** (specification) - Required - Actions specification, arbitrarily nested dictionary of action descriptions with attributes: type ("bool" | "int" | "float"), shape (int > 0 | iter[int > 0]), num_values (int > 0, required for type "int"), min_value/max_value (float, optional for type "float").
- **max_episode_timesteps** (int > 0) - Optional - Upper bound for number of timesteps per episode. Default: not given, better implicitly specified via environment argument.
- **batch_size** (parameter, int > 0) - Required - Number of episodes per update batch.
- **network** ("auto" | specification) - Optional - Policy network configuration. Default: "auto".
- **use_beta_distribution** (bool) - Optional - Whether to use the Beta distribution for bounded continuous actions. Default: false.
- **memory** (int > 0) - Optional - Batch memory capacity, must fit at least maximum batch_size + 1 episodes. Default: minimum capacity.
- **update_frequency** ("never" | parameter, int > 0 | 0.0 < float <= 1.0) - Optional - Frequency of updates, relative to batch_size if float. Default: batch_size.
- **learning_rate** (parameter, float > 0.0) - Optional - Optimizer learning rate. Default: 1e-3.
- **multi_step** (parameter, int >= 1) - Optional - Number of optimization steps. Default: 10.
- **subsampling_fraction** (parameter, int > 0 | 0.0 < float <= 1.0) - Optional - Absolute/relative fraction of batch timesteps to subsample. Default: 0.33.
- **likelihood_ratio_clipping** (parameter, float > 0.0) - Optional - Likelihood-ratio clipping threshold. Default: 0.25.
- **discount** (parameter, 0.0 <= float <= 1.0) - Optional - Discount factor for future rewards. Default: 0.99.
- **return_processing** (specification) - Optional - Return processing as layer or list of layers. Default: no return processing.
- **advantage_processing** (specification) - Optional - Advantage processing as layer or list of layers. Default: no advantage processing.
- **predict_terminal_values** (bool) - Optional - Whether to predict the value of terminal states. Default: false.
- **reward_processing** (specification) - Optional - Reward preprocessing as layer or list of layers. Default: no reward processing.
- **baseline** (specification) - Optional - Baseline network configuration. Default: none.
- **baseline_optimizer** (specification) - Optional - Optimizer for the baseline network. Default: none.
- **state_preprocessing** (specification) - Optional - State preprocessing configuration. Default: 'linear_normalization'.
- **exploration** (float) - Optional - Exploration rate. Default: 0.0.
- **variable_noise** (float) - Optional - Variable noise. Default: 0.0.
- **l2_regularization** (float) - Optional - L2 regularization coefficient. Default: 0.0.
- **entropy_regularization** (float) - Optional - Entropy regularization coefficient. Default: 0.0.
- **parallel_interactions** (int) - Optional - Number of parallel interactions. Default: 1.
- **config** (specification) - Optional - Configuration. Default: None.
- **saver** (specification) - Optional - Saver configuration. Default: None.
- **summarizer** (specification) - Optional - Summarizer configuration. Default: None.
- **tracking** (specification) - Optional - Tracking configuration. Default: None.
- **recorder** (specification) - Optional - Recorder configuration. Default: None.
- **kwargs** - Optional - Additional keyword arguments.

### Returns
An instance of the ProximalPolicyOptimization agent.

```

--------------------------------

### Environment.create

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Creates an environment from a specification. Supports various remote execution modes and reward shaping.

```APIDOC
## Environment.create

### Description
Creates an environment from a specification. In case of "socket-server" remote mode, runs environment in server communication loop until closed.

### Method
static

### Signature
`Environment.create(environment=None, max_episode_timesteps=None, reward_shaping=None, remote=None, blocking=False, host=None, port=None, **kwargs)`

### Parameters
- **environment** (specification | Environment class/object) - Required - JSON file, specification key, configuration dictionary, library module, `Environment` class/object, or gym.Env. Invalid for "socket-client" remote mode.
- **max_episode_timesteps** (int > 0) - Optional - Maximum number of timesteps per episode, overwrites the environment default if defined. Default: environment default. Invalid for "socket-client" remote mode.
- **reward_shaping** (callable [(s,a,t,r,s')->r|(r,t)] | str) - Optional - Reward shaping function mapping state, action, terminal, reward and next state to shaped reward and terminal, or a string expression with arguments "states", "actions", "terminal", "reward" and "next_states", e.g. "-1.0 if terminal else max(reward, 0.0)". Default: no reward shaping.
- **remote** ("multiprocessing" | "socket-client" | "socket-server") - Optional - Communication mode for remote environment execution of parallelized environment execution. "socket-client" mode requires a corresponding "socket-server" running, and "socket-server" mode runs environment in server communication loop until closed. Default: local execution.
- **blocking** (bool) - Optional - Whether remote environment calls should be blocking. Default: not blocking. Invalid unless "multiprocessing" or "socket-client" remote mode.
- **host** (str) - Optional - Socket server hostname or IP address. Required only for "socket-client" remote mode.
- **port** (int) - Optional - Socket server port. Required only for "socket-client/server" remote mode.
- **kwargs** - Optional - Additional arguments.

### Returns
None
```

--------------------------------

### Implement manual act-observe training loop

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Define a custom training loop using the act-observe interaction pattern for detailed control.

```python
# Create agent and environment
environment = Environment.create(
    environment='environment.json', max_episode_timesteps=500
)
agent = Agent.create(agent='agent.json', environment=environment)

# Train for 100 episodes
for _ in range(100):
    states = environment.reset()
    terminal = False
    while not terminal:
        actions = agent.act(states=states)
        states, terminal, reward = environment.execute(actions=actions)
        agent.observe(terminal=terminal, reward=reward)
```

--------------------------------

### Load custom environment object

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Load a custom environment by passing the class object directly to Environment.create.

```python
environment = Environment.create(
    environment=CustomEnvironment, max_episode_timesteps=100
)
```

--------------------------------

### Short-form policy specification with list or network='auto'

Source: https://tensorforce.readthedocs.io/en/latest/modules/policies.html

Two short-form policy specifications for Agent.create. The first uses a list of layer dictionaries to define a sequential network; the second uses the 'auto' network setting. Both are valid alternatives for the policy argument.

```python
Agent.create(
    ...
    policy=[
        dict(type='dense', size=64, activation='tanh'),
        dict(type='dense', size=64, activation='tanh')
    ],
    ...
)

```

```python
Agent.create(
    ...
    policy=dict(network='auto'),
    ...
)

```

--------------------------------

### VanillaPolicyGradient constructor

Source: https://tensorforce.readthedocs.io/en/latest/agents/vpg.html

Creates a Vanilla Policy Gradient (REINFORCE) agent with the specified configuration parameters.

```APIDOC
## VanillaPolicyGradient(states, actions, max_episode_timesteps, batch_size, network='auto', use_beta_distribution=False, memory='minimum', update_frequency=1.0, learning_rate=0.001, discount=0.99, reward_processing=None, return_processing=None, advantage_processing=None, predict_terminal_values=False, baseline=None, baseline_optimizer=None, state_preprocessing='linear_normalization', exploration=0.0, variable_noise=0.0, l2_regularization=0.0, entropy_regularization=0.0, parallel_interactions=1, config=None, saver=None, summarizer=None, tracking=None, recorder=None, **kwargs)

### Description
Creates a Vanilla Policy Gradient (REINFORCE) agent. This agent is used for reinforcement learning with policy gradient methods.

### Parameters
- **states** (specification) - Required - States specification, arbitrarily nested dictionary of state descriptions. Attributes: type ("bool"|"int"|"float") default "float", shape (int|iter[int]) required, num_values (int > 0) required for type "int", min_value/max_value (float) optional for type "float".
- **actions** (specification) - Required - Actions specification, arbitrarily nested dictionary of action descriptions. Attributes: type ("bool"|"int"|"float") required, shape (int > 0 | iter[int > 0]) default scalar, num_values (int > 0) required for type "int", min_value/max_value (float) optional for type "float".
- **max_episode_timesteps** (int > 0) - Optional - Upper bound for number of timesteps per episode. Default: not given, better implicitly specified via environment argument for Agent.create(...).
- **batch_size** (parameter, int > 0) - Required - Number of episodes per update batch.
- **network** ("auto" | specification) - Optional - Policy network configuration. Default: "auto", automatically configured network.
- **use_beta_distribution** (bool) - Optional - Whether to use the Beta distribution for bounded continuous actions by default. Default: false.
- **memory** (int > 0) - Optional - Batch memory capacity, has to fit at least maximum batch_size + 1 episodes. Default: minimum capacity.
- **update_frequency** ("never" | parameter, int > 0 | 0.0 < float <= 1.0) - Optional - Frequency of updates, relative to batch_size if float. Default: batch_size.
- **learning_rate** (parameter, float > 0.0) - Optional - Optimizer learning rate. Default: 1e-3.
- **discount** (parameter, 0.0 <= float <= 1.0) - Optional - Discount factor for future rewards of discounted-sum reward estimation. Default: 0.99.
- **return_processing** (specification) - Optional - Return processing as layer or list of layers. Default: no return processing.
- **advantage_processing** (specification) - Optional - Advantage processing as layer or list of layers. Default: no advantage processing.
- **predict_terminal_values** (bool) - Optional - Whether to predict the value of terminal states. Default: false.
- **reward_processing** (specification) - Optional - Reward preprocessing as layer or list of layers. Default: no reward processing.
- **baseline** (specification) - Optional - Baseline network configuration. Default: none.
- **baseline_optimizer** (float > 0.0 | specification) - Optional - Baseline optimizer configuration. Default: none.
- **l2_regularization** (parameter, float >= 0.0) - Optional - L2 regularization loss weight. Default: no L2 regularization.
- **entropy_regularization** (parameter, float >= 0.0) - Optional - Entropy regularization loss weight. Default: no entropy regularization.

### Notes
- The agent is also known by specification key `vpg` or `reinforce`.
- States and actions are better implicitly specified via the `environment` argument for `Agent.create(...)`.
- The `state_preprocessing`, `exploration`, `variable_noise`, `parallel_interactions`, `config`, `saver`, `summarizer`, `tracking`, and `recorder` parameters are listed in the constructor signature but not described in the source text.
```

--------------------------------

### run

Source: https://tensorforce.readthedocs.io/en/latest/execution/runner.html

The run method executes an experiment with configurable limits, parallelization, callbacks, progress display, and evaluation options.

```APIDOC
## run

### Description
Runs an experiment with the specified number of episodes, timesteps, or updates. Supports parallel environments, callbacks, progress bars, and evaluation mode.

### Method
run

### Parameters
- **num_episodes** (int > 0) - Optional - Number of episodes to run experiment, sum of episodes across all parallel/vectorized environment(s) / actors in a multi-actor environment (default: no episode limit).
- **num_timesteps** (int > 0) - Optional - Number of timesteps to run experiment, sum of timesteps across all parallel/vectorized environment(s) / actors in a multi-actor environment (default: no timestep limit).
- **num_updates** (int > 0) - Optional - Number of agent updates to run experiment (default: no update limit).
- **batch_agent_calls** (bool) - Optional - Whether to batch agent calls for parallel environment execution (default: false, separate call per environment).
- **sync_timesteps** (bool) - Optional - Whether to synchronize parallel environment execution on timestep-level, implied by batch_agent_calls (default: false, unless batch_agent_calls is true).
- **sync_episodes** (bool) - Optional - Whether to synchronize parallel environment execution on episode-level (default: false).
- **num_sleep_secs** (float) - Optional - Sleep duration if no environment is ready (default: one milliseconds).
- **callback** (callable[Runner, parallel] -> bool) - Optional - Callback function taking the runner instance plus parallel index and returning a boolean value indicating whether execution should continue (default: callback always true).
- **callback_episode_frequency** (int) - Optional - Episode interval between callbacks (default: every episode).
- **callback_timestep_frequency** (int) - Optional - Timestep interval between callbacks (default: not specified).
- **use_tqdm** (bool) - Optional - Whether to display a tqdm progress bar for the experiment run (default: true), with the following additional information (averaged over number of episodes given via mean_horizon):
  - return – cumulative episode return
  - ts/ep – timesteps per episode
  - sec/ep – seconds per episode
  - ms/ts – milliseconds per timestep
  - agent – percentage of time spent on agent computation
  - comm – if remote environment execution, percentage of time spent on communication
- **mean_horizon** (int) - Optional - Number of episodes progress bar values and evaluation score are averaged over (default: not averaged).
- **evaluation** (bool) - Optional - Whether to run in evaluation mode, only valid if single environment (default: no evaluation).
- **save_best_agent** (string) - Optional - Directory to save the best version of the agent according to the evaluation score (default: best agent is not saved).
- **evaluation_callback** (int | callable[Runner -> float]) - Optional - Callback function taking the runner instance and returning an evaluation score (default: cumulative evaluation return averaged over mean_horizon episodes).

### Returns
None explicitly documented.
```

--------------------------------

### Execute Distributed Environments via Socket

Source: https://tensorforce.readthedocs.io/en/latest/basics/features.html

Run environments across different machines using socket-server and socket-client configurations.

```bash
# Environment machine 1
python run.py --environment gym --level CartPole-v1 --remote socket-server \
    --port 65432

# Environment machine 2
python run.py --environment gym --level CartPole-v1 --remote socket-server \
    --port 65433

# Agent machine
python run.py --agent benchmarks/configs/ppo1.json --episodes 100 \
    --num-parallel 2 --remote socket-client --host 127.0.0.1,127.0.0.1 \
    --port 65432,65433 --batch-agent-calls
```

--------------------------------

### Import Runner utility

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Import the Runner class from the tensorforce.execution module.

```python
from tensorforce.execution import Runner
```

--------------------------------

### Full parametrized_distributions policy with custom distributions and decaying temperature

Source: https://tensorforce.readthedocs.io/en/latest/modules/policies.html

Full parametrized-distributions policy specification with a custom network, per-action distributions (gaussian for float, beta for bounded_action), and a decaying temperature schedule. Use when you need customized distributions and temperature decay.

```python
Agent.create(
    ...
    policy=dict(
        type='parametrized_distributions',
        network=[
            dict(type='dense', size=64, activation='tanh'),
            dict(type='dense', size=64, activation='tanh')
        ],
        distributions=dict(
            float=dict(type='gaussian', stddev_mode='global'),
            bounded_action=dict(type='beta')
        ),
        temperature=dict(
            type='decaying', decay='exponential', unit='episodes',
            num_steps=100, initial_value=0.01, decay_rate=0.5
        )
    )
    ...
)

```

--------------------------------

### Defining agent configuration in JSON

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

A sample JSON configuration file for a Tensorforce agent.

```json
{
    "agent": "tensorforce",
    "update": 64,
    "optimizer": {
        "optimizer": "adam",
        "learning_rate": 1e-3
    },
    "objective": "policy_gradient",
    "reward_estimation": {
        "horizon": 20
    }
}
```

--------------------------------

### Build Tensorforce Docker container

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Builds the Docker container from the Dockerfile in the current directory.

```bash
docker build .

```

--------------------------------

### Beta

Source: https://tensorforce.readthedocs.io/en/latest/modules/distributions.html

Beta distribution, for bounded continuous actions (specification key: `beta`).

```APIDOC
## Beta

### Description
Beta distribution, for bounded continuous actions (specification key: `beta`).

### Parameters
- **name** (string) - Optional - internal use.
- **action_spec** (specification) - Optional - internal use.
- **input_spec** (specification) - Optional - internal use.
```

--------------------------------

### Module path specification for Agent.create

Source: https://tensorforce.readthedocs.io/en/latest/basics/agent-specification.html

Specifies modules by module path. The policy network is given as 'my_module' and the memory type as the full path 'tensorforce.core.memories.Replay', with additional arguments (capacity) passed directly.

```python
Agent.create(...
    policy=dict(network='my_module'),
    memory=dict(type='tensorforce.core.memories.Replay', capacity=10000), ...
)
```

--------------------------------

### Environment.states

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Returns the state space specification.

```APIDOC
## Environment.states

### Description
Returns the state space specification.

### Method
instance

### Signature
`Environment.states()`

### Parameters
None

### Returns
- **specification** - Arbitrarily nested dictionary of state descriptions with the following attributes:
  - **type** ("bool" | "int" | "float") - state data type (default: "float").
  - **shape** (int | iter[int]) - state shape (required).
  - **num_states** (int > 0) - number of discrete state values (required for type "int").
  - **min_value/max_value** (float) - minimum/maximum state value (optional for type "float").
```

--------------------------------

### ConstantAgent constructor

Source: https://tensorforce.readthedocs.io/en/latest/agents/constant.html

Creates a ConstantAgent instance that returns constant action values. The constructor accepts states and actions specifications, optional max episode timesteps, constant action values, configuration, and recorder settings.

```APIDOC
## ConstantAgent(states, actions, max_episode_timesteps=None, action_values=None, config=None, recorder=None)

### Description
Agent returning constant action values (specification key: `constant`).

### Parameters
- **states** (specification) - Required - States specification, arbitrarily nested dictionary of state descriptions with attributes: type ("bool" | "int" | "float"), shape (int | iter[int]), num_values (int > 0, required for type "int"), min_value/max_value (float, optional for type "float").
- **actions** (specification) - Required - Actions specification, arbitrarily nested dictionary of action descriptions with attributes: type ("bool" | "int" | "float"), shape (int > 0 | iter[int > 0]), num_values (int > 0, required for type "int"), min_value/max_value (float, optional for type "float").
- **max_episode_timesteps** (int > 0) - Optional - Upper bound for number of timesteps per episode.
- **action_values** (dict[value]) - Optional - Constant value per action. Default: false for binary boolean actions, 0 for discrete integer actions, 0.0 for continuous actions.
- **config** (specification) - Optional - Additional configuration options:
  - **name** (string) - Agent name, used e.g. for TensorFlow scopes (default: "agent").
  - **device** (string) - Device name (default: TensorFlow default).
  - **seed** (int) - Random seed to set for Python, NumPy and TensorFlow (default: none).
  - **buffer_observe** (false | "episode" | int > 0) - Number of timesteps within an episode to buffer before calling the internal observe function (default: configuration-specific maximum).
  - **always_apply_exploration** (bool) - Whether to always apply exploration (default: false).
  - **always_apply_variable_noise** (bool) - Whether to always apply variable noise (default: false).
  - **enable_int_action_masking** (bool) - Whether int action options can be masked via an optional "[ACTION-NAME]_mask" state input (default: true).
  - **create_tf_assertions** (bool) - Whether to create internal TensorFlow assertion operations (default: true).
- **recorder** (path | specification) - Optional - Traces recordings directory or recorder configuration with attributes: directory (path, required), frequency (int > 0, default: every episode), start (int >= 0, default: 0), max-traces (int > 0, default: all).

### Returns
A ConstantAgent instance.
```

--------------------------------

### Dockerfile for stable Tensorforce

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Minimal Dockerfile to run Tensorforce in a container using the stable PyPI version.

```dockerfile
FROM python:3.8
RUN \
  pip3 install tensorforce

```