Tensorforce

https://tensorforce.readthedocs.io/en/latest/
Admin
Tensorforce is an open-source deep reinforcement learning framework built on TensorFlow, designed...
Tokens:59,124
Snippets:326
Trust Score:9.6
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install gym-retro

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_retro.html

Install the gym-retro package, which is required to use the OpenAI Retro environment adapter.

```bash
pip3 install gym-retro
```

--------------------------------

### Install OpenAI Gym dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_gym.html

Commands to install the OpenAI Gym library and its full suite of environments.

```bash
pip3 install gym
pip3 install gym[all]
```

--------------------------------

### Initializing a PPO agent

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Example of initializing a Proximal Policy Optimization (PPO) agent.

```python
agent = Agent.create(
    agent='ppo', environment=environment, batch_size=10, learning_rate=1e-3
)
```

--------------------------------

### Install ALE dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/ale.html

Install the system dependencies required by the Arcade Learning Environment adapter before using it.

```bash
sudo apt-get install libsdl1.2-dev libsdl-gfx1.2-dev libsdl-image1.2-dev cmake

```

--------------------------------

### Install stable Tensorforce from PyPI

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Installs the latest stable version of Tensorforce from PyPI.

```bash
pip3 install tensorforce

```

--------------------------------

### Install PyGame Learning Environment dependencies

Source: https://tensorforce.readthedocs.io/en/latest/environments/ple.html

Install the system dependencies and Python packages required to use the PyGame Learning Environment adapter. Run the apt-get command first, then the two pip3 install commands in order.

```bash
sudo apt-get install git python3-dev python3-setuptools python3-numpy python3-opengl     libsdl-image1.2-dev libsdl-mixer1.2-dev libsdl-ttf2.0-dev libsmpeg-dev libsdl1.2-dev     libportmidi-dev libswscale-dev libavformat-dev libavcodec-dev libtiff5-dev libx11-6     libx11-dev fluid-soundfont-gm timgm6mb-soundfont xfonts-base xfonts-100dpi xfonts-75dpi     xfonts-cyrillic fontconfig fonts-freefont-ttf libfreetype6-dev
```

```bash
pip3 install pygame
```

```bash
pip3 install git+https://github.com/ntasfi/PyGame-Learning-Environment.git
```

--------------------------------

### Install ViZDoom dependencies and package

Source: https://tensorforce.readthedocs.io/en/latest/environments/vizdoom.html

System dependencies and the vizdoom Python package required to use the ViZDoom environment adapter.

```bash
sudo apt-get install g++ build-essential libsdl2-dev zlib1g-dev libmpg123-dev libjpeg-dev     libsndfile1-dev nasm tar libbz2-dev libgtk2.0-dev make cmake git chrpath timidity     libfluidsynth-dev libgme-dev libopenal-dev timidity libwildmidi-dev unzip libboost-all-dev     liblua5.1-dev

pip3 install vizdoom
```

--------------------------------

### Specify state and reward preprocessing in Agent.create

Source: https://tensorforce.readthedocs.io/en/latest/modules/preprocessing.html

Example of how to define reward_estimation and state_preprocessing configurations within the Agent.create method.

```python
Agent.create(
    ...
    reward_estimation=dict(
        ...
        reward_processing=dict(type='clipping', lower=-1.0, upper=1.0)
    ),
    state_preprocessing=[
        dict(type='image', height=4, width=4, grayscale=True),
        dict(type='exponential_normalization', decay=0.999)
    ],
    ...
)
```

--------------------------------

### Install latest Tensorforce from GitHub

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Clones the Tensorforce repository and installs the latest version in editable mode.

```bash
git clone https://github.com/tensorforce/tensorforce.git
cd tensorforce
pip3 install -e .

```

--------------------------------

### Initializing a generic Tensorforce agent

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Example of initializing a generic agent with specific optimizer, objective, and reward estimation settings.

```python
agent = Agent.create(
    agent='tensorforce', environment=environment, update=64,
    optimizer=dict(optimizer='adam', learning_rate=1e-3),
    objective='policy_gradient', reward_estimation=dict(horizon=20)
)
```

--------------------------------

### Environment.reset

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Resets the environment to start a new episode.

```APIDOC
## Environment.reset

### Description
Resets the environment to start a new episode.

### Method
instance

### Signature
`Environment.reset(num_parallel=None)`

### Parameters
- **num_parallel** (int >= 1) - Optional - Number of environment instances executed in parallel, only valid if environment is vectorizable (no vectorization).

### Returns
- **(parallel,) dict[state]** - Dictionary containing initial state(s) and auxiliary information, and parallel index vector in case of vectorized execution.
```

--------------------------------

### Configure Agent memory with capacity

Source: https://tensorforce.readthedocs.io/en/latest/modules/memories.html

Example of specifying a Replay memory with a capacity of 10000 using an integer shorthand.

```python
Agent.create(
    ...
    memory=10000,
    ...
)
```

--------------------------------

### Configure Adam optimizer with OptimizerWrapper

Source: https://tensorforce.readthedocs.io/en/latest/modules/optimizers.html

Shows how to specify a customized Adam optimizer via the OptimizerWrapper by passing a dictionary to the optimizer argument of Agent.create. The example sets learning_rate, clipping_threshold, multi_step, subsampling_fraction, linesearch_iterations, and doublecheck_update.

```python
Agent.create(
    ...
    optimizer=dict(
        optimizer='adam', learning_rate=1e-3, clipping_threshold=1e-2,
        multi_step=10, subsampling_fraction=64, linesearch_iterations=5,
        doublecheck_update=True
    ),
    ...
)
```

--------------------------------

### Customize policy distributions with Agent.create

Source: https://tensorforce.readthedocs.io/en/latest/modules/distributions.html

Configure a policy with custom distributions via the `distributions` argument. The example sets a Gaussian distribution for continuous actions with global standard deviation mode and a Beta distribution for bounded actions.

```python
Agent.create(
    ...
    policy=dict(distributions=dict(
        float=dict(type='gaussian', stddev_mode='global'),
        bounded_action=dict(type='beta')
    ))
    ...
)

```

--------------------------------

### Install Tensorforce on M1 Macs without dependencies

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Installs Tensorforce on M1 Macs after manually installing its dependencies, ignoring the tensorflow dependency.

```bash
pip3 install tensorforce --no-deps

```

--------------------------------

### Loading agent configuration from a file

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

How to initialize an agent by passing the path to a JSON configuration file.

```python
agent = Agent.create(agent='agent.json', environment=environment)
```

--------------------------------

### Initialize OpenAI CartPole-v1 environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Access pre-defined Gym versions directly.

```python
environment = Environment.create(environment='gym', level='CartPole-v1')
```

--------------------------------

### Load environment from config file path

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Load an environment by passing the path to a configuration file.

```python
environment = Environment.create(
    environment='environment.json', max_episode_timesteps=500
)
```

--------------------------------

### ViZDoom

Source: https://tensorforce.readthedocs.io/en/latest/environments/vizdoom.html

Constructor for the ViZDoom environment adapter, which initializes a ViZDoom environment with configurable parameters.

```APIDOC
## ViZDoom

### Description
ViZDoom environment adapter (specification key: `vizdoom`).

### Constructor
`tensorforce.environments.ViZDoom(level, visualize=False, include_variables=False, factored_action=False, frame_skip=12, seed=None)`

### Parameters
- **level** (string) - Required - ViZDoom configuration file.
- **include_variables** (bool) - Optional - Whether to include game variables to state (default: false).
- **factored_action** (bool) - Optional - Whether to use factored action representation (default: false).
- **visualize** (bool) - Optional - Whether to visualize interaction (default: false).
- **frame_skip** (int > 0) - Optional - Number of times to repeat an action without observing (default: 12).
- **seed** (int) - Optional - Random seed (default: none).

### Installation Requirements
```
sudo apt-get install g++ build-essential libsdl2-dev zlib1g-dev libmpg123-dev libjpeg-dev libsndfile1-dev nasm tar libbz2-dev libgtk2.0-dev make cmake git chrpath timidity libfluidsynth-dev libgme-dev libopenal-dev timidity libwildmidi-dev unzip libboost-all-dev liblua5.1-dev

pip3 install vizdoom
```
```

--------------------------------

### Dockerfile for latest Tensorforce from GitHub

Source: https://tensorforce.readthedocs.io/en/latest/basics/installation.html

Dockerfile to run the latest Tensorforce version by cloning the GitHub repository and installing it in editable mode.

```dockerfile
FROM python:3.8
RUN \
  git clone https://github.com/tensorforce/tensorforce.git && \
  pip3 install -e tensorforce

```

--------------------------------

### OpenAIGym constructor

Source: https://tensorforce.readthedocs.io/en/latest/environments/openai_gym.html

Creates an OpenAI Gym environment adapter. The level parameter is required and can be a Gym id string or a Gym environment instance. Optional parameters control visualization, value clipping, terminal rewards, reward thresholds, state dropping, and additional Gym arguments.

```APIDOC
## OpenAIGym(level, visualize=False, import_modules=None, min_value=None, max_value=None, terminal_reward=0.0, reward_threshold=None, drop_states_indices=None, visualize_directory=None, **kwargs)

### Description
Creates an OpenAI Gym environment adapter for Tensorforce. The adapter allows Tensorforce agents to interact with OpenAI Gym environments.

### Method
Constructor

### Parameters
- **level** (string | gym.Env) - Required - Gym id or instance.
- **visualize** (bool) - Optional - Whether to visualize interaction (default: false).
- **import_modules** (list) - Optional - Modules to import (default: none).
- **min_value** (float) - Optional - Lower bound clipping for otherwise unbounded state values (default: no clipping).
- **max_value** (float) - Optional - Upper bound clipping for otherwise unbounded state values (default: no clipping).
- **terminal_reward** (float) - Optional - Additional reward for early termination (default: Gym default).
- **reward_threshold** (float) - Optional - Gym environment argument, the reward threshold before the task is considered solved (default: Gym default).
- **drop_states_indices** (list[int]) - Optional - Drop states indices (default: none).
- **visualize_directory** (string) - Optional - Visualization output directory (default: none).
- **kwargs** - Optional - Additional Gym environment arguments.

### Example
```python
from tensorforce.environments import OpenAIGym

environment = OpenAIGym(
    level='CartPole-v1',
    visualize=True,
    max_value=10.0
)
```
```

--------------------------------

### TensorforceAgent.observe

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Observes reward and whether a terminal state is reached, needs to be preceded by act(). See the act-observe script for an example application as part of the act-observe interface.

```APIDOC
## TensorforceAgent.observe

### Description
Observes reward and whether a terminal state is reached, needs to be preceded by act(). See the act-observe script for an example application as part of the act-observe interface.

### Method
Instance method

### Signature
`TensorforceAgent.observe(reward=0.0, terminal=False, parallel=0)`

### Parameters
- **reward** (float | iter[float]) - Optional - Reward. Default: 0.0.
- **terminal** (bool | 0 | 1 | 2 | iter[..]) - Optional - Whether a terminal state is reached, or 2 if the episode was aborted. Default: false.
- **parallel** (int, iter[int]) - Optional - Parallel execution index. Default: 0.

### Returns
- **int** - Number of performed updates.
```

--------------------------------

### TensorforceAgent.act

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Returns action(s) for the given state(s), needs to be followed by observe() unless independent mode. See the act-observe script for an example application as part of the act-observe interface.

```APIDOC
## TensorforceAgent.act

### Description
Returns action(s) for the given state(s), needs to be followed by observe() unless independent mode. See the act-observe script for an example application as part of the act-observe interface.

### Method
Instance method

### Signature
`TensorforceAgent.act(states, internals=None, parallel=0, independent=False, deterministic=True, evaluation=None)`

### Parameters
- **states** (dict[state] | iter[dict[state]]) - Required - Dictionary containing state(s) to be acted on.
- **internals** (dict[internal] | iter[dict[internal]]) - Optional - Dictionary containing current internal agent state(s), either given by initial_internals() at the beginning of an episode or as return value of the preceding act() call. Required if independent mode and agent has internal states.
- **parallel** (int | iter[int]) - Optional - Parallel execution index. Default: 0.
- **independent** (bool) - Optional - Whether this act() call is not part of the training agent-environment interaction and thus not followed by observe(), meaning its inputs/outputs/internals are not stored in memory and not used in updates, e.g. for independent evaluation episodes which should not be learned from. Default: false.
- **deterministic** (bool) - Optional - Whether action should be chosen deterministically, so no action distribution sampling and no exploration, only valid in independent mode. Default: true.
- **evaluation** - Optional - Not described in source.

### Returns
- If internals argument given: dict[action] | iter[dict[action]], dict[internal] | iter[dict[internal]] - Dictionary containing action(s), dictionary containing next internal agent state(s) if independent mode.
- Otherwise: dict[action] | iter[dict[action]] - Dictionary containing action(s).
```

--------------------------------

### Run basic experiment with Runner

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Configure and run a basic training and evaluation experiment using the Runner utility.

```python
runner = Runner(
    agent='agent.json',
    environment=dict(environment='gym', level='CartPole'),
    max_episode_timesteps=500
)

runner.run(num_episodes=200)

runner.run(num_episodes=100, evaluation=True)

runner.close()
```

--------------------------------

### PyGameLearningEnvironment.__init__

Source: https://tensorforce.readthedocs.io/en/latest/environments/ple.html

Initializes a PyGame Learning Environment adapter with the specified game level and configuration options. This is the constructor for the environment class.

```APIDOC
## PyGameLearningEnvironment.__init__

### Description
Initializes a PyGame Learning Environment adapter with the specified game level and configuration options. This is the constructor for the environment class.

### Method
Constructor

### Signature
`PyGameLearningEnvironment(level, visualize=False, frame_skip=1, fps=30)`

### Parameters
- **level** (string | subclass of `ple.games.base`) - Required - Game instance or name of class in `ple.games`, like "Catcher", "Doom", "FlappyBird", "MonsterKong", "Pixelcopter", "Pong", "PuckWorld", "RaycastMaze", "Snake", "WaterWorld".
- **visualize** (bool) - Optional - Whether to visualize interaction. Default: false.
- **frame_skip** (int > 0) - Optional - Number of times to repeat an action without observing. Default: 1.
- **fps** (int > 0) - Optional - The desired frames per second we want to run our game at. Default: 30.

### Usage Example
```python
from tensorforce.environments import PyGameLearningEnvironment

# Create a PyGame Learning Environment adapter
env = PyGameLearningEnvironment(
    level="Catcher",
    visualize=False,
    frame_skip=1,
    fps=30
)
```

### Notes
- The environment adapter uses the specification key `ple` or `pygame_learning_environment`.
- May require additional system packages and Python packages as listed in the documentation.
```

--------------------------------

### Initialize OpenAI CartPole environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Initialize the OpenAI CartPole environment using the Environment.create interface.

```python
environment = Environment.create(
    environment='gym', level='CartPole', max_episode_timesteps=500
)
```

--------------------------------

### Specify multi-input network with Register and Retrieve layers

Source: https://tensorforce.readthedocs.io/en/latest/modules/networks.html

Specifies a multi-input, non-sequential network using nested lists of layers. Each inner list forms a sequential component, and the special layers 'retrieve' and 'register' combine these components. The example uses two input states (observation and attributes) and concatenates their embeddings before a final dense layer.

```python
Agent.create(
    states=dict(
        observation=dict(type='float', shape=(16, 16, 3), min_value=-1.0, max_value=1.0),
        attributes=dict(type='int', shape=(4, 2), num_values=5)
    ),
    ...
    policy=[
        [
            dict(type='retrieve', tensors=['observation']),
            dict(type='conv2d', size=32),
            dict(type='flatten'),
            dict(type='register', tensor='obs-embedding')
        ],
        [
            dict(type='retrieve', tensors=['attributes']),
            dict(type='embedding', size=32),
            dict(type='flatten'),
            dict(type='register', tensor='attr-embedding')
        ],
        [
            dict(
                type='retrieve', aggregation='concat',
                tensors=['obs-embedding', 'attr-embedding']
            ),
            dict(type='dense', size=64)
        ]
    ],
    ...
)

```

--------------------------------

### Load custom environment by module path

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Load a custom environment by specifying its module path.

```python
environment = Environment.create(
    environment='envs.custom_env', max_episode_timesteps=100
)
```

--------------------------------

### Environment configuration file

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Define an environment using a JSON configuration file.

```json
{
    "environment": "gym",
    "level": "CartPole"
}
```

--------------------------------

### Import Environment class

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Import the Environment class to begin creating environments.

```python
from tensorforce.environments import Environment
```

--------------------------------

### Implement manual act-experience-update training loop

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Use the act-experience-update interface for more flexibility, noting that some stateful network layers may not update correctly in independent-mode.

```python
# Train for 100 episodes
for _ in range(100):
    episode_states = list()
    episode_internals = list()
    episode_actions = list()
    episode_terminal = list()
    episode_reward = list()

    states = environment.reset()
    internals = agent.initial_internals()
    terminal = False
    while not terminal:
        episode_states.append(states)
        episode_internals.append(internals)
        actions, internals = agent.act(
            states=states, internals=internals, independent=True
        )
        episode_actions.append(actions)
        states, terminal, reward = environment.execute(actions=actions)
        episode_terminal.append(terminal)
        episode_reward.append(reward)

    agent.experience(
        states=episode_states, internals=episode_internals,
        actions=episode_actions, terminal=episode_terminal,
        reward=episode_reward
    )
    agent.update()
```

--------------------------------

### Configure TensorBoard Summarizer

Source: https://tensorforce.readthedocs.io/en/latest/basics/features.html

Enable TensorBoard logging by providing a summarizer configuration with specific labels to track during training.

```python
Agent.create(...
    summarizer=dict(
        directory='data/summaries',
        # list of labels, or 'all'
        labels=['entropy', 'kl-divergence', 'loss', 'reward', 'update-norm']
    ), ...
)
```

--------------------------------

### Implement a custom environment

Source: https://tensorforce.readthedocs.io/en/latest/basics/getting-started.html

Create a custom environment by subclassing the Tensorforce Environment interface.

```python
class CustomEnvironment(Environment):

    def __init__(self):
        super().__init__()

    def states(self):
        return dict(type='float', shape=(8,))

    def actions(self):
        return dict(type='int', num_values=4)

    # Optional: should only be defined if environment has a natural fixed
    # maximum episode length; otherwise specify maximum number of training
    # timesteps via Environment.create(..., max_episode_timesteps=???)
    def max_episode_timesteps(self):
        return super().max_episode_timesteps()

    # Optional additional steps to close environment
    def close(self):
        super().close()

    def reset(self):
        state = np.random.random(size=(8,))
        return state

    def execute(self, actions):
        next_state = np.random.random(size=(8,))
        terminal = False  # Always False if no "natural" terminal state
        reward = np.random.random()
        return next_state, terminal, reward
```

--------------------------------

### TensorforceAgent.pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

Source: https://tensorforce.readthedocs.io/en/latest/agents/agent.html

Simple pretraining approach as a combination of experience() and update, akin to behavioral cloning, using experience traces obtained e.g. via recording agent interactions.

```APIDOC
## TensorforceAgent.pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

### Description
Simple pretraining approach as a combination of experience() and update, akin to behavioral cloning, using experience traces obtained e.g. via recording agent interactions.

### Method
pretrain(directory, num_iterations, num_traces=1, num_updates=1, extension='.npz')

### Parameters
- **directory** (path) - Required - Directory with experience traces, e.g. obtained via recorder; episode length has to be consistent with agent configuration.
- **num_iterations** (int > 0) - Required - Number of iterations consisting of loading new traces and performing multiple updates.
- **num_traces** (int > 0) - Optional (default: 1) - Number of traces to load per iteration; has to at least satisfy the update batch size.
- **num_updates** (int > 0) - Optional (default: 1) - Number of updates per iteration.
- **extension** (str) - Optional (default: ".npz") - Traces file extension to filter the given directory for.
```

--------------------------------

### OpenSim Constructor

Source: https://tensorforce.readthedocs.io/en/latest/environments/open_sim.html

Creates an OpenSim environment adapter instance. The level parameter is required and specifies the environment id, while visualize and integrator_accuracy are optional.

```APIDOC
## OpenSim(level, visualize=False, **kwargs)

### Description
Creates an OpenSim environment adapter instance. The level parameter is required and specifies the environment id, while visualize and integrator_accuracy are optional.

### Method
Constructor

### Signature
`tensorforce.environments.OpenSim(level, visualize=False, **kwargs)`

### Parameters
- **level** (_'Arm2D' | 'L2Run' | 'Prosthetics'_) - Required - Environment id.
- **visualize** (_bool_) - Optional - Whether to visualize interaction (default: false).
- **integrator_accuracy** (_float_) - Optional - Integrator accuracy (default: 5e-5).

### Specification Key
`osim`, `open_sim`
```

--------------------------------

### Environment.actions

Source: https://tensorforce.readthedocs.io/en/latest/environments/environment.html

Returns the action space specification.

```APIDOC
## Environment.actions

### Description
Returns the action space specification.

### Method
instance

### Signature
`Environment.actions()`

### Parameters
None

### Returns
- **specification** - Arbitrarily nested dictionary of action descriptions with the following attributes:
  - **type** ("bool" | "int" | "float") - action data type (required).
  - **shape** (int > 0 | iter[int > 0]) - action shape (default: scalar).
  - **num_actions** (int > 0) - number of discrete action values (required for type "int").
  - **min_value/max_value** (float) - minimum/maximum action value (optional for type "float").
```
Tokens:
Raw