d3rlpy

https://github.com/takuseno/d3rlpy
Admin
d3rlpy is an offline deep reinforcement learning library for practitioners and researchers.
Tokens:61,949
Snippets:523
Trust Score:9.7
License:MIT
Update:3 months ago
Show doc for...
Tokens:
Raw
### Setup Experience Replay Buffer and Explorer

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/online_rl.md

Create a First-In, First-Out (FIFO) replay buffer with a specified limit and an exploration strategy. The example uses a constant epsilon-greedy policy.

```python
# experience replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=100000, env=env)

# exploration strategy
# in this tutorial, epsilon-greedy policy with static epsilon=0.3
explorer = d3rlpy.algos.ConstantEpsilonGreedy(0.3)
```

--------------------------------

### Online Training Setup and Execution with DQN

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/online.md

Demonstrates the complete setup for online training using the DQN algorithm. This includes initializing the environment, algorithm, replay buffer, and an epsilon-greedy explorer. The `fit_online` method trains the agent for a specified number of steps, updating the policy periodically.

```python
import d3lpy
import gym

# setup environment
env = gym.make('CartPole-v1')
eval_env = gym.make('CartPole-v1')

# setup algorithm
dqn = d3rlpy.algos.DQN(
    batch_size=32,
    learning_rate=2.5e-4,
    target_update_interval=100,
).create(device="cuda:0")

# setup replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=100000, env=env)

# setup explorers
explorer = d3rlpy.algos.LinearDecayEpsilonGreedy(
    start_epsilon=1.0,
    end_epsilon=0.1,
    duration=10000,
)

# start training
dqn.fit_online(
    env,
    buffer,
    explorer=explorer, # you don't need this with probablistic policy algorithms
    eval_env=eval_env,
    n_steps=30000, # the number of total steps to train.
    n_steps_per_epoch=1000,
    update_interval=10, # update parameters every 10 steps.
)
```

--------------------------------

### Install D3RLPY and Atari Datasets

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/atari.ipynb

Install the D3RLPY library and the d4rl-atari dataset. This is a prerequisite for running the Atari environment examples.

```python
!pip install git+https://github.com/takuseno/d4rl-atari d3rlpy
```

--------------------------------

### Prepare Environment with Gym

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/data_collection.md

Initializes an OpenAI Gym environment. Ensure the 'gym' library is installed.

```python
import gym

env = gym.make("CartPole-v1")
```

--------------------------------

### Build d3rlpy from Source

Source: https://github.com/takuseno/d3rlpy/blob/master/CONTRIBUTING.md

Clone the repository and install the package in editable mode to start development.

```bash
git clone git@github.com:takuseno/d3rlpy
cd d3rlpy
pip install -e .
```

--------------------------------

### Finetune with Environment

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Sets up an experience replay buffer and an exploration strategy, then starts finetuning the pretrained policy directly with the environment.

```python
# setup experience replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=100000, env=env)

# setup exploration strategy if necessary
explorer = d3rlpy.algos.ConstantEpsilonGreedy(0.1)

# start finetuning
dqn.fit_online(env, buffer, explorer, n_steps=100000)
```

--------------------------------

### Install Rendering Dependencies

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/tpu.ipynb

Installs necessary dependencies for rendering in Google Colaboratory, such as xvfb and ffmpeg.

```bash
!apt-get install -y xvfb ffmpeg > /dev/null 2>&1
!pip install pyvirtualdisplay pygame moviepy > /dev/null 2>&1
```

--------------------------------

### Setup TPU Device and CQL Algorithm

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/tpu.ipynb

Configures the TPU device and sets up the DiscreteCQL algorithm for training. Training may be slow on small architectures.

```python
# get TPU device
import torch_xla.core.xla_model as xm
device = xm.xla_device()

# setup CQL algorithm
cql = d3rlpy.algos.DiscreteCQLConfig().create(device=str(device))

# start training
cql.fit(
    dataset,
    n_steps=10000,
    n_steps_per_epoch=1000,
    evaluators={
        'environment': d3rlpy.metrics.EnvironmentEvaluator(env), # evaluate with CartPole-v1 environment
    },
)
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/takuseno/d3rlpy/blob/master/CONTRIBUTING.md

Install additional dependencies required for development and testing.

```bash
pip install -r dev.requirements.txt
```

--------------------------------

### Manually Setup MinMaxActionScaler

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/preprocess_and_postprocess.md

Manually set up MinMaxActionScaler by computing the minimum and maximum action values from the dataset.

```python
# setup manually
actions = []
for episode in dataset.episodes:
    actions += episode.actions.tolist()
minimum_action = np.min(actions, axis=0)
maximum_action = np.max(actions, axis=0)
action_scaler = d3rlpy.preprocessing.MinMaxActionScaler(
    minimum=minimum_action,
    maximum=maximum_action,
)

# set as action scaler
sac = d3rlpy.algos.SACConfig(action_scaler=action_scaler).create()
```

--------------------------------

### Prepare Environment for Online RL

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/online_rl.md

Import the gym library and create instances of the training and evaluation environments. This setup is standard for RL tasks.

```python
import gym

# for training
env = gym.make("CartPole-v1")

# for evaluation
eval_env = gym.make("CartPole-v1")
```

--------------------------------

### Setup DQN Algorithm

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/getting_started.md

Configure and create a DQN algorithm instance. Specify the device for computation (e.g., 'cuda:0' for GPU). The algorithm can be built with a dataset to initialize neural networks.

```python
from d3rlpy.algos import DQNConfig

# if you don't use GPU, set device=None instead.
dqn = DQNConfig().create(device="cuda:0")

# initialize neural networks with the given observation shape and action size.
# this is not necessary when you directly call fit or fit_online method.
dqn.build_with_dataset(dataset)
```

--------------------------------

### Initialize CQL with Custom QRQFunctionFactory and Create Algorithm

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/q_functions.md

This example demonstrates initializing a QRQFunctionFactory with a specific number of quantiles and then creating a CQL algorithm instance with this custom factory.

```python
q_func = d3rlpy.models.QRQFunctionFactory(n_quantiles=32)

cql = d3rlpy.algos.CQLConfig(q_func_factory=q_func).create()

```

--------------------------------

### Setup and Train DiscreteCQL Algorithm

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/cartpole.ipynb

Configures and trains a DiscreteCQL algorithm on the loaded dataset. Training involves fitting the algorithm for a specified number of steps and includes environment evaluation.

```python
# setup CQL algorithm
cql = d3rlpy.algos.DiscreteCQLConfig().create(device='cuda:0')

# start training
cql.fit(
    dataset,
    n_steps=10000,
    n_steps_per_epoch=1000,
    evaluators={
        'environment': d3rlpy.metrics.EnvironmentEvaluator(env), # evaluate with CartPole-v1 environment
    },
)
```

--------------------------------

### Install d3rlpy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/index.md

Install d3rlpy using pip. This command installs the library and its dependencies.

```bash
pip install d3rlpy
```

--------------------------------

### Install d3rlpy and PyTorch with TPU Support

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/tpu.ipynb

Installs d3rlpy and PyTorch with XLA support for TPUs. May require restarting the runtime if XLA installation fails.

```bash
!pip install d3rlpy torch~=2.0.0 https://storage.googleapis.com/tpu-pytorch/wheels/colab/torch_xla-2.0-cp310-cp310-linux_x86_64.whl
```

--------------------------------

### Setup and Train DQN Algorithm

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/online.ipynb

Configures and trains a Deep Q-Network (DQN) algorithm using D3RLPy. This includes setting up the algorithm, an explorer, and a replay buffer.

```python
import d3rlpy

# setup DQN algorithm
dqn = d3rlpy.algos.DQNConfig(
    learning_rate=1e-3,
    target_update_interval=100,
).create(device='cuda:0')

# setup explorer
explorer = d3rlpy.algos.ConstantEpsilonGreedy(epsilon=0.3)

# setup replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=50000, env=env)

# start training
dqn.fit_online(
    env,
    buffer,
    explorer,
    eval_env=eval_env,
    n_steps=50000,
    n_steps_per_epoch=10000,
)
```

--------------------------------

### Install d3rlpy

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/cartpole.ipynb

Installs the d3rlpy library. This is a prerequisite for using the library's functionalities.

```python
!pip install d3rlpy
```

--------------------------------

### Pretrain with Dataset

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Initializes a DQN algorithm and starts offline training using the prepared dataset. This step generates a baseline policy before finetuning.

```python
# setup algorithm
dqn = d3rlpy.algos.DQNConfig().create()

# start offline training
dqn.fit(dataset, n_steps=100000)
```

--------------------------------

### Manually Setup StandardObservationScaler

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/preprocess_and_postprocess.md

Manually set up StandardObservationScaler by computing mean and standard deviation from the dataset.

```python
# setup manually
observations = []
for episode in dataset.episodes:
    observations += episode.observations.tolist()
mean = np.mean(observations, axis=0)
std = np.std(observations, axis=0)
observation_scaler = d3rlpy.preprocessing.StandardObservationScaler(mean=mean, std=std)

# set as observation_scaler
sac = d3rlpy.algos.SACConfig(observation_scaler=observation_scaler).create()
```

--------------------------------

### Start Offline Training

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/getting_started.md

Initiate offline training of the DQN algorithm using the prepared dataset. Evaluation metrics can be logged during training.

```python
dqn.fit(
    dataset,
    n_steps=10000,
    evaluators={
        'td_error': td_error_evaluator,
        'environment': env_evaluator,
    },
)
```

--------------------------------

### Manually Setup StandardRewardScaler

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/preprocess_and_postprocess.md

Manually set up StandardRewardScaler by computing the mean and standard deviation of rewards from the dataset.

```python
# setup manuall
rewards = []
for episode in dataset.episodes:
    rewards += episode.rewards.tolist()
mean = np.mean(rewards)
std = np.std(rewards)
reward_scaler = StandardRewardScaler(mean=mean, std=std)

# set as reward scaler
sac = d3rlpy.algos.SACConfig(reward_scaler=reward_scaler).create()
```

--------------------------------

### Start Online RL Training with DQN

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/online_rl.md

Initiate online training for the DQN agent using the configured environment, buffer, and explorer. Specify the total number of steps, evaluation frequency, and when parameter updates should begin.

```python
dqn.fit_online(
    env,
    buffer,
    explorer,
    n_steps=100000,  # train for 100K steps
    eval_env=eval_env,
    n_steps_per_epoch=1000,  # evaluation is performed every 1K steps
    update_start_step=1000,  # parameter update starts after 1K steps
)
```

--------------------------------

### Start online training loop

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Initiate an online training loop using the `fit_online` method. This method interacts directly with a Gym-like environment. It allows for configuration of exploration, updates, and evaluation.

```python
algo.fit_online(env, buffer=None, explorer=None, n_steps=1000000, n_steps_per_epoch=10000, update_interval=1, n_updates=1, update_start_step=0, random_steps=0, eval_env=None, eval_epsilon=0.0, eval_n_trials=10, save_interval=1, experiment_name=None, with_timestamp=True, logging_steps=500, logging_strategy=LoggingStrategy.EPOCH, logger_adapter=<d3rlpy.logging.file_adapter.FileAdapterFactory object>, show_progress=True, callback=None)
```

--------------------------------

### fit_online

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Starts the training loop for online deep reinforcement learning.

```APIDOC
## fit_online

Start training loop of online deep reinforcement learning.

### Method Signature
fit_online(env, buffer=None, explorer=None, n_steps=1000000, n_steps_per_epoch=10000, update_interval=1, n_updates=1, update_start_step=0, random_steps=0, eval_env=None, eval_epsilon=0.0, eval_n_trials=10, save_interval=1, experiment_name=None, with_timestamp=True, logging_steps=500, logging_strategy=LoggingStrategy.EPOCH, logger_adapter=<d3rlpy.logging.file_adapter.FileAdapterFactory object>, show_progress=True, callback=None)

### Parameters
* **env** (*Env* *[**Any* *,* *Any* *]*  *|* *Env* *[**Any* *,* *Any* *]*) – Gym-like environment.
* **buffer** (*ReplayBufferBase* *|* *None*) – Replay buffer.
* **explorer** (*Explorer* *|* *None*) – Action explorer.
* **n_steps** (*int*) – Number of total steps to train.
* **n_steps_per_epoch** (*int*) – Number of steps per epoch.
* **update_interval** (*int*) – Number of steps per update.
* **n_updates** (*int*) – Number of gradient steps at a time. The combination of `update_interval` and `n_updates` controls Update-To-Data (UTD) ratio.
* **update_start_step** (*int*) – Steps before starting updates.
* **random_steps** (*int*) – Steps for the initial random explortion.
* **eval_env** (*Env* *[**Any* *,* *Any* *]*  *|* *Env* *[**Any* *,* *Any* *]*  *|* *None*) – Gym-like environment. If None, evaluation is skipped.
* **eval_epsilon** (*float*) – $\epsilon$-greedy factor during evaluation.
* **save_interval** (*int*) – Number of epochs before saving models.
* **experiment_name** (*str* *|* *None*) – Experiment name for logging. If not passed, the directory name will be `{class name}_online_{timestamp}`.
* **with_timestamp** (*bool*) – Flag to add timestamp string to the last of directory name.
* **logging_steps** (*int*) – Number of steps to log metrics. This will be ignored if logging_strategy is EPOCH.
* **logging_strategy** (*LoggingStrategy*) – Logging strategy to use.
* **logger_adapter** (*LoggerAdapterFactory*) – LoggerAdapterFactory object.
* **show_progress** (*bool*) – Flag to show progress bar for iterations.
* **callback** (*Callable* *[* *[**Self* *,* *int* *,* *int* *]* *,* *None* *]*  *|* *None*) – Callable function that takes `(algo, epoch, total_step)`, which is called at the end of epochs.
* **eval_n_trials** (*int*)

### Returns
None
```

--------------------------------

### Setup DQN Algorithm for Online RL

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/online_rl.md

Configure and create a DQN algorithm instance using d3rlpy. Specify hyperparameters like batch size and learning rate. The `create` method handles device placement (e.g., 'cuda:0').

```python
import d3rlpy

# if you don't use GPU, set use_gpu=False instead.
dqn = d3rlpy.algos.DQNConfig(
    batch_size=32,
    learning_rate=2.5e-4,
    target_update_interval=100,
).create(device="cuda:0")

# initialize neural networks with the given environment object.
# this is not necessary when you directly call fit or fit_online method.
dqn.build_with_env(env)
```

--------------------------------

### Export Q-learning Policy as TorchScript

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/after_training_policies.md

Export a trained Q-learning policy to TorchScript format. This allows for deployment in environments that do not have d3rlpy installed.

```python
cql.save_policy("policy.pt")
```

--------------------------------

### Implement Custom TransitionPicker

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Example of a custom TransitionPicker that manually constructs a Transition object. This is useful for algorithms that require specific transition data handling.

```python
import d3rlpy

# Example TransitionPicker that simply picks transition
class CustomTransitionPicker(d3rlpy.dataset.TransitionPickerProtocol):
    def __call__(self, episode: d3rlpy.dataset.EpisodeBase, index: int) -> d3rlpy.dataset.Transition:
       observation = episode.observations[index]
       is_terminal = episode.terminated and index == episode.size() - 1
       if is_terminal:
           next_observation = d3rlpy.dataset.create_zero_observation(observation)
       else:
           next_observation = episode.observations[index + 1]
       return d3rlpy.dataset.Transition(
           observation=observation,
           action=episode.actions[index],
           reward=episode.rewards[index],
           next_observation=next_observation,
           terminal=float(is_terminal),
           interval=1,
       )
```

--------------------------------

### Implementing a custom LoggerAdapter

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/logging.md

Implement a custom LoggerAdapter to handle metric writing, parameter saving, and model checkpointing. This example saves parameters as JSON and metrics as CSV.

```python
import d3rlpy

class CustomAdapter(d3rlpy.logging.LoggerAdapter):
    def write_params(self, params: Dict[str, Any]) -> None:
        # save dictionary as json file
        with open("params.json", "w") as f:
            f.write(json.dumps(params, default=default_json_encoder, indent=2))

    def before_write_metric(self, epoch: int, step: int) -> None:
        pass

    def write_metric(
        self, epoch: int, step: int, name: str, value: float
    ) -> None:
        with open(f"{name}.csv", "a") as f:
            print(f"{epoch},{step},{value}", file=f)

    def after_write_metric(self, epoch: int, step: int) -> None:
        pass

    def save_model(self, epoch: int, algo: Any) -> None:
        algo.save(f"model_{epoch}.d3")

    def close(self) -> None:
        pass
```

--------------------------------

### Setup Environment Evaluator

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/getting_started.md

Initialize an EnvironmentEvaluator to assess algorithm performance directly within an environment. The environment object must be provided during initialization.

```python
from d3rlpy.metrics import EnvironmentEvaluator

# set environment in scorer function
env_evaluator = EnvironmentEvaluator(env)

# evaluate algorithm on the environment
rewards = env_evaluator(dqn, dataset=None)
```

--------------------------------

### Implement Custom WriterPreprocess

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Example of implementing a custom WriterPreprocess class by inheriting from d3rlpy.dataset.WriterPreprocessProtocol. This allows for custom processing of observations, actions, and rewards before they are stored in the replay buffer.

```python
import d3rlpy

class CustomWriterPreprocess(d3rlpy.dataset.WriterPreprocessProtocol):
    def process_observation(self, observation: d3rlpy.dataset.Observation) -> d3rlpy.dataset.Observation:
        return observation

    def process_action(self, action: np.ndarray) -> np.ndarray:
        return action

    def process_reward(self, reward: np.ndarray) -> np.ndarray:
        return reward
```

--------------------------------

### Predicting Action-Values with RandomPolicy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Example demonstrating the predict_value method to obtain action-values for given observations and actions. This method supports both continuous and discrete action spaces.

```python
import numpy as np

# 100 observations with shape of (10,)
x = np.random.random((10, 10))

# for continuous control
# 100 actions with shape of (2,)
actions = np.random.random((100, 2))

# for discrete control
# 100 actions in integer values
actions = np.random.randint(2, size=100)

values = algo.predict_value(x, actions)
# values.shape == (100,)
```

--------------------------------

### Load and Infer Action with TorchScript Decision Transformer Policy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/after_training_policies.md

Load a TorchScript Decision Transformer policy and infer actions using sequence inputs. This example shows how to prepare and feed sequential data for inference.

```python
import torch

# load TorchScript policy
policy = torch.jit.load("policy.pt")

# prepare sequence inputs
# context_size == 10, action_size=2
observations = torch.rand(10, 3)
actions = torch.rand(10, 2)
returns_to_go = torch.rand(10, 1)
timesteps = torch.zeros(10, dtype=torch.int32)

# infer the action
action = policy(observations, actions, returns_to_go, timesteps)
assert action.shape == (2,)
```

--------------------------------

### Online Training with SAC

Source: https://github.com/takuseno/d3rlpy/blob/master/README.md

This snippet demonstrates online training of a Soft Actor-Critic (SAC) agent using a FIFO replay buffer. It sets up the environment, algorithm with graph compilation, replay buffer, and then starts the online fitting process with evaluation.

```python
import d3rlpy
import gym

# prepare environment
env = gym.make('Hopper-v3')
eval_env = gym.make('Hopper-v3')

# prepare algorithm
sac = d3rlpy.algos.SACConfig(compile_graph=True).create(device='cuda:0')

# prepare replay buffer
buffer = d3rlpy.dataset.create_fifo_replay_buffer(limit=1000000, env=env)

# start training
sac.fit_online(env, buffer, n_steps=1000000, eval_env=eval_env)
```

--------------------------------

### Predicting Actions with RandomPolicy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Example of how to use the predict method to get greedy actions from observations. The shape of the output actions depends on whether the action space is continuous or discrete.

```python
import numpy as np

# 100 observations with shape of (10,)
x = np.random.random((10, 10))

actions = algo.predict(x)
# actions.shape == (100, action size) for continuous control
# actions.shape == (100,) for discrete control
```

--------------------------------

### Install Additional Packages

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/cli.md

Installs additional packages required by d3rlpy, such as environments or datasets. Specify the name of the package to install.

```bash
$ d3rlpy install <name>
```

```bash
# Install D4RL package
$ d3rlpy install d4rl
```

--------------------------------

### Install d3rlpy via Anaconda

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/installation.md

Install d3rlpy from the conda-forge channel.

```bash
$ conda install -c conda-forge d3rlpy
```

--------------------------------

### Prepare Pretrained Policies

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/after_training_policies.md

Prepare a dataset and environment, then set up and train a d3rlpy algorithm like CQL.

```python
import d3rlpy

# prepare dataset and environment
dataset, env = d3rlpy.datasets.get_dataset('pendulum-random')

# setup algorithm
cql_old = d3rlpy.algos.CQLConfig().create(device="cuda:0")

# start offline training
cql_old.fit(dataset, n_steps=100000)
```

--------------------------------

### IQLConfig.create

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Creates and returns an IQL algorithm object based on the configuration.

```APIDOC
#### create(device=False, enable_ddp=False)

Returns algorithm object.

* **Parameters:**
  * **device** (*Union* *[**int* *,* *str* *,* *bool* *]*) – device option. If the value is
    boolean and True, `cuda:0` will be used. If the value is
    integer, `cuda:<device>` will be used. If the value is string
    in torch device style, the specified device will be used.
  * **enable_ddp** (*bool*) – Flag to wrap models with DDP to enable Data
    Distributed Parallel training.
* **Returns:**
  algorithm object.
* **Return type:**
  [*IQL*](#d3rlpy.algos.IQL)
```

--------------------------------

### Initialize ReplayBuffer Components

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Demonstrates the initialization of individual modular components for the ReplayBuffer.

```python
import d3rlpy

# Buffer component
buffer = d3rlpy.dataset.FIFOBuffer(limit=100000)

# TransitionPicker component
transition_picker = d3rlpy.dataset.BasicTransitionPicker()

# TrajectorySlicer component
trajectory_slicer = d3rlpy.dataset.BasicTrajectorySlicer()

# WriterPreprocess component
writer_preprocessor = d3rlpy.dataset.BasicWriterPreprocess()
```

--------------------------------

### Decision Transformer Usage Example

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Example of how to use the Decision Transformer algorithm for offline training and interaction with an environment.

```APIDOC
## Decision Transformer

Decision Transformer-based algorithms usually require tricky interaction codes for evaluation.
In d3rlpy, those algorithms provide `as_stateful_wrapper` method to easily integrate the algorithm into your system.

```python
import d3rlpy

dataset, env = d3rlpy.datasets.get_pendulum()

dt = d3rlpy.algos.DecisionTransformerConfig().create(device="cuda:0")

# offline training
dt.fit(
   dataset,
   n_steps=100000,
   n_steps_per_epoch=1000,
   eval_env=env,
   eval_target_return=0,  # specify target environment return
)

# wrap as stateful actor for interaction
actor = dt.as_stateful_wrapper(target_return=0)

# interaction
observation, reward = env.reset(), 0.0
while True:
    action = actor.predict(observation, reward)
    observation, reward, done, truncated, _ = env.step(action)
    if done or truncated:
        break

# reset history
actor.reset()
```
```

--------------------------------

### Initialize Gym Environments

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/online.ipynb

Creates the training and evaluation environments using OpenAI Gym. Ensure the environment name is valid.

```python
import gym

env = gym.make('CartPole-v1')
eval_env = gym.make('CartPole-v1')
```

--------------------------------

### TransformerActionSampler Usage

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Example of using TransformerActionSampler with DecisionTransformer for offline training and stateful interaction.

```APIDOC
## TransformerActionSampler

### Description
`TransformerActionSampler` is an interface to sample actions from DecisionTransformer outputs. Basically, the default action-sampler will be used if you don’t explicitly specify one.

### Usage Example
```python
import d3rlpy

dataset, env = d3rlpy.datasets.get_pendulum()

dt = d3rlpy.algos.DecisionTransformerConfig().create(device="cuda:0")

# offline training
dt.fit(
   dataset,
   n_steps=100000,
   n_steps_per_epoch=1000,
   eval_env=env,
   eval_target_return=0,
   # manually specify action-sampler
   eval_action_sampler=d3rlpy.algos.IdentityTransformerActionSampler(),
)

# wrap as stateful actor for interaction with manually specified action-sampler
actor = dt.as_stateful_wrapper(
    target_return=0,
    action_sampler=d3rlpy.algos.IdentityTransformerActionSampler(),
)
```
```

--------------------------------

### Finetune with Different Algorithm (Discrete)

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Demonstrates finetuning a DQN policy after initial offline training with a different algorithm (DiscreteCQL). It involves transferring the Q-function.

```python
# setup offline RL algorithm
cql = d3rlpy.algos.DiscreteCQLConfig().create()

# train offline
cql.fit(dataset, n_steps=100000)

# transfer to DQN
dqn = d3rlpy.algos.DQNConfig().create()
dqn.build_with_env(env)
dqn.copy_q_function_from(cql)

# start finetuning
dqn.fit_online(env, buffer, explorer, n_steps=100000)
```

--------------------------------

### Prepare Dataset and Environment

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Sets up a built-in dataset and environment for CartPole-v0. This is the initial step for offline training and subsequent finetuning.

```python
import d3rlpy

# setup random CartPole-v0 dataset and environment
dataset, env = d3rlpy.datasets.get_dataset("cartpole-random")
```

--------------------------------

### Setup TD Error Evaluator

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/getting_started.md

Initialize a TDErrorEvaluator to compute metrics using the provided dataset episodes.

```python
from d3rlpy.metrics import TDErrorEvaluator

# calculate metrics with training dataset
td_error_evaluator = TDErrorEvaluator(episodes=dataset.episodes)
```

--------------------------------

### Finetune with Different Algorithm (Continuous)

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Shows how to finetune a continuous control policy (SAC) after offline training with another algorithm (CQL). This involves transferring both Q-function and policy.

```python
# offline RL
cql = d3rlpy.algos.CQLConfig().create()
cql.fit(dataset, n_steps=100000)

# transfer to SAC
sac = d3rlpy.algos.SACConfig().create()
sac.build_with_env(env)
sac.copy_q_function_from(cql)
sac.copy_policy_from(cql)

# online RL
sac.fit_online(env, buffer, n_steps=100000)
```

--------------------------------

### Reset Optimizer States

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Resets the internal states of the optimizer. This is useful for fine-tuning policies by ensuring a clean start for optimization.

```python
algo.reset_optimizer_states()
```

--------------------------------

### create

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Creates and returns a CQL algorithm object with the specified configuration.

```APIDOC
## create

### Description
Returns algorithm object.

### Parameters
* **device** (Union[int, str, bool]) - device option. If the value is boolean and True, `cuda:0` will be used. If the value is integer, `cuda:<device>` will be used. If the value is string in torch device style, the specified device will be used.
* **enable_ddp** (bool) - Flag to wrap models with DDP to enable Data Distributed Parallel training.

### Returns
algorithm object.

### Return type
[CQL](#d3rlpy.algos.CQL)
```

--------------------------------

### Initialize ReplayBuffer with Gym Environment

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Initializes the ReplayBuffer using an existing Gym environment. Ensure the environment is compatible with d3rlpy's requirements.

```python
import d3rlpy
import gym
env = gym.make("Pendulum-v1")
replay_buffer = d3rlpy.dataset.ReplayBuffer(
   buffer=buffer,
   transition_picker=transition_picker,
   trajectory_slicer=trajectory_slicer,
   writer_preprocessor=writer_preprocessor,
   env=env,
)
```

--------------------------------

### create(device, enable_ddp)

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Factory method to create a BCQ algorithm object.

```APIDOC
## create(device, enable_ddp)

### Description
Returns algorithm object.

### Method
create

### Parameters
#### Device
* **device** (*Union* *[**int* *,* *str* *,* *bool* *]*) – device option. If the value is boolean and True, `cuda:0` will be used. If the value is integer, `cuda:<device>` will be used. If the value is string in torch device style, the specified device will be used.
#### Enable DDP
* **enable_ddp** (*bool*) – Flag to wrap models with DDP to enable Data Distributed Parallel training.

### Returns
* algorithm object.

### Return type
*BCQ*
```

--------------------------------

### Combining multiple logger adapters

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/logging.md

Use CombineAdapterFactory to use multiple logger adapters simultaneously. This example combines file logging and TensorBoard logging.

```python
# combine FileAdapterFactory and TensorboardAdapterFactory
logger_adapter = d3rlpy.logging.CombineAdapterFactory([
   d3rlpy.logging.FileAdapterFactory(root_dir="d3rlpy_logs"),
   d3rlpy.logging.TensorboardAdapterFactory(root_dir="tensorboard_logs"),
])

dqn.fit(dataset=dataset, n_steps=100000, logger_adapter=logger_adapter)
```

--------------------------------

### RandomPolicy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

The RandomPolicy class for continuous control. It provides methods to get the action type, predict greedy actions, predict action-values, and sample actions.

```APIDOC
## class d3rlpy.algos.RandomPolicy

Bases: [`QLearningAlgoBase`](#d3rlpy.algos.QLearningAlgoBase)[`None`, [`RandomPolicyConfig`](#d3rlpy.algos.RandomPolicyConfig)]

* **Parameters:**
  **config** ([*RandomPolicyConfig*](#d3rlpy.algos.RandomPolicyConfig))

#### get_action_type()

Returns action type (continuous or discrete).

* **Returns:** action type.
* **Return type:** *ActionSpace*

#### predict(x)

Returns greedy actions.

```python
# 100 observations with shape of (10,)
x = np.random.random((100, 10))

actions = algo.predict(x)
# actions.shape == (100, action size) for continuous control
# actions.shape == (100,) for discrete control
```

* **Parameters:**
  **x** (*ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]*  *|* *Sequence* *[**ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]* *]*) – Observations
* **Returns:** Greedy actions
* **Return type:** *ndarray*[*Any*, *dtype*[*Any*]]

#### predict_value(x, action)

Returns predicted action-values.

```python
# 100 observations with shape of (10,)
x = np.random.random((100, 10))

# for continuous control
# 100 actions with shape of (2,)
actions = np.random.random((100, 2))

# for discrete control
# 100 actions in integer values
actions = np.random.randint(2, size=100)

values = algo.predict_value(x, actions)
# values.shape == (100,)
```

* **Parameters:**
  * **x** (*ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]*  *|* *Sequence* *[**ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]* *]*) – Observations
  * **action** (*ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]*) – Actions
* **Returns:** Predicted action-values
* **Return type:** *ndarray*[*Any*, *dtype*[*Any*]]

#### sample_action(x)

Returns sampled actions.

The sampled actions are identical to the output of predict method if
the policy is deterministic.

* **Parameters:**
  **x** (*ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]*  *|* *Sequence* *[**ndarray* *[**Any* *,* *dtype* *[**Any* *]* *]* *]*) – Observations.
* **Returns:** Sampled actions.
* **Return type:** *ndarray*[*Any*, *dtype*[*Any*]]
```

--------------------------------

### Configure and Create DiscreteCQL Agent

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/atari.ipynb

Configure and create a DiscreteCQL agent with specified hyperparameters. This includes setting learning rate, batch size, alpha, Q-function factory, observation scaler, and target update interval.

```python
cql = d3rlpy.algos.DiscreteCQLConfig(
    learning_rate=5e-5,
    optim_factory=d3rlpy.optimizers.AdamFactory(eps=1e-2 / 32),
    batch_size=32,
    alpha=4.0,
    q_func_factory=d3rlpy.models.q_functions.QRQFunctionFactory(
        n_quantiles=200
    ),
    observation_scaler=d3rlpy.preprocessing.PixelObservationScaler(),
    target_update_interval=2000,
    reward_scaler=d3rlpy.preprocessing.ClipRewardScaler(-1.0, 1.0),
).create(device="cuda:0")
```

--------------------------------

### Evaluate Agent with Video Recording

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/tpu.ipynb

Wraps the environment with RecordVideo and evaluates the trained Q-learning agent.

```python
import gym
from gym.wrappers import RecordVideo

# start virtual display
d3rlpy.notebook_utils.start_virtual_display()

# wrap RecordVideo wrapper
env = RecordVideo(gym.make("CartPole-v1", render_mode="rgb_array"), './video')

# evaluate
d3rlpy.metrics.evaluate_qlearning_with_environment(cql, env)
```

--------------------------------

### Initialize CQL with QRQFunctionFactory

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/q_functions.md

This snippet shows how to initialize a CQL algorithm with a QRQFunctionFactory. Ensure d3rlpy is imported.

```python
import d3rlpy

cql = d3rlpy.algos.CQLConfig(q_func_factory=d3rlpy.models.QRQFunctionFactory())

```

--------------------------------

### Setup Custom EncoderFactory

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/customize_neural_network.md

Define an `EncoderFactory` dataclass for your custom encoder. This factory is responsible for creating instances of your custom encoder and providing a type identifier for serialization.

```python
import dataclasses

@dataclasses.dataclass()
class CustomEncoderFactory(d3rlpy.models.EncoderFactory):
    feature_size: int

    def create(self, observation_shape):
        return CustomEncoder(observation_shape, self.feature_size)

    @staticmethod
    def get_type() -> str:
        return "custom"
```

--------------------------------

### Instantiate SAC Algorithm with Different Devices

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Instantiate the SAC algorithm with different device configurations (CPU, GPU).

```python
import d3rlpy

# instantiate algorithm with CPU
sac = d3rlpy.algos.SACConfig().create(device="cpu:0")
# instantiate algorithm with GPU
sac = d3rlpy.algos.SACConfig().create(device="cuda:0")
# instantiate algorithm with the 2nd GPU
sac = d3rlpy.algos.SACConfig().create(device="cuda:1")
```

--------------------------------

### Evaluate and Record Video

Source: https://github.com/takuseno/d3rlpy/blob/master/tutorials/online.ipynb

Evaluates the trained DQN agent and records the gameplay as a video. Requires the 'RecordVideo' wrapper from Gym.

```python
from gym.wrappers import RecordVideo

# start virtual display
d3rlpy.notebook_utils.start_virtual_display()

# wrap Monitor wrapper
env = RecordVideo(gym.make("CartPole-v1", render_mode="rgb_array"), './video')

# evaluate
d3rlpy.metrics.evaluate_qlearning_with_environment(dqn, env)
```

--------------------------------

### Implement Custom TrajectorySlicer

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Example of a custom TrajectorySlicer for Decision Transformer-based algorithms. It handles slicing and padding of trajectory data, including preparing terminal flags and returns-to-go.

```python
import d3rlpy

class CustomTrajectorySlicer(d3rlpy.dataset.TrajectorySlicerProtocol):
    def __call__(
        self, episode: d3rlpy.dataset.EpisodeBase, end_index: int, size: int
    ) -> d3rlpy.dataset.PartialTrajectory:
        end = end_index + 1
        start = max(end - size, 0)
        actual_size = end - start

        # prepare terminal flags
        terminals = np.zeros((actual_size, 1), dtype=np.float32)
        if episode.terminated and end_index == episode.size() - 1:
            terminals[-1][0] = 1.0

        # slice data
        observations = episode.observations[start:end]
        actions = episode.actions[start:end]
        rewards = episode.rewards[start:end]
        ret = np.sum(episode.rewards[start:])
        all_returns_to_go = ret - np.cumsum(episode.rewards[start:], axis=0)
        returns_to_go = all_returns_to_go[:actual_size].reshape((-1, 1))

        # prepare metadata
        timesteps = np.arange(start, end)
        masks = np.ones(end - start, dtype=np.float32)

        # compute backward padding size
        pad_size = size - actual_size

        if pad_size == 0:
            return d3rlpy.dataset.PartialTrajectory(
                observations=observations,
                actions=actions,
                rewards=rewards,
                returns_to_go=returns_to_go,
                terminals=terminals,
                timesteps=timesteps,
                masks=masks,
                length=size,
            )

        return d3rlpy.dataset.PartialTrajectory(
            observations=d3rlpy.dataset.batch_pad_observations(observations, pad_size),
            actions=d3rlpy.dataset.batch_pad_array(actions, pad_size),
            rewards=d3rlpy.dataset.batch_pad_array(rewards, pad_size),
            returns_to_go=d3rlpy.dataset.batch_pad_array(returns_to_go, pad_size),
            terminals=d3rlpy.dataset.batch_pad_array(terminals, pad_size),
            timesteps=d3rlpy.dataset.batch_pad_array(timesteps, pad_size),
            masks=d3rlpy.dataset.batch_pad_array(masks, pad_size),
            length=size,
        )
```

--------------------------------

### Manual Initialization of Min-Max Action Scaler

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/preprocessing.md

Initialize a MinMaxActionScaler with predefined minimum and maximum values.

```python
from d3rlpy.preprocessing import MinMaxActionScaler

action_scaler = MinMaxActionScaler(minimum=..., maximum=...)

cql = CQLConfig(action_scaler=action_scaler).create()
```

--------------------------------

### CalQL Algorithm Creation

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

This section details how to create an instance of the CalQL algorithm, including its configurable parameters.

```APIDOC
## create(device=False, enable_ddp=False)

### Description
Returns algorithm object.

### Parameters
* **device** (*Union* *[**int* *,* *str* *,* *bool* *]*) – device option. If the value is boolean and True, `cuda:0` will be used. If the value is integer, `cuda:<device>` will be used. If the value is string in torch device style, the specified device will be used.
* **enable_ddp** (*bool*) – Flag to wrap models with DDP to enable Data Distributed Parallel training.

### Returns
algorithm object.

### Return type
[*CalQL*](#d3rlpy.algos.CalQL)
```

--------------------------------

### QLearningAlgoBase.copy_q_function_optim_from

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Copies Q-function optimizer states from another Q-learning algorithm instance. This enables the transfer of optimizer states, which can be beneficial for resuming training or transferring complex training setups.

```APIDOC
## copy_q_function_optim_from

Copies Q-function optimizer states from the given algorithm.

```python
# pretrain with static dataset
cql = d3rlpy.algos.CQL()
cql.fit(dataset, n_steps=100000)

# transfer to online algorithm
sac = d3rlpy.algos.SAC()
sac.create_impl(cql.observation_shape, cql.action_size)
sac.copy_policy_optim_from(cql)
```

### Parameters
* **algo** ([*QLearningAlgoBase*](#d3rlpy.algos.QLearningAlgoBase) *[**QLearningAlgoImplBase* *,* *LearnableConfig* *]*) – Algorithm object.

### Return type
None
```

--------------------------------

### Configure SAC with Different Q-Function Factories

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/use_distributional_q_function.md

Demonstrates how to configure the SAC algorithm with different Q-function factories: the default MeanQFunction, Quantile Regression Q-Function, and Implicit Quantile Network Q-Function. This allows for experimenting with various Q-function architectures to potentially enhance performance.

```python
# default standard Q-function
mean_q_function = d3rlpy.models.MeanQFunctionFactory()
sac = d3rlpy.algos.SACConfig(q_func_factory=mean_q_function).create()
```

```python
# Quantile Regression Q-function
qr_q_function = d3rlpy.models.QRQFunctionFactory(n_quantiles=200)
sac = d3rlpy.algos.SACConfig(q_func_factory=qr_q_function).create()
```

```python
# Implicit Quantile Network Q-function
iqn_q_function = d3rlpy.models.IQNQFunctionFactory(
    n_quantiles=32,
    n_greedy_quantiles=64,
    embed_size=64,
)
sac = d3rlpy.algos.SACConfig(q_func_factory=iqn_q_function).create()
```

--------------------------------

### DiscreteSACConfig.create

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Factory method to create a DiscreteSAC algorithm instance from its configuration.

```APIDOC
#### create(device=False, enable_ddp=False)

Returns algorithm object.

* **Parameters:**
  * **device** (*Union* *[**int* *,* *str* *,* *bool* *]*) – device option. If the value is
    boolean and True, `cuda:0` will be used. If the value is
    integer, `cuda:<device>` will be used. If the value is string
    in torch device style, the specified device will be used.
  * **enable_ddp** (*bool*) – Flag to wrap models with DDP to enable Data
    Distributed Parallel training.
* **Returns:**
  algorithm object.
* **Return type:**
  [*DiscreteSAC*](#d3rlpy.algos.DiscreteSAC)
```

--------------------------------

### Load Trained Policies from .pt file

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/after_training_policies.md

Save a trained policy's model weights to a .pt file. Then, manually set up the algorithm and load the weights.

```python
# save pt file
cql_old.save_model("model.pt")
# setup algorithm manually
cql = d3rlpy.algos.CQLConfig().create()

# choose one of three to build PyTorch models

# if you have MDPDataset object
cql.build_with_dataset(dataset)
# or if you have Gym-styled environment object
cql.build_with_env(env)
# or manually set observation shape and action size
cql.create_impl((3,), 1)

# load pretrained model
cql.load_model("model.pt")
```

--------------------------------

### Finetune with Saved Policy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/finetuning.md

Loads a previously saved policy and finetunes it online with the environment. This is useful for continuing training or adapting a policy.

```python
# setup algorithm
dqn = d3rlpy.load_learnable("dqn_model.d3")

# start finetuning
dqn.fit_online(env, buffer, explorer, n_steps=100000)
```

--------------------------------

### Get Action Type for CRR

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/algos.md

Method to determine the action space type (continuous or discrete) handled by the CRR algorithm. This is useful for understanding the algorithm's compatibility with different environments.

```python
def get_action_type()
```

--------------------------------

### Initialize ReplayBuffer with Pre-collected Dataset

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/dataset.md

Initializes the ReplayBuffer using a pre-collected dataset. This is useful for offline reinforcement learning scenarios.

```python
import d3rlpy
dataset, _ = d3rlpy.datasets.get_pendulum()
replay_buffer = d3rlpy.dataset.ReplayBuffer(
   buffer=buffer,
   transition_picker=transition_picker,
   trajectory_slicer=trajectory_slicer,
   writer_preprocessor=writer_preprocessor,
   episodes=dataset.episodes,
)
```

--------------------------------

### Load and Infer Action with TorchScript Q-learning Policy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tutorials/after_training_policies.md

Load a TorchScript policy and use it to infer actions from random observations. This demonstrates the basic usage of an exported policy.

```python
import torch

# load TorchScript policy
policy = torch.jit.load("policy.pt")

# infer the action
action = policy(torch.rand(1, 3))
assert action.shape == (1, 1)
```

--------------------------------

### Configure Multi-Step Transitions and Replay Buffer in d3rlpy

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/tips.md

Utilize multi-step transitions for potentially better learning and configure a FIFO replay buffer with a custom transition picker.

```python
import d3rlpy

# multi-step transition sampling
transition_picker = d3rlpy.dataset.MultiStepTransitionPicker(
    n_steps=3,
    gamma=0.99,
)
# replay buffer for experience replay
buffer = d3rlpy.dataset.create_fifo_replay_buffer(
    limit=100000,
    env=env,
    transition_picker=transition_picker,
)
```

--------------------------------

### Manual Initialization of Min-Max Reward Scaler

Source: https://github.com/takuseno/d3rlpy/blob/master/docs/references/preprocessing.md

Initialize a MinMaxRewardScaler with predefined minimum and maximum reward values.

```python
from d3rlpy.preprocessing import MinMaxRewardScaler

reward_scaler = MinMaxRewardScaler(minimum=..., maximum=...)

cql = CQLConfig(reward_scaler=reward_scaler).create()
```
Tokens:
Raw