Hands-on Reinforcement Learning

https://hrl.boyuai.com/
Admin
Hands-on Reinforcement Learning is a comprehensive learning resource covering fundamental to...
Tokens:144,775
Snippets:334
Trust Score:6.4
Update:10 months ago
Show doc for...
Tokens:
Raw
### Training Setup and Hyperparameters

Source: https://hrl.boyuai.com/chapter/3/%E5%A4%9A%E6%99%BA%E8%83%BD%E4%BD%93%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0%E8%BF%9B%E9%98%B6

Configuration for training a multi-agent reinforcement learning setup. It defines hyperparameters such as the number of episodes, buffer size, learning rates, discount factor, and batch size. It also includes environment setup and replay buffer initialization.

```python
num_episodes = 5000
episode_length = 25  # 每条序列的最大长度
buffer_size = 100000
hidden_dim = 64
actor_lr = 1e-2
critic_lr = 1e-2
gamma = 0.95
tau = 1e-2
batch_size = 1024
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
update_interval = 100
minimal_size = 4000


env_id = "simple_adversary"
env = make_env(env_id)
replay_buffer = rl_utils.ReplayBuffer(buffer_size)


state_dims = []
action_dims = []
for action_space in env.action_space:
```

--------------------------------

### SAC Agent Initialization and Training Setup (Python)

Source: https://hrl.boyuai.com/chapter/2/sac%E7%AE%97%E6%B3%95

This Python code snippet sets up the hyperparameters and initializes the SAC agent, environment, and replay buffer. It configures the device for computation (CPU or GPU) and seeds random number generators for reproducibility.

```python
actor_lr = 1e-3
critic_lr = 1e-2
alpha_lr = 1e-2
num_episodes = 200
hidden_dim = 128
gamma = 0.98
tau = 0.005  # 软更新参数
buffer_size = 10000
minimal_size = 500
batch_size = 64
target_entropy = -1
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


env_name = 'CartPole-v0'
env = gym.make(env_name)
random.seed(0)
np.random.seed(0)
env.seed(0)
torch.manual_seed(0)
replay_buffer = rl_utils.ReplayBuffer(buffer_size)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
agent = SAC(state_dim, hidden_dim, action_dim, actor_lr, critic_lr, alpha_lr,
            target_entropy, tau, gamma, device)


return_list = rl_utils.train_off_policy_agent(env, agent, num_episodes,
                                              replay_buffer, minimal_size,
                                              batch_size)
```

--------------------------------

### Python GAIL Experiment Setup and Training Loop

Source: https://hrl.boyuai.com/chapter/3/%E6%A8%A1%E4%BB%BF%E5%AD%A6%E4%B9%A0

This Python code sets up the environment, initializes the PPO agent and GAIL algorithm, and runs the training loop for a specified number of episodes. It collects trajectory data and uses the GAIL.learn method to update the policy.

```python
env.seed(0)
torch.manual_seed(0)
lr_d = 1e-3
agent = PPO(state_dim, hidden_dim, action_dim, actor_lr,
              critic_lr, lmbda, epochs, eps, gamma, device)
gail = GAIL(agent, state_dim, action_dim, hidden_dim, lr_d)
n_episode = 500
return_list = []


with tqdm(total=n_episode, desc="进度条") as pbar:
    for i in range(n_episode):
        episode_return = 0
        state = env.reset()
        done = False
        state_list = []
        action_list = []
        next_state_list = []
        done_list = []
        while not done:
            action = agent.take_action(state)
            next_state, reward, done, _ = env.step(action)
            state_list.append(state)
            action_list.append(action)
            next_state_list.append(next_state)
            done_list.append(done)
            state = next_state
            episode_return += reward
        return_list.append(episode_return)
        gail.learn(expert_s, expert_a, state_list, action_list,
                   next_state_list, done_list)
        if (i + 1) % 10 == 0:
            pbar.set_postfix({'return': '%.3f' % np.mean(return_list[-10:])})
        pbar.update(1)
```

--------------------------------

### Install and Import MPE Environment Dependencies

Source: https://hrl.boyuai.com/chapter/3/%E5%A4%9A%E6%99%BA%E8%83%BD%E4%BD%93%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0%E8%BF%9B%E9%98%B6

Installs a specific version of the gym library and imports necessary modules for the multi-agent particle environment (MPE). This setup is required to resolve version conflicts and utilize the MPE functionalities.

```python
!pip install --upgrade gym==0.10.5 -q
import gym
from multiagent.environment import MultiAgentEnv
import multiagent.scenarios as scenarios
```

--------------------------------

### Dueling DQN Training Initialization and Execution (PyTorch)

Source: https://hrl.boyuai.com/chapter/2/dqn%E6%94%B9%E8%BF%9B%E7%AE%97%E6%B3%95

Demonstrates the setup and execution of the Dueling DQN training process. It initializes random seeds for reproducibility, sets up the replay buffer, instantiates the Dueling DQN agent, and then calls the `train_DQN` function. This section assumes `rl_utils`, `train_DQN`, and other necessary components are defined elsewhere.

```python
random.seed(0)
np.random.seed(0)
env.seed(0)
torch.manual_seed(0)
replay_buffer = rl_utils.ReplayBuffer(buffer_size) # Assuming rl_utils and buffer_size are defined
agent = DQN(state_dim, hidden_dim, action_dim, lr, gamma, epsilon, target_update, device, 'DuelingDQN')
return_list, max_q_value_list = train_DQN(agent, env, num_episodes, replay_buffer, minimal_size, batch_size) # Assuming train_DQN is defined

```

--------------------------------

### FrozenLake Environment Setup and Exploration with OpenAI Gym in Python

Source: https://hrl.boyuai.com/chapter/1/%E5%8A%A8%E6%80%81%E8%A7%84%E5%88%92%E7%AE%97%E6%B3%95

Sets up and explores the FrozenLake-v0 environment from OpenAI Gym. It demonstrates how to initialize the environment, access its state transition dynamics, and identify goal and hole states. This code is a prerequisite for applying reinforcement learning algorithms to this environment.

```python
import gym
env = gym.make("FrozenLake-v0")  # 创建环境
env = env.unwrapped  # 解封装才能访问状态转移矩阵P
env.render()  # 环境渲染,通常是弹窗显示或打印出可视化的环境

holes = set()
ends = set()
for s in env.P:
    for a in env.P[s]:
        for s_ in env.P[s][a]:
            if s_[2] == 1.0:  # 获得奖励为1,代表是目标
                ends.add(s_[1])
            if s_[3] == True:
                holes.add(s_[1])
holes = holes - ends
print("冰洞的索引:", holes)
print("目标的索引:", ends)


for a in env.P[14]:  # 查看目标左边一格的状态转移信息
    print(env.P[14][a])

```

--------------------------------

### Setup Multi-Agent Particle Environment (MPE) in Python

Source: https://hrl.boyuai.com/chapter/3/%E5%A4%9A%E6%99%BA%E8%83%BD%E4%BD%93%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0%E8%BF%9B%E9%98%B6

Installs and sets up the 'multiagent-particle-envs' environment for practicing MARL algorithms like MADDPG. This involves cloning the repository, installing it as an editable package, and updating the system path to access the environment modules.

```python
!git clone https://github.com/boyu-ai/multiagent-particle-envs.git --quiet
!pip install -e multiagent-particle-envs
import sys
sys.path.append("multiagent-particle-envs")

```

--------------------------------

### CQL Training Loop Setup and Execution (Python)

Source: https://hrl.boyuai.com/chapter/3/%E7%A6%BB%E7%BA%BF%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0

This Python script initializes the CQL agent, sets hyperparameters, and runs the training loop. It includes environment interaction for policy evaluation (not for training) and agent updates using data sampled from a replay buffer. Progress is tracked and displayed using tqdm. Dependencies include random, numpy, torch, and tqdm.

```python
random.seed(0)
np.random.seed(0)
env.seed(0)
torch.manual_seed(0)


beta = 5.0
num_random = 5
num_epochs = 100
num_trains_per_epoch = 500


agent = CQL(state_dim, hidden_dim, action_dim, action_bound, actor_lr,
            critic_lr, alpha_lr, target_entropy, tau, gamma, device, beta,
            num_random)


return_list = []
for i in range(10):
    with tqdm(total=int(num_epochs / 10), desc='Iteration %d' % i) as pbar:
        for i_epoch in range(int(num_epochs / 10)):
            # 此处与环境交互只是为了评估策略,最后作图用,不会用于训练
            epoch_return = 0
            state = env.reset()
            done = False
            while not done:
                action = agent.take_action(state)
                next_state, reward, done, _ = env.step(action)
                state = next_state
                epoch_return += reward
            return_list.append(epoch_return)


            for _ in range(num_trains_per_epoch):
                b_s, b_a, b_r, b_ns, b_d = replay_buffer.sample(batch_size)
                transition_dict = {
                    'states': b_s,
                    'actions': b_a,
                    'next_states': b_ns,
                    'rewards': b_r,
                    'dones': b_d
                }
                agent.update(transition_dict)


            if (i_epoch + 1) % 10 == 0:
                pbar.set_postfix({
                    'epoch':
                    '%d' % (num_epochs / 10 * i + i_epoch + 1),
                    'return':
                    '%.3f' % np.mean(return_list[-10:])
                })
            pbar.update(1)
```

--------------------------------

### MBPO Training and Visualization Setup in Python

Source: https://hrl.boyuai.com/chapter/3/%E5%9F%BA%E4%BA%8E%E6%A8%A1%E5%9E%8B%E7%9A%84%E7%AD%96%E7%95%A5%E4%BC%98%E5%8C%96

Sets up and runs the Model-Based Policy Optimization (MBPO) algorithm for the Pendulum-v0 environment. This code defines environment parameters, initializes the agent, model, and fake environment, and then proceeds to train the MBPO agent. Finally, it plots the training returns per episode.

```python
real_ratio = 0.5
env_name = 'Pendulum-v0'
env = gym.make(env_name)
num_episodes = 20
actor_lr = 5e-4
critic_lr = 5e-3
alpha_lr = 1e-3
hidden_dim = 128
gamma = 0.98
tau = 0.005  # 软更新参数
buffer_size = 10000
target_entropy = -1
model_alpha = 0.01  # 模型损失函数中的加权权重
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.shape[0]
action_bound = env.action_space.high[0]  # 动作最大值

rollout_batch_size = 1000
rollout_length = 1  # 推演长度k,推荐更多尝试
model_pool_size = rollout_batch_size * rollout_length

agent = SAC(state_dim, hidden_dim, action_dim, action_bound, actor_lr,
            critic_lr, alpha_lr, target_entropy, tau, gamma)
model = EnsembleDynamicsModel(state_dim, action_dim, model_alpha)
fake_env = FakeEnv(model)
env_pool = ReplayBuffer(buffer_size)
model_pool = ReplayBuffer(model_pool_size)
mbpo = MBPO(env, agent, fake_env, env_pool, model_pool, rollout_length,
            rollout_batch_size, real_ratio, num_episodes)

return_list = mbpo.train()

episodes_list = list(range(len(return_list)))
plt.plot(episodes_list, return_list)
plt.xlabel('Episodes')
plt.ylabel('Returns')
plt.title('MBPO on {}'.format(env_name))
plt.show()
```

--------------------------------

### DDPG Agent Initialization and Training Loop

Source: https://hrl.boyuai.com/chapter/3/%E7%9B%AE%E6%A0%87%E5%AF%BC%E5%90%91%E7%9A%84%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0

This snippet outlines the setup and training loop for a DDPG agent in a goal-conditioned environment using HER. It initializes hyperparameters, the environment, the replay buffer, and the DDPG agent. The training proceeds by interacting with the environment, storing trajectories, and periodically updating the agent using samples from the replay buffer with HER enabled.

```python
import torch
import numpy as np
import random
from tqdm import tqdm
# Assuming WorldEnv and DDPG are defined elsewhere
# from env import WorldEnv
# from ddpg import DDPG

actor_lr = 1e-3
critic_lr = 1e-3
hidden_dim = 128
state_dim = 4
action_dim = 2
action_bound = 1
sigma = 0.1
tau = 0.005
gamma = 0.98
num_episodes = 2000
n_train = 20
batch_size = 256
minimal_episodes = 200
buffer_size = 10000
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


random.seed(0)
np.random.seed(0)
torch.manual_seed(0)
env = WorldEnv() # Placeholder for environment definition
replay_buffer = ReplayBuffer_Trajectory(buffer_size)
agent = DDPG(state_dim, hidden_dim, action_dim, action_bound, actor_lr,
             critic_lr, sigma, tau, gamma, device) # Placeholder for DDPG agent definition


return_list = []
for i in range(10):
    with tqdm(total=int(num_episodes / 10), desc='Iteration %d' % i) as pbar:
        for i_episode in range(int(num_episodes / 10)):
            episode_return = 0
            state = env.reset()
            traj = Trajectory(state)
            done = False
            while not done:
                action = agent.take_action(state) # Placeholder for agent action selection
                state, reward, done = env.step(action)
                episode_return += reward
                traj.store_step(action, state, reward, done)
            replay_buffer.add_trajectory(traj)
            return_list.append(episode_return)
            if replay_buffer.size() >= minimal_episodes:
                for _ in range(n_train):
                    transition_dict = replay_buffer.sample(batch_size, True) # Use HER
                    agent.update(transition_dict) # Placeholder for agent update

```

--------------------------------

### DDPG Training Setup for Inverted Pendulum

Source: https://hrl.boyuai.com/chapter/2/ddpg%E7%AE%97%E6%B3%95

Configures the hyperparameters and initializes the environment and necessary components for training a DDPG agent on the Inverted Pendulum task. This includes setting learning rates, discount factor, buffer size, batch size, noise parameters, and device selection.

```python
import gym
import random
import numpy as np
import torch
import rl_utils

actor_lr = 3e-4
critic_lr = 3e-3
num_episodes = 200
hidden_dim = 64
gamma = 0.98
tau = 0.005  # 软更新参数
buffer_size = 10000
minimal_size = 1000
batch_size = 64
sigma = 0.01  # 高斯噪声标准差
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")


env_name = 'Pendulum-v0'
env = gym.make(env_name)
random.seed(0)
np.random.seed(0)
env.seed(0)
torch.manual_seed(0)
replay_buffer = rl_utils.ReplayBuffer(buffer_size)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.shape[0]
```

--------------------------------

### TRPO Training and Visualization Setup (Python)

Source: https://hrl.boyuai.com/chapter/2/trpo%E7%AE%97%E6%B3%95

This snippet sets up and executes the training of the TRPO agent on the CartPole-v0 environment. It defines hyperparameters, initializes the environment and agent, runs the training loop, and visualizes the results. Libraries used include gym, torch, and matplotlib.

```python
num_episodes = 500
hidden_dim = 128
gamma = 0.98
lmbda = 0.95
critic_lr = 1e-2
kl_constraint = 0.0005
alpha = 0.5
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


env_name = 'CartPole-v0'
env = gym.make(env_name)
env.seed(0)
torch.manual_seed(0)
agent = TRPO(hidden_dim, env.observation_space, env.action_space, lmbda,
             kl_constraint, alpha, critic_lr, gamma, device)
return_list = rl_utils.train_on_policy_agent(env, agent, num_episodes)


episodes_list = list(range(len(return_list)))
plt.plot(episodes_list, return_list)
plt.xlabel('Episodes')
plt.ylabel('Returns')
plt.title('TRPO on {}'.format(env_name))
plt.show()


mv_return = rl_utils.moving_average(return_list, 9)
plt.plot(episodes_list, mv_return)
plt.xlabel('Episodes')
plt.ylabel('Returns')
plt.title('TRPO on {}'.format(env_name))
plt.show()
```

--------------------------------

### Import Libraries for CQL Implementation

Source: https://hrl.boyuai.com/chapter/3/%E7%A6%BB%E7%BA%BF%E5%BC%BA%E5%8C%96%E5%AD%A6%E4%B9%A0

Imports essential libraries for numerical operations, environment interaction, progress visualization, random number generation, RL utilities, and PyTorch for deep learning components. These are foundational for building and training the CQL agent.

```python
import numpy as np
import gym
from tqdm import tqdm
import random
import rl_utils
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions import Normal
import matplotlib.pyplot as plt
```

--------------------------------

### PyTorch Device Setup and Swish Activation

Source: https://hrl.boyuai.com/chapter/3/%E6%A8%A1%E5%9E%8B%E9%A2%84%E6%B5%8B%E6%8E%A7%E5%88%B6

Sets up the PyTorch computation device (GPU or CPU) and defines a Swish activation function as a PyTorch nn.Module. The Swish function is implemented as x * sigmoid(x).

```python
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")




class Swish(nn.Module):
    ''' Swish激活函数 '''
    def __init__(self):
        super(Swish, self).__init__()


    def forward(self, x):
        return x * torch.sigmoid(x)
```

--------------------------------

### Training PPO on CartPole Environment in Python

Source: https://hrl.boyuai.com/chapter/2/ppo%E7%AE%97%E6%B3%95

This code snippet demonstrates how to set up and train the PPO agent within the CartPole environment. It initializes hyperparameters, defines the environment using OpenAI Gym, and creates an instance of the PPO agent. The training process is then initiated using a utility function 'rl_utils.train_on_policy_agent'. The code requires 'gym', 'torch', and 'rl_utils'.

```python
actor_lr = 1e-3
critic_lr = 1e-2
num_episodes = 500
hidden_dim = 128
gamma = 0.98
lmbda = 0.95
epochs = 10
eps = 0.2
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


env_name = 'CartPole-v0'
env = gym.make(env_name)
env.seed(0)
torch.manual_seed(0)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.n
agent = PPO(state_dim, hidden_dim, action_dim, actor_lr, critic_lr, lmbda,
            epochs, eps, gamma, device)


return_list = rl_utils.train_on_policy_agent(env, agent, num_episodes)

```

--------------------------------

### Import necessary libraries for MBPO

Source: https://hrl.boyuai.com/chapter/3/%E5%9F%BA%E4%BA%8E%E6%A8%A1%E5%9E%8B%E7%9A%84%E7%AD%96%E7%95%A5%E4%BC%98%E5%8C%96

Imports essential Python libraries for implementing the MBPO algorithm, including PyTorch for neural networks, Gym for environment interaction, and standard libraries for data manipulation and sampling.

```python
import gym
from collections import namedtuple
import itertools
from itertools import count
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.distributions.normal import Normal
import numpy as np
import collections
import random
import matplotlib.pyplot as plt
```

--------------------------------

### SAC Agent Initialization and Training Setup for Pendulum-v0 in Python

Source: https://hrl.boyuai.com/chapter/2/sac%E7%AE%97%E6%B3%95

This Python code sets up the Soft Actor-Critic (SAC) algorithm for the Pendulum-v0 environment. It initializes the environment, defines state and action dimensions, sets hyperparameters like learning rates, buffer size, and batch size, and instantiates the SAC agent and replay buffer. The training process is then initiated using a utility function for off-policy agents.

```python
env_name = 'Pendulum-v0'
env = gym.make(env_name)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.shape[0]
action_bound = env.action_space.high[0]  # 动作最大值
random.seed(0)
np.random.seed(0)
env.seed(0)
torch.manual_seed(0)


actor_lr = 3e-4
critic_lr = 3e-3
alpha_lr = 3e-4
num_episodes = 100
hidden_dim = 128
gamma = 0.99
tau = 0.005  # 软更新参数
buffer_size = 100000
minimal_size = 1000
batch_size = 64
target_entropy = -env.action_space.shape[0]
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


replay_buffer = rl_utils.ReplayBuffer(buffer_size)
agent = SACContinuous(state_dim, hidden_dim, action_dim, action_bound,
                      actor_lr, critic_lr, alpha_lr, target_entropy, tau,
                      gamma, device)


return_list = rl_utils.train_off_policy_agent(env, agent, num_episodes,
                                              replay_buffer, minimal_size,
                                              batch_size)
```

--------------------------------

### GAIL Training Loop and Visualization

Source: https://hrl.boyuai.com/chapter/3

This code snippet demonstrates the execution of the GAIL algorithm. It initializes the environment, agent, and GAIL instance, then runs the training loop for a specified number of episodes. After training, it plots the episode returns to visualize the learning performance.

```python
env.seed(0)
torch.manual_seed(0)
lr_d = 1e-3
agent = PPO(state_dim, hidden_dim, action_dim, actor_lr, critic_lr, lmbda,
            epochs, eps, gamma, device)
gail = GAIL(agent, state_dim, action_dim, hidden_dim, lr_d)
n_episode = 500
return_list = []


with tqdm(total=n_episode, desc="进度条") as pbar:
    for i in range(n_episode):
        episode_return = 0
        state = env.reset()
        done = False
        state_list = []
        action_list = []
        next_state_list = []
        done_list = []
        while not done:
            action = agent.take_action(state)
            next_state, reward, done, _ = env.step(action)
            state_list.append(state)
            action_list.append(action)
            next_state_list.append(next_state)
            done_list.append(done)
            state = next_state
            episode_return += reward
        return_list.append(episode_return)
        gail.learn(expert_s, expert_a, state_list, action_list,
                   next_state_list, done_list)
        if (i + 1) % 10 == 0:
            pbar.set_postfix({'return': '%.3f' % np.mean(return_list[-10:])})
        pbar.update(1)


```

```python
iteration_list = list(range(len(return_list)))
plt.plot(iteration_list, return_list)
plt.xlabel('Episodes')
plt.ylabel('Returns')
plt.title('GAIL on {}'.format(env_name))
plt.show()
```

--------------------------------

### Replay Buffer for Experience Storage

Source: https://hrl.boyuai.com/chapter/3/%E6%A8%A1%E5%9E%8B%E9%A2%84%E6%B5%8B%E6%8E%A7%E5%88%B6

The ReplayBuffer class stores transitions (state, action, reward, next_state, done) in a deque with a fixed capacity. It provides methods to add samples, get the current buffer size, and retrieve all stored samples, which is crucial for training the dynamics model.

```python
import collections
import numpy as np

class ReplayBuffer:
    def __init__(self, capacity):
        self.buffer = collections.deque(maxlen=capacity)


    def add(self, state, action, reward, next_state, done):
        self.buffer.append((state, action, reward, next_state, done))


    def size(self):
        return len(self.buffer)


    def return_all_samples(self):
        all_transitions = list(self.buffer)
        state, action, reward, next_state, done = zip(*all_transitions)
        return np.array(state), action, reward, np.array(next_state), done
```

--------------------------------

### Initialize DDPG Agent and Environment

Source: https://hrl.boyuai.com/chapter/2/ddpg%E7%AE%97%E6%B3%95

Initializes the DDPG agent with specified dimensions, learning rates, and other hyperparameters. It also sets up the environment and replay buffer for training. Dependencies include the DDPG class and utility functions.

```python
action_bound = env.action_space.high[0]  # 动作最大值
agent = DDPG(state_dim, hidden_dim, action_dim, action_bound, sigma, actor_lr, critic_lr, tau, gamma, device)
```

--------------------------------

### Initialize PPO Training Environment and Agent (Python)

Source: https://hrl.boyuai.com/chapter/2/ppo%E7%AE%97%E6%B3%95

Sets up hyperparameters, initializes the Pendulum-v0 environment with a specified seed, and creates a PPOContinuous agent. It ensures reproducibility by setting PyTorch's manual seed and determines the device (GPU or CPU) for computation. The code defines state and action dimensions based on the environment's observation and action spaces.

```python
actor_lr = 1e-4
critic_lr = 5e-3
num_episodes = 2000
hidden_dim = 128
gamma = 0.9
lmbda = 0.9
epochs = 10
eps = 0.2
device = torch.device("cuda") if torch.cuda.is_available() else torch.device(
    "cpu")


env_name = 'Pendulum-v0'
env = gym.make(env_name)
env.seed(0)
torch.manual_seed(0)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.shape[0]  # 连续动作空间
agent = PPOContinuous(state_dim, hidden_dim, action_dim, actor_lr, critic_lr,
                      lmbda, epochs, eps, gamma, device)


return_list = rl_utils.train_on_policy_agent(env, agent, num_episodes)
```
Tokens:
Raw