### Install HoST-pytorch

Source: https://github.com/lucidrains/host-pytorch/blob/main/README.md

Install the HoST-pytorch library using pip.

```bash
pip install HoST-pytorch
```

--------------------------------

### Usage Example for HoST-pytorch Agent

Source: https://github.com/lucidrains/host-pytorch/blob/main/README.md

Demonstrates how to initialize an Agent, interact with a mock environment, learn from memories, and save the trained policy. Ensure the environment and mock hyperparameters are correctly set up.

```python
import torch
from host_pytorch import Agent
from host_pytorch.mock_env import Env, mock_hparams

env = Env()

agent = Agent(
    num_actions = (10, 10, 20),
    actor = dict(
        dims = (env.dim_state, 256, 128),
    ),
    critics = dict(
        dims = (env.dim_state, 256),
    ),
    reward_hparams = mock_hparams()
)

memories = agent(env)

agent.learn(memories)

agent.save('./standing-up-policy.pt', overwrite = True)
```

--------------------------------

### Configure and Use Critics Network

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Initialize a Critics network with weights for different reward groups. Use it to get value estimates, calculate advantages, and compute critic loss for training, with optional past action conditioning.

```python
import torch
from host_pytorch import Critics

# Create critics with weights for each reward group
critics = Critics(
    weights = (2.5, 1.0, 0.1, 1.0),  # Weights for 4 reward groups
    num_critics = 4,
    num_actions = (10, 10, 20),
    dims = (512, 256),  # MLP dimensions
    dim_action_embed = 16,
    value_eps_clip = 0.4  # Value function clipping
)

# Get value estimates
state = torch.randn(4, 512)
values = critics(state)  # Shape: (4, 4) - batch x num_critics

# With past actions conditioning
past_actions = torch.randint(0, 10, (4, 3, 3))
values = critics(state, past_actions=past_actions)

# Compute advantages (weighted combination across critics)
rewards = torch.randn(4, 4)  # Rewards for each group
advantages = critics.calc_advantages(values, rewards=rewards)

# Compute critic loss for training
old_values = torch.randn(4, 4)
critic_loss = critics.forward_for_loss(
    state,
    old_values=old_values,
    rewards=rewards,
    past_actions=past_actions
)
critic_loss.backward()
```

--------------------------------

### Configure and Use Actor Network

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Instantiate an Actor network for action selection, supporting multiple action sets and past action conditioning. Use it for forward passes to get action probabilities or sample actions, and compute PPO loss.

```python
import torch
from host_pytorch import Actor

# Create actor with multiple action sets
actor = Actor(
    num_actions = (10, 10, 20),  # 3 action sets with different sizes
    dims = (512, 256, 128),  # Input dim, hidden layers
    eps_clip = 0.2,  # PPO clipping parameter
    entropy_weight = 0.01,  # Entropy bonus for exploration
    dim_action_embed = 4,  # Embedding size for past actions
    past_action_conv_kernel = 3,  # Kernel for action history processing
    norm_advantages = True
)

# Forward pass for action selection
state = torch.randn(4, 512)  # Batch of 4 states
past_actions = torch.randint(0, 10, (4, 3, 3))  # Past 3 actions for 3 action sets

# Get action probabilities
probs = actor(state, past_actions=past_actions, sample=False)

# Sample actions with log probabilities
actions, log_probs = actor(
    state,
    past_actions=past_actions,
    sample=True,
    sample_temperature=1.0
)

# Compute PPO loss for training
advantages = torch.randn(4)
policy_loss = actor.forward_for_loss(
    state,
    actions,
    log_probs,
    advantages,
    past_actions=past_actions
)
policy_loss.backward()
```

--------------------------------

### Mock Environment for Testing

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

A mock environment class for testing and development that generates random states. Replace this with actual physics simulation for real-world applications. It provides methods to get state dimensions, reset, and step the environment.

```python
import torch
from host_pytorch.mock_env import Env, mock_hparams, random_state

# Create mock environment
env = Env()

# Get state dimension for network initialization
print(f"State dimension: {env.dim_state}")

# Reset environment to get initial state
initial_state = env.reset()

# Step environment with actions
actions = torch.randint(0, 10, (3,))  # Actions for 3 action sets
next_state = env(actions)

# Generate mock hyperparameters for reward shaping
hparams = mock_hparams()

# Generate a random state for testing
test_state = random_state()
```

--------------------------------

### Initialize and Train Humanoid Standing Agent

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Use the Agent class to initialize the environment, create an actor-critic architecture, collect experience, train the policy using PPO, and save/load the trained model.

```python
import torch
from host_pytorch import Agent
from host_pytorch.mock_env import Env, mock_hparams

# Initialize environment
env = Env()

# Create agent with actor-critic architecture
agent = Agent(
    num_actions = (10, 10, 20),  # Multiple action sets for different joints
    actor = dict(
        dims = (env.dim_state, 256, 128),  # MLP dimensions for policy network
    ),
    critics = dict(
        dims = (env.dim_state, 256),  # MLP dimensions for value networks
    ),
    reward_hparams = mock_hparams(),  # Hyperparameters for reward shaping
    num_episodes = 5,
    max_episode_timesteps = 100,
    epochs = 2,
    batch_size = 16,
    gae_gamma = 0.99,
    gae_lam = 0.95
)

# Collect experience by running episodes
memories = agent(env)

# Train actor and critics using PPO
agent.learn(memories)

# Save trained policy
agent.save('./standing-up-policy.pt', overwrite=True)

# Load a saved policy
agent.load('./standing-up-policy.pt')
```

--------------------------------

### Initialize Robot State

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Create a State dataclass instance containing all required physical observations for the robot.

```python
import torch
from host_pytorch import State

# Create a robot state with all required observations
state = State(
    head_height = torch.tensor(1.5),
    angular_velocity = torch.tensor([0.1, 0.0, 0.05]),
    linear_velocity = torch.tensor([0.0, 0.0, 0.1]),
    orientation = torch.tensor([0.0, 0.0, 0.0]),
    projected_gravity_vector = torch.tensor(0.98),
    joint_velocity = torch.randn(20),
    joint_acceleration = torch.randn(20),
    joint_torque = torch.randn(20),
    joint_position = torch.randn(20),
    left_ankle_keypoint_z = torch.randn(4),
    right_ankle_keypoint_z = torch.randn(4),
    left_feet_height = torch.tensor(0.0),
    right_feet_height = torch.tensor(0.0),
    left_shank_angle = torch.tensor(0.1),
    right_shank_angle = torch.tensor(0.1),
    left_feet_pos = torch.tensor([0.0, 0.1, 0.0]),
    right_feet_pos = torch.tensor([0.0, -0.1, 0.0]),
    upper_body_posture = torch.randn(10),
    height_base = torch.tensor(0.8),
    contact_force = torch.tensor([0.0, 0.0, 100.0]),
    hip_joint_angle_lr = torch.tensor(0.5),
    robot_base_angle_q = torch.tensor([0.0, 0.0, 0.0]),
    feet_angle_q = torch.tensor([0.0, 0.0, 0.0]),
    knee_joint_angle_lr = torch.tensor([1.0, 1.0]),
    shoulder_joint_angle_l = torch.tensor(0.0),
    shoulder_joint_angle_r = torch.tensor(0.0),
    waist_yaw_joint_angle = torch.tensor(0.0)
)
```

--------------------------------

### Configure Reward Functions and Groups

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Organize custom reward functions into weighted groups for critic value estimation.

```python
from host_pytorch import RewardFunctionAndWeight, RewardGroup
import torch

# Define a custom reward function
def custom_balance_reward(state, hparam, derived_state, past_actions=None):
    """Reward for maintaining balance based on center of mass."""
    com_offset = (state.left_feet_pos + state.right_feet_pos) / 2
    return torch.exp(-com_offset.norm() ** 2)

# Create reward with weight
reward_entry = RewardFunctionAndWeight(
    function = custom_balance_reward,
    weight = 5.0
)

# Define a complete reward group
style_rewards = RewardGroup(
    name = 'style',
    weight = 1.0,  # Weight for this group's critic
    reward = [
        RewardFunctionAndWeight(custom_balance_reward, 5.0),
        # Add more reward functions here
    ]
)
```

--------------------------------

### Manage Custom Rewards in Agent

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Dynamically add or remove reward functions from an Agent instance during training.

```python
import torch
from host_pytorch import Agent
from host_pytorch.mock_env import Env, mock_hparams

env = Env()

agent = Agent(
    num_actions = (10, 10, 20),
    actor = dict(dims = (env.dim_state, 256, 128)),
    critics = dict(dims = (env.dim_state, 256)),
    reward_hparams = mock_hparams()
)

# Define a custom reward function
def custom_energy_penalty(state, hparam, derived_state, past_actions=None):
    """Penalize high energy consumption."""
    power = state.joint_torque * state.joint_velocity
    return -power.abs().sum()

# Add custom reward to the regularization group
agent.add_reward_function_(
    reward_fn = custom_energy_penalty,
    group_name = 'regularization',
    weight = 1e-4
)

# Train with the new reward
memories = agent(env)
agent.learn(memories)

# Remove the custom reward if needed
agent.delete_reward_function_(
    group_name = 'regularization',
    reward_fn_name = 'custom_energy_penalty'
)
```

--------------------------------

### Actor with Latent Conditioning

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Actors and Critics can be conditioned on latent vectors from a gene pool for evolutionary policy optimization. This enables population-based training with diverse behaviors. Ensure the gene pool and actor/critics are initialized with compatible latent dimensions.

```python
import torch
from host_pytorch import Actor, Critics
from evolutionary_policy_optimization import LatentGenePool

# Create a latent gene pool for evolutionary optimization
latent_gene_pool = LatentGenePool(
    num_latents = 32,  # Population size
    dim_latent = 64
)

# Actor with latent conditioning
actor = Actor(
    num_actions = 10,
    dims = (512, 256, 128),
    dim_latent = 64,  # Enable latent conditioning
    dim_action_embed = 4,
    past_action_conv_kernel = 3
)

# Critics with latent conditioning
critics = Critics(
    weights = (2.5, 1.0, 0.1, 1.0),
    num_critics = 4,
    num_actions = 10,
    dims = (512, 256),
    dim_latent = 64
)

# Get a latent vector from the gene pool
latent = latent_gene_pool(latent_id=4)

# Forward pass with latent conditioning
state = torch.randn(4, 512)
past_actions = torch.randint(0, 10, (4, 2, 1))

actions, log_probs = actor(
    state,
    latents = latent,
    past_actions = past_actions,
    sample = True
)

values = critics(
    state,
    latents = latent,
    past_actions = past_actions
)
```

--------------------------------

### MLP and GroupedMLP Building Blocks

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

MLP is a simple feedforward network. GroupedMLP efficiently computes multiple MLPs in parallel, useful for multi-critic architectures. Conditioning can be applied using latent vectors.

```python
import torch
from host_pytorch import MLP, GroupedMLP

# Simple MLP for actor backbone
mlp = MLP(512, 256, 128, 64)  # Variable number of dimensions
x = torch.randn(4, 512)
output = mlp(x)  # Shape: (4, 64)

# MLP with conditioning (e.g., from latent vectors)
cond = [torch.randn(4, 256)]  # Multiplicative conditioning
output = mlp(x, cond=cond)

# GroupedMLP for multiple critics
grouped_mlp = GroupedMLP(
    512, 256, 128, 1,  # Dimensions ending in 1 for value output
    num_mlps = 4  # 4 parallel MLPs for 4 critics
)
x = torch.randn(4, 512)
values = grouped_mlp(x)  # Shape: (4, 4, 1) - batch x num_mlps x output_dim
```

--------------------------------

### Define Reward Hyperparameters

Source: https://context7.com/lucidrains/host-pytorch/llms.txt

Configure reward shaping parameters such as thresholds and joint limits using the HyperParams dataclass.

```python
import torch
from host_pytorch.host import HyperParams

# Define reward hyperparameters
hparams = HyperParams(
    height_stage1_thres = torch.tensor(0.4),  # Height threshold for stage 1
    height_stage2_thres = torch.tensor(0.7),  # Height threshold for stage 2
    joint_velocity_abs_limit = torch.ones(20) * 10.0,
    joint_position_PD_target = torch.zeros(20),
    joint_position_lower_limit = torch.ones(20) * -3.14,
    joint_position_higher_limit = torch.ones(20) * 3.14,
    upper_body_posture_target = torch.zeros(10),
    height_base_target = torch.tensor(0.9),
    ankle_parallel_thres = 0.05,
    joint_power_T = 1.0,
    feet_parallel_min_height_diff = 0.02,
    feet_distance_thres = 0.9,
    waist_yaw_joint_angle_thres = 1.4,
    contact_force_ratio_is_foot_stumble = 3.0,
    max_hip_joint_angle_lr = 1.4,
    min_hip_joint_angle_lr = 0.9,
    knee_joint_angle_max_min = (2.85, -0.06),
    shoulder_joint_angle_max_min = (-0.02, 0.02)
)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.