### Install streaming-deep-rl

Source: https://github.com/lucidrains/streaming-deep-rl/blob/main/README.md

Install the library using pip. This is the first step before using any of the functionalities.

```bash
$ pip install streaming-deep-rl
```

--------------------------------

### Complete Training Loop Example with LunarLander

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Demonstrates a full training loop using Gymnasium's LunarLander environment with StreamingACLambda. This example highlights the complete streaming RL workflow, including agent initialization, environment interaction, and timestep-wise updates.

```python
import torch
import numpy as np
import gymnasium as gym
from collections import deque

from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.residual_normed_mlp import ResidualNormedMLP

# Hyperparameters
NUM_EPISODES = 1000
MAX_TIMESTEPS = 1000
DELAY_STEPS = 7

# Environment setup
env = gym.make('LunarLander-v3', continuous=False)
dim_state = int(env.observation_space.shape[0])
num_actions = int(env.action_space.n)

# Network architecture using ResidualNormedMLP for better training stability
actor = ResidualNormedMLP(128, dim_in=dim_state, depth=4, residual_every=1)
critic = ResidualNormedMLP(128, dim_in=dim_state, depth=4, residual_every=1)

# Initialize agent
agent = StreamingACLambda(
    actor=actor,
    critic=critic,
    dim_state=dim_state,
    dim_actor=128,
    dim_critic=128,
    num_discrete_actions=num_actions,
    actor_lr=1.0,
    critic_lr=1.0,
    entropy_weight=0.01,
    discount_factor=0.99,
    eligibility_trace_decay=0.8,
    delay_steps=DELAY_STEPS,
    adaptive=True,
    init_sparsity=0.9,
    use_critic_ema=True
)

# Metrics tracking
rolling_reward = deque(maxlen=100)

# Training loop
for episode in range(NUM_EPISODES):
    state_np, _ = env.reset()
    state = torch.from_numpy(state_np).float()

    episode_reward = 0.0
    agent.reset_trace_()  # Reset eligibility traces at episode start

    for timestep in range(MAX_TIMESTEPS):
        # Select action
        action_dist = agent.forward_action(state)
        action = agent.sample_action(action_dist)

        # Environment step
        next_state_np, reward, terminated, truncated, _ = env.step(int(action.item()))

        reward_t = torch.tensor(reward, dtype=torch.float32)
        is_terminal = torch.tensor(terminated, dtype=torch.bool)
        next_state = torch.from_numpy(next_state_np).float()

        # Streaming update - called at every timestep
        metrics = agent.update(
            state=state,
            action=action,
            next_state=next_state,
            reward=reward_t,
            is_terminal=is_terminal,
            drain=(terminated or truncated)  # Drain buffer at episode end
        )

        state = next_state
        episode_reward += reward

        if terminated or truncated:
            break

    rolling_reward.append(episode_reward)

    if (episode + 1) % 100 == 0:
        avg_reward = np.mean(rolling_reward)
        print(f"Episode {episode + 1}: Avg Reward (100 eps) = {avg_reward:.2f}")

env.close()
```

--------------------------------

### Continuous Control Setup with StreamingACLambda

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Sets up a StreamingACLambda agent for continuous control tasks, such as those found in MuJoCo environments. Ensure 'torch' is imported and 'MLP' is defined.

```python
dim_state = 17  # e.g., HalfCheetah observation dim
num_continuous_actions = 6  # e.g., HalfCheetah action dim

actor = MLP(dim_state, 256, 256, 256, norm_elementwise_affine=False, activate_last=True)
critic = MLP(dim_state, 256, 256, norm_elementwise_affine=False)

agent = StreamingACLambda(
    actor=actor,
    critic=critic,
    dim_state=dim_state,
    dim_actor=256,
    dim_critic=256,
    num_discrete_actions=0,
    num_continuous_actions=num_continuous_actions,
    discount_factor=0.99,
    delay_steps=5,
    entropy_weight=0.001  # Lower entropy for continuous actions
)

agent.reset_trace_()

state = torch.randn(dim_state)
action, action_dist = agent(state, sample=True)

# action_dist contains [mean, log_std] for each action dimension
print(f"Action shape: {action.shape}")  # (num_continuous_actions,)
print(f"Action dist shape: {action_dist.shape}")  # (num_continuous_actions, 2)
```

--------------------------------

### Initialize and Use StreamingACLambda Agent

Source: https://github.com/lucidrains/streaming-deep-rl/blob/main/README.md

Demonstrates initializing a StreamingACLambda agent with MLP actor and critic networks. The agent is then used to get an action from a state and update its parameters based on environment feedback. Note the `delay_steps` parameter, which is set to 7 for 7-step TD.

```python
import torch

from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

# actor and critic

actor = MLP(
    8, 128, 128, 128,
    norm_elementwise_affine = False,
    activate_last = True
)

critic = MLP(
    8, 128, 128,
    norm_elementwise_affine = False
)

# agent

agent = StreamingACLambda(
    actor = actor,
    critic = critic,
    dim_state = 8,
    dim_actor = 128,
    num_discrete_actions = 4,
    delay_steps = 7 # 7-step TD works well for me. next step TD tends to hit a performance wall
)

# get action from state and pass to environment or world model

state = torch.randn(8)
action, action_dist = agent(state, sample = True)

# environment or world model gives back

next_state = torch.randn(8)
reward = torch.tensor(1.)
done = torch.tensor(False)

# update at each timestep, "streaming"

agent.update(
    state = state,
    action = action,
    next_state = next_state,
    reward = reward,
    is_terminal = done
)
```

--------------------------------

### Configure Continuous Action Space

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Initializes the agent for continuous control tasks using Gaussian action distributions.

```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP
```

--------------------------------

### Initialize StreamingACLambda Agent

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Configure the actor-critic agent with networks and hyperparameters for streaming updates.

```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

# Define actor and critic networks
# Actor outputs embeddings that are transformed to action distributions
actor = MLP(
    8, 128, 128, 128,  # input_dim, hidden_dims..., output_dim
    norm_elementwise_affine=False,
    activate_last=True
)

# Critic outputs embeddings for value estimation
critic = MLP(
    8, 128, 128,  # input_dim, hidden_dims..., output_dim (must match dim_critic)
    norm_elementwise_affine=False
)

# Initialize the streaming agent
agent = StreamingACLambda(
    actor=actor,
    critic=critic,
    dim_state=8,                      # Observation space dimension
    dim_actor=128,                    # Actor output dimension
    dim_critic=128,                   # Critic output dimension
    num_discrete_actions=4,           # Number of discrete actions (use 0 for continuous)
    num_continuous_actions=0,         # Number of continuous action dimensions
    discount_factor=0.999,            # Gamma for TD learning
    eligibility_trace_decay=0.8,      # Lambda for eligibility traces
    actor_lr=1.0,                     # Actor learning rate
    critic_lr=1.0,                    # Critic learning rate
    entropy_weight=0.01,              # Entropy regularization coefficient
    delay_steps=7,                    # N-step TD delay (7-step works well)
    adaptive=True,                    # Use Adam-style adaptive learning rates
    init_sparsity=0.9,                # Sparse initialization ratio
    val_min=-3.0,                     # Min value for HL-Gauss distribution
    val_max=3.0,                      # Max value for HL-Gauss distribution
    num_bins=64,                      # Number of bins for distributional critic
    use_critic_ema=True,              # Use EMA for critic target
    l2_weight_decay=0.,               # L2 regularization weight
    enable_pilar=False,               # Enable PILAR (averaged n-step returns)
    use_delightful_pg=False           # Enable Delightful Policy Gradient
)

# Reset eligibility traces at episode start
agent.reset_trace_()

# Get action from state
state = torch.randn(8)
action, action_dist = agent(state, sample=True)  # sample=True returns (action, distribution)

# Or get just the action distribution
action_dist = agent(state, sample=False)
action = agent.sample_action(action_dist)

print(f"Action: {action}")
print(f"Action distribution shape: {action_dist.shape}")
```

--------------------------------

### Agent Update and Metrics

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Demonstrates how to update the agent at each timestep and access training metrics. The `update` method handles the core learning process, and the returned metrics provide insights into the agent's performance.

```APIDOC
## Agent Update and Metrics

### Description
Updates the agent with the latest experience (state, action, next_state, reward, terminal status) and returns training metrics. Set `drain=True` at the end of an episode to process any remaining data in the buffer.

### Method
`agent.update(state, action, next_state, reward, is_terminal, drain=False)`

### Parameters
- **state** (object) - The current state of the environment.
- **action** (object) - The action taken by the agent.
- **next_state** (object) - The state after taking the action.
- **reward** (float) - The reward received.
- **is_terminal** (bool) - Whether the episode has terminated.
- **drain** (bool) - Optional. If True, processes the remaining buffer at episode end.

### Accessing Metrics
Metrics are returned as an object with attributes like `td_error`, `value_pred`, `actor_grad_norm`, `critic_grad_norm`, `actor_trace_norm`, `critic_trace_norm`, `actor_scale`, and `critic_scale`.

### Metrics Example
```python
metrics = agent.update(
    state=state,
    action=action,
    next_state=next_state,
    reward=reward,
    is_terminal=is_terminal,
    drain=False
)

print(f"TD Error: {metrics.td_error:.4f}")
print(f"Value Prediction: {metrics.value_pred:.4f}")
print(f"Actor Gradient Norm: {metrics.actor_grad_norm:.4e}")
print(f"Critic Gradient Norm: {metrics.critic_grad_norm:.4e}")
print(f"Actor Trace Norm: {metrics.actor_trace_norm:.4f}")
print(f"Critic Trace Norm: {metrics.critic_trace_norm:.4f}")
print(f"Actor Scale: {metrics.actor_scale:.4e}")
print(f"Critic Scale: {metrics.critic_scale:.4e}")
```
```

--------------------------------

### StreamingACLambda Initialization

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Initializes the primary agent class for the Streaming Actor-Critic algorithm.

```APIDOC
## StreamingACLambda Initialization

### Description
Initializes the StreamingACLambda agent, which combines actor and critic networks for online reinforcement learning.

### Parameters
#### Request Body
- **actor** (nn.Module) - Required - Actor network (policy)
- **critic** (nn.Module) - Required - Critic network (value function)
- **dim_state** (int) - Required - Observation space dimension
- **dim_actor** (int) - Required - Actor output dimension
- **dim_critic** (int) - Required - Critic output dimension
- **num_discrete_actions** (int) - Optional - Number of discrete actions
- **num_continuous_actions** (int) - Optional - Number of continuous action dimensions
- **discount_factor** (float) - Optional - Gamma for TD learning
- **eligibility_trace_decay** (float) - Optional - Lambda for eligibility traces
- **actor_lr** (float) - Optional - Actor learning rate
- **critic_lr** (float) - Optional - Critic learning rate
- **delay_steps** (int) - Optional - N-step TD delay
- **adaptive** (bool) - Optional - Use Adam-style adaptive learning rates
- **init_sparsity** (float) - Optional - Sparse initialization ratio
- **use_critic_ema** (bool) - Optional - Use EMA for critic target
```

--------------------------------

### Train Lunar Lander

Source: https://github.com/lucidrains/streaming-deep-rl/blob/main/README.md

Command to run the training script for the Lunar Lander environment.

```bash
$ uv run train_lunar.py
```

--------------------------------

### Perform Timestep Update

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Updates the agent with a single transition. Set drain to True at the end of an episode to process remaining buffer data.

```python
metrics = agent.update(
    state=state,
    action=action,
    next_state=next_state,
    reward=reward,
    is_terminal=is_terminal,
    drain=False  # Set True at episode end to process remaining buffer
)

# Access training metrics
print(f"TD Error: {metrics.td_error:.4f}")
print(f"Value Prediction: {metrics.value_pred:.4f}")
print(f"Actor Gradient Norm: {metrics.actor_grad_norm:.4e}")
print(f"Critic Gradient Norm: {metrics.critic_grad_norm:.4e}")
print(f"Actor Trace Norm: {metrics.actor_trace_norm:.4f}")
print(f"Critic Trace Norm: {metrics.critic_trace_norm:.4f}")
print(f"Actor Scale: {metrics.actor_scale:.4e}")
print(f"Critic Scale: {metrics.critic_scale:.4e}")
```

--------------------------------

### Update StreamingACLambda Agent

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Perform an online update step using transition data from the environment.

```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

# Setup agent (abbreviated)
actor = MLP(8, 128, 128, 128, norm_elementwise_affine=False, activate_last=True)
critic = MLP(8, 128, 128, norm_elementwise_affine=False)
agent = StreamingACLambda(
    actor=actor, critic=critic,
    dim_state=8, dim_actor=128, dim_critic=128,
    num_discrete_actions=4, delay_steps=7
)

# Simulated environment interaction
state = torch.randn(8)
action, _ = agent(state, sample=True)

# Environment returns next state, reward, and termination flag
next_state = torch.randn(8)
reward = torch.tensor(1.0)
is_terminal = torch.tensor(False)
```

--------------------------------

### Normalize Rewards

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Scales rewards based on discounted return variance. Requires terminal flags to correctly track episode returns.

```python
import torch
from streaming_deep_rl import ScaleRewardNormalizer

# Create reward normalizer
reward_norm = ScaleRewardNormalizer(
    eps=1e-5,
    discount_factor=0.999,
    time_dilate_factor=1.0,
    mean_center=False  # Whether to mean-center rewards
)

reward_norm.train()

# Process rewards during training
rewards = [1.0, -0.5, 2.0, 0.1, -1.0, 3.0]
is_terminals = [False, False, False, False, True, False]

for reward, is_term in zip(rewards, is_terminals):
    reward_t = torch.tensor(reward)
    normalized_reward = reward_norm(reward_t, is_terminal=is_term, update=True)
    print(f"Raw: {reward:+.2f} -> Normalized: {normalized_reward.item():+.4f}")

print(f"\nReward variance estimate: {reward_norm.variance.item():.4f}")
```

--------------------------------

### Compute Action Distribution

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Generates action distributions using the actor network. Supports EMA smoothing for stable selection and optional sampling.

```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

# Setup agent with actor EMA enabled
actor = MLP(8, 128, 128, 128, norm_elementwise_affine=False, activate_last=True)
critic = MLP(8, 128, 128, norm_elementwise_affine=False)
agent = StreamingACLambda(
    actor=actor, critic=critic,
    dim_state=8, dim_actor=128, dim_critic=128,
    num_discrete_actions=4,
    actor_use_ema=True,
    actor_ema_beta=0.7
)

state = torch.randn(8)

# Get action distribution (uses EMA weights if available)
action_dist = agent.forward_action(state, use_ema=True, sample=False)

# Get action with sampling in one call
action, action_dist = agent.forward_action(state, use_ema=True, sample=True)

# Sample multiple actions from distribution
actions = agent.sample_action(action_dist)
print(f"Sampled action: {actions}")
```

--------------------------------

### ScaleRewardNormalizer

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Implements online reward scaling normalization by tracking discounted return variance for stable reward normalization during training.

```APIDOC
## ScaleRewardNormalizer

### Description
Online reward scaling normalizer that tracks discounted return variance for stable reward normalization during training.

### Initialization
`ScaleRewardNormalizer(eps=1e-5, discount_factor=0.999, time_dilate_factor=1.0, mean_center=False)`

### Parameters
- **eps** (float) - A small value to prevent division by zero. Defaults to 1e-5.
- **discount_factor** (float) - The discount factor for calculating discounted returns. Defaults to 0.999.
- **time_dilate_factor** (float) - Adjusts the adaptation speed. Defaults to 1.0.
- **mean_center** (bool) - Whether to mean-center rewards. Defaults to False.

### Usage
- **Training Mode**: Call `reward_norm.train()` to enable updates.
- **Evaluation Mode**: Call `reward_norm.eval()` to disable updates.

### Processing Rewards
`reward_norm(reward, is_terminal, update=True)`

### Parameters
- **reward** (torch.Tensor) - The reward to normalize.
- **is_terminal** (bool) - Whether the current step is terminal.
- **update** (bool) - Whether to update the running statistics. Defaults to True in train mode, False in eval mode.

### Returns
- **normalized_reward** (torch.Tensor) - The normalized reward.

### Example
```python
import torch
from streaming_deep_rl import ScaleRewardNormalizer

# Create reward normalizer
reward_norm = ScaleRewardNormalizer(
    eps=1e-5,
    discount_factor=0.999,
    time_dilate_factor=1.0,
    mean_center=False  # Whether to mean-center rewards
)

reward_norm.train()

# Process rewards during training
rewards = [1.0, -0.5, 2.0, 0.1, -1.0, 3.0]
is_terminals = [False, False, False, False, True, False]

for reward, is_term in zip(rewards, is_terminals):
    reward_t = torch.tensor(reward)
    normalized_reward = reward_norm(reward_t, is_terminal=is_term, update=True)
    print(f"Raw: {reward:+.2f} -> Normalized: {normalized_reward.item():+.4f}")

print(f"\nReward variance estimate: {reward_norm.variance.item():.4f}")
```
```

--------------------------------

### ObservationNormalizer

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Provides online observation normalization using Welford's algorithm. It normalizes inputs to zero mean and unit variance without requiring a pre-training phase.

```APIDOC
## ObservationNormalizer

### Description
Online observation normalization using Welford's algorithm for computing running mean and variance. Normalizes inputs to zero mean and unit variance without requiring a pre-training phase.

### Initialization
`ObservationNormalizer(dim, eps=1e-5, time_dilate_factor=1.0)`

### Parameters
- **dim** (int) - The dimensionality of the observations.
- **eps** (float) - A small value to prevent division by zero. Defaults to 1e-5.
- **time_dilate_factor** (float) - Adjusts the adaptation speed. Defaults to 1.0.

### Usage
- **Training Mode**: Call `normalizer.train()` to enable updates.
- **Evaluation Mode**: Call `normalizer.eval()` to disable updates.

### Processing Observations
`normalizer(obs, update=True)`

### Parameters
- **obs** (torch.Tensor) - The observation to normalize.
- **update** (bool) - Whether to update the running statistics. Defaults to True in train mode, False in eval mode.

### Returns
- **normalized_obs** (torch.Tensor) - The normalized observation.

### Example
```python
import torch
from streaming_deep_rl import ObservationNormalizer

# Create normalizer for 8-dimensional observations
normalizer = ObservationNormalizer(
    dim=8,
    eps=1e-5,
    time_dilate_factor=1.0  # Adjusts adaptation speed
)

# Set to training mode for online updates
normalizer.train()

# Process observations (normalizes and updates statistics)
for _ in range(100):
    obs = torch.randn(8) * 10 + 5  # Non-normalized observations
    normalized_obs = normalizer(obs, update=True)

print(f"Running mean: {normalizer.running_mean}")
print(f"Running variance: {normalizer.variance}")

# In evaluation mode, just normalize without updating
normalizer.eval()
test_obs = torch.randn(8) * 10 + 5
normalized_test = normalizer(test_obs)  # update defaults to False in eval mode

print(f"Normalized observation mean: {normalized_test.mean():.4f}")
```
```

--------------------------------

### StreamingACLambda.forward_action

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Computes the action distribution for a given state. It supports optional Exponential Moving Average (EMA) smoothing of actor weights for more stable action selection.

```APIDOC
## StreamingACLambda.forward_action

### Description
Computes the action distribution for a given state without gradients. Supports optional EMA smoothing of actor weights for more stable action selection.

### Method
`agent.forward_action(state, use_ema=True, sample=False)`

### Parameters
- **state** (torch.Tensor) - The input state.
- **use_ema** (bool) - Whether to use EMA weights for the actor. Defaults to True.
- **sample** (bool) - Whether to sample an action from the distribution. Defaults to False.

### Returns
- **action_dist** (object) - The action distribution.
- **action** (torch.Tensor, optional) - The sampled action, returned only if `sample=True`.

### Request Example (using EMA, not sampling)
```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

# Setup agent with actor EMA enabled
actor = MLP(8, 128, 128, 128, norm_elementwise_affine=False, activate_last=True)
critic = MLP(8, 128, 128, norm_elementwise_affine=False)
agent = StreamingACLambda(
    actor=actor, critic=critic,
    dim_state=8, dim_actor=128, dim_critic=128,
    num_discrete_actions=4,
    actor_use_ema=True,
    actor_ema_beta=0.7
)

state = torch.randn(8)

action_dist = agent.forward_action(state, use_ema=True, sample=False)
```

### Request Example (sampling an action)
```python
action, action_dist = agent.forward_action(state, use_ema=True, sample=True)
```

### Sampling Actions from Distribution
```python
actions = agent.sample_action(action_dist)
print(f"Sampled action: {actions}")
```
```

--------------------------------

### Compute Value Estimate

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Estimates state values using the critic network with HL-Gauss distributional output.

```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

actor = MLP(5, 128, norm_elementwise_affine=False, activate_last=False)
critic = MLP(5, 128, norm_elementwise_affine=False, activate_last=False)

agent = StreamingACLambda(
    actor=actor, critic=critic,
    dim_state=5, dim_actor=128, dim_critic=128,
    num_continuous_actions=1,
    val_min=-2.5, val_max=2.5, num_bins=127
)

state = torch.randn(5)
value = agent.forward_value(state)

print(f"State value estimate: {value.item():.4f}")
print(f"Value shape: {value.shape}")  # Shape: ()
```

--------------------------------

### Normalize Observations

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Applies online normalization to observations using Welford's algorithm. Use train() to update statistics and eval() to normalize without updates.

```python
import torch
from streaming_deep_rl import ObservationNormalizer

# Create normalizer for 8-dimensional observations
normalizer = ObservationNormalizer(
    dim=8,
    eps=1e-5,
    time_dilate_factor=1.0  # Adjusts adaptation speed
)

# Set to training mode for online updates
normalizer.train()

# Process observations (normalizes and updates statistics)
for _ in range(100):
    obs = torch.randn(8) * 10 + 5  # Non-normalized observations
    normalized_obs = normalizer(obs, update=True)

print(f"Running mean: {normalizer.running_mean}")
print(f"Running variance: {normalizer.variance}")

# In evaluation mode, just normalize without updating
normalizer.eval()
test_obs = torch.randn(8) * 10 + 5
normalized_test = normalizer(test_obs)  # update defaults to False in eval mode

print(f"Normalized observation mean: {normalized_test.mean():.4f}")
```

--------------------------------

### StreamingACLambda.update

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Updates the agent networks using transition data at each timestep.

```APIDOC
## StreamingACLambda.update

### Description
Updates the actor and critic networks using the streaming algorithm. This method should be called at every timestep with the transition data.

### Parameters
#### Request Body
- **state** (torch.Tensor) - Required - Current state observation
- **action** (torch.Tensor) - Required - Action taken
- **reward** (torch.Tensor) - Required - Reward received
- **next_state** (torch.Tensor) - Required - Next state observation
- **is_terminal** (bool) - Required - Termination flag
```

--------------------------------

### StreamingACLambda.forward_value

Source: https://context7.com/lucidrains/streaming-deep-rl/llms.txt

Computes the value estimate for a given state using the critic network. It supports HL-Gauss distributional output.

```APIDOC
## StreamingACLambda.forward_value

### Description
Computes the value estimate for a given state using the critic network with HL-Gauss distributional output.

### Method
`agent.forward_value(state)`

### Parameters
- **state** (torch.Tensor) - The input state.

### Returns
- **value** (torch.Tensor) - The estimated value of the state.

### Request Example
```python
import torch
from streaming_deep_rl import StreamingACLambda
from x_mlps_pytorch.normed_mlp import MLP

actor = MLP(5, 128, norm_elementwise_affine=False, activate_last=False)
critic = MLP(5, 128, norm_elementwise_affine=False, activate_last=False)

agent = StreamingACLambda(
    actor=actor, critic=critic,
    dim_state=5, dim_actor=128, dim_critic=128,
    num_continuous_actions=1,
    val_min=-2.5, val_max=2.5, num_bins=127
)

state = torch.randn(5)
value = agent.forward_value(state)

print(f"State value estimate: {value.item():.4f}")
print(f"Value shape: {value.shape}")  # Shape: ()
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.