### Setup Development Environment

Source: https://github.com/proroklab/popgym/blob/master/README.md

Commands to install pre-commit hooks and run the test suite for the project.

```python
pip install pre-commit
pre-commit install
pytest popgym/tests
```

--------------------------------

### Install POPGym from source

Source: https://github.com/proroklab/popgym/blob/master/docs/source/installation.md

Installation method for developers or users requiring the latest source code.

```bash
git clone https://github.com/smorad/popgym
cd popgym
pip install .
```

--------------------------------

### Install POPGym with baselines

Source: https://github.com/proroklab/popgym/blob/master/docs/source/installation.md

Installs the library along with memory models and RLlib dependencies.

```bash
pip install popgym[baselines]
```

```bash
git clone https://github.com/smorad/popgym
cd popgym
pip install ".[baselines]"
```

--------------------------------

### Install POPGym via pip

Source: https://github.com/proroklab/popgym/blob/master/docs/source/installation.md

Standard installation method for the stable version of the library.

```bash
pip install popgym
```

--------------------------------

### Install POPGym and Dependencies

Source: https://context7.com/proroklab/popgym/llms.txt

Install POPGym using pip. Choose the appropriate command based on whether you need navigation environments or memory baselines with RLlib. Installation from source is also supported.

```bash
# Install base environments (requires numpy and gymnasium)
pip install popgym

# Install with navigation environments (requires mazelib, Python <3.12)
pip install "popgym[navigation]"

# Install with memory baselines and RLlib
pip install "popgym[baselines]"

# Install from source
git clone https://github.com/smorad/popgym
cd popgym
pip install ".[baselines]"
```

--------------------------------

### Create and Interact with POPGym Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Environments can be instantiated directly or via the Gymnasium registry. They follow the standard Gymnasium interface. The example shows creating a HigherLower environment, resetting it, printing its spaces, and running a basic environment loop.

```python
import gymnasium as gym
import popgym
from popgym.envs.higher_lower import HigherLower
from popgym.envs.position_only_cartpole import PositionOnlyCartPoleEasy
from popgym.envs.concentration import ConcentrationEasy

# Direct instantiation with custom parameters
env = HigherLower(num_decks=2)
obs, info = env.reset(seed=42)
print(f"Observation space: {env.observation_space}")
print(f"Action space: {env.action_space}")

# Using Gymnasium registry
env = gym.make("popgym-HigherLowerEasy-v0")
obs, info = env.reset(seed=42)

# Create a position-only cartpole (POMDP version)
cartpole_env = PositionOnlyCartPoleEasy()
obs, info = cartpole_env.reset(seed=0)
print(f"Initial observation (position, angle only): {obs}")

# Run environment loop
done = False
total_reward = 0
while not done:
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    done = terminated or truncated
print(f"Episode finished with total reward: {total_reward}")
```

--------------------------------

### Install POPGym Base and Optional Environments

Source: https://github.com/proroklab/popgym/blob/master/README.md

Install the base POPGym environments using pip. Optional dependencies for navigation and baselines can be included with specific package extras. Note that navigation environments have a Python version constraint.

```python
# Install base environments, only requires numpy and gymnasium
pip install popgym 
# Also include navigation environments, which require mazelib
# NOTE: navigation envs require python <3.12 due to mazelib not supporting 3.12
pip install "popgym[navigation]" 
# Install memory baselines w/ RLlib 
pip install "popgym[baselines]" 
```

--------------------------------

### Initialize Diagnostic Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Demonstrates how to instantiate diagnostic environments and list all available configurations.

```python
env = gym.make("popgym-AutoencodeEasy-v0")
obs, info = env.reset(seed=0)
print(f"Autoencode - Obs space: {env.observation_space}")

# RepeatFirst: Agent must remember and repeat the first observation
env = gym.make("popgym-RepeatFirstEasy-v0")
obs, info = env.reset(seed=0)
print(f"RepeatFirst - Obs space: {env.observation_space}")

# RepeatPrevious: Agent must repeat observation from k steps ago
env = gym.make("popgym-RepeatPreviousEasy-v0")
obs, info = env.reset(seed=0)
print(f"RepeatPrevious - Obs space: {env.observation_space}")

# CountRecall: Agent must count occurrences of symbols
env = gym.make("popgym-CountRecallEasy-v0")
obs, info = env.reset(seed=0)
print(f"CountRecall - Obs space: {env.observation_space}")

# List all diagnostic environments
print("\nAll diagnostic environments:")
for env_class, config in ALL_DIAGNOSTIC.items():
    print(f"  {config['id']}")
```

--------------------------------

### Initialize Game Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Sets up memory-intensive game environments and demonstrates custom configuration for the Concentration game.

```python
import gymnasium as gym
import popgym
from popgym.envs.concentration import Concentration
from popgym.envs.battleship import Battleship
from popgym.envs.higher_lower import HigherLower
from popgym.envs.minesweeper import MineSweeper

# Concentration (Memory card game)
env = gym.make("popgym-ConcentrationEasy-v0")
obs, info = env.reset(seed=0)
print(f"Concentration - Action: flip card at index, Obs: visible cards")
# Agent must remember card positions to match pairs

# Battleship
env = gym.make("popgym-BattleshipEasy-v0")
obs, info = env.reset(seed=0)
print(f"Battleship - Agent must find hidden ships")

# Higher Lower (card game)
env = gym.make("popgym-HigherLowerEasy-v0")
obs, info = env.reset(seed=0)
print(f"HigherLower - Predict if next card is higher or lower")

# MineSweeper
env = gym.make("popgym-MineSweeperEasy-v0")
obs, info = env.reset(seed=0)
print(f"MineSweeper - Avoid mines using revealed numbers")

# Custom game setup
concentration = Concentration(num_decks=2, deck_type="colors")
obs, info = concentration.reset(seed=42)
print(f"Custom Concentration with 2 decks, matching colors")
```

--------------------------------

### Import popgym environments

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Initializes the environment library and lists all available environment classes.

```python
import popgym
from popgym.wrappers import PreviousAction, Antialias, Markovian, Flatten, DiscreteAction
from popgym.core.observability import Observability, STATE
env_classes = popgym.envs.ALL.keys()
print(env_classes)
```

--------------------------------

### Configure Environment Observability

Source: https://context7.com/proroklab/popgym/llms.txt

Demonstrates how to use the Markovian wrapper to set full or partial observability for environments.

```python
env_full = Markovian(
    Antialias(PreviousAction(PositionOnlyCartPoleEasy())),
    Observability.FULL
)
obs, info = env_full.reset(seed=0)
print(f"Full state observation: {obs}")
```

```python
env_both = Markovian(
    Antialias(PreviousAction(PositionOnlyCartPoleEasy())),
    Observability.FULL_AND_PARTIAL
)
obs, info = env_both.reset(seed=0)
print(f"Observation dict keys: {obs.keys()}")  # ['obs', 'state']
```

--------------------------------

### Initialize and Wrap a POPGym Environment

Source: https://github.com/proroklab/popgym/blob/master/README.md

Demonstrates initializing a PositionOnlyCartPoleEasy environment and applying several wrappers: PreviousAction, Flatten, and DiscreteAction. This sequence modifies the observation and action spaces for compatibility with Q-learning.

```python
import popgym
from popgym.wrappers import PreviousAction, Antialias, Flatten, DiscreteAction
env = popgym.envs.position_only_cartpole.PositionOnlyCartPoleEasy()
print(env.reset(seed=0))
wrapped = DiscreteAction(Flatten(PreviousAction(env))) # Append prev action to obs, flatten obs/action spaces, then map the multidiscrete action space to a single discrete action for Q learning
print(wrapped.reset(seed=0))
```

--------------------------------

### Initialize Control Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Initializes POMDP versions of classic control tasks where specific state information is hidden, and shows how to select different difficulty levels.

```python
import gymnasium as gym
import popgym
from popgym.envs import ALL_CONTROL

# PositionOnlyCartPole: CartPole with velocity hidden
env = gym.make("popgym-PositionOnlyCartPoleEasy-v0")
obs, info = env.reset(seed=0)
print(f"PositionOnlyCartPole obs (x, theta): {obs}")
# Must infer velocity from position history

# VelocityOnlyCartPole: CartPole with position hidden
env = gym.make("popgym-VelocityOnlyCartpoleEasy-v0")
obs, info = env.reset(seed=0)
print(f"VelocityOnlyCartPole obs: {obs}")

# PositionOnlyPendulum: Pendulum with angular velocity hidden
env = gym.make("popgym-PositionOnlyPendulumEasy-v0")
obs, info = env.reset(seed=0)
print(f"PositionOnlyPendulum obs: {obs}")

# Different difficulty levels have different episode lengths
easy = gym.make("popgym-PositionOnlyCartPoleEasy-v0")    # 200 steps
medium = gym.make("popgym-PositionOnlyCartPoleMedium-v0")  # 400 steps
hard = gym.make("popgym-PositionOnlyCartPoleHard-v0")    # 600 steps
```

--------------------------------

### Configure and run a GRU model with RLlib

Source: https://github.com/proroklab/popgym/blob/master/docs/source/baselines_quickstart.md

Demonstrates how to import a GRU model, inspect its configuration, and define a custom training configuration for RLlib.

```python
import popgym
import ray
from torch import nn
from popgym.baselines.ray_models.ray_gru import GRU
# See what GRU-specific hyperparameters we can set
print(GRU.MODEL_CONFIG)
# Show other settable model hyperparameters like
# what the actor/critic branches look like,
# what hidden size to use,
# whether to add a positional embedding, etc.
print(GRU.BASE_CONFIG)
# How long the temporal window for backprop is
# This doesn't need to be longer than 1024
bptt_size = 1024
config = {
"model": {
    "max_seq_len": bptt_size,
    "custom_model": GRU,
    "custom_model_config": {
    # Override the hidden_size from BASE_CONFIG
    # The input and output sizes of the MLP feeding the memory model
    "preprocessor_input_size": 128,
    "preprocessor_output_size": 64,
    "preprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
    # this is the size of the recurrent state in most cases
    "hidden_size": 128,
    # We should also change other parts of the architecture to use
    # this new hidden size
    # For the GRU, the output is of size hidden_size
    "postprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
    "postprocessor_output_size": 64,
    # Actor and critic networks
    "actor": nn.Linear(64, 64),
    "critic": nn.Linear(64, 64),
    # We can also override GRU-specific hyperparams
    "num_recurrent_layers": 1,
    },
},
# Some other rllib defaults you might want to change
# See https://docs.ray.io/en/latest/rllib/rllib-training.html#common-parameters
# for a full list of rllib settings
#
# These should be a factor of bptt_size
"sgd_minibatch_size": bptt_size * 4,
# Should be a factor of sgd_minibatch_size
"train_batch_size": bptt_size * 8,
# The environment we are training on
"env": "popgym-ConcentrationEasy-v0",
# You probably don't want to change these values
"rollout_fragment_length": bptt_size,
"framework": "torch",
"horizon": bptt_size,
"batch_mode": "complete_episodes",
}
# Stop after 50k environment steps
ray.tune.run("PPO", config=config, stop={"timesteps_total": 50_000})
```

--------------------------------

### Initialize Navigation Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Checks for mazelib dependency before initializing labyrinth-based navigation environments.

```python
import gymnasium as gym
import popgym
from popgym.envs import has_mazelib

# Check if navigation environments are available
if has_mazelib():
    from popgym.envs.labyrinth_escape import LabyrinthEscapeEasy
    from popgym.envs.labyrinth_explore import LabyrinthExploreEasy
    from popgym.envs import ALL_NAVIGATION

    # LabyrinthEscape: Find the exit
    env = gym.make("popgym-LabyrinthEscapeEasy-v0")
    obs, info = env.reset(seed=0)
    print(f"LabyrinthEscape - Navigate to exit")

    # LabyrinthExplore: Visit all cells
    env = gym.make("popgym-LabyrinthExploreEasy-v0")
    obs, info = env.reset(seed=0)
    print(f"LabyrinthExplore - Visit entire maze")

    print("\nAll navigation environments:")
    for env_class, config in ALL_NAVIGATION.items():
        print(f"  {config['id']}")
else:
    print("Navigation environments require mazelib: pip install popgym[navigation]")
    print("Note: mazelib requires Python <3.12")
```

--------------------------------

### PreviousAction Wrapper Usage

Source: https://context7.com/proroklab/popgym/llms.txt

Demonstrates how to include the previous action in the observation space and configure a custom null action for the initial reset.

```python
# Step returns observation with actual previous action
action = wrapped_env.action_space.sample()
obs, reward, terminated, truncated, info = wrapped_env.step(action)
print(f"Observation after step: {obs}")
# Output: (array([...], dtype=float32), <action_taken>)

# Custom null action for reset
custom_null = 1
wrapped_env = PreviousAction(env, null_action=custom_null)
obs, info = wrapped_env.reset()
print(f"Observation with custom null: {obs}")
```

--------------------------------

### Run environment step

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Executes a single step in the environment using a sampled action.

```python
wrapped_env.reset()
obs, reward, terminated, truncated, info = wrapped_env.step(wrapped_env.action_space.sample())
```

--------------------------------

### Apply Markovian wrapper

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Configures the environment to include the hidden Markov state in the info dictionary.

```python
wrapped_env = Markovian(wrapped_env, Observability.FULL_IN_INFO_DICT)
```

--------------------------------

### List All Available POPGym Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Access dictionaries provided by POPGym to list all available environments, categorized by type (diagnostic, control, noisy, game, navigation) and difficulty level. This helps in selecting appropriate environments for testing.

```python
import popgym
from popgym.envs import (
    ALL, ALL_EASY, ALL_MEDIUM, ALL_HARD,
    DIAGNOSTIC, CONTROL, NOISY, GAME, NAVIGATION,
    ALL_DIAGNOSTIC, ALL_CONTROL, ALL_NOISY, ALL_GAME, ALL_NAVIGATION
)

# List all environment classes
print("All environments:")
for env_class, config in ALL.items():
    print(f"  {config['id']}: {env_class.__name__}")

# List easy environments only
print("\nEasy environments:")
for env_class, config in ALL_EASY.items():
    print(f"  {config['id']}")

# List diagnostic environments (for testing memory)
print("\nDiagnostic environments:")
for env_class, config in ALL_DIAGNOSTIC.items():
    print(f"  {config['id']}")

# Example output:
# popgym-AutoencodeEasy-v0
# popgym-RepeatPreviousEasy-v0
# popgym-RepeatFirstEasy-v0
# popgym-CountRecallEasy-v0
# popgym-ConcentrationEasy-v0
# popgym-BattleshipEasy-v0
# ... and more
```

--------------------------------

### Apply PreviousAction wrapper

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Wraps the environment to include the previous action in the observation space.

```python
wrapped_env = PreviousAction(env)
```

--------------------------------

### Access Available Memory Baselines

Source: https://context7.com/proroklab/popgym/llms.txt

POPGym includes various pre-implemented models categorized by architecture. These can be iterated over for training or evaluation.

```python
from popgym.baselines.ray_models.ray_mlp import MLP, BasicMLP
from popgym.baselines.ray_models.ray_framestack import Framestack
from popgym.baselines.ray_models.ray_frameconv import Frameconv
from popgym.baselines.ray_models.ray_elman import Elman
from popgym.baselines.ray_models.ray_lstm import LSTM
from popgym.baselines.ray_models.ray_gru import GRU
from popgym.baselines.ray_models.ray_indrnn import IndRNN
from popgym.baselines.ray_models.ray_linear_attention import LinearAttention
from popgym.baselines.ray_models.ray_fwp import FastWeightProgrammer
from popgym.baselines.ray_models.ray_lmu import LMU
from popgym.baselines.ray_models.ray_s4d import S4D
from popgym.baselines.ray_models.ray_diffnc import DiffNC

# Dictionary of all available models with their categories
MEMORY_MODELS = {
    # Feedforward baselines (no memory)
    "MLP": BasicMLP,           # Basic MLP, no temporal awareness
    "PosMLP": MLP,             # MLP with positional encoding

    # Frame-based methods
    "Framestack": Framestack,  # Stack recent frames
    "TCN": Frameconv,          # Temporal Convolutional Networks

    # Recurrent models
    "Elman": Elman,            # Simple Elman RNN
    "LSTM": LSTM,              # Long Short-Term Memory
    "GRU": GRU,                # Gated Recurrent Unit
    "IndRNN": IndRNN,          # Independently Recurrent Neural Network

    # Attention-based models
    "FART": LinearAttention,   # Fast Autoregressive Transformers
    "FWP": FastWeightProgrammer,  # Fast Weight Programmers

    # State space models
    "LMU": LMU,                # Legendre Memory Units
    "S4D": S4D,                # Diagonal State Space Models

    # External memory
    "DNC": DiffNC,             # Differentiable Neural Computer
}

# Example: iterate and train all baselines
for name, model_class in MEMORY_MODELS.items():
    config = {
        "model": {
            "max_seq_len": 1024,
            "custom_model": model_class,
        },
        "env": "popgym-AutoencodeEasy-v0",
        "framework": "torch",
    }
    print(f"Model: {name}, Config: {model_class.MODEL_CONFIG}")
```

--------------------------------

### Create a stateless cartpole environment

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Instantiates a specific environment class for the position-only cartpole task.

```python
env = popgym.envs.position_only_cartpole.PositionOnlyCartPoleEasy()
```

--------------------------------

### Create a Custom Memory Model

Source: https://context7.com/proroklab/popgym/llms.txt

Inherit from BaseModel and implement initial_state and forward_memory to define custom memory logic. The model can then be registered in the RLlib configuration.

```python
import torch
from torch import nn
from typing import List, Tuple
import gymnasium as gym
from ray.rllib.utils.typing import ModelConfigDict, TensorType
from popgym.baselines.ray_models.base_model import BaseModel

class CustomMemoryModel(BaseModel):
    """Example custom memory model with simple attention."""

    MODEL_CONFIG = {
        "memory_size": 64,
        "num_heads": 4,
    }

    def __init__(
        self,
        obs_space: gym.spaces.Space,
        action_space: gym.spaces.Space,
        num_outputs: int,
        model_config: ModelConfigDict,
        name: str,
        **custom_model_kwargs,
    ):
        super().__init__(
            obs_space, action_space, num_outputs, model_config, name
        )
        # Define custom memory core
        self.core = nn.MultiheadAttention(
            embed_dim=self.cfg["preprocessor_output_size"],
            num_heads=self.cfg["num_heads"],
            batch_first=True,
        )
        self.memory_projection = nn.Linear(
            self.cfg["preprocessor_output_size"],
            self.cfg["hidden_size"]
        )

    def initial_state(self) -> List[TensorType]:
        """Return initial recurrent state (no batch dimension)."""
        return [
            torch.zeros(1, self.cfg["memory_size"]),
        ]

    def forward_memory(
        self,
        z: TensorType,  # [B, T, F]
        state: List[TensorType],
        t_starts: TensorType,
        seq_lens: TensorType,
    ) -> Tuple[TensorType, List[TensorType]]:
        """Process sequence through memory model."""
        B, T, F = z.shape

        # Self-attention over sequence
        attn_out, _ = self.core(z, z, z)

        # Project to hidden size
        out = self.memory_projection(attn_out)

        # Update state (last hidden for next forward pass)
        new_state = [out[:, -1:, :self.cfg["memory_size"]]]

        return out, new_state

# Use custom model in config
config = {
    "model": {
        "max_seq_len": 1024,
        "custom_model": CustomMemoryModel,
        "custom_model_config": {
            "memory_size": 64,
            "num_heads": 4,
            "hidden_size": 256,
        },
    },
    "env": "popgym-RepeatFirstEasy-v0",
    "framework": "torch",
}
```

--------------------------------

### Import Diagnostic Environments

Source: https://context7.com/proroklab/popgym/llms.txt

Diagnostic environments are used to evaluate memory capabilities. They can be accessed via the popgym.envs module.

```python
import gymnasium as gym
import popgym
from popgym.envs import ALL_DIAGNOSTIC
```

--------------------------------

### Print observation output

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Displays the observation returned by the environment step.

```python
print(obs)
>>> (array([0.0348076 , 0.02231686], dtype=float32), 1, 0)
```

--------------------------------

### Markovian Wrapper Usage

Source: https://context7.com/proroklab/popgym/llms.txt

Provides access to the hidden Markov state of the environment within the info dictionary.

```python
import popgym
from popgym.envs.position_only_cartpole import PositionOnlyCartPoleEasy
from popgym.wrappers import PreviousAction, Antialias, Markovian
from popgym.core.observability import Observability, STATE

# Create environment
env = PositionOnlyCartPoleEasy()
env = PreviousAction(env)
env = Antialias(env)

# Add hidden state to info dict
env = Markovian(env, Observability.FULL_IN_INFO_DICT)
obs, info = env.reset(seed=0)
obs, reward, terminated, truncated, info = env.step(env.action_space.sample())

print(f"Partial observation: {obs}")
print(f"Hidden Markov state: {info[STATE]}")
# Hidden state contains full cartpole state: [x, x_dot, theta, theta_dot]
```

--------------------------------

### Print Markov state

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Displays the underlying Markov state from the info dictionary.

```python
print(info[STATE])
>>> array([ 0.0348076 ,  0.14814377,  0.02231686, -0.31778395], dtype=float32)
```

--------------------------------

### Flatten observation and action spaces

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Flattens nested spaces and prints the resulting action space configuration.

```python
wrapped_env = Flatten(wrapped_env)
print(wrapped_env.action_space)
```

--------------------------------

### Cite POPGym

Source: https://github.com/proroklab/popgym/blob/master/README.md

BibTeX entry for referencing the POPGym benchmarking paper.

```bibtex
@inproceedings{
morad2023popgym,
title={{POPG}ym: Benchmarking Partially Observable Reinforcement Learning},
author={Steven Morad and Ryan Kortvelesy and Matteo Bettini and Stephan Liwicki and Amanda Prorok},
booktitle={The Eleventh International Conference on Learning Representations},
year={2023},
url={https://openreview.net/forum?id=chDrutUTs0K}
}
```

--------------------------------

### DiscreteAction Wrapper Usage

Source: https://context7.com/proroklab/popgym/llms.txt

Converts MultiDiscrete action spaces into a single Discrete space, often used in a wrapper chain for DQN algorithms.

```python
import popgym
from popgym.envs.concentration import ConcentrationEasy
from popgym.wrappers import PreviousAction, Flatten, DiscreteAction

# Create environment (Concentration has discrete action space already)
env = ConcentrationEasy()
print(f"Original action space: {env.action_space}")

# Example with MultiDiscrete (hypothetical)
# For environments with MultiDiscrete actions, DiscreteAction converts to single Discrete
env = PositionOnlyCartPoleEasy()
env = PreviousAction(env)
env = Flatten(env)
env = DiscreteAction(env)
print(f"Final action space: {env.action_space}")

# Complete wrapper chain for Q-learning algorithms
def prepare_env_for_dqn(env):
    """Prepare POMDP environment for DQN-style algorithms."""
    env = PreviousAction(env)  # Add previous action to observation
    env = Antialias(env)       # Add initial timestep indicator
    env = Flatten(env)         # Flatten nested spaces
    env = DiscreteAction(env)  # Convert to single discrete action
    return env

env = prepare_env_for_dqn(PositionOnlyCartPoleEasy())
obs, info = env.reset(seed=0)
print(f"Ready for DQN - Obs shape: {obs.shape}, Action space: {env.action_space}")
```

--------------------------------

### Train with GRU Baseline

Source: https://context7.com/proroklab/popgym/llms.txt

Configures a GRU-based model for RLlib training, including custom preprocessor and postprocessor layers.

```python
import popgym
import ray
from torch import nn
from popgym.baselines.ray_models.ray_gru import GRU

# View available GRU-specific hyperparameters
print("GRU config:", GRU.MODEL_CONFIG)
# Output: {'num_recurrent_layers': 1, 'benchmark': False}

# View base model hyperparameters
print("Base config:", GRU.BASE_CONFIG)
# Output: {'hidden_size': 256, 'preprocessor': ..., 'actor': ..., ...}

# Configure training
bptt_size = 1024  # Backprop through time window
config = {
    "model": {
        "max_seq_len": bptt_size,
        "custom_model": GRU,
        "custom_model_config": {
            # Architecture settings
            "preprocessor_input_size": 128,
            "preprocessor_output_size": 64,
            "preprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
            "hidden_size": 128,  # GRU hidden state size
            "postprocessor": nn.Sequential(nn.Linear(128, 64), nn.ReLU()),
            "postprocessor_output_size": 64,
            "actor": nn.Linear(64, 64),
            "critic": nn.Linear(64, 64),
            # GRU-specific settings
            "num_recurrent_layers": 1,
        },
    },
    # RLlib training settings
    "sgd_minibatch_size": bptt_size * 4,
    "train_batch_size": bptt_size * 8,
    "env": "popgym-ConcentrationEasy-v0",
    "rollout_fragment_length": bptt_size,
    "framework": "torch",
    "horizon": bptt_size,
    "batch_mode": "complete_episodes",
}

# Run training (requires Ray)
# ray.init()
# ray.tune.run("PPO", config=config, stop={"timesteps_total": 50_000})
```

--------------------------------

### Train with LSTM Baseline

Source: https://context7.com/proroklab/popgym/llms.txt

Configures an LSTM-based model for RLlib training with support for stacked recurrent layers.

```python
import popgym
import ray
from torch import nn
from popgym.baselines.ray_models.ray_lstm import LSTM

# LSTM-specific configuration
print("LSTM config:", LSTM.MODEL_CONFIG)
# Output: {'num_recurrent_layers': 1, 'benchmark': False}

bptt_size = 1024
config = {
    "model": {
        "max_seq_len": bptt_size,
        "custom_model": LSTM,
        "custom_model_config": {
            "hidden_size": 256,
            "num_recurrent_layers": 2,  # Stack 2 LSTM layers
            "preprocessor_input_size": 128,
            "preprocessor_output_size": 128,
            "preprocessor": nn.Sequential(
                nn.Linear(128, 128),
                nn.LayerNorm(128),
                nn.ReLU()
            ),
            "postprocessor_output_size": 256,
        },
    },
    "env": "popgym-PositionOnlyCartPoleEasy-v0",
    "sgd_minibatch_size": bptt_size * 4,
    "train_batch_size": bptt_size * 8,
    "rollout_fragment_length": bptt_size,
    "framework": "torch",
    "horizon": bptt_size,
    "batch_mode": "complete_episodes",
}

# ray.tune.run("PPO", config=config, stop={"timesteps_total": 100_000})
```

--------------------------------

### Apply Antialias wrapper

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Adds an indicator to the observation space to distinguish the initial timestep.

```python
wrapped_env = Antialias(wrapped_env)
```

--------------------------------

### Flatten and discretize action space

Source: https://github.com/proroklab/popgym/blob/master/docs/source/environment_quickstart.md

Converts nested observation and action spaces into a flat, discrete format for compatibility with standard RL libraries.

```python
DiscreteAction(Flatten(wrapped_env))
```

--------------------------------

### Flatten Wrapper Usage

Source: https://context7.com/proroklab/popgym/llms.txt

Converts nested observation and action spaces into flat 1D arrays for neural network compatibility.

```python
import popgym
from popgym.envs.position_only_cartpole import PositionOnlyCartPoleEasy
from popgym.wrappers import PreviousAction, Antialias, Flatten

# Create environment with nested observation space
env = PositionOnlyCartPoleEasy()
env = PreviousAction(env)
env = Antialias(env)
print(f"Before flatten - Obs space: {env.observation_space}")
# Output: Tuple(Tuple(Box(...), Discrete(2)), Discrete(2))

# Flatten for neural network compatibility
env = Flatten(env)
print(f"After flatten - Obs space: {env.observation_space}")
# Output: Box(shape=(5,), ...)

obs, info = env.reset(seed=0)
print(f"Flattened observation shape: {obs.shape}")
# Output: (5,)

# Flatten only observations, keep action space
env_obs_only = Flatten(
    Antialias(PreviousAction(PositionOnlyCartPoleEasy())),
    flatten_action=False,
    flatten_observation=True
)
print(f"Action space (not flattened): {env_obs_only.action_space}")
```

--------------------------------

### Antialias Wrapper Usage

Source: https://context7.com/proroklab/popgym/llms.txt

Adds a binary flag to observations to distinguish the initial timestep, preventing aliasing issues.

```python
import popgym
from popgym.envs.position_only_cartpole import PositionOnlyCartPoleEasy
from popgym.wrappers import PreviousAction, Antialias

# Create environment with PreviousAction wrapper
env = PositionOnlyCartPoleEasy()
env = PreviousAction(env)

# Add Antialias to distinguish initial timestep
env = Antialias(env)
print(f"Observation space: {env.observation_space}")
# Output: Tuple(Tuple(Box(...), Discrete(2)), Discrete(2))

# Reset returns obs with is_t0 flag = 1 (True)
obs, info = env.reset(seed=0)
print(f"Initial observation: {obs}")
# Output: ((array([...]), 0), 1)  # Last element is is_t0 flag

# After step, is_t0 flag = 0 (False)
action = env.action_space.sample()
obs, reward, terminated, truncated, info = env.step(action)
print(f"Observation after step: {obs}")
# Output: ((array([...]), <action>), 0)  # is_t0 = 0
```

--------------------------------

### Use PreviousAction Wrapper

Source: https://context7.com/proroklab/popgym/llms.txt

The PreviousAction wrapper modifies the observation space to include the previous action. This is useful for POMDPs where the agent's last action is important context for decision-making. The wrapper appends the previous action as a discrete value to the observation.

```python
import popgym
from popgym.envs.position_only_cartpole import PositionOnlyCartPoleEasy
from popgym.wrappers import PreviousAction

# Create base environment
env = PositionOnlyCartPoleEasy()
print(f"Original observation space: {env.observation_space}")
# Output: Box([-4.8, -0.419], [4.8, 0.419], (2,), float32)

# Wrap with PreviousAction
wrapped_env = PreviousAction(env)
print(f"Wrapped observation space: {wrapped_env.observation_space}")
# Output: Tuple(Box(...), Discrete(2))

# Reset returns observation tuple (obs, prev_action)
obs, info = wrapped_env.reset(seed=0)
print(f"Initial observation: {obs}")
# Output: (array([...], dtype=float32), 0)  # prev_action is null (0) at reset
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.