### Setup Local Development Environment in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tips.md

This snippet shows how to activate local development mode for the reinforcement learning project by developing specific packages. It also demonstrates how to add extra dependencies by activating the correct environment first.

```julia
using Pkg
Pkg.develop(path="src/ReinforcementLearningBase")
Pkg.develop(path="src/ReinforcementLearningCore")
Pkg.develop(path="src/ReinforcementLearningEnvironments")
Pkg.develop(path="src/ReinforcementLearningFarm") # optional
```

--------------------------------

### Simultaneous Multi-Agent Environment Example in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/an_introduction_to_reinforcement_learning_jl_design_implementations_thoughts/index.md

Illustrates a simultaneous multi-agent environment setup in Julia, where all agents play every turn. Basic examples like `TicTacToeEnv` and `RockPaperScissorsEnv` are provided.

```julia
using RLBase

abstract type SimultaneousEnv <: AbstreactEnv end

# Example for TicTacToeEnv
struct TicTacToeEnv <: SimultaneousEnv
    # ... board state and player information
end

# Example for RockPaperScissorsEnv
struct RockPaperScissorsEnv <: SimultaneousEnv
    # ... game state and player information
end

# In a MultiAgentPolicy for SimultaneousEnv, all agents are assumed to act each turn.
```

--------------------------------

### Simultaneous Action Execution in RockPaperScissorsEnv

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

This example illustrates simultaneous action execution in a multi-agent environment. It shows how to get the action space, plan an action using a policy, and execute the action in the RockPaperScissorsEnv.

```julia
using ReinforcementLearning

rps = RockPaperScissorsEnv();
action_space(rps)
action = plan!(RandomPolicy(), rps)
act!(rps, action)
```

--------------------------------

### Implement Custom Hook Stages in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Provides examples of how to implement custom hooks for reinforcement learning experiments in Julia. It shows the required methods to extend for different stages of an agent's interaction: `PreActStage`, `PostActStage`, `PreEpisodeStage`, and `PostEpisodeStage`. Subtyping `AbstractHook` provides default implementations.

```julia
# Example for extending specific stages:
# (hook::YourHook)(::PreActStage, agent, env, action)
# (hook::YourHook)(::PostActStage, agent, env)
# (hook::YourHook)(::PreEpisodeStage, agent, env)
# (hook::YourHook)(::PostEpisodeStage, agent, env)
```

--------------------------------

### Sequential Multi-Agent Environment Setup in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/an_introduction_to_reinforcement_learning_jl_design_implementations_thoughts/index.md

Details the requirements for setting up sequential multi-agent environments in Julia. It mandates the implementation of `RLBase.next_player!` and `current_player` for proper turn management within the `Base.run` loop.

```julia
using RLBase

abstract type SequentialEnv <: AbstreactEnv end

function RLBase.next_player!(env::SequentialEnv)
    # Logic to advance to the next player
end

function RLBase.current_player(env::SequentialEnv)
    # Returns the current player
end

# Example usage in a run loop:
# Base.run(policy, env)
# The run loop will use next_player! and current_player to manage turns.
```

--------------------------------

### Basic RL Experiment Setup with Random Policy

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Sets up a simple RL experiment by creating an environment and running a random policy for a specified number of steps. It then prints the collected rewards and their average. This demonstrates the basic usage of the ReinforcementLearning.jl framework.

```julia
using ReinforcementLearning

# Create environment and run random policy for 1000 steps
env = CartPoleEnv()
policy = RandomPolicy()
hook = TotalRewardPerEpisode()

run(policy, env, StopAfterNSteps(1_000), hook)

# Access collected rewards
println("Total rewards: ", hook.rewards)
println("Average reward: ", mean(hook.rewards))
```

--------------------------------

### Activate Julia Project and Display Status

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

This snippet demonstrates how to activate a specific Julia project environment and then display the status of the installed packages within that environment. It's crucial for managing project dependencies.

```julia
]
activate .

]
st
```

--------------------------------

### Load DM Dataset Example in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_final_term_report_210370741/index.md

This example demonstrates how to load a DM dataset, specifically 'fish_swim' from the DM control suite, using the `rl_unplugged_dm_dataset` function. It shows the expected output format, which is a RingBuffer containing transition data.

```julia
rl_unplugged_dm_dataset("fish_swim", [1]; type="dm_control_suite")
```

--------------------------------

### Initialize and Interact with RandomWalk1D Environment in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

This code initializes a RandomWalk1D environment and demonstrates basic interactions. It shows how to get the current state, the potential reward (before action), the action space, and how to apply an action to transition the environment to a new state.

```julia
env = RandomWalk1D()

state(env)
reward(env) # though it is meaningless since we haven't applied any action yet
action_space(env)

env(2)

state(env)
```

--------------------------------

### Interact with RandomWalk1D Environment in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tutorial.md

Demonstrates basic interaction with the `RandomWalk1D` environment. It shows how to get state and action spaces, current state, check termination, take random actions, and retrieve rewards. This is fundamental for understanding environment dynamics.

```julia
using ReinforcementLearning
env = RandomWalk1D()

S = state_space(env)
s = state(env)  # the initial position
A = action_space(env)

is_terminated(env)

while true
    act!(env, rand(A))
    is_terminated(env) && break
end

state(env)
reward(env)
```

--------------------------------

### Q-Approximator Forward Pass Examples

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

Demonstrates the forward pass of a TabularQApproximator in ReinforcementLearning.jl. It shows how to query Q-values for a specific state-action pair, all actions for a given state, and highlights the limitation that it only accepts Int states.

```julia
RLCore.forward(p.learner.approximator, 1, 1)  # Q(s, a)
RLCore.forward(p.learner.approximator, 1)     # [Q(s, a) for a in action_space(env)]
RLCore.forward(p.learner.approximator, false)
```

--------------------------------

### Multi-Threaded PPO Agent Setup in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Initializes a multi-threaded environment and prepares for training a Proximal Policy Optimization (PPO) agent. It creates multiple `CartPoleEnv` instances using `MultiThreadEnv` for parallel processing and sets up the actor-critic architecture for the PPO policy.

```julia
rng = StableRNG(123)
N_ENV = 8
UPDATE_FREQ = 32
env = MultiThreadEnv([
    CartPoleEnv(; T = Float32, rng = StableRNG(hash(123 + i))) for i in 1:N_ENV
]);
RLBase.reset!(env, is_force = true)

agent = Agent(
    policy = PPOPolicy(
        approximator = ActorCritic(
            actor = Chain(
                Dense(ns, 256, relu; init = glorot_uniform(rng)),
```

--------------------------------

### Enable and Manage Debug Timings in Julia RLCore

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tips.md

This snippet demonstrates how to enable debug timings for experiment runs within the RLCore module. It includes functions to enable, reset, and display timer results for hooks, policies, and optimization steps.

```julia
RLCore.TimerOutputs.enable_debug_timings(RLCore)
RLCore.TimerOutputs.reset_timer!(RLCore.timer)
RLCore.timer
```

--------------------------------

### Run Basic DQN CartPole Experiment in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/index.html

This snippet shows how to use the ReinforcementLearningExperiments package to run a basic Deep Q-Network (DQN) experiment on the CartPole environment. It requires the ReinforcementLearningExperiments package to be installed and loaded.

```julia
using ReinforcementLearningExperiments

run(E"JuliaRL_BasicDQN_CartPole")
```

--------------------------------

### MADDPG: SpeakerListenerEnv Setup and Policy Creation

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

Initializes the SpeakerListener environment, defines network parameters, and creates actor and critic networks for the MADDPG policy. It calculates dimensions based on environment states and action spaces, using glorot_uniform initialization.

```julia
env = SpeakerListenerEnv(max_steps = 25)
init = glorot_uniform(rng)
critic_dim = sum(length(state(env, p)) + length(action_space(env, p)) for p in (:Speaker, :Listener))
create_actor(player) = Chain(
    Dense(length(state(env, player)), 64, relu; init = init),
    Dense(64, 64, relu; init = init),
    Dense(64, length(action_space(env, player)); init = init)
    )
create_critic(critic_dim) = Chain(
    Dense(critic_dim, 64, relu; init = init),
    Dense(64, 64, relu; init = init),
    Dense(64, 1; init = init),
    )
```

--------------------------------

### Install and Use ReinforcementLearning.jl

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/README.md

This snippet demonstrates how to add the ReinforcementLearning.jl package to your Julia environment and how to run a basic reinforcement learning experiment using a RandomPolicy, CartPoleEnv, a stop condition, and a hook to collect rewards. It showcases the fundamental components of the package.

```julia
julia> ] add ReinforcementLearning

julia> using ReinforcementLearning

julia> run(
           RandomPolicy(),
           CartPoleEnv(),
           StopAfterNSteps(1_000),
           TotalRewardPerEpisode()
       )
```

--------------------------------

### Query Number of Agents in an Environment

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

This snippet shows how to determine the number of agents interacting with a reinforcement learning environment using NumAgentStyle. Examples are provided for a generic 'env' and the TicTacToeEnv.

```julia
using ReinforcementLearning

NumAgentStyle(env)
NumAgentStyle(ttt)
```

--------------------------------

### Handle TFRecord Example for DM Datasets in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_final_term_report_210370741/index.md

This function processes TFRecord examples from DM datasets, parsing features like observations, actions, and step types into a structured NamedTuple. It handles different feature types and sizes, including egocentric camera data, to create transition objects.

```julia
function make_transition(example::TFRecord.Example, feature_size::Dict{String, Tuple})
    f = example.features.feature
    
    observation_dict = Dict{Symbol, AbstractArray}()
    next_observation_dict = Dict{Symbol, AbstractArray}()
    transition_dict = Dict{Symbol, Any}()

    for feature in keys(feature_size)
        if split(feature, "/")[1] == "observation"
            ob_key = Symbol(chop(feature, head = length("observation")+1, tail=0))
            if split(feature, "/")[end] == "egocentric_camera"
                cam_feature_size = feature_size[feature]
                ob_size = prod(cam_feature_size)
                observation_dict[ob_key] = reshape(f[feature].bytes_list.value[1][1:ob_size], cam_feature_size...)
                next_observation_dict[ob_key] = reshape(f[feature].bytes_list.value[1][ob_size+1:end], cam_feature_size...)
            else
                if feature_size[feature] == ()
                    observation_dict[ob_key] = f[feature].float_list.value
                else
                    ob_size = feature_size[feature][1]
                    observation_dict[ob_key] = f[feature].float_list.value[1:ob_size]
                    next_observation_dict[ob_key] = f[feature].float_list.value[ob_size+1:end]
                end
            end
        elseif feature == "action"
            ob_size = feature_size[feature][1]
            action = f[feature].float_list.value
            transition_dict[:action] = action[1:ob_size]
            transition_dict[:next_action] = action[ob_size+1:end]
        elseif feature == "step_type"
            transition_dict[:terminal] = f[feature].float_list.value[1] == 2
        else
            ob_key = Symbol(feature)
            transition_dict[ob_key] = f[feature].float_list.value[1]
        end
    end
    state_nt = (state = NamedTuple(observation_dict),)
    next_state_nt = (next_state = NamedTuple(next_observation_dict),)
    transition = NamedTuple(transition_dict)

    merge(transition, state_nt, next_state_nt)
end
```

--------------------------------

### Custom Environment Implementation: GridWorld

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Provides a template for implementing a custom environment in Julia by defining a struct that adheres to the AbstractEnv interface. This example defines a GridWorld environment with customizable size, agent/goal positions, and termination conditions.

```julia
using ReinforcementLearning
using Random

Base.@kwdef mutable struct GridWorld <: AbstractEnv
    size::Int = 5
    agent_pos::Tuple{Int,Int} = (1, 1)
    goal_pos::Tuple{Int,Int} = (size, size)
    done::Bool = false
    reward::Float64 = 0.0
    max_steps::Int = 100
    current_step::Int = 0
    rng::AbstractRNG = Random.default_rng()
end
```

--------------------------------

### Display Reward Style of an Environment

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

This snippet demonstrates how to display the reward style of a given reinforcement learning environment. It shows examples for a generic type and a specific environment like MontyHallEnv.

```julia
using ReinforcementLearning

RewardStyle(tp)
RewardStyle(MontyHallEnv())
```

--------------------------------

### Instantiate OfflinePolicy with DQNLearner

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/phase1_technical_report_of_enriching_offline_reinforcement_learning_algorithms_in_reinforcement_learning_jl/index.md

Demonstrates how to instantiate an OfflinePolicy using a DQNLearner. This example highlights the configuration of the learner, dataset, action space continuity, and batch size, showcasing the practical application of the OfflinePolicy framework.

```julia
offline_dqn_policy = OfflinePolicy(
    learner = DQNLearner(
        # Omit specific code
    ),
    dataset = dataset,
    continuous = false,
    batchsize = 64,
)
```

--------------------------------

### Import Reinforcement Learning Libraries in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

This code imports essential libraries for reinforcement learning in Julia, including ReinforcementLearning itself, plotting tools, deep learning frameworks (Flux, CUDA, Zygote), and utility packages. This setup is necessary before defining or interacting with RL agents and environments.

```julia
using ReinforcementLearning
using StableRNGs
using Flux
using Flux.Losses
using IntervalSets
using CUDA
using Plots
```

--------------------------------

### Agent with Trajectory for Learning in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tutorial.md

Demonstrates the creation of an `Agent` that uses a `Trajectory` to store interaction data (State, Action, Reward, Terminal info). This agent is configured with a `RandomPolicy` and then run on the environment, illustrating the setup for policies in 'learner' mode.

```julia
using ReinforcementLearningTrajectories

trajectory = Trajectory(
    ElasticArraySARTSTraces(;
        state = Int64 => (),
        action = Int64 => (),
        reward = Float64 => (),
        terminal = Bool => (),
    ),
    DummySampler(),
    InsertSampleRatioController(),
)
agent = Agent(
    policy = RandomPolicy(),
    trajectory = trajectory
)
run(agent, env, StopAfterNEpisodes(10), TotalRewardPerEpisode())
```

--------------------------------

### Run Offline RL Experiments with Predefined Commands

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/phase2_technical_report_of_enriching_offline_reinforcement_learning_algorithms_in_reinforcement_learning_jl/index.md

Provides example commands to run various offline RL algorithms on Pendulum and CartPole environments. The `type` argument specifies the dataset collection method, which can be 'random', 'medium', or 'expert'. These commands are entry points for evaluating different offline RL strategies.

```julia
run(E`JuliaRL_BCQ_Pendulum(type)`)
run(E`JuliaRL_BCQD_CartPole(type)`)
run(E`JuliaRL_BEAR_Pendulum(type)`)
run(E`JuliaRL_CRR_Pendulum(type)`)
run(E`JuliaRL_CRR_CartPole(type)`)
run(E`JuliaRL_FisherBRC_Pendulum(type)`)
run(E`JuliaRL_PLAS_Pendulum(type)`)
```

```julia
run(E`JuliaRL_BCQ_Pendulum(medium)`)
```

--------------------------------

### Imperative Hook Example in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_use_hooks.md

A basic imperative programming approach to interact with an environment, where custom logic is explicitly written within a loop for tasks like saving parameters or evaluating policies.

```julia
while true
    action = plan!(policy, env)
    act!(env, action)

    # write your own logic here
    # like saving parameters, recording loss function, evaluating policy, etc.
    check!(stop_condition, env, policy) && break
    is_terminated(env) && reset!(env)
end
```

--------------------------------

### Log Training Loss with TensorBoard Hook in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Demonstrates how to log the training loss of an agent during reinforcement learning using a TensorBoard hook in Julia. It utilizes `DoEveryNSteps` to log the loss at regular intervals, incorporating the `with_logger` context manager and `@info` macro for logging.

```julia
DoEveryNSteps() do t, agent, env
    with_logger(lg) do
        @info "training" loss = agent.policy.learner.loss
    end
end,
```

--------------------------------

### UCB Explorer for Multi-Armed Bandits

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Initializes a 10-armed bandit problem environment using ReinforcementLearning.jl. This setup is intended for demonstrating and experimenting with exploration strategies like Upper Confidence Bound (UCB), although the UCB specific code is not shown in this snippet.

```julia
using ReinforcementLearning

# 10-armed bandit problem
env = MultiArmBanditsEnv(k=10, true_reward=0.0)

```

--------------------------------

### Recording Trajectory Data in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Demonstrates how to record trajectory data in a reinforcement learning context using Julia. The example shows pushing a SART (State, Action, Reward, Terminal) tuple into a trajectory object, highlighting the data structure used for storing agent experiences.

```julia
push!(t; reward=-1., terminal=true, state=1, action=1) # this action is meaningness
t
```

--------------------------------

### Load Google Research Atari DQN Replay Datasets

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_final_term_report_210370741/index.md

Provides an example of loading Google Research Atari DQN replay datasets using ReinforcementLearningDatasets.jl. Currently, datasets are loaded entirely into RAM, requiring significant memory (around 20 GB). Future support for lazy parallel loading is planned.

```julia
using ReinforcementLearningDatasets
ds = dataset(
        "pong",
        1,
        [1, 2]
    )
samples = Iterators.take(ds, 2)
```

--------------------------------

### Get Probability for Specific Action with EDPolicy

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

Retrieves the probability of a specific action given the environment and policy. It first computes the full probability distribution and then returns the probability corresponding to the given action. Includes error handling for actions not present in the action space. Dependencies: RLBase.

```julia
function RLBase.prob(π::EDPolicy, env::AbstractEnv, action)
    A = action_space(env)
    P = prob(π, env)
    @assert length(A) == length(P)
    if A isa Base.OneTo
        P[action]
    else
        for (a, p) in zip(A, P)
            if a == action
                return p
            end
        end
        @error "action[$action] is not found in action space[$(action_space(env))]"
    end
end
```

--------------------------------

### Define and Run QBasedPolicy in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tutorial.md

Sets up and runs a `QBasedPolicy` which uses a `TDLearner` with a `TabularQApproximator` and an `EpsilonGreedyExplorer`. This policy is then applied to the `RandomWalk1D` environment for simulation, demonstrating a more sophisticated learning approach.

```julia
NS = length(S)
NA = length(A)
policy = QBasedPolicy(
    learner = TDLearner(
            TabularQApproximator(
                n_state = NS,
                n_action = NA,
            ),
            :SARS
        ),
    explorer = EpsilonGreedyExplorer(0.1)
)

run(
    policy,
    RandomWalk1D(),
    StopAfterNEpisodes(10),
    TotalRewardPerEpisode()
)
```

--------------------------------

### Display Environment Versions - Julia REPL

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/FAQ.md

This code snippet demonstrates how to display the current Julia environment versions, including package statuses. It is useful for diagnosing compatibility issues or reporting bugs. It requires the `Pkg` and `Dates` modules.

```julia
using Pkg, Dates
today()
versioninfo()
buff = IOBuffer();Pkg.status(io=buff);println(String(take!(buff)))
```

--------------------------------

### Define Actor Network for MADDPG in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

Defines a neural network for the actor in a MADDPG setup. It uses Dense layers with ReLU and Tanh activations, initialized with glorot_uniform.

```julia
rng = StableRNG(123)
ns, na = 1, 1 # dimension of the state and action.
n_players = 2 # the number of players

create_actor() = Chain(
        Dense(ns, 64, relu; init = glorot_uniform(rng)),
        Dense(64, 64, relu; init = glorot_uniform(rng)),
        Dense(64, na, tanh; init = glorot_uniform(rng)),
    )
```

--------------------------------

### Run Q-Learning on GridWorld Environment

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Demonstrates running a Q-based policy with a TD learner and epsilon-greedy explorer on the custom GridWorld environment. It includes setting up the environment, policy, a hook for total reward, and running the experiment until a stop condition is met. Finally, it prints the average reward over the last 100 episodes.

```julia
# Use custom environment
env = GridWorld(size=5)
policy = QBasedPolicy(
    learner = TDLearner(
        TabularQApproximator(n_state=25, n_action=4),
        :SARS,
        γ=0.95,
        α=0.1
    ),
    explorer = EpsilonGreedyExplorer(ϵ_stable=0.1, decay_steps=5_000)
)

hook = TotalRewardPerEpisode()
run(policy, env, StopAfterNEpisodes(1_000), hook)
println("Average reward: ", mean(hook.rewards[end-99:end]))
```

--------------------------------

### Configure and Run Experiment with CartPoleEnv

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Sets up and runs a reinforcement learning experiment using the CartPole environment. It defines the environment, a Q-based policy with a TD learner and epsilon-greedy explorer, a stop condition, and hooks for collecting rewards and steps. The experiment is then encapsulated in an `Experiment` object and executed.

```julia
using ReinforcementLearning

# Define all experiment components
env = CartPoleEnv()
policy = QBasedPolicy(
    learner = TDLearner(
        TabularQApproximator(n_state=4, n_action=2),
        :SARS,
        γ=0.99,
        α=0.05
    ),
    explorer = EpsilonGreedyExplorer(0.1)
)

stop_condition = StopAfterNEpisodes(1_000)
hook = TotalRewardPerEpisode() + StepsPerEpisode()

# Create experiment object
experiment = Experiment(policy, env, stop_condition, hook)

# Run experiment (can be serialized/saved for reproducibility)
run(experiment)

# Access results
println("Rewards: ", hook[1][])
println("Steps: ", hook[2][])
```

--------------------------------

### Define Critic Network for MADDPG in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

Defines a neural network for the critic in a MADDPG setup. It takes the concatenated states and actions of all players as input and outputs a single value.

```julia
rng = StableRNG(123)
ns, na = 1, 1 # dimension of the state and action.
n_players = 2 # the number of players

create_critic() = Chain(
    Dense(n_players * ns + n_players * na, 64, relu; init = glorot_uniform(rng)),
    Dense(64, 64, relu; init = glorot_uniform(rng)),
    Dense(64, 1; init = glorot_uniform(rng)),
    )
```

--------------------------------

### Get Run Function Definition

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Provides the method signature for the `run` function when used with an abstract policy and environment, indicating it runs until the end of an episode. This is useful for understanding the underlying implementation.

```julia
@which run(policy, env)
```

--------------------------------

### Run a Full Episode Simulation

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Illustrates how to run a single episode of interaction between a policy and an environment until the episode naturally terminates. It also shows how to inspect the state, reward, and termination status of the environment after a run.

```julia
run(policy, env)

state(env), reward(env), is_terminated(env)
```

--------------------------------

### Custom Action Space Definition in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

This example defines a custom action space, SimplexSpace, for actions that lie on an N-dimensional simplex. It implements the `Base.in` and `Random.rand` methods to define membership and random sampling within this space.

```julia
using Random

struct SimplexSpace
    n::Int
end

function Base.in(x::AbstractVector, s::SimplexSpace)
    length(x) == s.n && all(>=(0), x) && isapprox(1, sum(x))
end

function Random.rand(rng::AbstractRNG, s::SimplexSpace)
    x = rand(rng, s.n)
    x ./= sum(x)
    x
end
```

--------------------------------

### Using Built-in Environments

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Shows how to instantiate and use various built-in environments provided by ReinforcementLearning.jl, including CartPole, MountainCar, RandomWalk1D, MultiArmBanditsEnv, and PendulumEnv. It demonstrates running a simple experiment with RandomPolicy on selected environments.

```julia
using ReinforcementLearning

# CartPole balancing
env1 = CartPoleEnv()
env1_continuous = CartPoleEnv(continuous=true)

# Mountain car
env2 = MountainCarEnv()
env2_continuous = ContinuousMountainCarEnv()

# Simple random walk
env3 = RandomWalk1D(N=7, rewards=-1.0 => 1.0)

# Multi-armed bandits
env4 = MultiArmBanditsEnv(k=10, true_reward=0.0)

# Pendulum
env5 = PendulumEnv()

# Run experiment on any environment
for (name, env) in [("CartPole", env1), ("MountainCar", env2), ("RandomWalk", env3)]
    hook = TotalRewardPerEpisode(is_display_on_exit=false)
    run(RandomPolicy(), env, StopAfterNEpisodes(10), hook)
    println("$name avg reward: ", mean(hook.rewards))
end
```

--------------------------------

### Basic DQN Agent Configuration in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Sets up a Deep Q-Network (DQN) agent for reinforcement learning. This involves defining the neural network approximator, the DQN learner with replay history and loss function, and an epsilon-greedy explorer for action selection. It uses `CartPoleEnv` as the environment and `CircularArraySARTTrajectory` for storing experience.

```julia
env = CartPoleEnv();
ns, na = length(state(env)), length(action_space(env))

policy = Agent(
    policy = QBasedPolicy(
        learner = BasicDQNLearner(
            approximator = NeuralNetworkApproximator(
                model = Chain(
                    Dense(ns, 128, relu; init = glorot_uniform(rng)),
                    Dense(128, 128, relu; init = glorot_uniform(rng)),
                    Dense(128, na; init = glorot_uniform(rng)),
                ) |> cpu,
                optimizer = Adam(),
            ),
            batchsize = 32,
            min_replay_history = 100,
            loss_func = huber_loss,
            rng = rng,
        ),
        explorer = EpsilonGreedyExplorer(
            kind = :exp,
            ϵ_stable = 0.01,
            decay_steps = 500,
            rng = rng,
        ),
    ),
    trajectory = CircularArraySARTTrajectory(
        capacity = 1000,
        state = Vector{Float32} => (ns,)
    ),
);
```

--------------------------------

### Get Available D4RL Policy Parameters in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_final_term_report_210370741/index.md

This function retrieves the parameters required for loading D4RL policies. It returns a set of available environments, agent types, and epoch ranges, which can be useful for understanding the scope of supported policies.

```julia
julia> d4rl_policy_params()
┌ Info: Set(["relocate", "maze2d_large", "antmaze_umaze", "hopper", "pen", "antmaze_medium", "walker", "hammer", "antmaze_large", "maze2d_umaze", "maze2d_medium", "ant", "door", "halfcheetah"])
│   agent = 2-element Vector{String}: …
└   epoch = 0:10
```

--------------------------------

### Define QBasedPolicy Structure and Plan Action in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/an_introduction_to_reinforcement_learning_jl_design_implementations_thoughts/index.md

Defines the QBasedPolicy struct, which encapsulates a Q-value learner and an action explorer. It also provides the plan! function to determine the next action based on the current environment state, utilizing the inner learner and explorer.

```julia
struct QBasedPolicy{L, E} <: AbstractPolicy
    learner::L
    explorer::E
end

RLBase.plan!((p::QBasedPolicy, env::AbstractEnv) = env |> p.learner |> p.explorer
```

--------------------------------

### Define Custom State Access for LotteryEnv

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

Defines a custom method for accessing the state of a LotteryEnv when the requested state style is Observation{String}. This allows the environment to return a string representation ('Game Over' or 'Game Start') based on its termination status.

```julia
RLBase.state(::Observation{String}, env::LotteryEnv) = is_terminated(env) ? "Game Over" : "Game Start"
```

--------------------------------

### Define and Instantiate a Random Policy

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Demonstrates how to create a random policy using the `RandomPolicy` constructor, which requires the action space of the environment. It shows the structure of the instantiated policy, including its action space and random number generator.

```julia
policy = RandomPolicy(action_space(env))

typename(RandomPolicy)
├─ action_space => 2-element Base.OneTo{Int64}
└─ rng => typename(Random._GLOBAL_RNG)
```

--------------------------------

### Q-Learning with Tabular Approximator

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Implements a complete Q-learning algorithm on a discrete environment using a tabular approximator. It sets up the environment, creates a Q-based policy with epsilon-greedy exploration, trains the policy for a number of episodes, and then evaluates the learned policy.

```julia
using ReinforcementLearning

# Setup environment
env = RandomWalk1D(N=7, rewards=-1.0 => 1.0)
NS = length(state_space(env))  # Number of states
NA = length(action_space(env))  # Number of actions

# Create Q-learning policy with epsilon-greedy exploration
policy = QBasedPolicy(
    learner = TDLearner(
        TabularQApproximator(n_state=NS, n_action=NA, init=0.0),
        :SARS,              # Q-learning method
        γ=0.95,             # Discount factor
        α=0.1               # Learning rate
    ),
    explorer = EpsilonGreedyExplorer(
        ϵ_stable=0.01,      # Final epsilon
        ϵ_init=1.0,         # Initial epsilon
        kind=:linear,
        warmup_steps=0,
        decay_steps=5_000
    )
)

# Train for 10,000 episodes
hook = TotalRewardPerEpisode()
run(policy, env, StopAfterNEpisodes(10_000), hook)

# Evaluate learned policy with greedy explorer
test_policy = QBasedPolicy(policy.learner, GreedyExplorer())
test_hook = TotalRewardPerEpisode(is_display_on_exit=false)
run(test_policy, env, StopAfterNEpisodes(100), test_hook)

println("Training average (last 100): ", mean(hook.rewards[end-99:end]))
println("Test average: ", mean(test_hook.rewards))
```

--------------------------------

### Advanced Stop Conditions for Training

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Demonstrates combining multiple stop criteria for training using StopIfAny and StopIfAll. Examples include stopping after a certain number of episodes, seconds, or when performance plateaus. It uses TotalRewardPerEpisode hook to track performance for the StopAfterNoImprovement condition.

```julia
using ReinforcementLearning
using Statistics

env = CartPoleEnv()
agent = Agent(...)

# Track performance
hook = TotalRewardPerEpisode(is_display_on_exit=false)

# Complex stop condition: stop if any condition met
stop_condition = StopIfAny(
    StopAfterNEpisodes(10_000),          # Max episodes
    StopAfterNSeconds(3600.0),           # Max 1 hour
    StopAfterNoImprovement(
        () -> mean(hook.rewards[max(1, end-99):end]),
        patience=500,                     # Wait 500 episodes
        δ=0.1                            # Min improvement
    )
)

run(agent, env, stop_condition, hook)
println("Stopped after $(length(hook.rewards)) episodes")

# Alternative: all conditions must be met
stop_all = StopIfAll(
    StopAfterNEpisodes(1000),
    StopIfEnvTerminated()
)
```

--------------------------------

### Schedule ApeXOptimizer Actor Initialization

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/wiki/[TEMP]-A-Very-Draft-Proposal

Demonstrates how to schedule and wrap the ApeXOptimizer actor for distributed execution using actor model abstractions. This typically involves configuration parameters.

```julia
@schedule @wrap_optimizer_actor(ApeXOptimizer(configs))
```

--------------------------------

### Environment and EDManager Initialization for Kuhn Poker

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

Sets up the environment and the EDManager for the Kuhn Poker experiment. It wraps the OpenSpiel environment, transforms actions and state representations, and defines the neural network architecture and learner for the ED policy. The EDManager is then initialized with policies for each player, and the stop condition and custom hook are instantiated. Dependencies include StableRNG, OpenSpielEnv, ActionTransformedEnv, DefaultStateStyleEnv, Flux, and RLBase.

```julia
# set random seed.
rng = StableRNG(123)
# wrap and initial the OpenSpiel environment.
env = OpenSpielEnv(game)
wrapped_env = ActionTransformedEnv(
    env,
    action_mapping = a -> RLBase.current_player(env) == chance_player(env) ? a : Int(a - 1),
    action_space_mapping = as -> RLBase.current_player(env) == chance_player(env) ? 
        as : Base.OneTo(num_distinct_actions(env.game)),
)
wrapped_env = DefaultStateStyleEnv{InformationSet{Array}()}(wrapped_env)
player = 0 # or 1
ns, na = length(state(wrapped_env, player)), length(action_space(wrapped_env, player))
# construct the `EDmanager`.
create_network() = Chain(
    Dense(ns, 64, relu;init = glorot_uniform(rng)),
    Dense(64, na;init = glorot_uniform(rng))
)
create_learner() = NeuralNetworkApproximator(
    model = create_network(),
    # set the l2-regularization and use gradient descent optimizer.
    optimizer = Flux.Optimise.Optimiser(WeightDecay(0.001), Descent())
)
EDmanager = EDManager(
    Dict(
        player => EDPolicy(
            1 - player, # opponent
            create_learner(), # neural network learner
            WeightedSoftmaxExplorer(), # explorer
        )
        for player in players(env) if player != chance_player(env)
    )
)
# initialize the `stop_condition` and `hook`.
stop_condition = StopAfterNEpisodes(100_000, is_show_progress=!haskey(ENV, "CI"))
hook = KuhnOpenNewEDHook(0, 100, [], [])
```

--------------------------------

### Q-Based Policy Implementation in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/rlcore.md

Demonstrates how to implement a Q-based policy using an existing learner and explorer. This approach avoids manually coding the interaction between the policy and the explorer by defining an AbstractLearner subtype and specializing the `optimise!` function.

```julia
struct YourLearnerType <: AbstractLearner end

function RLBase.optimise!(::YourLearnerType, ::Stage, ::Trajectory)
    # Implementation specific to your algorithm
end
```

--------------------------------

### Run RL Agent with Greedy Explorer and Plot Steps

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

This code snippet demonstrates running a reinforcement learning agent with a Q-based policy and a greedy explorer for a fixed number of episodes. It then plots the number of steps taken per episode using a StepsPerEpisode hook.

```julia
hook = StepsPerEpisode()
run(
    QBasedPolicy(
        learner=policy.learner,
        explorer=GreedyExplorer()
    ),
    env,
    StopAfterNEpisodes(10),
    hook
)
plot(hook.steps[1:end-1])
```

--------------------------------

### Initialize Q-Based Policy with TD Learner

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

Initializes a Q-Based Policy using a TD Learner with a Tabular Q-Approximator. This policy estimates state-action values for reinforcement learning tasks. It requires state and action space information from the environment. The default explorer is EpsilonGreedy.

```julia
p = QBasedPolicy(
    learner = TDLearner(
        TabularQApproximator(
            n_state = length(state_space(env)),
            n_action = length(action_space(env)),
        ),
        :SARS
    ),
    explorer = EpsilonGreedyExplorer(0.1)
)
plan!(p, env)
```

--------------------------------

### Create an Environment Actor in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/wiki/[TEMP]-A-Very-Draft-Proposal

Demonstrates how to wrap an `AbstractEnv` into an asynchronous actor using Julia's actor model. The actor continuously receives and processes messages for interacting, observing, and resetting the environment. It highlights the use of `@actor` and `@match` for message handling and suggests a `@wrap_actor` macro for simplification.

```julia
env_actor = @actor begin
    env = ExampleEnv(init_configs)
    while true
        sender, msg = receive()
        @match msg
            (:interact!, actions) => interact!(env, actions)
            (:observe, role) => tell(sender, observe(env, role))
            (:reset!,) => reset!(env)
            # do something else
            (:ping,) => tell(sender, :pong)
        end
    end
end

# The code above can be further simplified by introducing an `@wrap_actor` macro
env_actors = @wrap_environment_actor ExampleEnv(init_configs)
```

--------------------------------

### Define Legal Action Space Mask in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Demonstrates how to define a legal action space mask for environments with a full action set. This is often used in neural network-based algorithms to filter actions at the output layer. It requires `legal_action_space` and `action_space` to be defined for the environment.

```julia
RLBase.legal_action_space_mask(env::YourEnv) = map(action_space(env)) do action
    action in legal_action_space(env)
end
```

--------------------------------

### Action Style and Legal Actions in TicTacToeEnv

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/How_to_write_a_customized_environment.md

This code demonstrates how to query the action style and retrieve legal actions for the TicTacToe environment. It shows the output of ActionStyle, legal_action_space, and legal_action_space_mask.

```julia
using ReinforcementLearning

ttt = TicTacToeEnv();
ActionStyle(ttt)
legal_action_space(ttt)
legal_action_space_mask(ttt)
```

--------------------------------

### Implement Custom Reward Clipping Wrapper in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Shows how to create a custom environment wrapper in Julia that clips the reward. This wrapper inherits from `AbstractEnvWrapper` and overrides the `reward` function to clamp the reward value between -0.1 and 0.1. It requires the original environment to be stored in the `env` field.

```julia
struct ClipRewardWrapper{T} <: AbstractEnvWrapper
    env::T
end

RLBase.reward(env::ClipRewardWrapper) = clamp(reward(env.env), -0.1, 0.1)
```

--------------------------------

### Run NFSP Experiment in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

This snippet demonstrates how to execute a reinforcement learning experiment using the NFSP algorithm. It requires defining a stop condition and a hook for logging/monitoring. The `run` function orchestrates the training process.

```Julia
run(nfsp, wrapped_env, stop_condition, hook)
```

--------------------------------

### Evaluate Agent During Training using Nested Run in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Shows how to evaluate an agent's performance during training by running a nested experiment within the main training loop in Julia. This is achieved by using `DoEveryNSteps` to trigger a secondary `run` call with specific evaluation conditions and hooks.

```julia
run(
    agent,
    env,
    stop_condition,
    DoEveryNSteps(EVALUATION_FREQ) do t, agent, env
        run(agent, env, eval_stop_condition, eval_hook)
    end
)
```

--------------------------------

### Run Simulation with Random Policy in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/src/tutorial.md

Executes a simulation using a `RandomPolicy` on the `RandomWalk1D` environment for a specified number of episodes. It utilizes the `run` function and collects `TotalRewardPerEpisode`. This serves as a benchmark for policy performance.

```julia
run(
    RandomPolicy(),
    RandomWalk1D(),
    StopAfterNEpisodes(10),
    TotalRewardPerEpisode()
)
```

--------------------------------

### Define Custom Stop Condition Function in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/guide/index.md

Illustrates the basic structure for defining a custom stop condition function in Julia for reinforcement learning experiments. This function is executed after interacting with the environment and should return a boolean indicating whether to stop the experiment. It typically takes the agent and environment as arguments.

```julia
function hook(agent, env)::Bool
    # ...
end
```

--------------------------------

### Interact Policy with Environment for Multiple Steps

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/a_practical_introduction_to_RL.jl/index.html

Shows how to generate multiple actions from a policy given an environment. This is useful for observing the behavior of a policy over several timesteps within an episode.

```julia
[policy(env) for _ in 1:10]
```

--------------------------------

### Add ReinforcementLearningExperiments Package in Julia

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/index.html

This snippet demonstrates how to add the ReinforcementLearningExperiments package to your Julia environment. This package provides access to various built-in experiments for reinforcement learning.

```julia
]
add ReinforcementLearningExperiments
```

--------------------------------

### Define GridWorld Environment Interface for RLBase

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Implements the required RLBase interface methods for a GridWorld environment, including state and action spaces, state, reward, termination conditions, and reset/act functions. Also includes optional trait definitions for agent and state styles. This setup is crucial for integrating custom environments into the ReinforcementLearning.jl framework.

```julia
RLBase.state_space(env::GridWorld) = (Base.OneTo(env.size), Base.OneTo(env.size))
RLBase.action_space(env::GridWorld) = Base.OneTo(4)  # up, down, left, right
RLBase.state(env::GridWorld, ::Observation, ::DefaultPlayer) = [env.agent_pos...]
RLBase.reward(env::GridWorld) = env.reward
RLBase.is_terminated(env::GridWorld) = env.done

function RLBase.reset!(env::GridWorld)
    env.agent_pos = (1, 1)
    env.done = false
    env.reward = 0.0
    env.current_step = 0
    nothing
end

function RLBase.act!(env::GridWorld, action::Int)
    env.current_step += 1

    # Move: 1=up, 2=down, 3=left, 4=right
    x, y = env.agent_pos
    if action == 1 && y < env.size
        y += 1
    elseif action == 2 && y > 1
        y -= 1
    elseif action == 3 && x > 1
        x -= 1
    elseif action == 4 && x < env.size
        x += 1
    end
    env.agent_pos = (x, y)

    # Rewards
    if env.agent_pos == env.goal_pos
        env.reward = 10.0
        env.done = true
    elseif env.current_step >= env.max_steps
        env.reward = -1.0
        env.done = true
    else
        env.reward = -0.1
    end
    nothing
end

# Optional trait definitions
RLBase.NumAgentStyle(::GridWorld) = SINGLE_AGENT
RLBase.ActionStyle(::GridWorld) = MINIMAL_ACTION_SET
RLBase.StateStyle(::GridWorld) = Observation{Vector{Int}}()
RLBase.ChanceStyle(::GridWorld) = DETERMINISTIC
Random.seed!(env::GridWorld, seed) = Random.seed!(env.rng, seed)
```

--------------------------------

### Initialize Reinforcement Learning Datasets

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_final_term_report_210370741/index.md

This function initializes various RL datasets by calling their respective init functions. It's typically called when the project is first imported.

```julia
function __init__()
    RLDatasets.d4rl_init()
    RLDatasets.d4rl_pybullet_init()
    RLDatasets.atari_init()
    RLDatasets.rl_unplugged_atari_init()
end
```

--------------------------------

### Agent with Trajectory Storage and Experience Buffer

Source: https://context7.com/juliareinforcementlearning/reinforcementlearning.jl/llms.txt

Demonstrates creating a learning agent that utilizes an experience buffer for storing trajectories. It sets up an environment, an agent with a Q-based policy and a trajectory buffer, and then runs an experiment, accessing and printing the collected rewards and steps per episode.

```julia
using ReinforcementLearning
using ReinforcementLearningTrajectories

env = CartPoleEnv()

# Create agent with policy and trajectory buffer
agent = Agent(
    policy = QBasedPolicy(
        learner = TDLearner(
            TabularQApproximator(n_state=4, n_action=2),
            :SARS,
            γ=0.99,
            α=0.01
        ),
        explorer = EpsilonGreedyExplorer(0.1)
    ),
    trajectory = Trajectory(
        CircularArraySARTSTraces(
            capacity = 10_000,
            state = Int64 => (),
            action = Int64 => (),
            reward = Float64 => (),
            terminal = Bool => ()
        ),
        BatchSampler(32),
        InsertSampleRatioController(n_inserted = -1)
    )
)

# Run experiment
hook = TotalRewardPerEpisode() + StepsPerEpisode()
run(agent, env, StopAfterNEpisodes(500), hook)

# Access results
rewards = hook[1][]
steps = hook[2][]
println("Average reward: ", mean(rewards))
println("Average steps: ", mean(steps))
```

--------------------------------

### Integrate OpenSpiel Environments with ReinforcementLearning.jl

Source: https://github.com/juliareinforcementlearning/reinforcementlearning.jl/blob/main/docs/homepage/blog/ospp_report_210370190/index.md

This code configures an OpenSpiel environment for use with ReinforcementLearning.jl. It includes crucial action and state space transformations to ensure compatibility between OpenSpiel's 0-based indexing and Julia's 1-based indexing, especially for chance players. It also adapts the state style for training.

```Julia
env = OpenSpielEnv("kuhn_poker")
wrapped_env = ActionTransformedEnv(
    env,
    # action is `0-based` in OpenSpiel, while `1-based` in Julia.
    action_mapping = a -> RLBase.current_player(env) == chance_player(env) ? a : Int(a - 1),
    action_space_mapping = as -> RLBase.current_player(env) == chance_player(env) ? 
        as : Base.OneTo(num_distinct_actions(env.game)),
)
# `InformationSet{String}()` is not supported when trainning.
wrapped_env = DefaultStateStyleEnv{InformationSet{Array}()}(wrapped_env)
```