### Setup EscapeRoomba POMDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/gallery.md

Installs necessary packages and loads the RoombaPOMDPs module for the EscapeRoomba example. This is a setup step before running the main simulation.

```julia
using Pkg
Pkg.add(name="ParticleFilters", version="0.5.7")
Pkg.add(url="https://github.com/sisl/RoombaPOMDPs.git")
```

--------------------------------

### Conversion Examples

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/common_rl.md

Examples demonstrating the conversion of a POMDP to a CommonRLInterface environment and vice-versa using the `convert` function.

```APIDOC
## Converting POMDP to CommonRLInterface Environment

### Description
Converts a POMDP object to a CommonRLInterface environment.

### Method
`convert(AbstractEnv, pomdp_object)`

### Example
```julia
using POMDPs
using POMDPTools
using POMDPModels
using CommonRLInterface

env = convert(AbstractEnv, BabyPOMDP())
r = act!(env, true)
observe(env)
```

## Converting CommonRLInterface Environment to POMDP

### Description
Converts a CommonRLInterface environment to a POMDP object.

### Method
`convert(POMDP, rl_environment)`

### Example
```julia
using BasicPOMCP

m = convert(POMDP, env)
planner = solve(POMCPSolver(), m)
a = action(planner, initialstate(m))
```
```

--------------------------------

### Example Grid World Transitions

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Demonstrates the output of the `transition` function for various states and actions in the Grid World MDP.

```julia
@show transition(mdp, GridWorldState(1, 1), :up)
```

```julia
@show transition(mdp, GridWorldState(1, 1), :left)
```

```julia
@show transition(mdp, GridWorldState(9, 3), :right)
```

```julia
@show transition(mdp, GridWorldState(-1, -1), :down)
```

--------------------------------

### Example: GenerativeBeliefMDP Usage

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/model.md

An example demonstrating the creation and usage of `GenerativeBeliefMDP` with a `BabyPOMDP` and `DiscreteUpdater`, showing how to step through the belief MDP.

```jldoctest
using POMDPs
using POMDPModels
using POMDPTools

pomdp = BabyPOMDP()
updater = DiscreteUpdater(pomdp)

belief_mdp = GenerativeBeliefMDP(pomdp, updater)
@show statetype(belief_mdp) # POMDPModels.BoolDistribution

for (a, r, sp) in stepthrough(belief_mdp, RandomPolicy(belief_mdp), "a,r,sp", max_steps=5)
    @show a, r, sp
end
```

--------------------------------

### TagPOMDP Problem Setup and Solving

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/gallery.md

Installs the Plots package, defines the TagPOMDP, solves it with SARSOPSolver, and simulates a policy to generate a GIF. Requires POMDPs, POMDPTools, POMDPGifs, NativeSARSOP, Random, and TagPOMDPProblem.

```julia
using Pkg
Pkg.add("Plots")
using Plots
```

```julia
using POMDPs
using POMDPTools
using POMDPGifs
using NativeSARSOP
using Random
using TagPOMDPProblem

pomdp = TagPOMDP()
solver = SARSOPSolver(; max_time=20.0)
policy = solve(solver, pomdp)
sim = GifSimulator(; filename="examples/TagPOMDP.gif", max_steps=50, rng=MersenneTwister(1), show_progress=false)
saved_gif = simulate(sim, pomdp, policy)

println("gif saved to: $(saved_gif.filename)")
```

```julia
using Pkg
Pkg.rm("Plots")
```

--------------------------------

### Install POMDPs.jl and Solvers

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Install the POMDPs.jl package and common solver packages like QMDP from the Julia REPL.

```julia
using Pkg
Pkg.add("POMDPs")
Pkg.add("QMDP")
Pkg.add("POMDPTools")
Pkg.add("QuickPOMDPs")
```

--------------------------------

### Get Action from MCTS Planner

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Demonstrates how to use the MCTS planner to determine the action for a given state.

```julia
s = GridWorldState(9, 2)
@show action(mcts_planner, s)
```

```julia
s = GridWorldState(8, 3)
@show action(mcts_planner, s)
```

--------------------------------

### Install POMDPs.jl Package

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/install.md

Use this command to install the main POMDPs.jl package from the Julia REPL.

```julia
import Pkg
Pkg.add("POMDPs") # installs the POMDPs.jl package
```

--------------------------------

### Prepare for Parallel Simulations

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_simulations.md

Set up parameters for running multiple simulations in parallel, including the number of simulations, maximum steps, and starting seed for reproducibility.

```julia
using DataFrames 
using StatsBase: std

# Defining paramters for the simulations
number_of_sim_to_run = 100
max_steps = 20
starting_seed = 1

```

--------------------------------

### QuickMDP Example with Incorrect Transition

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/faq.md

Demonstrates a common error where the `transition` function returns a state directly instead of a distribution. This can lead to cryptic `MethodError` exceptions.

```julia
using POMDPs, QuickPOMDPs, POMDPTools

m = QuickMDP(
    states = [:a, :b],
    actions = [:left, :right],
    initialstate = Deterministic(:a),
    discount = 0.9,
    transition = (s, a) -> a == :right ? :b : :a,  # BUG: returns a state
    reward = (s, a) -> s == :b ? 1.0 : 0.0,
)

simulate(RolloutSimulator(max_steps=5), m, RandomPolicy(m))
```

--------------------------------

### QuickMountainCar Example with GIF Simulation

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/gallery.md

Defines the Mountain Car problem using QuickPOMDPs and simulates a policy to generate a GIF. Requires POMDPs, POMDPTools, POMDPGifs, Random, QuickPOMDPs, Compose, and Cairo.

```julia
using POMDPs
using POMDPTools
using POMDPGifs
using Random
using QuickPOMDPs
using Compose
import Cairo

mountaincar = QuickMDP(
    function (s, a, rng)        
        x, v = s
        vp = clamp(v + a*0.001 + cos(3*x)*-0.0025, -0.07, 0.07)
        xp = x + vp
        if xp > 0.5
            r = 100.0
        else
            r = -1.0
        end
        return (sp=(xp, vp), r=r)
    end,
    actions = [-1., 0., 1.],
    initialstate = Deterministic((-0.5, 0.0)),
    discount = 0.95,
    isterminal = s -> s[1] > 0.5,

    render = function (step) 
        cx = step.s[1]
        cy = 0.45*sin(3*cx)+0.5
        car = (context(), Compose.circle(cx, cy+0.035, 0.035), fill("blue"))
        track = (context(), line([(x, 0.45*sin(3*x)+0.5) for x in -1.2:0.01:0.6]), Compose.stroke("black"))
        goal = (context(), star(0.5, 1.0, -0.035, 5), fill("gold"), Compose.stroke("black"))
        bg = (context(), Compose.rectangle(), fill("white"))
        ctx = context(0.7, 0.05, 0.6, 0.9, mirror=Mirror(0, 0, 0.5))
        return compose(context(), (ctx, car, track, goal), bg)
    end
)

energize = FunctionPolicy(s->s[2] < 0.0 ? -1.0 : 1.0)
sim = GifSimulator(; filename="examples/QuickMountainCar.gif", max_steps=200, fps=20, rng=MersenneTwister(1), show_progress=false)
saved_gif = simulate(sim, mountaincar, energize)

println("gif saved to: $(saved_gif.filename)")
```

--------------------------------

### Example GridWorldMDP Rewards

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Shows how to call the reward function for different states and actions in the GridWorldMDP.

```julia
@show reward(mdp, GridWorldState(1, 1), :up)
```

```julia
@show reward(mdp, GridWorldState(1, 1), :left)
```

```julia
@show reward(mdp, GridWorldState(9, 3), :right)
```

```julia
@show reward(mdp, GridWorldState(-1, -1), :down)
```

```julia
@show reward(mdp, GridWorldState(2, 3), :up)
```

--------------------------------

### Example State, Action, Reward, and Next State

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/model.md

Demonstrates the output of a state transition, including the action taken, reward received, and the resulting belief state. This is useful for understanding the dynamics of a POMDP.

```julia
statetype(belief_mdp) = DiscreteBelief{POMDPModels.BabyPOMDP, Bool}Bool}
(a, r, sp) = (true, -5.0, DiscreteBelief{POMDPModels.BabyPOMDP, Bool}(POMDPModels.BabyPOMDP(-5.0, -10.0, 0.1, 0.8, 0.1, 0.9), Bool[0, 1], [1.0, 0.0]))
(a, r, sp) = (true, -5.0, DiscreteBelief{POMDPModels.BabyPOMDP, Bool}(POMDPModels.BabyPOMDP(-5.0, -10.0, 0.1, 0.8, 0.1, 0.9), Bool[0, 1], [1.0, 0.0]))
(a, r, sp) = (true, -5.0, DiscreteBelief{POMDPModels.BabyPOMDP, Bool}(POMDPModels.BabyPOMDP(-5.0, -10.0, 0.1, 0.8, 0.1, 0.9), Bool[0, 1], [1.0, 0.0]))
(a, r, sp) = (false, 0.0, DiscreteBelief{POMDPModels.BabyPOMDP, Bool}(POMDPModels.BabyPOMDP(-5.0, -10.0, 0.1, 0.8, 0.1, 0.9), Bool[0, 1], [0.9759036144578314, 0.02409638554216867]))
(a, r, sp) = (false, 0.0, DiscreteBelief{POMDPModels.BabyPOMDP, Bool}(POMDPModels.BabyPOMDP(-5.0, -10.0, 0.1, 0.8, 0.1, 0.9), Bool[0, 1], [0.9701315984030756, 0.029868401596924433]))
```

--------------------------------

### Add and Update POMDPs.jl Dependencies

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Installs or updates necessary Julia packages for POMDPs.jl examples.

```julia
using Pkg

Pkg.add("POMDPs")
Pkg.add("POMDPTools")
Pkg.add("DiscreteValueIteration")
Pkg.add("MCTS")
```

```julia
Pkg.update()
```

--------------------------------

### Define Tiger POMDP using QuickPOMDPs

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Defines a Tiger POMDP concisely using keyword arguments with QuickPOMDPs. This is recommended for quick setup. It also shows how to solve and simulate the POMDP.

```julia
using QuickPOMDPs, POMDPTools, QMDP

# Classic Tiger POMDP
m = QuickPOMDP(
    states       = ["left", "right"],
    actions      = ["left", "right", "listen"],
    observations = ["left", "right"],
    initialstate = Uniform(["left", "right"]),
    discount     = 0.95,

    transition = function (s, a)
        a == "listen" ? Deterministic(s) : Uniform(["left", "right"])
    end,

    observation = function (a, sp)
        if a == "listen"
            sp == "left" ? SparseCat(["left","right"], [0.85, 0.15]) :
                           SparseCat(["right","left"], [0.85, 0.15])
        else
            Uniform(["left", "right"])
        end
    end,

    reward = function (s, a)
        a == "listen" ? -1.0 : (s == a ? -100.0 : 10.0)
    end
)

solver = QMDPSolver()
policy = solve(solver, m)

rsum = 0.0
for (s, b, a, o, r) in stepthrough(m, policy, "s,b,a,o,r", max_steps=10)
    println("s: $s, a: $a, o: $o, r: $r")
    global rsum += r
end
println("Total undiscounted reward: $rsum")
```

--------------------------------

### Install POMDPs.jl and QMDP Solver

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/README.md

Use Julia's package manager to add POMDPs.jl and the QMDP solver package to your environment.

```julia
using Pkg; Pkg.add("POMDPs"); Pkg.add("QMDP")
```

--------------------------------

### Get Action from Value Iteration Policy

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Demonstrates how to use the obtained Value Iteration policy to determine the optimal action for a given state.

```julia
s = GridWorldState(9, 2)
@show action(vi_policy, s)
```

```julia
s = GridWorldState(8, 3)
@show action(vi_policy, s)
```

--------------------------------

### Online Planning with BasicPOMCP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_solvers.md

Perform online planning using the BasicPOMCP solver. Define the solver with parameters, call POMDPs.solve to get an online planner, and then query the planner using the action function, which triggers action computation.

```julia
using BasicPOMCP

pomcp_solver = POMCPSolver(; c=5.0, tree_queries=1000, rng=MersenneTwister(1))
pomcp_planner = POMDPs.solve(pomcp_solver, quick_crying_baby_pomdp)

b = initialstate(quick_crying_baby_pomdp)
a = action(pomcp_planner, b)

@show a
```

--------------------------------

### Saving a Policy Object with JLD2

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/faq.md

Provides an example of how to save a trained policy object to a file using the JLD2.jl library for later use.

```julia
using JLD2
save("my_policy.jld2", "policy", policy)
```

--------------------------------

### State-Dependent Actions in MDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

Implement state-dependent action spaces by providing a function to the `actions` argument. This example shows how to return a subset of actions for a specific state.

```jldoctest
actions = function (s = nothing) 
    if s == 1
        return [1]      #<--- return state-dep-actions
    else
        return [1,2,3]  #<--- return full action space here
    end
end
```

--------------------------------

### Verify Greedy Policy for MDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/offline_solver.md

Verify the greedy policy generated by the solver on an example from POMDPModels. This checks if the policy produces the expected greedy action for a given state.

```jldoctest offline
using POMDPModels

gw = SimpleGridWorld(size=(2,1), rewards=Dict(GWPos(2,1)=>1.0))
policy = solve(GreedyOfflineSolver(), gw)

action(policy, GWPos(1,1))
```

--------------------------------

### Initialize Belief for HistoryUpdater

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_updater.md

Implement `initialize_belief` for `HistoryUpdater`. It takes the initial state distribution `d` and returns it as the first element in a new `Any` vector, representing the start of the belief history.

```julia
POMDPs.initialize_belief(up::HistoryUpdater, d) = Any[d]
```

--------------------------------

### Generic Solver Using Type Inference Utilities

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

An example of how to use statetype, actiontype, and obstype to write generic solver code that can work with different POMDP models without needing to know their specific type parameters.

```julia
# Useful in generic solver code:
function generic_solve(m)
    S = statetype(m)
    A = actiontype(m)
    value_table = Dict{Tuple{S,A}, Float64}()
    return value_table
end
```

--------------------------------

### Add JuliaPOMDP Registry

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/install.md

Run this command to add the JuliaPOMDP registry, which contains auxiliary packages and older solvers. Ensure you have a Julia distribution (0.4 or greater) installed.

```julia
using Pkg
pkg"registry add https://github.com/JuliaPOMDP/Registry"
```

--------------------------------

### Solve and Evaluate Tiger POMDP with QMDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/get_started.md

This snippet solves the Tiger POMDP using QMDP, computes a policy, and evaluates its performance through a simulated history. Ensure QMDP, POMDPModels, and POMDPTools are installed.

```julia
using POMDPs, QMDP, POMDPModels, POMDPTools

# initialize problem and solver
pomdp = TigerPOMDP() # from POMDPModels
solver = QMDPSolver() # from QMDP

# compute a policy
policy = solve(solver, pomdp)

#evaluate the policy
belief_updater = updater(policy) # the default QMDP belief updater (discrete Bayesian filter)
init_dist = initialstate(pomdp) # from POMDPModels
hr = HistoryRecorder(max_steps=100) # from POMDPTools
hist = simulate(hr, pomdp, policy, belief_updater, init_dist) # run 100 step simulation
println("reward: $(discounted_reward(hist))")
```

--------------------------------

### Enumerate state, action, and observation spaces

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Shows how to collect and print the full state, action, and observation spaces for a QuickPOMDP. Also demonstrates state-dependent action spaces.

```julia
using POMDPs, POMDPTools, QuickPOMDPs

m = QuickPOMDP(
    states       = [:s1, :s2, :s3],
    actions      = [:a1, :a2],
    observations = [:o1, :o2],
    discount     = 0.95,
    initialstate = Uniform([:s1, :s2, :s3]),
    transition   = (s, a) -> Uniform([:s1, :s2, :s3]),
    observation  = (a, sp) -> Uniform([:o1, :o2]),
    reward       = (s, a) -> s == :s3 ? 1.0 : 0.0
)

println(collect(states(m)))        # [:s1, :s2, :s3]
println(collect(actions(m)))       # [:a1, :a2]
println(collect(observations(m)))  # [:o1, :o2]

# State-dependent action space
m2 = QuickPOMDP(
    states   = 1:4,
    actions  = s -> s == 1 ? [1] : [1, 2, 3],
    discount = 0.9,
    initialstate = Uniform(1:4),
    transition   = (s, a) -> Uniform(1:4),
    reward       = (s, a) -> 0.0
)
println(collect(actions(m2, 1)))   # [1]
println(collect(actions(m2, 3)))   # [1, 2, 3]
```

--------------------------------

### Initialize and use DisplaySimulator

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

Instantiate `DisplaySimulator` and use it with a POMDP model and policy to simulate and visualize steps in real-time. Requires POMDPs, POMDPModels, POMDPTools, and ElectronDisplay.

```julia
using POMDPs
using POMDPModels
using POMDPTools
using ElectronDisplay
ElectronDisplay.CONFIG.single_window = true

ds = DisplaySimulator()
m = SimpleGridWorld()
simulate(ds, m, RandomPolicy(m))
```

--------------------------------

### GridWorldMDP State Space Helper Functions

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Defines helper functions for the GridWorldMDP state space: isterminal to check for the terminal state, initialstate for the starting state distribution, and stateindex to get the index of a given state.

```julia
# Check if a state is the terminal state
POMDPs.isterminal(mdp::GridWorldMDP, s::GridWorldState) = s == GridWorldState(-1, -1)

# Define the initial state distribution (always start in the bottom left)
POMDPs.initialstate(mdp::GridWorldMDP) = Deterministic(GridWorldState(1, 1))

# Function that returns the index of a state in the state space
function POMDPs.stateindex(mdp::GridWorldMDP, s::GridWorldState)
    if isterminal(mdp, s)
        return length(states(mdp))
    end
    
    @assert 1 <= s.x <= mdp.size_x "Invalid state"
    @assert 1 <= s.y <= mdp.size_y "Invalid state"
    
    si = (s.x - 1) * mdp.size_y + s.y
    return si
end
```

--------------------------------

### Fast Single-Trajectory Rollout with RolloutSimulator

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Runs a single episode and returns the total discounted reward. Can be configured with max_steps and optionally take an explicit initial belief. Requires POMDPs, POMDPTools, POMDPModels, and optionally QMDP.

```julia
using POMDPs, POMDPTools, POMDPModels, QMDP

pomdp  = TigerPOMDP()
policy = solve(QMDPSolver(), pomdp)

rs = RolloutSimulator(max_steps=50)
r  = simulate(rs, pomdp, policy)
println("Discounted reward: $r")

# With explicit initial state / belief
up = DiscreteUpdater(pomdp)
b0 = initialize_belief(up, initialstate(pomdp))
r2 = simulate(rs, pomdp, policy, up, b0)
println("Discounted reward (explicit): $r2")
```

--------------------------------

### Run fast rollout simulations with RolloutSimulator

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

Use `RolloutSimulator` for simple, fast simulations of MDP or POMDP environments. It simulates a single trajectory and returns the total discounted reward.

```julia
rs = RolloutSimulator()
mdp = GridWorld()
policy = RandomPolicy(mdp)

r = simulate(rs, mdp, policy)
```

--------------------------------

### Cleanup EscapeRoomba Dependencies

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/gallery.md

Removes the ParticleFilters and RoombaPOMDPs packages after the EscapeRoomba example has been run. This is a cleanup step to revert package changes.

```julia
Pkg.rm("ParticleFilters")
Pkg.rm("RoombaPOMDPs")
```

--------------------------------

### Sample from generative model using @gen macro

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Demonstrates sampling next state, observation, and reward from a QuickPOMDP using the @gen macro. Ensure QuickPOMDP is imported.

```julia
using POMDPs, POMDPTools, QuickPOMDPs

m = QuickPOMDP(
    states       = 1:3,
    actions      = 1:2,
    observations = 1:3,
    discount     = 0.9,
    initialstate = Uniform(1:3),
    transition   = (s, a) -> Uniform(1:3),
    observation  = (a, sp) -> Deterministic(sp),
    reward       = (s, a) -> Float64(s),
    isterminal   = s -> false
)

s = 2
a = 1
# Sample next state and reward only
sp, r = @gen(:sp, :r)(m, s, a)
println("sp=$sp, r=$r")

# Sample next state, observation, and reward
sp, o, r = @gen(:sp, :o, :r)(m, s, a)
println("sp=$sp, o=$o, r=$r")

# Sample only next state
sp = @gen(:sp)(m, s, a)
println("sp=$sp")
```

--------------------------------

### MountainCar Terminal State Example

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

This function defines terminal states where the position, s[1], is greater than 0.5. The value of a terminal state is always zero.

```julia
isterminal = s -> s[1] > 0.5
```

--------------------------------

### Schedule for Exploration Policies

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/policies.md

Defines decay schedules for exploration parameters like epsilon or temperature. An example shows an exponential decay schedule for an EpsGreedyPolicy.

```julia
```julia 
    m # your mdp or pomdp model
    exploration_policy = EpsGreedyPolicy(m, k->0.05*0.9^(k/10))
```
```

```julia
```@docs
    LinearDecaySchedule
```
```

--------------------------------

### Run Rollout Simulation

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_simulations.md

Use RolloutSimulator for a standard simulation run. It returns the total discounted reward for the simulation. Configure with max_steps.

```julia
function run_rollout_simulation() # hide
policy = RandomPolicy(explicit_crying_baby_pomdp)
sim = RolloutSimulator(max_steps=10)
r_sum = simulate(sim, explicit_crying_baby_pomdp, policy)
println("Total discounted reward: $r_sum")
end # hide
run_rollout_simulation() # hide
```

--------------------------------

### Ordered States, Actions, and Observations

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/model.md

Provides functions to get ordered lists of states, actions, and observations, ensuring consistency with the `index` function from POMDPs.jl.

```julia
using POMDPs
using POMDPTools

@doc ordered_states
@doc ordered_actions
@doc ordered_observations
```

--------------------------------

### Policy and Solver Functions

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/api.md

Functions related to solving POMDPs and interacting with policies, including solving the POMDP, getting the policy and updater, and taking actions.

```APIDOC
## solve

### Description
Solves the POMDP and returns a policy.

### Method
`solve(solver, pomdp)`

### Parameters
- **solver** (Solver) - The solver.
- **pomdp** (POMDP) - The POMDP model.

### Returns
- The solved policy.

## updater

### Description
Returns the belief updater associated with a policy.

### Method
`updater(policy)`

### Parameters
- **policy** (Policy) - The policy.

### Returns
- The belief updater.

## action

### Description
Returns the action to take given the current belief.

### Method
`action(policy, belief)`

### Parameters
- **policy** (Policy) - The policy.
- **belief** (Belief) - The current belief.

### Returns
- The recommended action.

## value

### Description
Returns the value of a belief state.

### Method
`value(policy, belief)`

### Parameters
- **policy** (Policy) - The policy.
- **belief** (Belief) - The current belief.

### Returns
- The value of the belief state.
```

--------------------------------

### Solve GridWorld MDP with MCTS

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Initializes the GridWorldMDP and an MCTSSolver, then solves the MDP to obtain a planner.

```julia
# Initialize the problem (we have already done this, but just calling it again for completeness in the example)
mdp = GridWorldMDP()

# Initialize the solver with desired parameters 
solver = MCTSSolver(n_iterations=1000, depth=20, exploration_constant=10.0)

# Now we construct a planner by calling POMDPs.solve. For online planners, the computation for the
# optimal action occurs in the call to `action`.
mcts_planner = POMDPs.solve(solver, mdp)
nothing # hide
```

--------------------------------

### Create and Populate Simulation List

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_simulations.md

This code initializes a list of simulation objects for different policies (SARSOP, POMCP, heuristic, random) with specified seeds and parameters. Each policy is added multiple times with unique seeds for robust analysis.

```julia
rand_policy = RandomPolicy(quick_crying_baby_pomdp, rng=MersenneTwister(1))

# Create the list of Sim objects
sim_list = []

# Add 100 Sim objects of each policy to the list.
for sim_number in 1:number_of_sim_to_run
    seed = starting_seed + sim_number

    # Add the SARSOP policy
    push!(sim_list, Sim(
        quick_crying_baby_pomdp, 
        rng=MersenneTwister(seed),
        sarsop_policy,
        max_steps=max_steps,
        metadata=Dict(:policy => "sarsop", :seed => seed))
    )

    # Add the POMCP policy
    push!(sim_list, Sim(
        quick_crying_baby_pomdp, 
        rng=MersenneTwister(seed),
        pomcp_planner, 
        max_steps=max_steps,
        metadata=Dict(:policy => "pomcp", :seed => seed))
    )

    # Add the heuristic policy
    push!(sim_list, Sim(
        quick_crying_baby_pomdp, 
        rng=MersenneTwister(seed),
        heuristic_policy, 
        max_steps=max_steps,
        metadata=Dict(:policy => "heuristic", :seed => seed))
    )

    # Add the random policy
    push!(sim_list, Sim(
        quick_crying_baby_pomdp, 
        rng=MersenneTwister(seed),
        rand_policy, 
        max_steps=max_steps,
        metadata=Dict(:policy => "random", :seed => seed))
    )
end
```

--------------------------------

### RolloutSimulator

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

The `RolloutSimulator` is designed for fast rollout simulations. When `simulate` is called, it generates a single trajectory and returns the total discounted reward.

```APIDOC
## RolloutSimulator

### Description
Simulates a single trajectory of an MDP or POMDP and returns the discounted reward. Ideal for fast rollout evaluations.

### Constructor
```julia
RolloutSimulator()
```

### Usage Example
```julia
rs = RolloutSimulator()
mdp = GridWorld()
policy = RandomPolicy(mdp)

r = simulate(rs, mdp, policy)
```
```

--------------------------------

### State-Dependent Actions in POMDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

Define state-dependent action spaces for POMDPs where allowable actions depend on the belief. This example restricts actions based on the probability of a specific state.

```jldoctest
actions = function (b) 
    if pdf(b, 1) > 0.0
        return [1,2,3]
    else
        return [2,3]
    end
end
```

--------------------------------

### Environment Wrapper Types

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/common_rl.md

Explains the different types of (PO)MDP wrappers used when converting from CommonRLInterface environments.

```APIDOC
## Generative Model Wrappers

### Description
Wraps an RL environment in a `RLEnvMDP` or `RLEnvPOMDP` if `state` and `setstate!` functions are provided. This makes the POMDPs.jl generative model interface available.

## Opaque Wrappers

### Description
Wraps an RL environment in `OpaqueRLEnvPOMDP` or `OpaqueRLEnvMDP` if `state` and `setstate!` are not provided. The resulting (PO)MDP can only be simulated, and the state is an `OpaqueRLEnvState` tracking the environment's age.
```

--------------------------------

### DisplaySimulator

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

The `DisplaySimulator` visualizes each step of a simulation in real-time using multimedia displays like Jupyter notebooks or ElectronDisplay. It leverages `POMDPTools.render` and Julia's `display` function.

```APIDOC
## `DisplaySimulator`

### Description
A simulator that displays each step of a simulation in real time.

### Method
`DisplaySimulator(; predisplay=nothing, kwargs...)`

### Parameters
- `predisplay`: A function to call before displaying each step, useful for clearing output in environments like Jupyter notebooks.

### Example
```julia
using POMDPTools
using POMDPModels

ds = DisplaySimulator()
m = SimpleGridWorld()
simulate(ds, m, RandomPolicy(m))
```
```

--------------------------------

### Analyze simulation results from DataFrame

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

After running simulations with `run_parallel`, the results are stored in a DataFrame. This example shows how to compute the average reward per step from the collected statistics.

```julia
mean(df[:reward]./df[:n_steps])
```

--------------------------------

### transition(m, s, a)

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Returns a distribution over next states given the current state `s` and action `a`. This function must be implemented by all (PO)MDP problem definitions.

```APIDOC
### `transition(m, s, a)` — state transition distribution

Returns a distribution over next states given the current state `s` and action `a`. Must be implemented by all (PO)MDP problem definitions.

```julia
using POMDPs, POMDPTools

struct GridMDP <: MDP{Int, Symbol}
    size::Int
end

POMDPs.states(m::GridMDP)   = 1:m.size
POMDPs.actions(m::GridMDP)  = [:left, :right]
POMDPs.discount(m::GridMDP) = 0.95
POMDPs.isterminal(m::GridMDP, s) = s == m.size

function POMDPs.transition(m::GridMDP, s, a)
    if a == :right
        s < m.size ? Deterministic(s + 1) : Deterministic(s)
    else
        s > 1 ? Deterministic(s - 1) : Deterministic(s)
    end
end

POMDPs.reward(m::GridMDP, s, a) = s == m.size ? 1.0 : -0.01
POMDPs.initialstate(m::GridMDP) = Uniform(1:m.size)
POMDPs.stateindex(m::GridMDP, s) = s
POMDPs.actionindex(m::GridMDP, a) = a == :left ? 1 : 2

m = GridMDP(5)
d = transition(m, 3, :right)
println(rand(d))   # 4
println(pdf(d, 4)) # 1.0
println(pdf(d, 3)) # 0.0
```
```

--------------------------------

### Verify Solver with SimpleGridWorld

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/online_solver.md

Verifies the Monte Carlo greedy solver by creating a SimpleGridWorld instance and using the planner to determine an action from a given state.

```jldoctest
using POMDPModels

gw = SimpleGridWorld(size=(2,1), rewards=Dict(GWPos(2,1)=>1.0))
solver = MonteCarloGreedySolver(1000)
planner = solve(solver, gw)

action(planner, GWPos(1,1))

# output

:right

```

--------------------------------

### Interactive Environment Interface with sim()

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Provides a flexible way to interact with a (PO)MDP environment using Julia's do block syntax. The block receives observations (POMDP) or states (MDP) and must return an action. Requires POMDPs, POMDPTools, and POMDPModels.

```julia
using POMDPs, POMDPTools, POMDPModels

pomdp = TigerPOMDP()

# Interact via observations
history = sim(pomdp, max_steps=10) do obs
    println("Heard: $obs")
    return TIGER_LISTEN   # always listen
end
println("Reward: ", discounted_reward(history))

# With a belief updater - block receives beliefs instead of raw observations
up = DiscreteUpdater(pomdp)
history2 = sim(pomdp, up, max_steps=10) do b
    # b is a DiscreteBelief
    p_left = pdf(b, TIGER_LEFT)
    p_left > 0.9 ? TIGER_OPEN_RIGHT : TIGER_LISTEN
end
println("Reward: ", discounted_reward(history2))
```

--------------------------------

### RockSample Problem with SARSOPSolver and GIF Simulation

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/gallery.md

Defines the RockSample POMDP and solves it using SARSOPSolver, then simulates a policy to create a GIF. Requires POMDPs, POMDPTools, POMDPGifs, NativeSARSOP, Random, RockSample, and Cairo.

```julia
using POMDPs
using POMDPTools
using POMDPGifs
using NativeSARSOP
using Random
using RockSample 
using Cairo 

pomdp = RockSamplePOMDP(rocks_positions=[
    (2,3), (4,4), (4,2)
], 
                        sensor_efficiency=20.0,
                        discount_factor=0.95, 
                        good_rock_reward = 20.0)

solver = SARSOPSolver(precision=1e-3; max_time=10.0)
policy = solve(solver, pomdp)

sim = GifSimulator(; filename="examples/RockSample.gif", max_steps=30, rng=MersenneTwister(1), show_progress=false)
saved_gif = simulate(sim, pomdp, policy)

println("gif saved to: $(saved_gif.filename)")
```

--------------------------------

### Sparse Categorical Observation Distribution

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

Represent discrete observation spaces with specified probabilities using `SparseCat`. This example shows how to define an observation distribution for a discrete POMDP, common in grid-world problems.

```julia
SparseCat([:left, :right], [0.85, 0.15])
```

--------------------------------

### QuickPOMDP Implementation of Tiger POMDP

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

This snippet shows a complete QuickPOMDP implementation for the Tiger problem, defining states, actions, observations, discount factor, transition, observation, reward, and initial state distributions.

```julia
using QuickPOMDPs: QuickPOMDP
using POMDPTools: Deterministic, Uniform, SparseCat

m = QuickPOMDP(
    states = ["left", "right"],
    actions = ["left", "right", "listen"],
    observations = ["left", "right"],
    discount = 0.95,

    transition = function (s, a)
        if a == "listen"
            return Deterministic(s) # tiger stays behind the same door
        else # a door is opened
            return Uniform(["left", "right"])
        end
    end,

    observation = function (a, sp)
        if a == "listen"
            if sp == "left"
                return SparseCat(["left", "right"], [0.85, 0.15]) # sparse categorical
            else
                return SparseCat(["right", "left"], [0.85, 0.15])
            end
        else
            return Uniform(["left", "right"])
        end
    end,

    reward = function (s, a)
        if a == "listen"
            return -1.0
        elseif s == a # the tiger was found
            return -100.0
        else # the tiger was escaped
            return 10.0
        end
    end,

    initialstate = Uniform(["left", "right"]),
);
```

--------------------------------

### Query policy and value function

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Demonstrates querying a policy for an action given a belief state and calculating the value of a belief state. Requires QMDPSolver and POMDPModels.

```julia
using POMDPs, POMDPTools, POMDPModels, QMDP

tiger  = TigerPOMDP()
solver = QMDPSolver()
policy = solve(solver, tiger)

# Query with a belief (DiscreteBelief)
up = DiscreteUpdater(tiger)
b0 = initialize_belief(up, initialstate(tiger))
a  = action(policy, b0)
println("Initial action: $a")   # typically "listen"

# Update belief and query again
b1 = update(up, b0, "listen", "left")
a1 = action(policy, b1)
println("After hearing left: $a1")

# Query value
v = value(policy, b0)
println("Value at initial belief: $v")
```

--------------------------------

### Define Tiger POMDP with TabularPOMDP

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Define a POMDP using numeric arrays for transition (T), reward (R), and observation (O) probabilities. States, actions, and observations are represented by integers. This example sets up the Tiger problem.

```julia
using POMDPModels: TabularPOMDP

# Tiger POMDP: states/actions/obs mapped to integers 1=left, 2=right, 3=listen
T = zeros(2, 3, 2)
T[:,:,1] = [1.  0.5  0.5;  0.  0.5  0.5]
T[:,:,2] = [0.  0.5  0.5;  1.  0.5  0.5]

O = zeros(2, 3, 2)
O[:,:,1] = [0.85  0.5  0.5;  0.15  0.5  0.5]
O[:,:,2] = [0.15  0.5  0.5;  0.85  0.5  0.5]

R = [-1.  -100.  10.;
     -1.   10.  -100.]

m = TabularPOMDP(T, R, O, 0.95)
println(discount(m))         # 0.95
println(states(m))           # 1:2
println(transition(m, 1, 1)) # distribution over next states
```

--------------------------------

### Configure DisplaySimulator for Jupyter notebooks

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/POMDPTools/simulators.md

To achieve an animated visualization in Jupyter notebooks where the image is overwritten at each step, initialize `DisplaySimulator` with a `predisplay` function that clears the output.

```julia
DisplaySimulator(predisplay=(d)->IJulia.clear_output(true))
```

--------------------------------

### Solve GridWorld MDP with Value Iteration

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/example_gridworld_mdp.md

Initializes the GridWorldMDP and a ValueIterationSolver, then solves the MDP to obtain an optimal policy.

```julia
# Initialize the problem (we have already done this, but just calling it again for completeness in the example)
mdp = GridWorldMDP()

# Initialize the solver with desired parameters
solver = ValueIterationSolver(; max_iterations=100, belres=1e-3, verbose=true)

# Solve for an optimal policy
vi_policy = POMDPs.solve(solver, mdp)
nothing # hide
```

--------------------------------

### MountainCar Reward Function Example

Source: https://github.com/juliapomdp/pomdps.jl/blob/master/docs/src/def_pomdp.md

This reward function returns a large positive reward if the resulting position is beyond 0.5, and a small negative reward otherwise. Implement the version with the fewest arguments possible.

```julia
reward = function (s, a, sp)
    if sp[1] > 0.5
        return 100.0
    else
        return -1.0
    end
end
```

--------------------------------

### Implement minimal reward function for MyMDP

Source: https://context7.com/juliapomdp/pomdps.jl/llms.txt

Implement the minimal `reward` function signature `reward(m, s, a)` for an MDP. POMDPs.jl automatically derives more complex signatures if needed. This example defines a simple reward based on the state.

```julia
using POMDPs

struct MyMDP <: MDP{Int, Int} end

# Minimal signature; POMDPs.jl auto-provides reward(m,s,a,sp) and reward(m,s,a,sp,o)
POMDPs.reward(m::MyMDP, s::Int, a::Int) = s == 5 ? 10.0 : -1.0

m = MyMDP()
println(reward(m, 5, 1))        # 10.0
println(reward(m, 3, 2))        # -1.0
println(reward(m, 3, 2, 4))     # -1.0  (auto-derived)
```