### Game Simulation and Plotting Example

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/game.md

This example demonstrates how to initialize and run a game simulation using the Game class and a GreedyPolicy. It then calls a plotting function to visualize the average reward per step.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.game import Game

n_bandits = 10

policy = GreedyPolicy(optimistic_initialization=1, n_bandits=n_bandits)

game = Game(n_episodes=2000,
         n_steps=1000,
         policies=[policy],
         n_bandits=n_bandits)

game.game_loop()

game.plot_average_reward_by_step()
```

--------------------------------

### Setup and Run Game Simulation

Source: https://github.com/danielalopes/pymab/blob/main/examples/bayesian_ucb_bernoulli.ipynb

Initializes the Game object with simulation parameters and the defined policies, then executes the game loop.

```python
# Setup the game
game = Game(n_episodes=2000, 
            n_steps=1000, 
            Q_values=Q_values,
            policies=[
                    ucb_policy_0,
                    ucb_policy_1,
                    ucb_policy_2,
                    bayesian_ucb
                ], 
            n_bandits=n_bandits,
            )

# Run the game
game.game_loop()
```

--------------------------------

### Setup the Game Environment

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Configures the multi-armed bandit game with the number of episodes, steps, and the defined policies and bandit parameters.

```python
# Setup the game
game = Game(n_episodes=100, 
            n_steps=1000, 
            policies=[
                    greedy,
                    thomson_sampling
                ], 
            n_bandits=n_bandits,
            Q_values=Q_values,
            )
```

--------------------------------

### Setup Stationary Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Configures a standard game environment with no changes in reward distributions. This serves as a baseline to compare the performance of policies in non-stationary settings.

```python
stationary_game = Game(
    n_episodes=n_episodes,
    n_steps=n_steps,
    policies=policies,
    n_bandits=n_bandits,
)
```

--------------------------------

### Get Q-values

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Retrieves the estimated Q-values for all actions.

```APIDOC
## Q_values

### Description
Get the true Q-values for all actions.

### Returns
List of true Q-values.

### Raises
**ValueError** – If Q-values haven’t been set.

### Return type
*List[float]*
```

--------------------------------

### UCB Policy Action Selection Example

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Demonstrates how to initialize and use the UCB policy for action selection over multiple steps. This is useful for exploring different actions and gathering reward data in a multi-armed bandit problem.

```python
policy = UCBPolicy(n_bandits=3)
for _ in range(100):
    action, reward = policy.select_action()
    # Use the action and reward as needed
```

--------------------------------

### Setup Gradual Non-Stationary Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Configures a game environment with gradual changes in reward distributions. The `change_rate` parameter controls the magnitude of these small, step-by-step changes.

```python
non_stationary_gradual_game = Game(
    n_episodes=n_episodes,
    n_steps=n_steps,
    policies=policies,
    n_bandits=n_bandits,
    environment_change=EnvironmentChangeType.GRADUAL,
    change_params={'change_rate': 0.01}
)
```

--------------------------------

### Plot Initial Bandit Distributions

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Assigns the true Q-values to each policy and plots their initial reward distributions before the game starts.

```python
for policy in game.policies:
    policy.Q_values = game.Q_values
    policy.plot_distribution()
```

--------------------------------

### Setup Abrupt Non-Stationary Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Configures a game environment with abrupt changes in reward distributions. `change_frequency` and `change_magnitude` parameters define how often and how significantly the rewards change.

```python
non_stationary_abrupt_game = Game(
    n_episodes=n_episodes,
    n_steps=n_steps,
    policies=policies,
    n_bandits=n_bandits,
    environment_change=EnvironmentChangeType.ABRUPT,
    change_params={'change_frequency': 100, 'change_magnitude': 0.5},
)
```

--------------------------------

### Q_values property

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Gets the true Q-values for all actions. This property returns a list of the estimated Q-values for each action.

```APIDOC
## Q_values

### Description
Get the true Q-values for all actions.

### Returns
- List of true Q-values.

### Raises
- **ValueError** – If Q-values haven’t been set.
```

--------------------------------

### RewardDistribution.generate_Q_values

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Abstract method to get a set of Q values. This is an abstract method that must be implemented by subclasses.

```APIDOC
## RewardDistribution.generate_Q_values

### Description
Abstract method to get a set of Q values. This method defines the interface for generating multiple Q-values.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean or central value of the distribution.
- **variance** (*float*) – The variance or spread of the distribution.
- **size** (*int*) – The number of Q values to generate.

### Returns
A list of Q-values sampled from the distribution.

### Return type
np.ndarray[*float*]
```

--------------------------------

### Setup Random Arm Swapping Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Configures a game environment where arm reward distributions are swapped randomly at certain steps. The `shift_probability` controls the likelihood of such a swap occurring.

```python
non_stationary_swapping_game = Game(
    n_episodes=n_episodes,
    n_steps=n_steps,
    policies=policies,
    n_bandits=n_bandits,
    environment_change=EnvironmentChangeType.RANDOM_ARM_SWAPPING,
    change_params={'shift_probability': 0.2},
)
```

--------------------------------

### UniformRewardDistribution.generate_Q_values

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a set of Q values sampled from a Uniform distribution. This method implements the abstract generate_Q_values method for Uniform distributions.

```APIDOC
## UniformRewardDistribution.generate_Q_values

### Description
Get a set of Q values sampled from a Uniform distribution. This method implements the abstract generate_Q_values method for Uniform distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean or central value of the distribution.
- **variance** (*float*) – The variance or spread of the distribution.
- **size** (*int*) – The number of Q values to generate.

### Returns
A list of Q-values sampled from the distribution.

### Return type
np.ndarray[*float*]
```

--------------------------------

### GaussianRewardDistribution.generate_Q_values

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a set of Q values sampled from a Gaussian distribution. This method implements the abstract generate_Q_values method for Gaussian distributions.

```APIDOC
## GaussianRewardDistribution.generate_Q_values

### Description
Get a set of Q values sampled from a Gaussian distribution. This method implements the abstract generate_Q_values method for Gaussian distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean or central value of the distribution.
- **variance** (*float*) – The variance or spread of the distribution.
- **size** (*int*) – The number of Q values to generate.

### Returns
A list of Q-values sampled from the distribution.

### Return type
np.ndarray[*float*]
```

--------------------------------

### BernoulliRewardDistribution.generate_Q_values

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a set of Q values sampled from a Bernoulli distribution. This method implements the abstract generate_Q_values method for Bernoulli distributions.

```APIDOC
## BernoulliRewardDistribution.generate_Q_values

### Description
Get a set of Q values sampled from a Bernoulli distribution. This method implements the abstract generate_Q_values method for Bernoulli distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean or central value of the distribution.
- **variance** (*float*) – The variance or spread of the distribution.
- **size** (*int*) – The number of Q values to generate.

### Returns
A list of Q-values sampled from the distribution.

### Return type
np.ndarray[*float*]
```

--------------------------------

### Get Reward Distribution Class

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Retrieves the reward distribution class based on its name. Supports 'gaussian', 'bernoulli', and 'uniform' distributions. Raises ValueError for unknown names.

```python
gaussian_dist = UCBPolicy.get_reward_distribution('gaussian')
```

--------------------------------

### Policy Initialization and Game Simulation

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Shows how to instantiate various policy types (Greedy, Epsilon-Greedy, Bayesian UCB, Thompson Sampling) and set up a game simulation with these policies to compare their performance.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.policies.epsilon_greedy import EpsilonGreedyPolicy
from pymab.policies.bayesian_ucb import BayesianUCBPolicy

from pymab.policies.thompson_sampling import ThompsonSamplingPolicy
from pymab.game import Game

n_bandits = 10

greedy_policy = GreedyPolicy(n_bandits=n_bandits,
                           optimistic_initialization=0)

greedy_policy_optimistic_initialization_1 = GreedyPolicy(n_bandits=n_bandits,
                                                      optimistic_initialization=1)

greedy_policy_optimistic_initialization_5 = GreedyPolicy(n_bandits=n_bandits,
                                                      optimistic_initialization=5)

epsilon_greedy_policy_0_01 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                       epsilon=0.01)

epsilon_greedy_policy_0_1 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                       epsilon=0.1)

epsilon_greedy_policy_0_5 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                       epsilon=0.5)

ucb_policy_0 = BayesianUCBPolicy(n_bandits=n_bandits,
                     c=0)

ucb_policy_1 = BayesianUCBPolicy(n_bandits=n_bandits,
                     c=1)

ucb_policy_2 = BayesianUCBPolicy(n_bandits=n_bandits,
                     c=2)

thomson_sampling = ThompsonSamplingPolicy(n_bandits=n_bandits)

n_bandits = 10

# Setup the game
game = Game(n_episodes=200,
            n_steps=100,
            policies=[
greedy_policy,
greedy_policy_optimistic_initialization_1,
greedy_policy_optimistic_initialization_5,
epsilon_greedy_policy_0_01,
epsilon_greedy_policy_0_1,
epsilon_greedy_policy_0_5,
ucb_policy_0,
ucb_policy_1,
ucb_policy_2,
thomson_sampling
           ],
            n_bandits=n_bandits,
            )

game.game_loop()

game.plot_average_reward_by_step()
```

--------------------------------

### Get Reward Distribution

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Retrieves a reward distribution class based on its name.

```APIDOC
## static get_reward_distribution

### Description
Get the reward distribution class based on the given name.

### Parameters
* **name** (*str*) – Name of the reward distribution.

### Returns
The corresponding reward distribution class.

### Raises
**ValueError** – If an unknown distribution name is provided.

### Return type
*Type[[RewardDistribution](reward_distribution.md#pymab.reward_distribution.RewardDistribution)]

### Supported distributions
- ‘gaussian’: Normal distribution
- ‘bernoulli’: Binary distribution
- ‘uniform’: Uniform distribution
```

--------------------------------

### Setting Up and Running the Bandit Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/comparing_policies_gaussian.ipynb

Configures a Game instance with specified simulation parameters and a list of initialized policies, then executes the game loop to collect data.

```python
# Setup the game
game = Game(n_episodes=200, 
            n_steps=100, 
            policies=[
greedy_policy,
greedy_policy_optimistic_initialization_1,
greedy_policy_optimistic_initialization_5,
epsilon_greedy_policy_0_01,
epsilon_greedy_policy_0_1,
epsilon_greedy_policy_0_5,
ucb_policy_0,
ucb_policy_1,
ucb_policy_2,
thomson_sampling
            ], 
            n_bandits=n_bandits,
            )

# Run the game
game.game_loop()
```

--------------------------------

### Get Actual Reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Retrieves the actual reward for a given action, sampled from its distribution.

```APIDOC
## _get_actual_reward

### Description
Get the actual reward for a given action.

### Parameters
* **action_index** (*int*) – Index of the chosen action.

### Returns
The reward value sampled from the distribution.

### Return type
*float*
```

--------------------------------

### Get UCB Value

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Calculates the Upper Confidence Bound (UCB) value for a given action.

```APIDOC
## _get_ucb_value

### Description
Calculate the Upper Confidence Bound (UCB) value for a given action. This method combines the estimated reward with the confidence interval and handles unselected actions by returning infinity.

### Parameters
* **action_index** (*int*) – The index of the action to calculate the UCB value for.

### Returns
The calculated UCB value, or infinity for unselected actions.

### Return type
*float*

### Theory
UCB = Q(a) + U(a), where Q(a) is the estimated reward and U(a) is the confidence interval.
```

--------------------------------

### Run a Multi-Armed Bandit Experiment

Source: https://github.com/danielalopes/pymab/blob/main/README.md

This snippet demonstrates how to set up and run a basic experiment comparing Greedy and Thompson Sampling policies over multiple episodes and steps. It includes defining policies, initializing the game, running the simulation, and plotting the results.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.policies.thompson_sampling import ThompsonSamplingPolicy
from pymab.game import Game

n_bandits = 5

# Define the policies
greedy_policy = GreedyPolicy(
                    optimistic_initialization=1,
                    n_bandits=n_bandits
                )
ts_policy = ThompsonSamplingPolicy(n_bandits=n_bandits)

# Define the game
game = Game(
       n_episodes=2000,
       n_steps=1000,
       policies=[greedy_policy, ts_policy],
       n_bandits=n_bandits
  )

# Run the game
game.game_loop()

# Plot the results
game.plot_average_reward_by_step()
```

--------------------------------

### StationaryUCBPolicy._get_actual_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Get the actual reward for a given action. This method samples the reward from the action's distribution.

```APIDOC
## StationaryUCBPolicy._get_actual_reward(action_index)

### Description
Get the actual reward for a given action.

### Parameters
#### Path Parameters
- **action_index** (int) - Required - Index of the chosen action.

### Returns
- **float** - The reward value sampled from the distribution.
```

--------------------------------

### Initialize Policies and Game Simulation

Source: https://github.com/danielalopes/pymab/blob/main/examples/contextual_bandits_proxies.ipynb

Initializes a GreedyPolicy and multiple ContextualBanditPolicies with different learning rates. It then sets up a Game simulation with these policies and runs the game loop.

```python
# Define Q-values, which are the true values of the bandits
Q_values = np.array([0.1, 0.5, 0.9])
n_bandits = 3

# Initialize the GreedyPolicy
greedy_policy = GreedyPolicy(optimistic_initialization=1,
                             n_bandits=n_bandits,)
contextual_policy_lr_0_01 = ContextualBanditPolicy(n_bandits=n_bandits,
                                          context_dim=6,
                                            learning_rate=0.01,
                                   context_func=partial(generate_random_context, quality_levels=Q_values))
contextual_policy_lr_0_1 = ContextualBanditPolicy(n_bandits=n_bandits,
                                           context_dim=6,
                                             learning_rate=0.1,
                                    context_func=partial(generate_random_context, quality_levels=Q_values))
contextual_policy_lr_0_5 = ContextualBanditPolicy(n_bandits=n_bandits,
                                           context_dim=6,
                                             learning_rate=0.5,
                                    context_func=partial(generate_random_context, quality_levels=Q_values))
contextual_policy_lr_0_9 = ContextualBanditPolicy(n_bandits=n_bandits,
                                           context_dim=6,
                                             learning_rate=0.9,
                                    context_func=partial(generate_random_context, quality_levels=Q_values))

# Setup the game
game = Game(n_episodes=2000, 
            n_steps=1000,
            Q_values=Q_values,
            policies=[greedy_policy, contextual_policy_lr_0_01, contextual_policy_lr_0_1, contextual_policy_lr_0_5, contextual_policy_lr_0_9], 
            n_bandits=n_bandits,
            is_stationary=False,)

# Run the game
game.game_loop()
```

--------------------------------

### Initializing Bandit Policies with Different Parameters

Source: https://github.com/danielalopes/pymab/blob/main/examples/comparing_policies_gaussian.ipynb

Initializes various bandit policies, including Greedy, Epsilon-Greedy, and Bayesian UCB, with different parameter configurations to prepare for game simulation.

```python
n_bandits = 10

# Initialize the policies to compare
greedy_policy = GreedyPolicy(n_bandits=n_bandits,
                            optimistic_initialization=0)

greedy_policy_optimistic_initialization_1 = GreedyPolicy(n_bandits=n_bandits,
                                                       optimistic_initialization=1)

greedy_policy_optimistic_initialization_5 = GreedyPolicy(n_bandits=n_bandits,
                                                       optimistic_initialization=5)

epsilon_greedy_policy_0_01 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                         epsilon=0.01)

epsilon_greedy_policy_0_1 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                         epsilon=0.1)

epsilon_greedy_policy_0_5 = EpsilonGreedyPolicy(n_bandits=n_bandits,
                                         epsilon=0.5)

ucb_policy_0 = BayesianUCBPolicy(n_bandits=n_bandits,
                      c=0)

ucb_policy_1 = BayesianUCBPolicy(n_bandits=n_bandits,
                      c=1)

ucb_policy_2 = BayesianUCBPolicy(n_bandits=n_bandits,
                      c=2)

thomson_sampling = ThompsonSamplingPolicy(n_bandits=n_bandits)
```

--------------------------------

### RewardDistribution.get_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Abstract method to get a reward based on the distribution. This is an abstract method that must be implemented by subclasses.

```APIDOC
## RewardDistribution.get_reward

### Description
Abstract method to get a reward based on the distribution. This method defines the interface for sampling a single reward.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean or central value of the distribution.
- **variance** (*float*) – The variance or spread of the distribution.

### Returns
A reward sampled from the distribution.

### Return type
*float*
```

--------------------------------

### UniformRewardDistribution.get_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a reward sampled from a Uniform distribution. This method implements the abstract get_reward method for Uniform distributions.

```APIDOC
## UniformRewardDistribution.get_reward

### Description
Get a reward sampled from a Uniform distribution. This method implements the abstract get_reward method for Uniform distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The central value around which the uniform distribution is centered.
- **variance** (*float*) – The half-range of the uniform distribution.

### Returns
A reward sampled from the uniform distribution.

### Return type
*float*
```

--------------------------------

### Initialize Bandit Policies

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Sets up the number of bandits and reward distribution, then initializes Thompson Sampling and Greedy policies with specified parameters.

```python
n_bandits = 3

reward_distribution = 'bernoulli'

thomson_sampling = ThompsonSamplingPolicy(n_bandits=n_bandits, reward_distribution=reward_distribution)
greedy = GreedyPolicy(optimistic_initialization=1,
                      n_bandits=n_bandits, reward_distribution=reward_distribution)

# Define Q-values, which are the true values of the bandits
#Q_values = np.array([0.1, 0.5, -0.2, 0.4, 0.7, 0.45, 0.3, 0.2, 0.05, -0.1])
Q_values = [0.3, 0.7, 0.1]
```

--------------------------------

### BernoulliRewardDistribution.get_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a reward sampled from a Bernoulli distribution. This method implements the abstract get_reward method for Bernoulli distributions.

```APIDOC
## BernoulliRewardDistribution.get_reward

### Description
Get a reward sampled from a Bernoulli distribution. This method implements the abstract get_reward method for Bernoulli distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The probability of success in the Bernoulli distribution.
- **variance** (*float*) – Not used in Bernoulli distribution, included for interface compatibility.

### Returns
A reward sampled from the Bernoulli distribution.

### Return type
*float*
```

--------------------------------

### Policy Initialization

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes a policy with specified parameters. This constructor sets up the policy for a multi-armed bandit problem.

```APIDOC
## __init__

### Description
Initializes a policy with specified parameters.

### Parameters
* **n_bandits** (*int*) - The number of arms/actions available.
* **optimistic_initialization** (*float*) - The initial optimistic value for Q-values. Defaults to 0.0.
* **variance** (*float*) - The variance parameter, used in some algorithms. Defaults to 1.0.
* **reward_distribution** (*str*) - The type of reward distribution to use (e.g., 'gaussian', 'bernoulli'). Defaults to 'gaussian'.
* **c** (*float*) - The exploration parameter, often used in UCB algorithms. Defaults to 1.0.

### Return type
*None*
```

--------------------------------

### GaussianRewardDistribution.get_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Get a reward sampled from a Gaussian distribution. This method implements the abstract get_reward method for Gaussian distributions.

```APIDOC
## GaussianRewardDistribution.get_reward

### Description
Get a reward sampled from a Gaussian distribution. This method implements the abstract get_reward method for Gaussian distributions.

### Parameters
#### Path Parameters
- **q_value** (*float*) – The mean of the Gaussian distribution.
- **variance** (*float*) – The standard deviation of the Gaussian distribution.

### Returns
A reward sampled from the Gaussian distribution.

### Return type
*float*
```

--------------------------------

### Initialize and Run a Bandit Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/basic_usage.ipynb

This snippet shows how to initialize the GreedyPolicy with optimistic initialization, set up a Game instance with specified parameters, and then run the game loop. Finally, it plots the average reward per step.

```python
# Define Q-values, which are the true values of the bandits
#Q_values = np.array([0.1, 0.5, -0.2, 0.4, 0.7, 0.45, 0.3, 0.2, 0.05, -0.1])
n_bandits = 10

# Initialize the GreedyPolicy
policy = GreedyPolicy(optimistic_initialization=1,
                      n_bandits=n_bandits)

# Setup the game
game = Game(n_episodes=2000, 
            n_steps=1000, 
            policies=[policy], 
            n_bandits=n_bandits)

# Run the game
game.game_loop()

# Plot the results
game.plot_average_reward_by_step()
```

--------------------------------

### StationaryUCBPolicy.get_reward_distribution

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Get the reward distribution class based on the given name. Supported distributions include 'gaussian', 'bernoulli', and 'uniform'.

```APIDOC
## StationaryUCBPolicy.get_reward_distribution(name)

### Description
Get the reward distribution class based on the given name.

### Parameters
#### Path Parameters
- **name** (str) - Required - Name of the reward distribution.

### Returns
- **Type[[RewardDistribution](reward_distribution.md#pymab.reward_distribution.RewardDistribution)]** - The corresponding reward distribution class.

### Raises
- **ValueError** – If an unknown distribution name is provided.

### Supported Distributions
- ‘gaussian’
- ‘bernoulli’
- ‘uniform’
```

--------------------------------

### Initialize UCB Policies

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Initializes various UCB policies including Stationary, Sliding Window with different window sizes, and Discounted UCB with different discount factors. These policies are then collected into a list for use in the game simulations.

```python
ucb_stationary_policy = StationaryUCBPolicy(
    n_bandits=10,
    c=1,
)

ucb_sliding_window_policy_50 = SlidingWindowUCBPolicy(
    n_bandits=10,
    c=1,
    window_size=50,
)
ucb_sliding_window_policy_100 = SlidingWindowUCBPolicy(
    n_bandits=10,
    c=1,
    window_size=100,
)

ucb_sliding_window_policy_200 = SlidingWindowUCBPolicy(
    n_bandits=10,
    c=1,
    window_size=200,
)

ucb_discounted_policy_0_9 = DiscountedUCBPolicy(
    n_bandits=10,
    c=1,
    discount_factor=0.9,
)
ucb_discounted_policy_0_5 = DiscountedUCBPolicy(
    n_bandits=10,
    c=1,
    discount_factor=0.5,
)
ucb_discounted_policy_0_1 = DiscountedUCBPolicy(
    n_bandits=10,
    c=1,
    discount_factor=0.1,
)

policies = [
        ucb_stationary_policy, 
        ucb_sliding_window_policy_50,
        ucb_sliding_window_policy_100, 
        ucb_sliding_window_policy_200, 
        ucb_discounted_policy_0_1, 
        ucb_discounted_policy_0_5, 
        ucb_discounted_policy_0_9
    ]
```

--------------------------------

### DiscountedUCBPolicy.__init__()

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes a Discounted UCB policy with specified parameters. The discount factor influences the weighting of past observations, allowing adaptation to non-stationary environments.

```APIDOC
### *class* DiscountedUCBPolicy

Bases: `DiscountedMixin`, [`UCBPolicy`](#pymab.policies.ucb.UCBPolicy)

#### __init__(n_bandits, optimistic_initialization=0, variance=1.0, reward_distribution='gaussian', c=1.0, discount_factor=0.9)

### Description
Initialize a Discounted UCB policy.

### Parameters:
* **n_bandits** (*int*) – Number of bandits (arms) in the problem.
* **optimistic_initialization** (*float*) – Initial value for estimated rewards.
* **variance** (*float*) – Variance of the reward distribution.
* **reward_distribution** (*str*) – Type of reward distribution (“gaussian” or “bernoulli”).
* **c** (*float*) – Exploration parameter controlling confidence bound width.
* **discount_factor** (*float*) – Factor for discounting past rewards (between 0 and 1).

The discount factor determines how much weight is given to past observations,
with values closer to 1 giving more weight to historical data.
```

--------------------------------

### Run a Simple Multi-Armed Bandit Game

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/index.md

This snippet demonstrates how to set up and run a game with two policies (Greedy and Thompson Sampling) and then plot the average reward per step. Ensure the necessary policies and the Game class are imported.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.policies.thompson_sampling import ThompsonSamplingPolicy
from pymab.game import Game

n_bandits = 5

# Define the policies
greedy_policy = GreedyPolicy(
                    optimistic_initialization=1,
                    n_bandits=n_bandits
                )
ts_policy = ThompsonSamplingPolicy(n_bandits=n_bandits)

# Define the game
game = Game(
     n_episodes=2000,
     n_steps=1000,
     policies=[greedy_policy, ts_policy],
     n_bandits=n_bandits
)

# Run the game
game.game_loop()

# Plot the results
game.plot_average_reward_by_step()
```

--------------------------------

### Greedy Policy - _get_actual_reward

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Gets the actual reward for a given action index. This is a private helper method used internally by the policy.

```APIDOC
## Greedy Policy - _get_actual_reward

### Description
Get the actual reward for a given action.

### Parameters
#### Path Parameters
- **action_index** (int) - Required - Index of the chosen action.

### Returns
The reward value sampled from the distribution.

### Return type
float
```

--------------------------------

### Game Class Initialization

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/game.md

Initializes a Game instance for multi-armed bandit simulations. It sets up the environment, policies, and simulation parameters.

```APIDOC
## Game

### Description

Initializes a Game instance for multi-armed bandit simulations. It sets up the environment, policies, and simulation parameters.

### Parameters

* **n_episodes** (*int*) - Number of episodes to run.
* **n_steps** (*int*) - Number of steps per episode.
* **policies** (*List* *[*[*Policy*](policies.md#pymab.policies.policy.Policy) *]*) - List of policies to evaluate.
* **n_bandits** (*int*) - Number of bandit arms.
* **Q_values** (*List* *[**float* *]*) - Optional initial Q-values for arms. If None, generated randomly. Defaults to None.
* **Q_values_mean** (*float*) - Mean for random Q-value generation. Defaults to 0.0.
* **Q_values_variance** (*float*) - Variance for random Q-value generation. Defaults to 1.0.
* **environment_change** (*Union* *[*[*EnvironmentChangeType*](#pymab.game.EnvironmentChangeType) *,* *Type* *[*[*EnvironmentChangeMixin*](#pymab.game.EnvironmentChangeMixin) *]* *]*) - Type of environment dynamics. Defaults to STATIONARY.
* **change_params** (*dict*) - Parameters for non-stationary environments. Defaults to None.
* **results_folder** (*Path*) - Path to save visualization results. Defaults to DEFAULT_RESULTS_FOLDER.

### Return type

*None*
```

--------------------------------

### EpsilonGreedyPolicy.__init__

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the EpsilonGreedyPolicy with specified parameters for multi-armed bandit problems.

```APIDOC
## __init__(self, n_bandits, optimistic_initialization=0, variance=1.0, reward_distribution='gaussian', epsilon=0.1)

### Parameters
#### Path Parameters
- **n_bandits** (int) – Required
- **optimistic_initialization** (float) – Optional - Defaults to 0.
- **variance** (float) – Optional - Defaults to 1.0.
- **reward_distribution** (str) – Optional - Defaults to "gaussian".
- **epsilon** (float) – Optional - Defaults to 0.1.

### Return type
None
```

--------------------------------

### Initialize Bandit Policies

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_gaussian.ipynb

Sets up Thompson Sampling and Greedy policies for a specified number of bandits and reward distribution type. The Q-values represent the true expected rewards of each bandit.

```python
n_bandits = 3
reward_distribution = 'gaussian'

thomson_sampling = ThompsonSamplingPolicy(n_bandits=n_bandits, reward_distribution=reward_distribution)
greedy = GreedyPolicy(optimistic_initialization=1,
                      n_bandits=n_bandits, reward_distribution=reward_distribution)

# Define Q-values, which are the true values of the bandits
#Q_values = np.array([0.1, 0.5, -0.2, 0.4, 0.7, 0.45, 0.3, 0.2, 0.05, -0.1])
Q_values = [-0.3, 0.7, 0.1]
```

--------------------------------

### Generate Rewards using BernoulliDistribution

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/reward_distribution.md

Demonstrates how to generate rewards using BernoulliDistribution. This is useful for simulating reward signals in a multi-armed bandit setting where rewards are binary (success/failure) or follow a Bernoulli distribution. Ensure necessary classes are imported.

```python
from pymab.reward_distribution import BernoulliDistribution, GaussianDistribution
from pymab.game import Game
from pymab.policies.epsilon_greedy import EpsilonGreedyPolicy

Q_values = [0.3, 0.5, 0.7]
rewards = []
for q_value in Q_values:
   rewards.append(BernoulliDistribution.get_reward(q_value=q_value, variance=2))
```

--------------------------------

### Define Bandit Parameters and Policies

Source: https://github.com/danielalopes/pymab/blob/main/examples/bayesian_ucb_bernoulli.ipynb

Sets up the true Q-values for Bernoulli bandits and initializes various UCB policies, including StationaryUCBPolicy with different 'c' values and BayesianUCBPolicy.

```python
# Define Q-values, which are the true values of the bandits
# For bernoulli distribution, the values should always be between 0 and 1.
Q_values = [0.1, 0.8, 0.3, 0.4, 0.9, 0.2, 0.25, 0.6, 0.5, 0.35]
n_bandits = 10

reward_distribution = 'bernoulli'

ucb_policy_0 = StationaryUCBPolicy(n_bandits=n_bandits,
                      c=0, reward_distribution=reward_distribution)

ucb_policy_1 = StationaryUCBPolicy(n_bandits=n_bandits,
                      c=1, reward_distribution=reward_distribution)

ucb_policy_2 = StationaryUCBPolicy(n_bandits=n_bandits,
                      c=2, reward_distribution=reward_distribution)

bayesian_ucb = BayesianUCBPolicy(n_bandits=n_bandits, reward_distribution=reward_distribution)
```

--------------------------------

### Initialize UCB Policy

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the UCB policy with a specified number of bandits and optional parameters for exploration and reward distribution.

```python
policy = UCBPolicy(n_bandits=3)
```

--------------------------------

### BernoulliBayesianUCBPolicy.__init__

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the Bernoulli Bayesian UCB policy with specified parameters for bandit count, initialization, variance, reward distribution, and exploration.

```APIDOC
## __init__(self, n_bandits, optimistic_initialization=0.0, variance=1.0, reward_distribution='bernoulli', c=1.0)

### Description
Initializes the Bernoulli Bayesian UCB policy.

### Parameters
* **n_bandits** (*int*) – Number of bandits (actions) available.
* **optimistic_initialization** (*float*) – Initial Q-value for all actions. Defaults to 0.0.
* **variance** (*float*) – Variance of the reward distribution. Defaults to 1.0.
* **reward_distribution** (*str*) – Type of reward distribution. Must be “bernoulli”. Defaults to “bernoulli”.
* **c** (*float*) – Exploration parameter for UCB calculation. Defaults to 1.0.

### Returns
*None*
```

--------------------------------

### plot_Q_values_evolution_by_bandit_first_episode

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/game.md

Plots the evolution of Q-values for each bandit during the first episode. This provides insight into early learning dynamics.

```APIDOC
## plot_Q_values_evolution_by_bandit_first_episode(save=True, plot_name='', plot_config={})

### Description
Shows the progression of Q-values for each bandit specifically during the first episode of the simulation.

### Parameters
#### Query Parameters
- **save** (bool) - Optional - Whether to save the plot to a file. Defaults to True.
- **plot_name** (str) - Optional - The name to use when saving the plot. Defaults to an empty string.
- **plot_config** (dict[str, Any]) - Optional - A dictionary for custom plot configurations.
```

--------------------------------

### GreedyPolicy.__init__

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the Greedy policy with specified parameters, including the number of arms and optional optimistic initialization.

```APIDOC
## GreedyPolicy.__init__(n_bandits, optimistic_initialization=0, variance=1.0, reward_distribution='gaussian')

### Parameters
* **n_bandits** (*int*)
* **optimistic_initialization** (*float*)
* **variance** (*float*)
* **reward_distribution** (*str*)

### Return type
*None*
```

--------------------------------

### Import necessary libraries

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Imports the required classes and functions from numpy and pymab for setting up and running the bandit simulation.

```python
import numpy as np
from pymab.policies.thompson_sampling import ThompsonSamplingPolicy
from pymab.policies.greedy import GreedyPolicy
from pymab.game import Game
```

--------------------------------

### Random Context Generation for Proxies

Source: https://github.com/danielalopes/pymab/blob/main/examples/contextual_bandits_proxies.ipynb

Generates a semi-random context for proxy server selection, considering quality levels. It simulates metrics like latency, bandwidth, downtime, success rate, proximity, and load.

```python
def generate_random_context(quality_levels: List[float]) -> np.array:
    """
    Generates a semi-random context for the proxy server selection considering the quality level.

    Args:
        quality_levels (List[float]): The quality levels of each proxy.

    Returns:
        np.array: Array of shape (context_dim,) with semi-random context values.
    """
    latency = [np.random.uniform(10, 200 / quality_level) for quality_level in quality_levels] # Lower latency for higher quality
    bandwidth = [np.random.uniform(10 * quality_level, 1000 * quality_level) for quality_level in quality_levels] # Higher bandwidth for higher quality
    downtime_rate = [np.random.uniform(0, 5 / quality_level) for quality_level in quality_levels] # Lower downtime for higher quality
    success_rate = [np.random.uniform(90 + quality_level * 10, 100) for quality_level in quality_levels] # Success rate range
    proximity = [np.random.uniform(1, 5000 - (1 - quality_level) * 1000) for quality_level in quality_levels] # Proximity range
    load = [np.random.uniform(0, 100 - quality_level * 100) for quality_level in quality_levels] # Load range
    # print("latency", latency)
    # print("bandwidth", bandwidth)
    # print("downtime_rate_range", downtime_rate)
    # print("success_rate_range", success_rate)
    # print("proximity_range", proximity)
    # print("load_range", load)

    context = np.array([latency, bandwidth, downtime_rate, success_rate, proximity, load])

    return normalize_context(context)
```

--------------------------------

### Import necessary classes

Source: https://github.com/danielalopes/pymab/blob/main/examples/basic_usage.ipynb

Import the GreedyPolicy and Game classes from the pymab library. These are essential for setting up and running bandit simulations.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.game import Game
```

--------------------------------

### Importing Bandit Policies and Game Class

Source: https://github.com/danielalopes/pymab/blob/main/examples/comparing_policies_gaussian.ipynb

Imports necessary policy classes (Greedy, Epsilon-Greedy, Bayesian UCB, Thompson Sampling) and the Game class from the PyMAB library.

```python
from pymab.policies.greedy import GreedyPolicy
from pymab.policies.epsilon_greedy import EpsilonGreedyPolicy
from pymab.policies.bayesian_ucb import BayesianUCBPolicy

from pymab.policies.thompson_sampling import ThompsonSamplingPolicy
from pymab.game import Game
```

--------------------------------

### Run the Bandit Game Simulation

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Executes the main game loop for the configured number of episodes and steps, allowing policies to interact with the bandits.

```python
# Run the game
game.game_loop()
```

--------------------------------

### Import necessary libraries

Source: https://github.com/danielalopes/pymab/blob/main/examples/contextual_bandits_proxies.ipynb

Imports required modules from functools, numpy, typing, and pymab for policy and game simulation.

```python
from functools import partial
import numpy as np
from typing import List

from pymab.policies.greedy import GreedyPolicy
from pymab.policies.contextual_bandits import ContextualBanditPolicy
from pymab.game import Game
```

--------------------------------

### Plot Rate of Optimal Actions by Step

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Visualizes the proportion of times each policy chose the optimal action at each step, indicating exploration vs. exploitation efficiency.

```python
game.plot_rate_optimal_actions_by_step()
```

--------------------------------

### Run Stationary Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Executes the game simulation loop for the stationary environment. This establishes the performance benchmark for the selected policies under stable conditions.

```python
stationary_game.game_loop()
```

--------------------------------

### UCBPolicy Initialization

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the UCB Policy with specified parameters for multi-armed bandit problems.

```APIDOC
## __init__ ##

### Description
Initializes the UCB Policy with specified parameters for multi-armed bandit problems.

### Parameters
* **n_bandits** (int) - The number of actions (bandits) available.
* **optimistic_initialization** (float) - Initial value for optimistic initialization. Defaults to 0.0.
* **variance** (float) - The variance parameter, used in some reward distribution calculations. Defaults to 1.0.
* **reward_distribution** (str) - The type of reward distribution to assume ('gaussian', 'bernoulli', 'uniform'). Defaults to 'gaussian'.
* **c** (float) - The exploration parameter, controlling the trade-off between exploration and exploitation. Defaults to 1.0.

### Returns
None
```

--------------------------------

### Import Policies and Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/bayesian_ucb_bernoulli.ipynb

Imports necessary classes from the pymab library for UCB policies and game simulation.

```python
from pymab.policies.ucb import StationaryUCBPolicy
from pymab.policies.bayesian_ucb import BayesianUCBPolicy
from pymab.game import Game
```

--------------------------------

### Plot Average Reward by Step

Source: https://github.com/danielalopes/pymab/blob/main/examples/thomson_sampling_bernoulli.ipynb

Visualizes the average reward obtained by the policies at each step of the game, showing performance trends over time.

```python
# Plot the results
game.plot_average_reward_by_step()
```

--------------------------------

### UCBPolicy Initialization

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the UCBPolicy with specified parameters for multi-armed bandit problems. It sets up the bandit environment, optimistic initialization values, and variance for confidence interval calculations.

```APIDOC
## UCBPolicy

### Description
Initializes the UCBPolicy with specified parameters for multi-armed bandit problems. It sets up the bandit environment, optimistic initialization values, and variance for confidence interval calculations.

### Parameters
* **n_bandits** (*int*) - The number of arms (actions) in the bandit problem.
* **optimistic_initialization** (*float*) - The initial value assigned to each arm to encourage exploration. Defaults to 0.0.
* **variance** (*float*) - The variance parameter used in confidence interval calculations. Defaults to 1.0.
* **reward_distribution** (*str*) - The name of the reward distribution to use ('gaussian', 'bernoulli', 'uniform'). Defaults to 'gaussian'.
* **c** (*float*) - A constant used in the UCB formula for exploration. Defaults to 1.0.
```

--------------------------------

### Plotting for Stationary Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Generates performance plots for the stationary environment. These visualizations provide a baseline for evaluating how policies perform when the reward distributions remain constant.

```python
stationary_game.plot_average_reward_by_step()
```

```python
stationary_game.plot_average_reward_by_step_smoothed()
```

```python
stationary_game.plot_rate_optimal_actions_by_step()
```

```python
stationary_game.plot_cumulative_regret_by_step()
```

--------------------------------

### Define Simulation Parameters

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Sets the number of bandits, episodes, and steps for the simulation. These parameters control the scale and duration of the experiments.

```python
n_bandits = 10
n_episodes = 500
n_steps = 1000
```

--------------------------------

### EpsilonGreedyPolicy.select_action

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Selects an action based on the Epsilon-Greedy policy, balancing exploration (probability ε) and exploitation (probability 1-ε).

```APIDOC
## select_action(self, *args, **kwargs)

### Description
Select an action based on the Epsilon-Greedy policy.

### Returns
- Index of the chosen action (int)
- Reward received for the action (float)

### Return type
A tuple containing
```

--------------------------------

### select_action

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Selects the next action using the UCB algorithm. It first ensures each action is selected once for initial estimates, then chooses the action with the highest UCB value.

```APIDOC
## select_action(*args, **kwargs)

### Description
Select the next action based on the UCB algorithm. Initially, it selects each action once to gather initial estimates. After that, it chooses the action with the highest UCB value.

### Returns
A tuple containing the index of the chosen action and the reward obtained from taking that action.

### Return type
*Tuple*[*int*, *float*]

### Example Usage
```python
policy = UCBPolicy(n_bandits=3)
for _ in range(100):
    action, reward = policy.select_action()
    # Use the action and reward as needed
```
```

--------------------------------

### Run Random Arm Swapping Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Executes the game simulation loop for the environment with random arm swapping. This tests policy robustness against unpredictable changes in arm reward associations.

```python
non_stationary_swapping_game.game_loop()
```

--------------------------------

### ThompsonSamplingPolicy Constructor

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/policies.md

Initializes the Thompson Sampling policy. This constructor sets up the policy with the specified number of bandit arms and configuration for optimistic initialization, variance, and reward distribution.

```APIDOC
## __init__

### Description
Initializes the Thompson Sampling policy.

### Parameters
* **n_bandits** (*int*) - The number of bandit arms available.
* **optimistic_initialization** (*float*) - The initial value for all action estimates. Defaults to 0.0.
* **variance** (*float*) - The variance of the reward distribution. Defaults to 1.0.
* **reward_distribution** (*str*) - The type of reward distribution ('gaussian', 'bernoulli', or 'uniform'). Defaults to 'gaussian'.
```

--------------------------------

### Plotting for Random Arm Swapping Game

Source: https://github.com/danielalopes/pymab/blob/main/examples/non_stationary_policies.ipynb

Generates performance plots for the random arm swapping environment. These plots help analyze how policies cope with sudden, random reassignments of reward distributions to arms.

```python
non_stationary_swapping_game.plot_average_reward_by_step()
```

```python
non_stationary_swapping_game.plot_average_reward_by_step_smoothed()
```

```python
non_stationary_swapping_game.plot_rate_optimal_actions_by_step()
```

```python
non_stationary_swapping_game.plot_cumulative_regret_by_step()
```

```python
non_stationary_swapping_game.plot_optimal_arm_evolution()
```

--------------------------------

### AbruptChangeEnvironmentMixin Class

Source: https://github.com/danielalopes/pymab/blob/main/docs/source/game.md

Mixin for non-stationary environments where reward distributions change abruptly and periodically.

```APIDOC
## Class AbruptChangeEnvironmentMixin

### Description
Represents a non-stationary environment where reward distributions change abruptly and periodically.

### Methods
#### __init__(change_frequency, change_magnitude)
- **Description**: Initializes the mixin with change frequency and magnitude.
- **Parameters**:
  - **change_frequency** (int) - The number of steps between abrupt changes.
  - **change_magnitude** (float) - The standard deviation of the normal distribution used for abrupt changes.

#### apply_change(Q_values, step)
- **Description**: Applies abrupt changes to the Q-values at specified intervals.
- **Parameters**:
  - **Q_values** (ndarray) - The current Q-values.
  - **step** (int) - The current simulation step.
- **Return type**:
  - ndarray: The modified Q-values.
```