MABWiser

https://fidelity.github.io/mabwiser/
Admin
MABWiser is a research library for fast prototyping of multi-armed bandit algorithms supporting...
Tokens:32,049
Snippets:210
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Quick Start: UCB1 Learning Policy Example

Source: https://fidelity.github.io/mabwiser/index.html

Demonstrates how to use the UCB1 learning policy for decision-making between arms based on expected rewards. Requires importing MAB, LearningPolicy, and NeighborhoodPolicy from MABWiser.

```python
# An example that shows how to use the UCB1 learning policy
# to make decisions between two arms based on their expected rewards.

# Import MABWiser Library
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model
mab = MAB(arms, LearningPolicy.UCB1(alpha=1.25))

# Train
mab.fit(decisions, rewards)

# Test
mab.predict()

```

--------------------------------

### Initialize Simulator Data

Source: https://fidelity.github.io/mabwiser/api.html

Setup the initial arms, decision history, and reward history required for simulation.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
```

--------------------------------

### Setup and Data Preparation for MABWiser

Source: https://fidelity.github.io/mabwiser/_sources/examples.rst.txt

Sets up the MABWiser environment by importing necessary libraries and preparing sample data for an advertisement optimization scenario. Includes data scaling.

```python
import pandas as pd
from sklearn.preprocessing import StandardScaler

from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Arms
ads = [1, 2, 3, 4, 5]

# Historical data of ad decisions with corresponding revenues and context information
train_df = pd.DataFrame({'ad': [1, 1, 1, 2, 4, 5, 3, 3, 2, 1, 4, 5, 3, 2, 5],
							 'revenues': [10, 17, 22, 9, 4, 20, 7, 8, 20, 9, 50, 5, 7, 12, 10],
							 'age': [22, 27, 39, 48, 21, 20, 19, 37, 52, 26, 18, 42, 55, 57, 38],
							 'click_rate': [0.2, 0.6, 0.99, 0.68, 0.15, 0.23, 0.75, 0.17,
											0.33, 0.65, 0.56, 0.22, 0.19, 0.11, 0.83],
							'subscriber': [1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0]})

# Arm features for warm start
arm_to_features = {1: [0, 1, 1], 2: [0, 0.5, 0.5], 3: [1, 1, 0.5], 4: [0.2, 1, 0], 5: [0, 1, 0.1], 6: [0, 0.5, 0.5]}

# Test data to for new prediction
test_df = pd.DataFrame({'age': [37, 52], 'click_rate': [0.5, 0.6], 'subscriber': [0, 1]})
test_df_revenue = pd.Series([7, 13])

# Scale the training and test data
scaler = StandardScaler()
train = scaler.fit_transform(train_df[['age', 'click_rate', 'subscriber']])
test = scaler.transform(test_df)
```

--------------------------------

### Import MABWiser classes

Source: https://fidelity.github.io/mabwiser/_sources/installation.rst.txt

Verify the installation by importing the core library components.

```python
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
```

--------------------------------

### Recompile Documentation

Source: https://fidelity.github.io/mabwiser/new_bandit.html

Run this command to recompile documentation after making changes. Ensure MABWiser is uninstalled or in development mode if installed.

```bash
make github
```

--------------------------------

### Initialize and Train UCB1 Policy

Source: https://fidelity.github.io/mabwiser/_sources/quick.rst.txt

Demonstrates the basic setup for a MAB model using the UCB1 learning policy with provided historical decisions and rewards.

```python
# An example that shows how to use the UCB1 learning policy 
# to choose between two arms based on their expected rewards.

# Import MABWiser Library
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model 
mab = MAB(arms, LearningPolicy.UCB1(alpha=1.25))

# Train
mab.fit(decisions, rewards)

# Test
mab.predict()
```

--------------------------------

### Run MABWiser tests

Source: https://fidelity.github.io/mabwiser/_sources/installation.rst.txt

Execute the project's test suite to verify the installation.

```bash
git clone https://github.com/fidelity/mabwiser.git
cd mabwiser
python -m unittest discover tests
```

--------------------------------

### UCB1 Learning Policy Example

Source: https://fidelity.github.io/mabwiser/quick.html

This example shows how to use the UCB1 learning policy to select between two arms based on their expected rewards. It covers importing the library, defining data, initializing the model, training it with historical data, and making predictions.

```python
# An example that shows how to use the UCB1 learning policy
# to choose between two arms based on their expected rewards.

# Import MABWiser Library
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model
mab = MAB(arms, LearningPolicy.UCB1(alpha=1.25))

# Train
mab.fit(decisions, rewards)

# Test
mab.predict()

```

--------------------------------

### UCB1 Learning Policy Example

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates initializing and using the UCB1 learning policy with the MAB class. Requires arms, decisions, and rewards for fitting.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(list_of_arms, LearningPolicy.UCB1(alpha=1.25))
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm2'
```

--------------------------------

### Parallel MAB Setup

Source: https://fidelity.github.io/mabwiser/examples.html

Provides the necessary imports and setup for using MABWiser in a parallel processing scenario, including generating synthetic data for user contexts and rewards. This is relevant for large-scale recommendation systems.

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

from mabwiser.mab import MAB, LearningPolicy

# Seed
seed = 111

# Arms
arms = list(np.arange(100))
```

--------------------------------

### Usage Example: Creating and Training a MAB Model with a New Bandit Policy

Source: https://fidelity.github.io/mabwiser/_sources/new_bandit.rst.txt

Demonstrates how to instantiate and train a MAB model using a custom learning policy. Ensure MABWiser library is imported and data is prepared before use.

```python
# Import MABWiser Library
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model 
mab = MAB(arms, LearningPolicy.MyCoolPolicy(my_parameter=42))

# Train
mab.fit(decisions, rewards)

# Test
mab.predict()
```

--------------------------------

### Test Warm Start

Source: https://fidelity.github.io/mabwiser/new_bandit.html

Placeholder for testing the execution of a warm start and asserting that its behavior is as expected.

```python
    def test_warm_start(self):
        # Test warm start can be executed
        # Assert warm start behavior is as expected
```

--------------------------------

### Install MABWiser in Development Mode

Source: https://fidelity.github.io/mabwiser/new_bandit.html

Use this command to install MABWiser in development mode, which is necessary for Sphinx to use the local package during documentation recompilation.

```bash
pip install -e
```

--------------------------------

### LinUCB Learning Policy Example

Source: https://fidelity.github.io/mabwiser/examples.html

Demonstrates initializing and using the LinUCB learning policy for ad optimization. It includes fitting the model with historical data, making predictions, and updating the model with new data or arms.

```python
linucb = MAB(arms=ads,
                             learning_policy=LearningPolicy.LinUCB(alpha=1.25, l2_lambda=1))

linucb.fit(decisions=train_df['ad'], rewards=train_df['revenues'], contexts=train)

prediction = linucb.predict(test)

expectations = linucb.predict_expectations(test)

print("LinUCB: ", prediction, " ", expectations)
assert(prediction == [5, 2])

linucb.partial_fit(decisions=prediction, rewards=test_df_revenue, contexts=test)

linucb.add_arm(6)

linucb.warm_start(arm_to_features, distance_quantile=0.75)
```

--------------------------------

### Usage Example for a New Bandit Policy

Source: https://fidelity.github.io/mabwiser/new_bandit.html

Demonstrates how a user would initialize and train a custom bandit policy named MyCoolPolicy.

```python
# Import MABWiser Library
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

# Data
arms = ['Arm1', 'Arm2']
decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
rewards = [20, 17, 25, 9]

# Model
mab = MAB(arms, LearningPolicy.MyCoolPolicy(my_parameter=42))

# Train
mab.fit(decisions, rewards)

# Test
mab.predict()
```

--------------------------------

### LSHNearest Neighborhood Policy Example

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates fitting and predicting using the LSHNearest neighborhood policy with MAB. Requires specifying the number of dimensions and hash tables.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> list_of_arms = [1, 2, 3, 4]
>>> decisions = [1, 1, 1, 2, 2, 3, 3, 3, 3, 3]
>>> rewards = [0, 1, 1, 0, 0, 0, 0, 1, 1, 1]
>>> contexts = [[0, 1, 2, 3, 5], [1, 1, 1, 1, 1], [0, 0, 1, 0, 0],[0, 2, 2, 3, 5], [1, 3, 1, 1, 1],                             [0, 0, 0, 0, 0], [0, 1, 4, 3, 5], [0, 1, 2, 4, 5], [1, 2, 1, 1, 3], [0, 2, 1, 0, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.EpsilonGreedy(epsilon=0),                           NeighborhoodPolicy.LSHNearest(5, 3))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[0, 1, 2, 3, 5], [1, 1, 1, 1, 1]])
[3, 1]
```

--------------------------------

### Copy Attributes for Warm Starting

Source: https://fidelity.github.io/mabwiser/_sources/new_bandit.rst.txt

The `_copy_arms` method defines how attributes are copied from a warm arm to a cold arm during the warm-starting process. This method should be implemented to transfer relevant policy-specific data.

```python
def _copy_arms(self, cold_arm_to_warm_arm: Dict[Arm, Arm]) -> None:
            # TODO:
            # This method tells the policy how to warm start cold arms, given a cold arm to warm arm mapping.
            # It will typically involve copying attributes from a warm arm to a cold arm, e.g.
            for cold_arm, warm_arm in cold_arm_to_warm_arm.items():
                self.my_value_to_arm[cold_arm] = deepcopy(self.my_value_to_arm[cold_arm])
```

--------------------------------

### Build MABWiser from source

Source: https://fidelity.github.io/mabwiser/_sources/installation.rst.txt

Use these commands to clone the repository and build a wheel package locally.

```bash
git clone https://github.com/fidelity/mabwiser.git
cd mabwiser
pip install setuptools wheel # if wheel is not installed
python setup.py sdist bdist_wheel
pip install dist/mabwiser-X.X.X-py3-none-any.whl
```

--------------------------------

### Initialize and Run Simulator

Source: https://fidelity.github.io/mabwiser/api.html

Initializes two EpsilonGreedy Multi-Armed Bandits with different epsilon values and a simulator. The simulator is then run with provided decisions and rewards data.

```python
mab1 = MAB(arms, LearningPolicy.EpsilonGreedy(epsilon=0.25), seed=123456)
mab2 = MAB(arms, LearningPolicy.EpsilonGreedy(epsilon=0.30), seed=123456)
bandits = [('EG 25%', mab1), ('EG 30%', mab2)]
offline_sim = Simulator(bandits, decisions, rewards, test_size=0.5, batch_size=0)
offline_sim.run()
offline_sim.bandit_to_arm_to_stats_avg['EG 30%']['Arm1']
```

--------------------------------

### Initialize and use EpsilonGreedy policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates setting up the MAB class with the EpsilonGreedy policy and performing a prediction.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(arms, LearningPolicy.EpsilonGreedy(epsilon=0.25), seed=123456)
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm1'
```

--------------------------------

### Contextual Bandit with EpsilonGreedy and KNearest

Source: https://fidelity.github.io/mabwiser/api.html

Illustrates setting up a contextual multi-armed bandit using EpsilonGreedy and KNearest policies. Includes fitting with contexts, predicting, adding arms, and partial fitting with contexts.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1', 'Arm2']
>>> rewards = [20, 17, 25, 9, 11]
>>> contexts = [[0, 0, 0], [1, 0, 1], [0, 1, 1], [0, 0, 0], [1, 1, 1]]
>>> contextual_mab = MAB(arms, LearningPolicy.EpsilonGreedy(), NeighborhoodPolicy.KNearest(k=3))
>>> contextual_mab.fit(decisions, rewards, contexts)
>>> contextual_mab.predict([[1, 1, 0], [1, 1, 1], [0, 1, 0]])
['Arm2', 'Arm2', 'Arm2']
>>> contextual_mab.add_arm('Arm3')
>>> contextual_mab.partial_fit(['Arm3'], [30], [[1, 1, 1]])
>>> contextual_mab.predict([[1, 1, 1]])
'Arm3'
```

--------------------------------

### Initialize and Predict with KNearest Neighborhood Policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates setting up a MAB instance with a KNearest neighborhood policy and performing a prediction.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> list_of_arms = [1, 2, 3, 4]
>>> decisions = [1, 1, 1, 2, 2, 3, 3, 3, 3, 3]
>>> rewards = [0, 1, 1, 0, 0, 0, 0, 1, 1, 1]
>>> contexts = [[0, 1, 2, 3, 5], [1, 1, 1, 1, 1], [0, 0, 1, 0, 0],[0, 2, 2, 3, 5], [1, 3, 1, 1, 1],                             [0, 0, 0, 0, 0], [0, 1, 4, 3, 5], [0, 1, 2, 4, 5], [1, 2, 1, 1, 3], [0, 2, 1, 0, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.EpsilonGreedy(epsilon=0),                           NeighborhoodPolicy.KNearest(2, "euclidean"))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[0, 1, 2, 3, 5], [1, 1, 1, 1, 1]])
[1, 1]
```

--------------------------------

### Initialize and use LinGreedy policy

Source: https://fidelity.github.io/mabwiser/api.html

Shows how to use the LinGreedy policy with context-aware data for arm selection.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> contexts = [[0, 1, 2, 3], [1, 2, 3, 0], [2, 3, 1, 0], [3, 2, 1, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.LinGreedy(epsilon=0.5))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[3, 2, 0, 1]])
'Arm2'
```

--------------------------------

### Initialize and use Softmax policy

Source: https://fidelity.github.io/mabwiser/api.html

Shows the Softmax policy configuration using a temperature parameter to control exploration.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(list_of_arms, LearningPolicy.Softmax(tau=1))
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm2'
```

--------------------------------

### Initialize and Predict with Clusters Neighborhood Policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates setting up a MAB instance with a Clusters neighborhood policy and performing a prediction.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> list_of_arms = [1, 2, 3, 4]
>>> decisions = [1, 1, 1, 2, 2, 3, 3, 3, 3, 3]
>>> rewards = [0, 1, 1, 0, 0, 0, 0, 1, 1, 1]
>>> contexts = [[0, 1, 2, 3, 5], [1, 1, 1, 1, 1], [0, 0, 1, 0, 0],[0, 2, 2, 3, 5], [1, 3, 1, 1, 1],                             [0, 0, 0, 0, 0], [0, 1, 4, 3, 5], [0, 1, 2, 4, 5], [1, 2, 1, 1, 3], [0, 2, 1, 0, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.EpsilonGreedy(epsilon=0), NeighborhoodPolicy.Clusters(3))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[0, 1, 2, 3, 5], [1, 1, 1, 1, 1]])
[3, 1]
```

--------------------------------

### Context-Free MAB with Epsilon Greedy Policy

Source: https://fidelity.github.io/mabwiser/examples.html

Demonstrates initializing a MAB model with an Epsilon Greedy policy, fitting historical data, and performing online updates.

```python
    from mabwiser.mab import MAB, LearningPolicy

######################################################################################
#
# MABWiser
# Scenario: A/B Testing for Website Layout Design
#
# An e-commerce website experiments with 2 different layouts options for their homepage
# Each layouts decision leads to generating different revenues
#
# What should the choice of layouts be based on historical data?
#
######################################################################################

# Arms
options = [1, 2]

# Historical data of layouts decisions and corresponding rewards
layouts = [1, 1, 1, 2, 1, 2, 2, 1, 2, 1, 2, 2, 1, 2, 1]
revenues = [10, 17, 22, 9, 4, 0, 7, 8, 20, 9, 50, 5, 7, 12, 10]

# Arm to features
arm_to_features = {1: [0, 0, 1], 2: [1, 1, 0], 3: [1, 1, 0]}

###################################
# Epsilon Greedy Learning Policy
###################################

# Epsilon Greedy learning policy with random exploration set to 15%
greedy = MAB(arms=options,
             learning_policy=LearningPolicy.EpsilonGreedy(epsilon=0.15),
             seed=123456)

# Learn from previous layouts decisions and revenues generated
greedy.fit(decisions=layouts, rewards=revenues)

# Predict the next best layouts decision
prediction = greedy.predict()

# Expected revenues of each layouts learnt from historical data based on epsilon greedy policy
expectations = greedy.predict_expectations()

# Results
print("Epsilon Greedy: ", prediction, " ", expectations)
assert(prediction == 2)

# Additional historical data becomes available which allows online learning
additional_layouts = [1, 2, 1, 2]
additional_revenues = [0, 12, 7, 19]

# Online updating of the model
greedy.partial_fit(additional_layouts, additional_revenues)

# Adding a new layout option
greedy.add_arm(3)

# Warm start new arm
greedy.warm_start(arm_to_features, distance_quantile=0.5)
```

--------------------------------

### Radius Neighborhood Policy Example

Source: https://fidelity.github.io/mabwiser/api.html

Illustrates using the Radius neighborhood policy for MAB. This policy selects observations within a specified radius using a given distance metric.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> list_of_arms = [1, 2, 3, 4]
>>> decisions = [1, 1, 1, 2, 2, 3, 3, 3, 3, 3]
>>> rewards = [0, 1, 1, 0, 0, 0, 0, 1, 1, 1]
>>> contexts = [[0, 1, 2, 3, 5], [1, 1, 1, 1, 1], [0, 0, 1, 0, 0],[0, 2, 2, 3, 5], [1, 3, 1, 1, 1],                             [0, 0, 0, 0, 0], [0, 1, 4, 3, 5], [0, 1, 2, 4, 5], [1, 2, 1, 1, 3], [0, 2, 1, 0, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.EpsilonGreedy(epsilon=0),                           NeighborhoodPolicy.Radius(2, "euclidean"))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[0, 1, 2, 3, 5], [1, 1, 1, 1, 1]])
[3, 1]
```

--------------------------------

### Get Arm Expectations

Source: https://fidelity.github.io/mabwiser/_sources/new_bandit.rst.txt

The `predict_expectations` method returns a copy of the internal dictionary containing the expected value for each arm. Returning a copy prevents accidental modification of the policy's internal state.

```python
def predict_expectations(self, contexts: np.ndarray = None) -> Dict[Arm, Num]:
            # This method returns a copy of the expectations dictionary.
            # Make sure to return a copy of the internal object,
            # so that the user cannot accidentally break your policy.
            return self.arm_to_expectation.copy()
```

--------------------------------

### Epsilon Greedy Policy for Website Layout A/B Testing

Source: https://fidelity.github.io/mabwiser/_sources/examples.rst.txt

Demonstrates using the Epsilon Greedy learning policy to optimize website layout choices based on historical revenue data. Requires importing MAB and LearningPolicy. The model can be updated online and new arms can be added.

```python
from mabwiser.mab import MAB, LearningPolicy

######################################################################################
#
# MABWiser
# Scenario: A/B Testing for Website Layout Design
#
# An e-commerce website experiments with 2 different layouts options for their homepage
# Each layouts decision leads to generating different revenues
#
# What should the choice of layouts be based on historical data?
#
######################################################################################

# Arms
options = [1, 2]

# Historical data of layouts decisions and corresponding rewards
layouts = [1, 1, 1, 2, 1, 2, 2, 1, 2, 1, 2, 2, 1, 2, 1]
revenues = [10, 17, 22, 9, 4, 0, 7, 8, 20, 9, 50, 5, 7, 12, 10]

# Arm to features
arm_to_features = {1: [0, 0, 1], 2: [1, 1, 0], 3: [1, 1, 0]}

###################################
# Epsilon Greedy Learning Policy
###################################

# Epsilon Greedy learning policy with random exploration set to 15%
greedy = MAB(arms=options,
             learning_policy=LearningPolicy.EpsilonGreedy(epsilon=0.15),
             seed=123456)

# Learn from previous layouts decisions and revenues generated
greedy.fit(decisions=layouts, rewards=revenues)

# Predict the next best layouts decision
prediction = greedy.predict()

# Expected revenues of each layouts learnt from historical data based on epsilon greedy policy
expectations = greedy.predict_expectations()

# Results
print("Epsilon Greedy: ", prediction, " ", expectations)
assert(prediction == 2)

# Additional historical data becomes available which allows online learning
additional_layouts = [1, 2, 1, 2]
additional_revenues = [0, 12, 7, 19]

# Online updating of the model
greedy.partial_fit(additional_layouts, additional_revenues)

# Adding a new layout option
greedy.add_arm(3)

# Warm start new arm
greedy.warm_start(arm_to_features, distance_quantile=0.5)
```

--------------------------------

### Initialize and Fit TreeBandit Policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates initializing a MAB instance with the TreeBandit neighborhood policy and fitting it with historical data.

```python
>>> from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> contexts = [[0, 1, 2, 3], [1, 2, 3, 0], [2, 3, 1, 0], [3, 2, 1, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.EpsilonGreedy(epsilon=0), NeighborhoodPolicy.TreeBandit())
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[3, 2, 0, 1]])
'Arm2'
```

--------------------------------

### Initialize and train a UCB1 bandit model

Source: https://fidelity.github.io/mabwiser/_sources/index.rst.txt

Demonstrates the basic workflow of importing the library, defining arms and rewards, fitting the model, and making predictions.

```python
# An example that shows how to use the UCB1 learning policy 
	# to make decisions between two arms based on their expected rewards.

	# Import MABWiser Library
	from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

	# Data
	arms = ['Arm1', 'Arm2']
	decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
	rewards = [20, 17, 25, 9]

	# Model 
	mab = MAB(arms, LearningPolicy.UCB1(alpha=1.25))

	# Train
	mab.fit(decisions, rewards)

	# Test
	mab.predict()
```

--------------------------------

### Simulator for Hyper-Parameter Tuning

Source: https://fidelity.github.io/mabwiser/examples.html

Demonstrates using the MABWiser Simulator to perform hyper-parameter tuning for multiple bandit configurations. It shows how to set up bandits, prepare data, configure the simulator with various parameters like scaling, test size, and batch size, and then run and analyze the simulation results.

```python
import random
from sklearn.preprocessing import StandardScaler
from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy
from mabwiser.simulator import Simulator

######################################################################################
#
# MABWiser
# Scenario: Hyper-Parameter Tuning using the built-in Simulator capability
#
######################################################################################

# Data
size = 1000
decisions = [random.randint(0, 2) for _ in range(size)]
rewards = [random.randint(0, 1000) for _ in range(size)]
contexts = [[random.random() for _ in range(50)] for _ in range(size)]

# Bandits to simulate
n_jobs = 2
hyper_parameter_tuning = []
for radius in range(6, 10):
    hyper_parameter_tuning.append(('Radius'+str(radius),
                                  MAB([0, 1], LearningPolicy.UCB1(1),
                                      NeighborhoodPolicy.Radius(radius),
                                      n_jobs=n_jobs)))

# Simulator with given bandits and data
# The parameters uses standard scaler,
# Test split size set to 0.5
# The split is not order dependent, i.e., random split
# Online training with batch size 10, i.e., bandits will re-train at each batch
# Offline training can be run with batch_size 0, i.e., no re-training during test phase
sim = Simulator(hyper_parameter_tuning, decisions, rewards, contexts,
                scaler=StandardScaler(), test_size=0.5, is_ordered=False, batch_size=10, seed=123456)

# Run the simulator
sim.run()

# Save the results with a prefix
sim.save_results("my_results_")

# You can probe the fields of the simulator for other statisics
for mab_name, mab in sim.bandits:
    print(mab_name + "\n")

    # Since the simulation is online, print the 'total' stats
    print('Worst Case Scenario:', sim.bandit_to_arm_to_stats_min[mab_name]['total'])
    print('Average Case Scenario:', sim.bandit_to_arm_to_stats_avg[mab_name]['total'])
    print('Best Case Scenario:', sim.bandit_to_arm_to_stats_max[mab_name]['total'], "\n\n")

# Plot the average case results per every arm for each bandit
sim.plot(metric='avg', is_per_arm=True)
```

--------------------------------

### Initialize and use Random policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates the Random policy which selects arms uniformly at random.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(list_of_arms, LearningPolicy.Random())
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm2'
```

--------------------------------

### Warm-Start Untrained Arms

Source: https://fidelity.github.io/mabwiser/_sources/new_bandit.rst.txt

The `warm_start` method initializes cold arms using data from warm arms based on feature similarity and a distance quantile. It calls the base class's `_copy_arms` method to facilitate this process.

```python
def warm_start(self, arm_to_features: Dict[Arm, List[Num]], distance_quantile: float) -> None:
            # This method warm starts untrained (cold) arms for which no decisions has been observed
            # A cold arm is warm started using a warm arm that is within some minimum distance from the cold arm
            # based on the given arm_to_features and distance_quantile inputs.

            # Calculate pairwise distances between arms and determine cold arm to warm arm mapping
            # Call _copy_arms from the base class.
            return super().warm_start(arm_to_features, distance_quantile)
```

--------------------------------

### Initialize and use Thompson Sampling policy

Source: https://fidelity.github.io/mabwiser/api.html

Demonstrates Thompson Sampling with binary rewards and with a custom binarizer function for non-binary rewards.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [1, 1, 1, 0]
>>> mab = MAB(list_of_arms, LearningPolicy.ThompsonSampling())
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm2'
```

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> arm_to_threshold = {'Arm1':10, 'Arm2':10}
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [10, 20, 15, 7]
>>> def binarize(arm, reward): return reward > arm_to_threshold[arm]
>>> mab = MAB(list_of_arms, LearningPolicy.ThompsonSampling(binarizer=binarize))
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm2'
```

--------------------------------

### Initialize and use Popularity policy

Source: https://fidelity.github.io/mabwiser/api.html

Shows the usage of the Popularity policy which selects arms based on their mean reward probability.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(list_of_arms, LearningPolicy.Popularity())
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm1'
```

--------------------------------

### Initialize and use LinTS policy

Source: https://fidelity.github.io/mabwiser/api.html

Illustrates the usage of the LinTS policy, which utilizes Thompson Sampling for exploration.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> list_of_arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> contexts = [[0, 1, 2, 3], [1, 2, 3, 0], [2, 3, 1, 0], [3, 2, 1, 0]]
>>> mab = MAB(list_of_arms, LearningPolicy.LinTS(alpha=0.25))
>>> mab.fit(decisions, rewards, contexts)
>>> mab.predict([[3, 2, 0, 1]])
'Arm2'
```

--------------------------------

### Upgrade mabwiser from Source

Source: https://fidelity.github.io/mabwiser/installation.html

Instructions for upgrading a custom build of the library. This involves pulling the latest changes, rebuilding the wheel, and reinstalling.

```bash
git pull origin master
python setup.py sdist bdist_wheel
pip install --upgrade --no-cache-dir dist/mabwiser-X.X.X-py3-none-any.whl
```

--------------------------------

### Initialize and Use LinUCB Policy

Source: https://fidelity.github.io/mabwiser/_sources/examples.rst.txt

Initializes the LinUCB learning policy with specified alpha and l2_lambda. It demonstrates fitting the model with historical data, predicting the next best ad, and updating the model online.

```python
linucb = MAB(arms=ads,
				 learning_policy=LearningPolicy.LinUCB(alpha=1.25, l2_lambda=1))

linucb.fit(decisions=train_df['ad'], rewards=train_df['revenues'], contexts=train)

prediction = linucb.predict(test)

expectations = linucb.predict_expectations(test)

print("LinUCB: ", prediction, " ", expectations)
assert(prediction == [5, 2])

linucb.partial_fit(decisions=prediction, rewards=test_df_revenue, contexts=test)

linucb.add_arm(6)

linucb.warm_start(arm_to_features, distance_quantile=0.75)
```

--------------------------------

### EpsilonGreedy Policy with MAB

Source: https://fidelity.github.io/mabwiser/api.html

Shows how to use the EpsilonGreedy policy with the MAB class, including fitting, predicting, adding arms, and partial fitting.

```python
>>> from mabwiser.mab import MAB, LearningPolicy
>>> arms = ['Arm1', 'Arm2']
>>> decisions = ['Arm1', 'Arm1', 'Arm2', 'Arm1']
>>> rewards = [20, 17, 25, 9]
>>> mab = MAB(arms, LearningPolicy.EpsilonGreedy(epsilon=0.25), seed=123456)
>>> mab.fit(decisions, rewards)
>>> mab.predict()
'Arm1'
>>> mab.add_arm('Arm3')
>>> mab.partial_fit(['Arm3'], [30])
>>> mab.predict()
'Arm3'
```

--------------------------------

### Data Preparation for MABWiser

Source: https://fidelity.github.io/mabwiser/examples.html

Illustrates the necessary data preparation steps, including creating DataFrames for historical data and context features, and scaling the data using StandardScaler. This is a prerequisite for training MAB models.

```python
import pandas as pd
from sklearn.preprocessing import StandardScaler

# Arms
ads = [1, 2, 3, 4, 5]

# Historical data of ad decisions with corresponding revenues and context information
train_df = pd.DataFrame({'ad': [1, 1, 1, 2, 4, 5, 3, 3, 2, 1, 4, 5, 3, 2, 5],
                                                     'revenues': [10, 17, 22, 9, 4, 20, 7, 8, 20, 9, 50, 5, 7, 12, 10],
                                                     'age': [22, 27, 39, 48, 21, 20, 19, 37, 52, 26, 18, 42, 55, 57, 38],
                                                     'click_rate': [0.2, 0.6, 0.99, 0.68, 0.15, 0.23, 0.75, 0.17,
                                                                                    0.33, 0.65, 0.56, 0.22, 0.19, 0.11, 0.83],
                                                    'subscriber': [1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0]})

# Arm features for warm start
arm_to_features = {1: [0, 1, 1], 2: [0, 0.5, 0.5], 3: [1, 1, 0.5], 4: [0.2, 1, 0], 5: [0, 1, 0.1], 6: [0, 0.5, 0.5]}

# Test data to for new prediction
test_df = pd.DataFrame({'age': [37, 52], 'click_rate': [0.5, 0.6], 'subscriber': [0, 1]})
test_df_revenue = pd.Series([7, 13])

# Scale the training and test data
scaler = StandardScaler()
train = scaler.fit_transform(train_df[['age', 'click_rate', 'subscriber']])
test = scaler.transform(test_df)
```

--------------------------------

### Parametric Contextual MAB for Advertisement Optimization

Source: https://fidelity.github.io/mabwiser/_sources/examples.rst.txt

Illustrates using a parametric contextual MAB for ad optimization, considering user context like age, click rate, and subscriber status. Requires pandas and scikit-learn for data manipulation and scaling. Arm features are used for warm-starting.

```python
import pandas as pd
from sklearn.preprocessing import StandardScaler

from mabwiser.mab import MAB, LearningPolicy, NeighborhoodPolicy

######################################################################################
#
# MABWiser
# Scenario: Advertisement Optimization
#
# An e-commerce website needs to solve the problem of which ad to display to online users
# Each advertisement decision leads to generating different revenues
#
# What should the choice of advertisement be given the context of an online user
# based on customer data such as age, click rate, subscriber?
#
######################################################################################

# Arms
ads = [1, 2, 3, 4, 5]

# Historical data of ad decisions with corresponding revenues and context information
train_df = pd.DataFrame({'ad': [1, 1, 1, 2, 4, 5, 3, 3, 2, 1, 4, 5, 3, 2, 5],
							 'revenues': [10, 17, 22, 9, 4, 20, 7, 8, 20, 9, 50, 5, 7, 12, 10],
							 'age': [22, 27, 39, 48, 21, 20, 19, 37, 52, 26, 18, 42, 55, 57, 38],
							 'click_rate': [0.2, 0.6, 0.99, 0.68, 0.15, 0.23, 0.75, 0.17,
											0.33, 0.65, 0.56, 0.22, 0.19, 0.11, 0.83],
							 'subscriber': [1, 0, 1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0]} 
							 )

# Arm features for warm start
arm_to_features = {1: [0, 1, 1], 2: [0, 0.5, 0.5], 3: [1, 1, 0.5], 4: [0.2, 1, 0], 5: [0, 1, 0.1], 6: [0, 0.5, 0.5]}

# Test data to for new prediction
test_df = pd.DataFrame({'age': [37, 52], 'click_rate': [0.5, 0.6], 'subscriber': [0, 1]})
test_df_revenue = pd.Series([7, 13])

# Scale the training and test data
scaler = StandardScaler()
train = scaler.fit_transform(train_df[['age', 'click_rate', 'subscriber']])
test = scaler.transform(test_df)
```

--------------------------------

### Mabwiser Methods (P, R, W)

Source: https://fidelity.github.io/mabwiser/genindex.html

Details on methods like partial_fit, plot, predict, predict_expectations, remove_arm, reset, and warm_start.

```APIDOC
## Methods (P, R, W)

### Methods
- **partial_fit()** (mabwiser.base_mab.BaseMAB method, mabwiser.mab.MAB method)
- **plot()** (mabwiser.simulator.Simulator method)
- **predict()** (mabwiser.base_mab.BaseMAB method, mabwiser.mab.MAB method)
- **predict_expectations()** (mabwiser.base_mab.BaseMAB method, mabwiser.mab.MAB method)
- **remove_arm()** (mabwiser.base_mab.BaseMAB method, mabwiser.mab.MAB method)
- **reset()** (in module mabwiser.utils)
- **run()** (mabwiser.simulator.Simulator method)
- **warm_start()** (mabwiser.base_mab.BaseMAB method, mabwiser.mab.MAB method)

### Properties
- **trained_arms** (mabwiser.base_mab.BaseMAB property)

### Attributes
- **rewards** (mabwiser.simulator.Simulator attribute)
- **rng** (mabwiser.base_mab.BaseMAB attribute)
```
Tokens:
Raw