### Install dependencies

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Install the required packages to run the examples.

```bash
uv add polars tqdm
```

--------------------------------

### Launch the marimo notebook server

Source: https://github.com/e10v/tea-tasting/blob/main/examples/README.md

Start the marimo editor to access and run the example notebooks.

```bash
uv run marimo edit examples
```

--------------------------------

### Install dependencies with uv

Source: https://github.com/e10v/tea-tasting/blob/main/examples/README.md

Create a virtual environment and install the required packages: marimo, tea-tasting, polars, and ibis-framework with duckdb support.

```bash
uv venv && uv pip install marimo tea-tasting polars 'ibis-framework[duckdb]'
```

--------------------------------

### Install Polars dependency

Source: https://github.com/e10v/tea-tasting/blob/main/docs/multiple-testing.md

Install the Polars data backend required to reproduce the examples.

```bash
uv add polars
```

--------------------------------

### Install tea-tasting with uv

Source: https://github.com/e10v/tea-tasting/blob/main/README.md

Use this command to install the package using the uv package manager.

```bash
uv add tea-tasting
```

--------------------------------

### Install tea-tasting with pip

Source: https://github.com/e10v/tea-tasting/blob/main/README.md

Use this command to install the package using pip.

```bash
pip install tea-tasting
```

--------------------------------

### Install Ibis backend with uv

Source: https://github.com/e10v/tea-tasting/blob/main/README.md

Install the Ibis framework with a specific backend, such as duckdb, to enable data backend support.

```bash
uv add 'ibis-framework[duckdb]'
```

--------------------------------

### Clone the tea-tasting repository

Source: https://github.com/e10v/tea-tasting/blob/main/examples/README.md

Clone the repository and navigate into the project directory to begin local setup.

```bash
git clone https://github.com/e10v/tea-tasting.git && cd tea-tasting
```

--------------------------------

### Run basic experiment analysis

Source: https://github.com/e10v/tea-tasting/blob/main/README.md

A basic example showing how to generate user data, define an experiment with various metrics, and analyze the results.

```pycon
>>> import tea_tasting as tt

>>> data = tt.make_users_data(rng=42)
>>> experiment = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions"),
...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
...     orders_per_user=tt.Mean("orders"),
...     revenue_per_user=tt.Mean("revenue"),
... )
>>> result = experiment.analyze(data)
>>> result
metric             control treatment rel_effect_size rel_effect_size_ci pvalue
sessions_per_user     2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
orders_per_user      0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user      5.24      5.73            9.3%       [-2.4%, 22%]  0.123
```

--------------------------------

### Install dependencies for DuckDB and Polars

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Install the necessary packages to use DuckDB and Polars as backends with tea-tasting.

```bash
uv add 'ibis-framework[duckdb]' polars
```

--------------------------------

### Define Experiment Metrics

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Example of initializing an Experiment with various Mean and RatioOfMeans metrics using custom parameters.

```pycon
>>> another_experiment = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions", alternative="greater"),
...     orders_per_session=tt.RatioOfMeans("orders", "sessions", confidence_level=0.9),
...     orders_per_user=tt.Mean("orders", equal_var=True),
...     revenue_per_user=tt.Mean("revenue", use_t=False),
... )
```

--------------------------------

### Get all global options with get_config

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Retrieve a dictionary containing all current global configuration options by calling get_config without parameters.

```pycon
>>> global_config = tt.get_config()
```

--------------------------------

### Parallelize simulations with ProcessPoolExecutor

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Speed up simulations by passing an executor.map function to the map_ parameter. This example uses the fork context, which is not supported on Windows or WASM notebooks.

```pycon
>>> import concurrent.futures
>>> import multiprocessing as mp

>>> ctx = mp.get_context("fork")  # doctest: +SKIP
>>> with concurrent.futures.ProcessPoolExecutor(mp_context=ctx) as executor:
...     results_parallel = experiment.simulate(
...         data,
...         100,
...         rng=42,
...         treat=treat,
...         batch_size=10,
...         map_=executor.map,
...         progress=tqdm.tqdm,
...     )  # doctest: +SKIP
... 
100%|█████████████████████████████████████| 100/100 [00:00<00:00, 251.60it/s]
```

--------------------------------

### Get a global option value with get_config

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Retrieve the current value of a specific global configuration parameter by passing its name to get_config.

```pycon
>>> tt.get_config("equal_var")
False
```

--------------------------------

### Prepare demo database with DuckDB

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Initialize a DuckDB connection and populate it with sample experimental data using Ibis.

```pycon
>>> import duckdb
>>> import ibis
>>> import polars as pl
>>> import tea_tasting as tt

>>> users_data = tt.make_users_data(rng=42)
>>> con = ibis.connect("duckdb://")
>>> con.create_table("users_data", users_data)
DatabaseTable: memory.main.users_data
  user     int64
  variant  int64
  sessions int64
  orders   int64
  revenue  float64
```

--------------------------------

### Import modules and prepare data

Source: https://github.com/e10v/tea-tasting/blob/main/docs/custom-metrics.md

Initializes the environment by importing necessary libraries and preparing a PyArrow table for experiment analysis.

```pycon
>>> from typing import Literal, NamedTuple
>>> import numpy as np
>>> import pyarrow as pa
>>> import pyarrow.compute as pc
>>> import scipy.stats
>>> import tea_tasting as tt
>>> import tea_tasting.aggr
>>> import tea_tasting.config
>>> import tea_tasting.metrics
>>> import tea_tasting.utils

>>> data = tt.make_users_data(rng=42)
>>> data = data.append_column(
...     "has_order",
...     pc.greater(data["orders"], 0).cast(pa.int64()),
... )
>>> data
pyarrow.Table
user: int64
variant: int64
sessions: int64
orders: int64
revenue: double
has_order: int64
----
user: [[0,1,2,3,4,...,3995,3996,3997,3998,3999]]
variant: [[1,0,1,1,0,...,0,0,0,0,0]]
sessions: [[2,2,2,2,1,...,2,2,3,1,5]]
orders: [[1,1,1,1,1,...,0,0,0,0,2]]
revenue: [[9.17,6.43,7.94,15.93,7.14,...,0,0,0,0,17.16]]
has_order: [[1,1,1,1,1,...,0,0,0,0,1]]
```

--------------------------------

### Analyze multiple metric types in one experiment

Source: https://github.com/e10v/tea-tasting/blob/main/docs/custom-metrics.md

Demonstrates initializing an experiment with a mix of aggregated and granular metrics and performing the analysis. The output shows the resulting metrics table with p-values and effect sizes.

```pycon
>>> experiment = tt.Experiment(
...     prop_users_with_orders=Proportion("has_order"),
...     mean_users_with_orders=tt.Mean("has_order"),
...     mwu_orders=MannWhitneyU("orders"),
...     mwu_revenue=MannWhitneyU("revenue"),
... )
>>> experiment.analyze(data)
metric                 control treatment rel_effect_size rel_effect_size_ci pvalue
prop_users_with_orders   0.345     0.384             11%             [-, -] 0.0117
mean_users_with_orders   0.345     0.384             11%        [2.5%, 21%] 0.0106
mwu_orders                   -         -               -             [-, -] 0.0263
mwu_revenue                  -         -               -             [-, -] 0.0300
```

--------------------------------

### Analyze experiment metrics with tea-tasting

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Demonstrates how to define metrics using tt.Mean and tt.RatioOfMeans and analyze experiment data with a single query.

```pycon
>>> experiment = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions"),
...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
...     orders_per_user=tt.Mean("orders"),
...     revenue_per_user=tt.Mean("revenue"),
... )
>>> result = experiment.analyze(data)
>>> result
metric             control treatment rel_effect_size rel_effect_size_ci pvalue
sessions_per_user     2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
orders_per_user      0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user      5.24      5.73            9.3%       [-2.4%, 22%]  0.123
```

--------------------------------

### Calculate null hypothesis rejection rates

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Defines a function to calculate the proportion of rejected null hypotheses across multiple significance levels (alpha) for A/A test results. The example shows the function definition and the resulting output table for various metrics.

```pycon
>>> def null_rejected(
...     results_data: pl.DataFrame,
...     alphas: tuple[float, ...] = (0.01, 0.02, 0.05),
... ) -> pl.DataFrame:
...     return results_data.group_by("metric", maintain_order=True).agg(
...         pl.col("pvalue").le(alpha).mean().alias(f"null_rejected_{alpha}")
...         for alpha in alphas
...     )
... 
>>> null_rejected(results_data)
shape: (5, 4)
┌────────────────────┬────────────────────┬────────────────────┬────────────────────┐
│ metric             ┆ null_rejected_0.01 ┆ null_rejected_0.02 ┆ null_rejected_0.05 │
│ ---                ┆ ---                ┆ ---                ┆ ---                │
│ str                ┆ f64                ┆ f64                ┆ f64                │
╞════════════════════╪════════════════════╪════════════════════╪════════════════════╡
│ sessions_per_user  ┆ 0.01               ┆ 0.02               ┆ 0.05               │
│ orders_per_session ┆ 0.02               ┆ 0.02               ┆ 0.06               │
│ orders_per_user    ┆ 0.01               ┆ 0.02               ┆ 0.05               │
│ revenue_per_user   ┆ 0.02               ┆ 0.03               ┆ 0.06               │
│ n_users            ┆ 0.01               ┆ 0.01               ┆ 0.04               │
└────────────────────┴────────────────────┴────────────────────┴────────────────────┘
```

--------------------------------

### Track simulation progress with tqdm

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Use the progress parameter in experiment.simulate to integrate with tqdm for real-time progress monitoring.

```pycon
>>> import tqdm

>>> results_progress = experiment.simulate(
...     data,
...     100,
...     rng=42,
...     progress=tqdm.tqdm,
... )  # doctest: +SKIP
100%|██████████████████████████████████████| 100/100 [00:01<00:00, 64.47it/s]
```

--------------------------------

### Run A/A tests

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Define experiment metrics and execute simulations using the simulate method.

```pycon
>>> experiment = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions"),
...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
...     orders_per_user=tt.Mean("orders"),
...     revenue_per_user=tt.Mean("revenue"),
...     n_users=tt.SampleRatio(),
... )
>>> results = experiment.simulate(data, 100, rng=42)
>>> results_data = results.to_polars()
>>> results_data.select(
...     "metric",
...     "control",
...     "treatment",
...     "rel_effect_size",
...     "rel_effect_size_ci_lower",
...     "rel_effect_size_ci_upper",
...     "pvalue",
... )  # doctest: +SKIP
shape: (500, 7)
┌────────────────────┬──────────┬───────────┬─────────────────┬────────────────────┬────────────────────┬──────────┐
│ metric             ┆ control  ┆ treatment ┆ rel_effect_size ┆ rel_effect_size_ci ┆ rel_effect_size_ci ┆ pvalue   │
│ ---                ┆ ---      ┆ ---       ┆ ---             ┆ _lower             ┆ _upper             ┆ ---      │
│ str                ┆ f64      ┆ f64       ┆ f64             ┆ ---                ┆ ---                ┆ f64      │
│                    ┆          ┆           ┆                 ┆ f64                ┆ f64                ┆          │
╞════════════════════╪══════════╪═══════════╪═════════════════╪════════════════════╪════════════════════╪══════════╡
│ sessions_per_user  ┆ 1.98004  ┆ 1.998998  ┆ 0.009575        ┆ -0.021272          ┆ 0.041393           ┆ 0.547091 │
│ orders_per_session ┆ 0.263105 ┆ 0.258647  ┆ -0.016945       ┆ -0.108177          ┆ 0.083621           ┆ 0.730827 │
│ orders_per_user    ┆ 0.520958 ┆ 0.517034  ┆ -0.007532       ┆ -0.102993          ┆ 0.098087           ┆ 0.883462 │
│ revenue_per_user   ┆ 5.446662 ┆ 5.14521   ┆ -0.055346       ┆ -0.162811          ┆ 0.065914           ┆ 0.356327 │
│ n_users            ┆ 2004.0   ┆ 1996.0    ┆ null            ┆ null               ┆ null               ┆ 0.91187  │
│ …                  ┆ …        ┆ …         ┆ …               ┆ …                  ┆ …                  ┆ …        │
│ sessions_per_user  ┆ 1.993624 ┆ 1.985212  ┆ -0.00422        ┆ -0.034685          ┆ 0.027207           ┆ 0.78959  │
│ orders_per_session ┆ 0.269373 ┆ 0.251991  ┆ -0.064527       ┆ -0.151401          ┆ 0.03124            ┆ 0.179445 │
│ orders_per_user    ┆ 0.537028 ┆ 0.500255  ┆ -0.068475       ┆ -0.158141          ┆ 0.030742           ┆ 0.169217 │
│ revenue_per_user   ┆ 5.511967 ┆ 5.071928  ┆ -0.079833       ┆ -0.184806          ┆ 0.038656           ┆ 0.177868 │
│ n_users            ┆ 2039.0   ┆ 1961.0    ┆ null            ┆ null               ┆ null               ┆ 0.223423 │
└────────────────────┴──────────┴───────────┴─────────────────┴────────────────────┴────────────────────┴──────────┘
```

--------------------------------

### Create an Ibis Table from an SQL query

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Uses the con.sql method to execute a SQL query and return an Ibis Table object.

```pycon
>>> data = con.sql("select * from users_data")
>>> data
SQLQueryResult
  query:
    select * from users_data
  schema:
    user     int64
    variant  int64
    sessions int64
    orders   int64
    revenue  float64
```

--------------------------------

### Set global option values with set_config

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Update global configuration settings using set_config. Note that settings are captured at the time of Experiment initialization.

```pycon
>>> tt.set_config(equal_var=True, use_t=False)
>>> experiment_with_config = tt.Experiment(
...     sessions_per_user=tt.Mean("sessions"),
...     orders_per_session=tt.RatioOfMeans("orders", "sessions"),
...     orders_per_user=tt.Mean("orders"),
...     revenue_per_user=tt.Mean("revenue"),
... )
>>> tt.set_config(equal_var=False, use_t=True)
>>> orders_per_user = experiment_with_config.metrics["orders_per_user"]
>>> print(
...     f"orders_per_user.equal_var: {orders_per_user.equal_var}\n"
...     f"orders_per_user.use_t: {orders_per_user.use_t}"
... )
orders_per_user.equal_var: True
orders_per_user.use_t: False
```

--------------------------------

### Analyze experiment with custom metric

Source: https://github.com/e10v/tea-tasting/blob/main/docs/custom-metrics.md

Demonstrates using the custom Proportion metric alongside a standard Mean metric within an Experiment object.

```pycon
>>> experiment_prop = tt.Experiment(
...     prop_users_with_orders=Proportion("has_order"),
...     mean_users_with_orders=tt.Mean("has_order", use_t=False),
... )
>>> experiment_prop.analyze(data)
metric                 control treatment rel_effect_size rel_effect_size_ci pvalue
prop_users_with_orders   0.345     0.384             11%             [-, -] 0.0117
mean_users_with_orders   0.345     0.384             11%        [2.5%, 21%] 0.0106
```

--------------------------------

### Execute Mann-Whitney U Test

Source: https://github.com/e10v/tea-tasting/blob/main/docs/custom-metrics.md

Demonstrates how to use the custom MannWhitneyU metric within an Experiment and analyze the results.

```pycon
>>> experiment_mwu = tt.Experiment(
...     mwu_orders=MannWhitneyU("orders"),
...     mwu_revenue=MannWhitneyU("revenue"),
... )
>>> result_mwu = experiment_mwu.analyze(data)
>>> result_mwu.with_keys(("metric", "pvalue", "statistic"))
metric      pvalue statistic
mwu_orders  0.0263   2069092
mwu_revenue 0.0300   2068060
```

--------------------------------

### Prepare data for simulation

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Generate user data without uplift and remove the variant column to prepare for A/A testing.

```pycon
>>> import polars as pl
>>> import tea_tasting as tt

>>> data = (
...     tt.make_users_data(rng=42, orders_uplift=0, revenue_uplift=0)
...     .drop_columns("variant")
... )
>>> data
pyarrow.Table
user: int64
sessions: int64
orders: int64
revenue: double
----
user: [[0,1,2,3,4,...,3995,3996,3997,3998,3999]]
sessions: [[2,2,2,2,1,...,2,2,3,1,5]]
orders: [[1,1,1,0,1,...,0,1,1,0,4]]
revenue: [[19.06,12.09,8.84,0,9.9,...,0,4.8,9.63,0,12.7]]
```

--------------------------------

### Analyze data with DuckDB relation

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Demonstrates how to convert arrow data to a DuckDB relation and run an experiment analysis.

```pycon
>>> data_duckdb = duckdb.from_arrow(users_data)
>>> experiment.analyze(data_duckdb)
metric             control treatment rel_effect_size rel_effect_size_ci pvalue
sessions_per_user     2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
orders_per_user      0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user      5.24      5.73            9.3%       [-2.4%, 22%]  0.123
```

--------------------------------

### Define experiment with dictionary metrics

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Define an experiment using a dictionary for metrics and specifying a custom variant column name.

```pycon
>>> new_experiment = tt.Experiment(
...     {
...         "sessions per user": tt.Mean("sessions"),
...         "orders per session": tt.RatioOfMeans("orders", "sessions"),
...         "orders per user": tt.Mean("orders"),
...         "revenue per user": tt.Mean("revenue"),
...     },
...     variant="variant",
... )
```

--------------------------------

### Analyze power for all experiment metrics

Source: https://github.com/e10v/tea-tasting/blob/main/docs/power-analysis.md

Performs power analysis for all metrics defined in an experiment using a configuration context for sample sizes.

```pycon
>>> with tt.config_context(n_obs=(10_000, 20_000)):
...     experiment = tt.Experiment(
...         sessions_per_user=tt.Mean("sessions", "sessions_covariate"),
...         orders_per_session=tt.RatioOfMeans(
...             numer="orders",
...             denom="sessions",
...             numer_covariate="orders_covariate",
...             denom_covariate="sessions_covariate",
...         ),
...         orders_per_user=tt.Mean("orders", "orders_covariate"),
...         revenue_per_user=tt.Mean("revenue", "revenue_covariate"),
...     )
... 
>>> power_result = experiment.solve_power(data)
>>> power_result
metric             power effect_size rel_effect_size n_obs
sessions_per_user    80%      0.0458            2.3% 10000
sessions_per_user    80%      0.0324            1.6% 20000
orders_per_session   80%      0.0177            6.8% 10000
orders_per_session   80%      0.0125            4.8% 20000
orders_per_user      80%      0.0374            7.2% 10000
orders_per_user      80%      0.0264            5.1% 20000
revenue_per_user     80%       0.488            9.2% 10000
revenue_per_user     80%       0.345            6.5% 20000
```

--------------------------------

### Compile Ibis table to SQL

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Generates and prints the SQL query corresponding to the Ibis table expression.

```pycon
>>> print(aggr_data.compile(pretty=True))
SELECT
  "t0"."variant",
  AVG("t0"."sessions") AS "sessions_per_user",
  AVG("t0"."orders") / AVG("t0"."sessions") AS "orders_per_session",
  AVG("t0"."orders") AS "orders_per_user",
  AVG("t0"."revenue") AS "revenue_per_user"
FROM (
  SELECT
    *
  FROM users_data
) AS "t0"
GROUP BY
  1
```

--------------------------------

### Query and fetch data with Ibis interactive mode

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Enables interactive mode to print the first 5 rows of the Ibis Table, then disables it.

```pycon
>>> ibis.options.interactive = True
>>> print(data.head(5))
┏━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━━┓
┃ user  ┃ variant ┃ sessions ┃ orders ┃ revenue ┃
┡━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━━┩
│ int64 │ int64   │ int64    │ int64  │ float64 │
├───────┼─────────┼──────────┼────────┼─────────┤
│     0 │       1 │        2 │      1 │    9.17 │
│     1 │       0 │        2 │      1 │    6.43 │
│     2 │       1 │        2 │      1 │    7.94 │
│     3 │       1 │        2 │      1 │   15.93 │
│     4 │       0 │        1 │      1 │    7.14 │
└───────┴─────────┴──────────┴────────┴─────────┘

>>> ibis.options.interactive = False
```

--------------------------------

### Prepare experiment data

Source: https://github.com/e10v/tea-tasting/blob/main/docs/multiple-testing.md

Generate and concatenate user data for a control and two treatment variants using Polars.

```pycon
>>> import polars as pl
>>> import tea_tasting as tt

>>> data = pl.concat((
...     tt.make_users_data(
...         rng=42,
...         orders_uplift=0.10,
...         revenue_uplift=0.15,
...         return_type="polars",
...     ),
...     tt.make_users_data(
...         rng=21,
...         orders_uplift=0.15,
...         revenue_uplift=0.20,
...         return_type="polars",
...     )
...         .filter(pl.col("variant").eq(1))
...         .with_columns(variant=pl.lit(2, pl.Int64)),
... ))
>>> data
shape: (6_046, 5)
┌──────┬─────────┬──────────┬────────┬─────────┐
│ user ┆ variant ┆ sessions ┆ orders ┆ revenue │
│ ---  ┆ ---     ┆ ---      ┆ ---    ┆ ---     │
│ i64  ┆ i64     ┆ i64      ┆ i64    ┆ f64     │
╞══════╪═════════╪══════════╪════════╪═════════╡
│ 0    ┆ 1       ┆ 2        ┆ 1      ┆ 9.58    │
│ 1    ┆ 0       ┆ 2        ┆ 1      ┆ 6.43    │
│ 2    ┆ 1       ┆ 2        ┆ 1      ┆ 8.3     │
│ 3    ┆ 1       ┆ 2        ┆ 1      ┆ 16.65   │
│ 4    ┆ 0       ┆ 1        ┆ 1      ┆ 7.14    │
│ …    ┆ …       ┆ …        ┆ …      ┆ …       │
│ 3989 ┆ 2       ┆ 4        ┆ 4      ┆ 34.93   │
│ 3991 ┆ 2       ┆ 1        ┆ 0      ┆ 0.0     │
│ 3992 ┆ 2       ┆ 3        ┆ 3      ┆ 27.96   │
│ 3994 ┆ 2       ┆ 2        ┆ 1      ┆ 17.22   │
│ 3998 ┆ 2       ┆ 3        ┆ 0      ┆ 0.0     │
└──────┴─────────┴──────────┴────────┴─────────┘
```

--------------------------------

### Simulate experiment with dynamic data function

Source: https://github.com/e10v/tea-tasting/blob/main/docs/simulated-experiments.md

Uses the experiment.simulate method with a data generation function and displays the resulting null hypothesis rejection rates in a Polars DataFrame.

```pycon
>>> results_data_gen = experiment.simulate(tt.make_users_data, 100, rng=42)
>>> null_rejected(results_data_gen.to_polars())
shape: (5, 4)
┌────────────────────┬────────────────────┬────────────────────┬────────────────────┐
│ metric             ┆ null_rejected_0.01 ┆ null_rejected_0.02 ┆ null_rejected_0.05 │
│ ---                ┆ ---                ┆ ---                ┆ ---                │
│ str                ┆ f64                ┆ f64                ┆ f64                │
╞════════════════════╪════════════════════╪════════════════════╪════════════════════╡
│ sessions_per_user  ┆ 0.01               ┆ 0.01               ┆ 0.06               │
│ orders_per_session ┆ 0.27               ┆ 0.36               ┆ 0.54               │
│ orders_per_user    ┆ 0.24               ┆ 0.32               ┆ 0.49               │
│ revenue_per_user   ┆ 0.17               ┆ 0.26               ┆ 0.39               │
│ n_users            ┆ 0.01               ┆ 0.01               ┆ 0.04               │
└────────────────────┴────────────────────┴────────────────────┴────────────────────┘
```

--------------------------------

### Fetch and display Ibis table results

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Enables interactive mode to print the contents of the aggregated table.

```pycon
>>> ibis.options.interactive = True
>>> print(aggr_data)  # doctest: +SKIP
┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
┃ variant ┃ sessions_per_user ┃ orders_per_session ┃ orders_per_user ┃ revenue_per_user ┃
┡━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━┩
│ int64   │ float64           │ float64            │ float64         │ float64          │
├─────────┼───────────────────┼────────────────────┼─────────────────┼──────────────────┤
│       0 │          1.996045 │           0.265726 │        0.530400 │         5.241028 │
│       1 │          1.982802 │           0.289031 │        0.573091 │         5.730111 │
└─────────┴───────────────────┴────────────────────┴─────────────────┴──────────────────┘

>>> ibis.options.interactive = False
```

--------------------------------

### Aggregate data with group_by

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Groups data by the 'variant' column and calculates mean metrics for sessions, orders, and revenue.

```pycon
>>> aggr_data = data.group_by("variant").aggregate(
...     sessions_per_user=data.sessions.mean(),
...     orders_per_session=data.orders.mean() / data.sessions.mean(),
...     orders_per_user=data.orders.mean(),
...     revenue_per_user=data.revenue.mean(),
... )
>>> aggr_data
r0 := SQLQueryResult
  query:
    select * from users_data
  schema:
    user     int64
    variant  int64
    sessions int64
    orders   int64
    revenue  float64
<BLANKLINE>
Aggregate[r0]
  groups:
    variant: r0.variant
  metrics:
    sessions_per_user:  Mean(r0.sessions)
    orders_per_session: Mean(r0.orders) / Mean(r0.sessions)
    orders_per_user:    Mean(r0.orders)
    revenue_per_user:   Mean(r0.revenue)
```

--------------------------------

### Sample ratio mismatch check

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Shows how to use the SampleRatio class to detect imbalances in A/B test variant sizes. By default, it expects an equal number of observations.

```pycon
>>> experiment_sample_ratio = tt.Experiment(
...     orders_per_user=tt.Mean("orders"),
...     revenue_per_user=tt.Mean("revenue"),
...     sample_ratio=tt.SampleRatio(),
... )
>>> result_sample_ratio = experiment_sample_ratio.analyze(data)
>>> result_sample_ratio
metric           control treatment rel_effect_size rel_effect_size_ci pvalue
orders_per_user    0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user    5.24      5.73            9.3%       [-2.4%, 22%]  0.123
sample_ratio        2023      1977               -             [-, -]  0.477
```

--------------------------------

### Analyze data with Polars DataFrame

Source: https://github.com/e10v/tea-tasting/blob/main/docs/data-backends.md

Demonstrates how to convert arrow data to a Polars DataFrame and run an experiment analysis.

```pycon
>>> data_polars = pl.from_arrow(users_data)
>>> experiment.analyze(data_polars)
metric             control treatment rel_effect_size rel_effect_size_ci pvalue
sessions_per_user     2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
orders_per_user      0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user      5.24      5.73            9.3%       [-2.4%, 22%]  0.123
```

--------------------------------

### Generating a Markdown table with to_markdown

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

The to_markdown method provides a formatted Markdown table representation of the experiment results.

```pycon
>>> print(result.to_markdown())
| metric             | control | treatment | rel_effect_size | rel_effect_size_ci | pvalue |
| :----------------- | ------: | --------: | --------------: | -----------------: | -----: |
| sessions_per_user  |    2.00 |      1.98 |          -0.66% |      [-3.7%, 2.5%] |  0.674 |
| orders_per_session |   0.266 |     0.289 |            8.8% |      [-0.89%, 19%] | 0.0762 |
| orders_per_user    |   0.530 |     0.573 |            8.0% |       [-2.0%, 19%] |  0.118 |
| revenue_per_user   |    5.24 |      5.73 |            9.3% |       [-2.4%, 22%] |  0.123 |
```

--------------------------------

### Displaying ExperimentResult in a console

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Evaluating the result object in a terminal or Python console displays a plain-text table.

```pycon
>>> result
metric             control treatment rel_effect_size rel_effect_size_ci pvalue
sessions_per_user     2.00      1.98          -0.66%      [-3.7%, 2.5%]  0.674
orders_per_session   0.266     0.289            8.8%      [-0.89%, 19%] 0.0762
orders_per_user      0.530     0.573            8.0%       [-2.0%, 19%]  0.118
revenue_per_user      5.24      5.73            9.3%       [-2.4%, 22%]  0.123
```

--------------------------------

### Analyze power for multiple sample sizes

Source: https://github.com/e10v/tea-tasting/blob/main/docs/power-analysis.md

Demonstrates calculating the relative effect size for a Mean metric across multiple specified sample sizes.

```pycon
>>> orders_per_user = tt.Mean("orders", alpha=0.1, power=0.7, n_obs=(10_000, 20_000))
>>> orders_per_user.solve_power(data, "rel_effect_size")
power effect_size rel_effect_size n_obs
  70%      0.0367            7.1% 10000
  70%      0.0260            5.0% 20000
```

--------------------------------

### Define Proportion metric class

Source: https://github.com/e10v/tea-tasting/blob/main/docs/custom-metrics.md

Implements the Proportion metric by subclassing MetricBaseAggregated, defining initialization, aggregation columns, and the analysis logic using chi-squared contingency tests.

```pycon
>>> class Proportion(tea_tasting.metrics.MetricBaseAggregated[ProportionResult]):
...     def __init__(
...         self,
...         column: str,
...         *,
...         correction: bool = True,
...         method: Literal["g-test", "pearson"] = "g-test",
...     ) -> None:
...         self.column = tea_tasting.utils.check_scalar(column, "column", typ=str)
...         self.correction = tea_tasting.utils.auto_check(correction, "correction")
...         self.method = tea_tasting.utils.check_scalar(
...             method, "method", typ=str, in_={"g-test", "pearson"})
...     @property
...     def aggr_cols(self) -> tea_tasting.aggr.AggrCols:
...         return tea_tasting.aggr.AggrCols(
...             has_count=True,
...             mean_cols=(self.column,),
...         )
...     def analyze_aggregates(
...         self,
...         control: tea_tasting.aggr.Aggregates,
...         treatment: tea_tasting.aggr.Aggregates,
...     ) -> ProportionResult:
...         observed = np.empty(shape=(2, 2), dtype=np.int64)
...         observed[0, 0] = round(control.count() * control.mean(self.column))
...         observed[1, 0] = control.count() - observed[0, 0]
...         observed[0, 1] = round(treatment.count() * treatment.mean(self.column))
...         observed[1, 1] = treatment.count() - observed[0, 1]
...         res = scipy.stats.chi2_contingency(
...             observed=observed,
...             correction=self.correction,
...             lambda_=int(self.method == "pearson"),
...         )
...         return ProportionResult(
...             control=control.mean(self.column),
...             treatment=treatment.mean(self.column),
...             effect_size=treatment.mean(self.column) - control.mean(self.column),
...             rel_effect_size=treatment.mean(self.column)/control.mean(self.column) - 1,
...             pvalue=res.pvalue,
...             statistic=res.statistic,
...         )
... 
```

--------------------------------

### Perform Benjamini-Yekutieli procedure

Source: https://github.com/e10v/tea-tasting/blob/main/docs/multiple-testing.md

Set the arbitrary_dependence parameter to True to perform the Benjamini-Yekutieli procedure for cases with arbitrary dependence between hypotheses.

```pycon
>>> tt.adjust_fdr(results, metrics, arbitrary_dependence=True)
comparison metric           control treatment rel_effect_size  pvalue pvalue_adj
(0, 1)     orders_per_user    0.530     0.573            8.0%   0.118      0.245
(0, 1)     revenue_per_user    5.24      5.99             14%  0.0211     0.0592
(0, 2)     orders_per_user    0.530     0.594             12%  0.0213     0.0592
(0, 2)     revenue_per_user    5.24      6.25             19% 0.00218     0.0182
```

--------------------------------

### Temporarily set global options with config_context

Source: https://github.com/e10v/tea-tasting/blob/main/docs/user-guide.md

Use config_context to apply configuration changes only within a specific block, ensuring global settings revert afterward.

```pycon
>>> with tt.config_context(equal_var=True, use_t=False):
...     experiment_within_context = tt.Experiment(
...         sessions_per_user=tt.Mean("sessions"),
...         orders_per_session=tt.RatioOfMeans("orders", "sessions"),
...         orders_per_user=tt.Mean("orders"),
...         revenue_per_user=tt.Mean("revenue"),
...     )
... 
>>> orders_per_user_context = experiment_with_config.metrics["orders_per_user"]
>>> print(
...     f"global_config.equal_var: {tt.get_config('equal_var')}\n"
...     f"global_config.use_t: {tt.get_config('use_t')}\n\n"
...     f"orders_per_user_context.equal_var: {orders_per_user_context.equal_var}\n"
...     f"orders_per_user_context.use_t: {orders_per_user_context.use_t}"
... )
global_config.equal_var: False
global_config.use_t: True
<BLANKLINE>
orders_per_user_context.equal_var: True
orders_per_user_context.use_t: False
```