### Setup Development Environment with uv

Source: https://github.com/dustalov/evalica/blob/master/README.md

Installs Evalica and its development dependencies using the uv package manager, including setting up a virtual environment and activating it.

```console
$ uv venv
$ uv pip install maturin
$ source .venv/bin/activate
$ maturin develop --uv --extras dev,docs,gradio
```

--------------------------------

### Importing Dependencies

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Initial setup for the environment including Evalica, pandas, and plotting libraries.

```python
from __future__ import annotations

from typing import TYPE_CHECKING

import evalica
import numpy as np
import pandas as pd
import plotly.express as px
from tqdm.auto import trange

if TYPE_CHECKING:
    from plotly.graph_objects import Figure

%config InlineBackend.figure_formats = ['svg']
```

--------------------------------

### Setup Development Environment without uv

Source: https://github.com/dustalov/evalica/blob/master/README.md

Installs Evalica and its development dependencies using standard Python venv and pip, including setting up a virtual environment and activating it.

```console
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install maturin
$ maturin develop --extras dev,docs,gradio
```

--------------------------------

### Install Evalica dependencies

Source: https://github.com/dustalov/evalica/blob/master/coling2025/README.md

List of required Python packages for running the experiments.

```text
evalica==0.3.2
numpy==2.2.0
pandas==2.2.3
pyarrow==18.1.0
scikit-learn==1.6.0
tqdm==4.67.1
```

--------------------------------

### Checking Evalica Version

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Verify the installed version of the Evalica library.

```python
evalica.__version__
```

--------------------------------

### Load and Prepare Food Dataset

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Loads the food dataset from a CSV file and maps the 'winner' column to Evalica's Winner enum. Displays the first 5 rows.

```python
df_food = pd.read_csv(
    "https://raw.githubusercontent.com/dustalov/evalica/0893fd0f1e8107b2d62fd6c5816b55b417c1a050/food.csv",
    dtype=str,
)

df_food["winner"] = df_food["winner"].map(
    {
        "left": Winner.X,
        "right": Winner.Y,
        "tie": Winner.Draw,
    },
)

df_food.head(5)
```

--------------------------------

### CLI for Pairwise Comparison (Bradley-Terry)

Source: https://github.com/dustalov/evalica/blob/master/docs/index.md

Use the Evalica command-line interface to perform pairwise comparisons using the Bradley-Terry model. Input is a CSV file (`-i food.csv`), and the command is `pairwise bradley-terry`.

```console
$ evalica -i food.csv pairwise bradley-terry
```

--------------------------------

### Solver Selection

Source: https://context7.com/dustalov/evalica/llms.txt

Configures the solver backend, choosing between the fast Rust-based pyo3 implementation or the pure Python naive fallback.

```python
from evalica import bradley_terry, alpha, Winner, PYO3_AVAILABLE, SOLVER

# Check if Rust extension is available
print(f"Rust extension available: {PYO3_AVAILABLE}")
print(f"Default solver: {SOLVER}")

# Explicitly use naive (pure Python) solver
result = bradley_terry(
    ['A', 'B', 'C'],
    ['B', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw],
    solver='naive',
)

# Explicitly use pyo3 (Rust) solver - faster for large datasets
result = bradley_terry(
    ['A', 'B', 'C'],
    ['B', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw],
    solver='pyo3',
)
```

--------------------------------

### Initialize Evalica Wasm

Source: https://github.com/dustalov/evalica/blob/master/examples/wasm/index.html

Import and initialize the Evalica WebAssembly module. This must be awaited before using any Evalica functions.

```javascript
import init from './pkg/evalica.js';

async function run() {
  await init();
  // Wasm is loaded and ready
}
```

--------------------------------

### Set Up Event Listeners for UI Buttons

Source: https://github.com/dustalov/evalica/blob/master/examples/wasm/index.html

Assigns click event handlers to 'run-button' and 'alpha-button' to trigger respective methods. Also includes initial calls to run the methods.

```javascript
document.getElementById('run-button').onclick = runMethod;
document.getElementById('alpha-button').onclick = runAlpha;
runMethod();
runAlpha();
} catch (e) {
  document.getElementById('status').innerText = 'Error: ' + e;
  console.error(e);
}
}
```

--------------------------------

### Pairwise Ranking CLI

Source: https://github.com/dustalov/evalica/blob/master/README.md

Executes pairwise ranking using the Evalica command-line interface with a CSV input file. The output includes item scores and ranks.

```console
$ evalica -i food.csv pairwise bradley-terry
item,score,rank
Tacos,2.509025136024378,1
Sushi,1.1011561298265815,2
Burger,0.8549063627182466,3
Pasta,0.7403814336665869,4
Pizza,0.5718366915548537,5
```

--------------------------------

### Downloading Arena Data

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Fetch the public battle dataset using curl.

```bash
!curl -LOC - 'https://storage.googleapis.com/arena_external_data/public/clean_battle_20240814_public.json'
```

--------------------------------

### Run Pairwise Comparison Methods

Source: https://github.com/dustalov/evalica/blob/master/examples/wasm/index.html

Execute various pairwise comparison algorithms like Counting, Bradley-Terry, and PageRank. Requires pre-defined items, comparison data (xs, ys, winners), and weights.

```javascript
const items = ['pizza', 'burger', 'sushi'];
const xs = new Uint32Array([0, 1, 0]);
const ys = new Uint32Array([1, 2, 2]);
const winners = new Uint8Array([1, 2, 0]); // 1=X, 2=Y, 0=Draw
const weights = new Float64Array([1.0, 1.0, 1.0]);
const total = items.length;
const method = document.getElementById('method-select').value;
let results;
const win_weight = 1.0;
const tie_weight = 0.5;

switch (method) {
  case 'counting':
    results = counting(xs, ys, winners, weights, total, win_weight, tie_weight);
    break;
  case 'average_win_rate':
    results = averageWinRate(xs, ys, winners, weights, total, win_weight, tie_weight);
    break;
  case 'bradley_terry':
    results = bradleyTerry(xs, ys, winners, weights, total, win_weight, tie_weight, 1e-6, 100);
    break;
  case 'newman':
    results = newman(xs, ys, winners, weights, total, 1.0, 1e-6, 100);
    break;
  case 'eigen':
    results = eigen(xs, ys, winners, weights, total, win_weight, tie_weight, 1e-6, 100);
    break;
  case 'pagerank':
    results = pagerank(xs, ys, winners, weights, total, win_weight, tie_weight, 0.85, 1e-6, 100);
    break;
  case 'elo':
    results = elo(xs, ys, winners, weights, total, 1000.0, 10.0, 400.0, 32.0, win_weight, tie_weight);
    break;
}
```

--------------------------------

### Command-Line Interface Usage

Source: https://context7.com/dustalov/evalica/llms.txt

Executes ranking methods and reliability metrics directly from the terminal.

```bash
# Pairwise ranking with Bradley-Terry
evalica -i comparisons.csv pairwise bradley-terry
# Output:
# item,score,rank
# Tacos,2.509025136024378,1
# Sushi,1.1011561298265815,2
# Burger,0.8549063627182466,3

# Available pairwise methods: counting, average-win-rate, bradley-terry,
# elo, eigen, pagerank, newman

# Elo ranking
evalica -i comparisons.csv pairwise elo

# PageRank
evalica -i comparisons.csv pairwise pagerank

# Save output to file
evalica -i comparisons.csv -o results.csv pairwise bradley-terry

# Krippendorff's alpha with different distance metrics
evalica -i ratings.csv alpha --distance=nominal
evalica -i ratings.csv alpha --distance=ordinal
evalica -i ratings.csv alpha --distance=interval
evalica -i ratings.csv alpha --distance=ratio

# Output:
# metric,value
# alpha,0.743421052631579
# observed,7.999999999999999
# expected,31.179487179487182
```

--------------------------------

### Generate Comparison Matrices

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Creates win and tie matrices from the indexed pairwise comparison data. These matrices summarize the outcomes of all comparisons.

```python
matrices = evalica.matrices(df_food["left_id"], df_food["right_id"], df_food["winner"], index)
```

```python
pd.DataFrame(matrices.win_matrix, index=index, columns=index)  # win matrix
```

```python
pd.DataFrame(matrices.tie_matrix, index=index, columns=index)  # tie matrix
```

--------------------------------

### Visualizing Win Matrices

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Generate heatmaps for raw win counts and win fractions.

```python
%%time
xs_indexed, ys_indexes, index = evalica.indexing(df_arena["model_a"], df_arena["model_b"])

matrices = evalica.matrices(
    xs_indexed,
    ys_indexes,
    df_arena["winner"],
    index,
)

df_matrix = pd.DataFrame.from_records(
    matrices.win_matrix,
    index=index,
    columns=index,
)

visualize(
    df_matrix.loc[
        average_win_scores_no_ties.index[:15].tolist(),
        average_win_scores_no_ties.index[:15].tolist(),
    ],
    title="Win Counts",
)
```

```python
df_matrix_proba = df_matrix / (df_matrix + df_matrix.T)
df_matrix_proba = df_matrix_proba.loc[
    average_win_scores_no_ties.index[:15].tolist(),
    average_win_scores_no_ties.index[:15].tolist(),
]

visualize(df_matrix_proba, title="Win Fractions")
```

--------------------------------

### Import Libraries

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Imports necessary libraries for Evalica, pandas, and plotly. Sets up the plotting backend for SVG output.

```python
import evalica
import pandas as pd
import plotly.express as px
from evalica import Winner, alpha_bootstrap, bootstrap, bradley_terry

%config InlineBackend.figure_formats = ['svg']
```

--------------------------------

### Aggregate pairwise comparisons with Bradley-Terry

Source: https://github.com/dustalov/evalica/blob/master/docs/migration.md

Compare Crowd-Kit's DataFrame-based approach with Evalica's flexible input options using Winner constants.

```pycon
>>> import pandas as pd
>>> from crowdkit.aggregation import BradleyTerry
>>> df = pd.DataFrame(
...     [
...         ['item1', 'item2', 'item1'],
...         ['item3', 'item2', 'item2']
...     ],
...     columns=['left', 'right', 'label']
... )
>>> agg_bt = BradleyTerry(n_iter=100).fit_predict(df)
```

```pycon
>>> import pandas as pd
>>> from evalica import bradley_terry, Winner
>>> df = pd.DataFrame(
...     [
...         ['item1', 'item2', Winner.X],
...         ['item2', 'item3', Winner.Y]
...     ],
...     columns=['left', 'right', 'label']
... )
>>> scores = bradley_terry(df['left'], df['right'], df['label'], limit=100)
```

```pycon
>>> from evalica import bradley_terry, Winner
>>> scores = bradley_terry(
...     ['item1', 'item2'],
...     ['item2', 'item3'],
...     [Winner.X, Winner.Y],
...     limit=100,
... )
```

--------------------------------

### CLI for Krippendorff's Alpha Calculation

Source: https://github.com/dustalov/evalica/blob/master/docs/index.md

Use the Evalica command-line interface to calculate Krippendorff's alpha. Input is a CSV file (`-i codings.csv`) in matrix format, and the command is `alpha --distance=nominal`.

```console
$ evalica -i codings.csv alpha --distance=nominal
```

--------------------------------

### Win and Tie Matrices

Source: https://context7.com/dustalov/evalica/llms.txt

Constructs win and tie matrices from pairwise comparison data.

```python
import pandas as pd
from evalica import indexing, matrices, Winner

# Prepare data
xs = ['pizza', 'burger', 'pizza', 'sushi']
ys = ['burger', 'sushi', 'sushi', 'pizza']
winners = [Winner.X, Winner.Y, Winner.Draw, Winner.X]

# Get indexed representations
xs_idx, ys_idx, index = indexing(xs, ys)

# Build matrices
result = matrices(xs_idx, ys_idx, winners, index)

# Convert to DataFrames for visualization
win_df = pd.DataFrame(result.win_matrix, index=index, columns=index)
tie_df = pd.DataFrame(result.tie_matrix, index=index, columns=index)

print("Win Matrix:")
print(win_df)
# Shows wins[i,j] = number of times i beat j

print("\nTie Matrix:")
print(tie_df)
# Shows ties[i,j] = number of ties between i and j (symmetric)
```

--------------------------------

### Calculating Average Win Rates

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Compute win rates with and without tie weights.

```python
%%time
average_win_rates = evalica.average_win_rate(
    df_arena["model_a"],
    df_arena["model_b"],
    df_arena["winner"],
)

average_win_rates.scores.to_frame()
```

```python
%%time
average_win_rates_no_ties = evalica.average_win_rate(
    df_arena["model_a"],
    df_arena["model_b"],
    df_arena["winner"],
    tie_weight=0,  # LMSYS' leaderboard excludes ties
)

average_win_scores_no_ties = average_win_rates_no_ties.scores
average_win_scores_no_ties.to_frame()
```

--------------------------------

### Compute Krippendorff's alpha

Source: https://github.com/dustalov/evalica/blob/master/docs/migration.md

Migrate from NLTK's AnnotationTask to Evalica's DataFrame-based alpha function.

```pycon
>>> from nltk.metrics import binary_distance
>>> from nltk.metrics.agreement import AnnotationTask
>>> data = [
...     ('coder1', 'item1', 1),
...     ('coder1', 'item2', 2),
...     ('coder2', 'item1', 1),
...     ('coder2', 'item2', 3),
...     ('coder3', 'item1', 2),
...     ('coder3', 'item2', 2),
... ]
>>> task = AnnotationTask(data, distance=binary_distance)
>>> task.alpha()
```

```pycon
>>> import pandas as pd
>>> from evalica import alpha
>>> df = pd.DataFrame(
...     [[1, 2], [1, 3], [2, 2]],
... )
>>> result = alpha(df, distance="nominal")
>>> result.alpha
```

--------------------------------

### Generate chatbot_arena.csv

Source: https://github.com/dustalov/evalica/blob/master/coling2025/README.md

Executes the chatbot_arena module to generate the corresponding CSV file.

```shell
python3 -m chatbot_arena
```

--------------------------------

### Generate scale.csv

Source: https://github.com/dustalov/evalica/blob/master/coling2025/README.md

Executes the scale_data and scale_compute modules to generate the scale.csv file.

```shell
python3 -m scale_data
python3 -m scale_compute
```

--------------------------------

### Evalica Core Components

Source: https://github.com/dustalov/evalica/blob/master/docs/utils.md

Overview of the primary classes and constants available in the evalica package.

```APIDOC
## Evalica Core Components

### Description
This module provides the core functionality for evaluation, including bootstrapping, ranking methods, and solver configurations.

### Components
- **BootstrapResult**: Class representing the result of a bootstrap operation.
- **Winner**: Class representing a winner in a comparison.
- **WINNERS**: Constant containing winner definitions.
- **PYO3_AVAILABLE**: Boolean flag indicating if PyO3 bindings are available.
- **SOLVER**: Constant for default solver configuration.
- **bootstrap**: Function to perform bootstrapping.
- **indexing**: Module for indexing operations.
- **matrices**: Module for matrix-based evaluation.
- **MatricesResult**: Class representing the result of matrix operations.
- **RankingMethod**: Enum or class defining supported ranking methods.
- **Result**: Class representing a general evaluation result.
- **SolverName**: Enum or class defining available solver names.
- **pairwise_frame**: Function to generate a pairwise comparison frame.
- **pairwise_scores**: Function to calculate pairwise scores.
- **__version__**: String representing the current library version.
```

--------------------------------

### Newman Model

Source: https://github.com/dustalov/evalica/blob/master/docs/bradley-terry.md

Documentation for the Newman ranking model implementation.

```APIDOC
## Newman Model

### Description
Provides the implementation for the Newman ranking model.

### Module
`evalica.newman`
```

--------------------------------

### Krippendorff's Alpha Input Format

Source: https://context7.com/dustalov/evalica/llms.txt

Defines the matrix format for alpha computation and demonstrates loading it in Python.

```csv
1,2,3,1
1,2,3,2
2,3,3,2
1,2,,1
```

```python
import pandas as pd
from evalica import alpha

# Read ratings matrix (no header, observers as rows)
df = pd.read_csv('ratings.csv', header=None, dtype=str)

# Missing values are handled automatically
result = alpha(df, distance='nominal')
print(f"Alpha: {result.alpha:.4f}")
```

--------------------------------

### Bradley-Terry Model

Source: https://github.com/dustalov/evalica/blob/master/docs/bradley-terry.md

Documentation for the Bradley-Terry ranking model implementation.

```APIDOC
## Bradley-Terry Model

### Description
Provides the implementation for the Bradley-Terry model used for pairwise comparison ranking.

### Module
`evalica.bradley_terry`
```

--------------------------------

### Compute Krippendorff's Alpha and Confidence Intervals

Source: https://github.com/dustalov/evalica/blob/master/README.md

Demonstrates calculating Krippendorff's alpha with nominal distance and its bootstrap confidence intervals using pandas DataFrames. Ensure data is structured with rows as raters and columns as units.

```python
import pandas as pd
from evalica import alpha
data = pd.DataFrame([
    [1, 1, None, 1],
    [2, 2, 3, 2],
    [3, 3, 3, 3],
    [3, 3, 3, 3],
    [2, 2, 2, 2],
    [1, 2, 3, 4],
    [4, 4, 4, 4],
    [1, 1, 2, 1],
    [2, 2, 2, 2],
    [None, 5, 5, 5],
    [None, None, 1, 1],
]).T
result = alpha(data, distance='nominal')
print(result.alpha)
```

```python
from evalica import alpha_bootstrap
bootstrap_result = alpha_bootstrap(data, distance='nominal', n_resamples=1000, random_state=42)
print((bootstrap_result.low, bootstrap_result.high))
```

--------------------------------

### Krippendorff's Alpha CLI

Source: https://github.com/dustalov/evalica/blob/master/README.md

Calculates Krippendorff's alpha using the command-line interface with a CSV file containing ratings in a matrix format. Specify the distance metric using the --distance flag.

```console
$ evalica -i codings.csv alpha --distance=nominal
metric,value
alpha,0.743421052631579
observed,7.999999999999999
expected,31.179487179487182
```

--------------------------------

### Preprocessing Arena Data

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Load and clean the battle dataset, mapping winners to Evalica types.

```python
df_arena = pd.read_json("clean_battle_20240629_public.json")
df_arena = df_arena[df_arena["anony"]]
df_arena = df_arena[df_arena["dedup_tag"].apply(lambda x: x.get("sampled", False))]
df_arena["winner"] = df_arena["winner"].map(
    {
        "model_a": evalica.Winner.X,
        "model_b": evalica.Winner.Y,
        "tie": evalica.Winner.Draw,
        "tie (bothbad)": evalica.Winner.Draw,
    },
)
df_arena = df_arena[~df_arena["winner"].isna()]
df_arena
```

--------------------------------

### Compute and Visualize Alpha Bootstrap Confidence Intervals

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Calculates confidence intervals for Krippendorff's alpha using bootstrapping and visualizes the distribution with a histogram. Requires Plotly.

```python
alpha_bootstrap_result = alpha_bootstrap(
    df_codings,
    distance="nominal",
    n_resamples=1000,
    confidence_level=0.95,
    random_state=42,
)

fig = px.histogram(
    alpha_bootstrap_result.distribution,
    nbins=50,
    title="Krippendorff's Alpha Bootstrap Distribution",
    labels={"value": "Alpha", "count": "Frequency"},
)

fig.add_vline(x=alpha_bootstrap_result.alpha, line_dash="dash", line_color="red", annotation_text="Point Estimate")
fig.add_vline(x=alpha_bootstrap_result.low, line_dash="dot", line_color="blue", annotation_text="Lower Bound")
fig.add_vline(x=alpha_bootstrap_result.high, line_dash="dot", line_color="blue", annotation_text="Upper Bound")
fig.show()
```

--------------------------------

### Pairwise Comparison CSV Format

Source: https://context7.com/dustalov/evalica/llms.txt

Defines the expected CSV structure for pairwise methods and demonstrates how to load and map it in Python.

```csv
left,right,winner
pizza,burger,left
burger,sushi,right
pizza,sushi,tie
tacos,pizza,left
sushi,tacos,right
```

```python
import pandas as pd
from evalica import Winner

# Read and convert CSV data
df = pd.read_csv('comparisons.csv')

# Map winner strings to Winner enum
winner_map = {'left': Winner.X, 'right': Winner.Y, 'tie': Winner.Draw}
df['winner'] = df['winner'].map(winner_map)

# Use with any ranking method
from evalica import bradley_terry
result = bradley_terry(df['left'], df['right'], df['winner'])
```

--------------------------------

### Calculating Elo Ratings

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Compute Elo scores and visualize the resulting win probabilities.

```python
%%time
elo = evalica.elo(
    df_arena["model_a"],
    df_arena["model_b"],
    df_arena["winner"],
)

elo.scores.to_frame()
```

```python
df_elo = evalica.pairwise_frame(elo.scores[:15])

visualize(df_elo, title="Elo Win Probabilities")
```

--------------------------------

### Calculate Krippendorff's Alpha and Confidence Interval

Source: https://github.com/dustalov/evalica/blob/master/examples/wasm/index.html

Calculates Krippendorff's alpha, observed and expected disagreement, and a confidence interval using bootstrap samples. Updates the UI with the results or an error message.

```javascript
load.slice(3).sort((a, b) => a - b);
const alphaTail = (1.0 - confidenceLevel) / 2.0;
const low = quantile(distribution, alphaTail);
const high = quantile(distribution, 1.0 - alphaTail);
let outputText = `Krippendorff's alpha: ${alphaValue.toFixed(6)}\n
`;
outputText += `Observed disagreement: ${observed.toFixed(6)}\n
`;
outputText += `Expected disagreement: ${expected.toFixed(6)}\n
`;
outputText += `${(confidenceLevel * 100).toFixed(0)}% CI: [${low.toFixed(6)}, ${high.toFixed(6)}]\n
`;
outputText += `Bootstrap samples: ${distribution.length}\n
`;
document.getElementById('alpha-output').innerText = outputText;
document.getElementById('alpha-status').innerText = 'Done!';
} catch (err) {
  document.getElementById('alpha-status').innerText = 'Error: ' + err;
  console.error(err);
}
```

--------------------------------

### Evalica Exception and Warning Classes

Source: https://github.com/dustalov/evalica/blob/master/docs/errors.md

Overview of the error handling classes provided by the evalica module.

```APIDOC
## Evalica Exception Classes

### Description
These classes represent specific error conditions that may be raised during the execution of evalica functions.

- **InsufficientRatingsError**: Raised when the provided input data lacks the minimum number of ratings required for computation.
- **LengthMismatchError**: Raised when input sequences or arrays do not match in length where equality is expected.
- **ScoreDimensionError**: Raised when the dimensionality of the score input is invalid for the requested operation.
- **SolverError**: Raised when the underlying solver fails to converge or encounters a computational error.
- **UnknownDistanceError**: Raised when an unsupported or unrecognized distance metric is specified.

## Evalica Warning Classes

### Description
These classes represent non-fatal issues that may occur during execution.

- **RustExtensionWarning**: Issued when there are issues or specific conditions related to the Rust extension component of the library.
```

--------------------------------

### Bootstrapping Bradley-Terry

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Perform bootstrap resampling to estimate confidence intervals for Bradley-Terry ratings.

```python
%%time
BOOTSTRAP_ROUNDS = 10

bt_bootstrap = []

for seed in trange(BOOTSTRAP_ROUNDS, desc="Bootstrap"):
    df_sample = df_arena.sample(frac=1.0, replace=True, random_state=seed)

    result = evalica.bradley_terry(
        df_sample["model_a"],
        df_sample["model_b"],
        df_sample["winner"],
        index=index,  # we safely save some time by not reindexing the elements
    )

    bt_bootstrap.append(result.scores)

df_bootstrap = pd.DataFrame(bt_bootstrap)
df_bootstrap = df_bootstrap[df_bootstrap.median().index]

df_bootstrap
```

```python
df_bootstrap.median().to_frame(name="bradley_terry")
```

```python
df_bootstrap_ci = (
    pd.DataFrame(
        {
            "lower": df_bootstrap.quantile(0.025),
            "rating": df_bootstrap.quantile(0.5),
            "upper": df_bootstrap.quantile(0.975),
        },
    )
    .reset_index(names="model")
    .sort_values("rating", ascending=False)
)
```

--------------------------------

### Perform Bootstrap Resampling for Bradley-Terry Model

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Conducts bootstrap resampling on the Bradley-Terry model to estimate the distribution of scores. Requires `bootstrap` function from Evalica.

```python
bootstrap_result = bootstrap(
    bradley_terry,
    df_food["left"],
    df_food["right"],
    df_food["winner"],
    n_resamples=10,
    random_state=42,
)

df_melted = bootstrap_result.distribution.melt(var_name="Item", value_name="Score")

df_melted.head(5)
```

--------------------------------

### Calculating Bradley-Terry Ratings

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Compute Bradley-Terry scores and visualize the resulting win probabilities.

```python
%%time
bt = evalica.bradley_terry(
    df_arena["model_a"],
    df_arena["model_b"],
    df_arena["winner"],
)

bt.scores.to_frame()
```

```python
df_bt = evalica.pairwise_frame(bt.scores[:15])

visualize(df_bt, title="Bradley–Terry Win Probabilities")
```

--------------------------------

### Compute Elo ratings with Python

Source: https://github.com/dustalov/evalica/blob/master/README.md

Calculates Elo scores from pairwise comparison data using the elo function and Winner enum.

```pycon
>>> from evalica import elo, Winner
>>> result = elo(
...     ['pizza', 'burger', 'pizza'],
...     ['burger', 'sushi', 'sushi'],
...     [Winner.X, Winner.Y, Winner.Draw],
... )
>>> result.scores
pizza     1014.972058
burger     970.647200
sushi     1014.380742
Name: elo, dtype: float64
```

--------------------------------

### Counting Method

Source: https://context7.com/dustalov/evalica/llms.txt

Calculates weighted win counts for items based on pairwise comparison results.

```python
from evalica import counting, Winner

result = counting(
    ['A', 'B', 'C', 'A', 'B'],
    ['B', 'C', 'A', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Y],
    win_weight=1.0,  # weight for wins
    tie_weight=0.5,  # weight for ties
)

print(result.scores)
# Shows raw win counts weighted by parameters
```

--------------------------------

### Generate rust_python.csv

Source: https://github.com/dustalov/evalica/blob/master/coling2025/README.md

Executes the rust_python module to generate the corresponding CSV file.

```shell
python3 -m rust_python
```

--------------------------------

### Convert Ranking Scores to Pairwise Probability Matrix

Source: https://context7.com/dustalov/evalica/llms.txt

Uses the Bradley-Terry model to generate a win probability matrix from comparison data.

```python
from evalica import bradley_terry, pairwise_frame, Winner

# Get Bradley-Terry scores
result = bradley_terry(
    ['A', 'B', 'C', 'A', 'B'],
    ['B', 'C', 'A', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Y],
)

# Convert to pairwise win probability matrix
pairwise_df = pairwise_frame(result.scores)
print(pairwise_df)
# Shows P(row beats column) based on Bradley-Terry model
# Values on diagonal are 0.5, matrix rows sum to ~0.5 * n
```

--------------------------------

### Bootstrap Confidence Intervals for Rankings

Source: https://context7.com/dustalov/evalica/llms.txt

Computes bootstrap confidence intervals for ranking methods like Bradley-Terry.

```python
from evalica import bootstrap, bradley_terry, Winner

result = bootstrap(
    method=bradley_terry,
    xs=['A', 'B', 'C', 'A', 'B', 'C'],
    ys=['B', 'C', 'A', 'C', 'A', 'B'],
    winners=[Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Y, Winner.X],
    n_resamples=1000,
    confidence_level=0.95,
    bootstrap_method='BCa',  # 'percentile', 'basic', or 'BCa'
    random_state=42,
)

print("Point estimates:")
print(result.result.scores)

print("\nConfidence intervals:")
for item in result.index:
    print(f"{item}: [{result.low[item]:.4f}, {result.high[item]:.4f}]")

print("\nStandard errors:")
print(result.stderr)

# Access full bootstrap distribution
print(f"\nDistribution shape: {result.distribution.shape}")
```

--------------------------------

### Visualize Pairwise Comparison Results

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Generates an image plot visualizing the pairwise comparison results, showing the fraction of wins between items. Requires Plotly.

```python
fig = px.imshow(df_bt_pairwise, color_continuous_scale="RdBu", text_auto=".2f")
fig.update_layout(xaxis_title="Loser", yaxis_title="Winner", xaxis_side="top")
fig.update_traces(hovertemplate="Winner: %{y}<br>Loser: %{x}<br>Fraction of Wins: %{z}")
fig.show()
```

--------------------------------

### Calculate Krippendorff's Alpha and CI

Source: https://github.com/dustalov/evalica/blob/master/examples/wasm/index.html

Compute Krippendorff's alpha with a bootstrap confidence interval. Requires rater data, unit data, distance metric, number of resamples, and confidence level.

```javascript
const distance = document.getElementById('distance-select').value;
const nResamples = Number.parseInt(document.getElementById('alpha-resamples').value, 10);
const confidenceLevel = Number.parseFloat(document.getElementById('alpha-confidence-level').value);

const codes = new BigInt64Array([
  0n, 0n, -1n, 0n, // Unit 1
  1n, 1n, 2n, 1n, // Unit 2
  2n, 2n, 2n, 2n, // Unit 3
  2n, 2n, 2n, 2n, // Unit 4
  1n, 1n, 1n, 1n, // Unit 5
  0n, 1n, 2n, 3n, // Unit 6
  3n, 3n, 3n, 3n, // Unit 7
  0n, 0n, 1n, 0n, // Unit 8
  1n, 1n, 1n, 1n, // Unit 9
  -1n, 4n, 4n, 4n, // Unit 10
  -1n, -1n, 0n, 0n // Unit 11
]);
const uniqueValues = new Float64Array([1.0, 2.0, 3.0, 4.0, 5.0]);
const nUnits = 11;
const nRaters = 4;
const seed = 12345n;

const pointEstimate = alpha(codes, uniqueValues, nUnits, nRaters, distance);
const [alphaValue, observed, expected] = Array.from(pointEstimate);

const bootstrap = alphaBootstrap(codes, uniqueValues, nUnits, nRaters, distance, nResamples, seed);
const payload = Array.from(bootstrap);
```

--------------------------------

### Compute Elo Ratings

Source: https://context7.com/dustalov/evalica/llms.txt

Calculate Elo scores for pairwise comparisons with support for custom K-factor, initial rating, base, and scale parameters.

```python
from evalica import elo, Winner

# Basic usage with food preference data
result = elo(
    ['pizza', 'burger', 'pizza'],      # left items
    ['burger', 'sushi', 'sushi'],      # right items
    [Winner.X, Winner.Y, Winner.Draw], # outcomes
)

print(result.scores)
# pizza     1014.972058
# burger     970.647200
# sushi     1014.380742
# Name: elo, dtype: float64

# Advanced usage with custom parameters
result = elo(
    xs=['A', 'B', 'A', 'C'],
    ys=['B', 'C', 'C', 'A'],
    winners=[Winner.X, Winner.X, Winner.Y, Winner.Draw],
    initial=1500.0,  # starting rating
    k=32.0,          # K-factor
    base=10.0,       # base of exponent
    scale=400.0,     # scale factor
)

print(f"Scores: {result.scores.to_dict()}")
print(f"Initial rating: {result.initial}")
print(f"K-factor: {result.k}")
```

--------------------------------

### Compute Elo Scores for Pairwise Comparisons

Source: https://github.com/dustalov/evalica/blob/master/docs/index.md

Use the `elo` function to compute Elo scores from pairwise comparison data. Requires importing `elo` and `Winner` from `evalica`. The function takes lists for 'Item X', 'Item Y', and 'Winner' (Winner.X, Winner.Y, or Winner.Draw).

```python
from evalica import elo, Winner
result = elo(
    ['pizza', 'burger', 'pizza'],
    ['burger', 'sushi', 'sushi'],
    [Winner.X, Winner.Y, Winner.Draw],
 )
result.scores
```

--------------------------------

### Elo Rating API

Source: https://context7.com/dustalov/evalica/llms.txt

Computes Elo scores for pairwise comparisons using the classic chess rating algorithm.

```APIDOC
## Elo Rating

### Description
Computes Elo scores for pairwise comparisons using the classic chess rating algorithm with configurable K-factor, initial rating, base, and scale parameters.

### Parameters
#### Request Body
- **xs** (list) - Required - List of left items in comparisons
- **ys** (list) - Required - List of right items in comparisons
- **winners** (list) - Required - List of Winner enum values (X, Y, or Draw)
- **initial** (float) - Optional - Starting rating (default: 1000.0)
- **k** (float) - Optional - K-factor (default: 32.0)
- **base** (float) - Optional - Base of exponent (default: 10.0)
- **scale** (float) - Optional - Scale factor (default: 400.0)

### Response
- **scores** (pandas.Series) - Calculated Elo scores
- **initial** (float) - Initial rating used
- **k** (float) - K-factor used
```

--------------------------------

### Alpha Bootstrap Confidence Intervals

Source: https://context7.com/dustalov/evalica/llms.txt

Computes bootstrap confidence intervals for Krippendorff's alpha using KALPHA-style resampling.

```python
import pandas as pd
from evalica import alpha_bootstrap

data = pd.DataFrame([
    [1, 2, 1],
    [1, 2, 2],
    [2, 3, 2],
    [1, 2, 1],
]).T

result = alpha_bootstrap(
    data,
    distance='nominal',
    n_resamples=5000,
    confidence_level=0.95,
    random_state=42,
)

print(f"Alpha: {result.alpha:.4f}")
print(f"95% CI: [{result.low:.4f}, {result.high:.4f}]")
print(f"Bootstrap samples: {result.n_resamples}")

# Access full distribution for visualization
print(f"Distribution mean: {result.distribution.mean():.4f}")
print(f"Distribution std: {result.distribution.std():.4f}")
```

--------------------------------

### Visualize Bootstrap Score Distribution

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Creates a box plot visualizing the distribution of scores obtained from the bootstrap resampling of the Bradley-Terry model. Requires Plotly.

```python
fig = px.box(df_melted, x="Score", y="Item", color="Item", title="Bradley–Terry Bootstrap Scores")
fig.update_traces(hovertemplate="<b>%{y}</b><br>Score: %{x:.3f}<extra></extra>")
fig.show()
```

--------------------------------

### Create Pairwise Frame from Bradley-Terry Scores

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Transforms the Bradley-Terry model scores into a pairwise comparison frame for easier visualization or further analysis.

```python
df_bt_pairwise = evalica.pairwise_frame(bt_result.scores)

df_bt_pairwise
```

--------------------------------

### Apply Weights to Comparisons

Source: https://context7.com/dustalov/evalica/llms.txt

Allows weighting individual comparisons to handle importance or repeated data, supporting both lists and numpy arrays.

```python
from evalica import bradley_terry, Winner
import numpy as np

# Weights can emphasize certain comparisons
result = bradley_terry(
    xs=['A', 'B', 'C', 'A'],
    ys=['B', 'C', 'A', 'C'],
    winners=[Winner.X, Winner.Y, Winner.Draw, Winner.X],
    weights=[1.0, 2.0, 1.0, 3.0],  # weight each comparison
)

print(result.scores)

# Also works with numpy arrays
weights = np.array([1.0, 2.0, 1.0, 3.0])
result = bradley_terry(
    xs=['A', 'B', 'C', 'A'],
    ys=['B', 'C', 'A', 'C'],
    winners=[Winner.X, Winner.Y, Winner.Draw, Winner.X],
    weights=weights,
)
```

--------------------------------

### Create Indexed IDs for Pairwise Comparisons

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Generates unique IDs for the left and right items in each pairwise comparison, along with an index for the entire dataset.

```python
df_food["left_id"], df_food["right_id"], index = evalica.indexing(df_food["left"], df_food["right"])
```

--------------------------------

### Calculate Scores using Elo Rating System

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Applies the Elo rating system to estimate item strengths, commonly used in competitive games.

```python
elo_result = evalica.elo(df_food["left"], df_food["right"], df_food["winner"])
elo_result.scores.to_frame()
```

--------------------------------

### Calculate Scores using Counting Method

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Computes scores based on a simple counting of wins, losses, and ties for each item.

```python
count_result = evalica.counting(df_food["left"], df_food["right"], df_food["winner"])
count_result.scores.to_frame()
```

--------------------------------

### Calculate Scores using Bradley-Terry Model

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Applies the Bradley-Terry model to estimate item strengths based on pairwise comparison outcomes.

```python
bt_result = evalica.bradley_terry(df_food["left"], df_food["right"], df_food["winner"])
bt_result.scores.to_frame()
```

--------------------------------

### Newman's Algorithm API

Source: https://context7.com/dustalov/evalica/llms.txt

Computes rankings using Newman's algorithm, extending Bradley-Terry to handle ties.

```APIDOC
## Newman's Algorithm

### Description
Computes rankings using Newman's algorithm which extends Bradley-Terry to better handle ties through an explicit tie parameter.

### Parameters
#### Request Body
- **xs** (list) - Required - List of left items
- **ys** (list) - Required - List of right items
- **winners** (list) - Required - List of Winner enum values
- **v_init** (float) - Optional - Initial tie parameter
- **tolerance** (float) - Optional - Convergence tolerance
- **limit** (int) - Optional - Max iterations

### Response
- **scores** (pandas.Series) - Calculated scores
- **v** (float) - Final tie parameter
- **v_init** (float) - Initial tie parameter
- **iterations** (int) - Number of iterations
```

--------------------------------

### Calculate Krippendorff's Alpha for Different Distances

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Computes Krippendorff's alpha for the codings dataset using 'nominal', 'ordinal', 'interval', and 'ratio' distance metrics. Requires the `alpha` function from Evalica.

```python
distances = ["nominal", "ordinal", "interval", "ratio"]
alpha_values = {dist: evalica.alpha(df_codings, distance=dist).alpha for dist in distances}  # type: ignore[arg-type]

pd.Series(alpha_values, name="alpha").to_frame()
```

--------------------------------

### Compute Eigenvalue-Based Rankings

Source: https://context7.com/dustalov/evalica/llms.txt

Rank items using the principal eigenvector of the win matrix for a spectral approach.

```python
from evalica import eigen, Winner

result = eigen(
    ['A', 'B', 'C', 'A', 'B'],
    ['B', 'C', 'A', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Y],
    tolerance=1e-6,
    limit=100,
)

print(result.scores)
print(f"Iterations: {result.iterations}")
```

--------------------------------

### Compute Newman's Algorithm Rankings

Source: https://context7.com/dustalov/evalica/llms.txt

Rank items using Newman's algorithm, which extends Bradley-Terry to handle ties explicitly via a tie parameter.

```python
from evalica import newman, Winner

# Newman handles ties explicitly with a tie parameter
result = newman(
    ['A', 'B', 'C', 'A', 'B'],
    ['B', 'C', 'A', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Draw],
    v_init=0.5,      # initial tie parameter
    tolerance=1e-6,
    limit=100,
)

print(result.scores)
print(f"Tie parameter v: {result.v}")
print(f"Initial v: {result.v_init}")
print(f"Iterations: {result.iterations}")
```

--------------------------------

### Defining Visualization Function

Source: https://github.com/dustalov/evalica/blob/master/Chatbot-Arena.ipynb

Helper function to create a heatmap visualization for pairwise data.

```python
def visualize(df_pairwise: pd.DataFrame, title: str | None = None) -> Figure:
    fig = px.imshow(df_pairwise, color_continuous_scale="RdBu", text_auto=".2f")

    fig.update_layout(
        title=title,
        title_x=0.5,
        title_y=0.075,
        xaxis_title="Loser",
        yaxis_title="Winner",
        xaxis_side="top",
        width=800,
        height=640,
    )

    fig.update_traces(hovertemplate="Winner: %{y}<br>Loser: %{x}<br>Fraction of Wins: %{z}")

    return fig
```

--------------------------------

### Custom Distance Functions

Source: https://context7.com/dustalov/evalica/llms.txt

Allows defining custom distance functions for Krippendorff's alpha calculation.

```python
import pandas as pd
from evalica import alpha

# Define a custom binary distance function
def binary_distance(a, b):
    return 0.0 if a == b else 1.0

# Define a custom squared difference distance
def squared_distance(a, b):
    return (float(a) - float(b)) ** 2

data = pd.DataFrame([
    [1, 2, 1],
    [1, 2, 2],
    [2, 3, 2],
]).T

result = alpha(data, distance=binary_distance)
print(f"Alpha with custom distance: {result.alpha:.4f}")
```

--------------------------------

### Calculate Scores using Average Win Rate

Source: https://github.com/dustalov/evalica/blob/master/docs/tutorial.ipynb

Calculates scores based on the average win rate for each item across all comparisons.

```python
avr_result = evalica.average_win_rate(df_food["left"], df_food["right"], df_food["winner"])
avr_result.scores.to_frame()
```

--------------------------------

### Compute Bradley-Terry Scores

Source: https://context7.com/dustalov/evalica/llms.txt

Perform maximum likelihood estimation for probabilistic ranking of items using the Bradley-Terry model.

```python
from evalica import bradley_terry, Winner

# Rank items based on pairwise comparisons
result = bradley_terry(
    ['Tacos', 'Sushi', 'Burger', 'Pasta', 'Pizza'],
    ['Sushi', 'Burger', 'Pasta', 'Pizza', 'Tacos'],
    [Winner.X, Winner.X, Winner.X, Winner.X, Winner.Y],
    tolerance=1e-6,  # convergence tolerance
    limit=100,       # max iterations
)

print(result.scores)
# Tacos     2.509025
# Sushi     1.101156
# Burger    0.854906
# Pasta     0.740381
# Pizza     0.571837

print(f"Converged in {result.iterations} iterations")
print(f"Tolerance: {result.tolerance}")
```

--------------------------------

### Calculate Krippendorff's alpha with pandas

Source: https://context7.com/dustalov/evalica/llms.txt

Computes inter-rater reliability using a pandas DataFrame with nominal distance and a naive solver.

```python
import pandas as pd
data = pd.DataFrame([[1, 2], [1, 3], [2, 2]]).T
result = alpha(data, distance='nominal', solver='naive')
```

--------------------------------

### Average Win Rate

Source: https://context7.com/dustalov/evalica/llms.txt

Computes the average pairwise win rate for each item across all comparisons.

```python
from evalica import average_win_rate, Winner

result = average_win_rate(
    ['A', 'B', 'C', 'A', 'B'],
    ['B', 'C', 'A', 'C', 'A'],
    [Winner.X, Winner.Y, Winner.Draw, Winner.X, Winner.Y],
    win_weight=1.0,
    tie_weight=0.5,
)

print(result.scores)
# Shows average win rate (0-1) for each item
```