### Install Ranx

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Install the Ranx library using pip. Note that the first run may take longer due to compilation.

```python
!
```

--------------------------------

### Install Ranx Library

Source: https://github.com/amenra/ranx/blob/master/notebooks/7_plot.ipynb

Install the ranx library using pip. Note that the first time ranx functions are run, they may take longer due to compilation.

```python
!pip install ranx
```

--------------------------------

### Install ranx

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Install the ranx library using pip. Note that the first run of ranx functions may require compilation.

```python
!pip install -U ranx
```

--------------------------------

### Install Ranx Package

Source: https://github.com/amenra/ranx/blob/master/README.md

Install the Ranx library using pip. Requires Python 3.8 or higher.

```bash
pip install ranx
```

--------------------------------

### Download and Compare Runs from Ranxhub

Source: https://github.com/amenra/ranx/blob/master/notebooks/6_ranxhub.ipynb

Download pre-computed runs from ranxhub and compare them against Qrels from ir-datasets using specified metrics. Ensure you have the 'ranx' library installed.

```python
from ranx import Qrels, Run, compare

runs = [
    Run.from_ranxhub("msmarco-passage/trec-dl-2020/ranxhub/ance"),
    Run.from_ranxhub("msmarco-passage/trec-dl-2020/ranxhub/bm25-pyserini"),
    Run.from_ranxhub("msmarco-passage/trec-dl-2020/ranxhub/distilbert-kd"),
    Run.from_ranxhub("msmarco-passage/trec-dl-2020/ranxhub/distilbert-kd-tasb"),
    Run.from_ranxhub("msmarco-passage/trec-dl-2020/ranxhub/unicoil"),
]

# Import Qrels from ir-datasets
qrels = Qrels.from_ir_datasets("msmarco-passage/trec-dl-2020/judged")

# "-l2" changes the relevance level for the binary metrics
# as by instructions https://trec.nist.gov/data/deep2020.html
compare(qrels, runs, ["ndcg@10", "map@1000-l2", "recall@1000-l2"])
```

--------------------------------

### Optimize Fusion and Get Optimization Report

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Optimize fusion parameters and retrieve a detailed report of all evaluated configurations. This helps in understanding the performance across different parameter settings.

```python
from ranx import fuse, evaluate, optimize_fusion

best_params, optimization_report = optimize_fusion(
    qrels=qrels,
    runs=[run_4, run_5],
    norm="min-max",
    method="wsum",
    metric="ndcg@100",
    return_optimization_report=True,
)

# The optimization results are saved in a OptimizationReport instance,
# which provides handy functionalities such as tabular formatting
optimization_report.to_table()
```

--------------------------------

### Create Qrels and Run from Dictionaries

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Demonstrates how to create Qrels and Run objects by converting Python dictionaries. This is useful for small, in-memory datasets.

```python
# The standard way of creating Qrels and Run is converting Python Dictionaries
qrels_dict = {"q_1": {"d_12": 5, "d_25": 3}, "q_2": {"d_11": 6, "d_22": 1}}

run_dict = {
    "q_1": {
        "d_12": 0.9,
        "d_23": 0.8,
        "d_25": 0.7,
        "d_36": 0.6,
        "d_32": 0.5,
        "d_35": 0.4,
    },
    "q_2": {
        "d_12": 0.9,
        "d_11": 0.8,
        "d_25": 0.7,
        "d_36": 0.6,
        "d_22": 0.5,
        "d_35": 0.4,
    },
}

qrels = Qrels(qrels_dict)
run = Run(run_dict)
```

--------------------------------

### Incrementally Build Qrels and Run Objects

Source: https://context7.com/amenra/ranx/llms.txt

Demonstrates how to incrementally build `Qrels` and `Run` objects by adding data query by query or in batches. This is useful for streaming data or when dealing with large datasets that cannot fit into memory.

```python
from ranx import Qrels, Run

# Incrementally build a Qrels
qrels = Qrels(name="streaming_qrels")
qrels.add(q_id="q_1", doc_ids=["d_1", "d_2"], scores=[2, 1])
qrels.add_multi(
    q_ids=["q_2", "q_3"],
    doc_ids=[["d_3", "d_4"], ["d_5"]],
    scores=[[3, 1], [2]]
)
qrels.add_score(q_id="q_1", doc_id="d_3", score=3)  # update/add single score

# Incrementally build a Run
run = Run(name="my_model")
run.add(q_id="q_1", doc_ids=["d_1", "d_2", "d_3"], scores=[0.9, 0.7, 0.5])
run.add_multi(
    q_ids=["q_2", "q_3"],
    doc_ids=[["d_4", "d_5"], ["d_6", "d_7"]],
    scores=[[0.8, 0.6], [0.95, 0.4]]
)

# Convert back to dict or DataFrame for downstream use
qrels_dict = qrels.to_dict()
run_df = run.to_dataframe()
```

--------------------------------

### Import Qrels and Run classes

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Import the necessary Qrels and Run classes from the ranx library.

```python
from ranx import Qrels, Run
```

--------------------------------

### Get Raw Plot Data as DataFrame

Source: https://github.com/amenra/ranx/blob/master/notebooks/7_plot.ipynb

Retrieve the raw data for the interpolated precision at recall graph as a pandas DataFrame. This allows for custom plotting with other libraries. Set `return_graph=False` to get the DataFrame.

```python
df = plot(qrels, runs, graph="iprec_at_recall", return_graph=False)
df
```

--------------------------------

### Create Qrels and Run Objects

Source: https://github.com/amenra/ranx/blob/master/README.md

Create Qrels and Run objects from Python dictionaries. Ensure the dictionary structure matches the expected format for queries and documents.

```python
from ranx import Qrels, Run

qrels_dict = { "q_1": { "d_12": 5, "d_25": 3 },
               "q_2": { "d_11": 6, "d_22": 1 } }

run_dict = { "q_1": { "d_12": 0.9, "d_23": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_32": 0.5, "d_35": 0.4  },
             "q_2": { "d_12": 0.9, "d_11": 0.8, "d_25": 0.7,
                      "d_36": 0.6, "d_22": 0.5, "d_35": 0.4  } }

qrels = Qrels(qrels_dict)
run = Run(run_dict)
```

--------------------------------

### Inspect Qrels

Source: https://context7.com/amenra/ranx/llms.txt

Inspect Qrels objects to get the number of queries, list query IDs, or convert to a Pandas DataFrame.

```python
# --- Inspect ---
print(len(qrels))            # number of queries
print(qrels.get_query_ids()) # list of query IDs
df = qrels.to_dataframe()    # convert to pandas DataFrame
```

--------------------------------

### Download Sample Data

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Downloads TREC formatted qrels and run files from a GitHub repository to a local 'notebooks/data' directory.

```python
import os
import requests

for file in ["qrels", "run_1", "run_2", "run_3", "run_4", "run_5"]:
    os.makedirs("notebooks/data", exist_ok=True)

    with open(f"notebooks/data/{file}.trec", "w") as f:
        master = f"https://raw.githubusercontent.com/AmenRa/ranx/master/notebooks/data/{file}.trec"
        f.write(requests.get(master).text)
```

--------------------------------

### Set and Get Qrels/Run names

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Modify or provide a name for Qrels and Run objects using the 'name' attribute or during initialization. The name is used in other ranx functionalities.

```python
# Qrels and Run have a `name` attribute which is used by other functionalities
# of the library, you can change a Qrels or a Run name by changing its value
# or providing it during creation (recommended)

qrels.name = "dataset_a"
run.name = "bm25"

print(qrels.name)
print(run.name)

qrels = Qrels(qrels_dict, name="dataset_a")
run = Run(run_dict, name="bm25")

print(qrels.name)
print(run.name)
```

--------------------------------

### Load Qrels and Runs from Files

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Loads Qrels and Run objects from TREC formatted files. Ensure the files are correctly path-ed.

```python
# Let's load qrels and runs from files and compare them
qrels = Qrels.from_file("notebooks/data/qrels.trec", kind="trec")

run_1 = Run.from_file("notebooks/data/run_1.trec", kind="trec")
run_2 = Run.from_file("notebooks/data/run_2.trec", kind="trec")
run_3 = Run.from_file("notebooks/data/run_3.trec", kind="trec")
run_4 = Run.from_file("notebooks/data/run_4.trec", kind="trec")
run_5 = Run.from_file("notebooks/data/run_5.trec", kind="trec")
```

--------------------------------

### Download Qrels files for import

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Download sample Qrels files in JSON and TREC formats from a remote URL to a local directory for subsequent import.

```python
# Download Qrels in JSON format and Qrels TREC-Style file
import os
import requests

os.makedirs("notebooks/data", exist_ok=True)

with open("notebooks/data/small_qrels.json", "w") as f:
    master = "https://raw.githubusercontent.com/AmenRa/ranx/master/notebooks/data/small_qrels.json"
    f.write(requests.get(master).text)

with open("notebooks/data/small_qrels.trec", "w") as f:
    master = "https://raw.githubusercontent.com/AmenRa/ranx/master/notebooks/data/small_qrels.trec"
    f.write(requests.get(master).text)
```

--------------------------------

### Load Qrels and Run Files

Source: https://github.com/amenra/ranx/blob/master/notebooks/3_evaluation.ipynb

Load ground truth (Qrels) and retrieval results (Run) from TREC-formatted files. Ensure the files are correctly specified.

```python
qrels = Qrels.from_file("notebooks/data/qrels.test", kind="trec")
run = Run.from_file("notebooks/data/results.test", kind="trec")
```

--------------------------------

### Make Run Comparable

Source: https://github.com/amenra/ranx/blob/master/docs/run.md

Prepare a run for comparison with ground truth (qrels) by adding empty results for missing queries and removing documents not present in the qrels.

```python
run.make_comparable(qrels)
```

--------------------------------

### Load Qrels and Runs

Source: https://github.com/amenra/ranx/blob/master/notebooks/4_comparison_and_report.ipynb

Loads ground truth data (qrels) and multiple retrieval runs from TREC files into ranx objects.

```python
# Let's load qrels and runs from files
qrels = Qrels.from_file("notebooks/data/qrels.trec")

run_1 = Run.from_file("notebooks/data/run_1.trec")
run_2 = Run.from_file("notebooks/data/run_2.trec")
run_3 = Run.from_file("notebooks/data/run_3.trec")
run_4 = Run.from_file("notebooks/data/run_4.trec")
run_5 = Run.from_file("notebooks/data/run_5.trec")
```

--------------------------------

### Optimize Fusion Parameters and Apply to Test Set

Source: https://context7.com/amenra/ranx/llms.txt

This snippet demonstrates optimizing fusion parameters on a training set and then applying the optimized parameters to a test set for evaluation. Ensure Qrels and Run objects are loaded correctly before proceeding.

```python
from ranx import Qrels, Run, optimize_fusion, fuse, evaluate

# Training data
train_qrels = Qrels.from_file("train_qrels.trec")
train_runs  = [Run.from_file(f"train_run_{i}.trec") for i in range(1, 4)]

# Test data
test_runs = [Run.from_file(f"test_run_{i}.trec") for i in range(1, 4)]
test_qrels = Qrels.from_file("test_qrels.trec")

# Optimize Weighted Sum on the training set
best_params = optimize_fusion(
    qrels=train_qrels,
    runs=train_runs,
    norm="min-max",
    method="wsum",          # supports all optimizable algorithms (rrf, rbc, gmnz, etc.)
    metric="ndcg@100",      # metric to maximize
)
# best_params = {"weights": [0.3, 0.5, 0.2]}  (example)

# Apply optimized parameters to test set
combined_test_run = fuse(
    runs=test_runs,
    norm="min-max",
    method="wsum",
    params=best_params,
)

# Evaluate
print(evaluate(test_qrels, combined_test_run, ["ndcg@10", "map@100", "mrr@100"]))
# >>> {"ndcg@10": 0.534, "map@100": 0.412, "mrr@100": 0.571}
```

--------------------------------

### Import Ranx Components

Source: https://github.com/amenra/ranx/blob/master/notebooks/3_evaluation.ipynb

Import necessary classes and functions from the ranx library, including Qrels, Run, and evaluate.

```python
from ranx import Qrels, Run, evaluate
```

--------------------------------

### Load Qrels from ir-datasets

Source: https://context7.com/amenra/ranx/llms.txt

Load Qrels directly from the `ir-datasets` catalog. The library will automatically download the dataset if it is not already present.

```python
# --- From ir-datasets (auto-downloads if missing) ---
qrels = Qrels.from_ir_datasets("msmarco-document/dev")
```

--------------------------------

### Load Qrels and Runs

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Loads ground truth (qrels) and search result (runs) data from TREC files into ranx objects.

```python
from ranx import Qrels, Run

# Let's load qrels and runs from files
qrels = Qrels.from_file("notebooks/data/qrels.trec")

run_4 = Run.from_file("notebooks/data/run_4.trec")
run_4.name = "System A"
run_5 = Run.from_file("notebooks/data/run_5.trec")
run_5.name = "System B"
```

--------------------------------

### Qrels / Run — Add Data Incrementally

Source: https://context7.com/amenra/ranx/llms.txt

Supports adding data to Qrels and Run objects one query or batch at a time, enabling streaming or incremental construction without loading everything into memory.

```APIDOC
## Qrels / Run — Incremental Data Addition

### Description
Both `Qrels` and `Run` support adding data one query or batch at a time, enabling streaming or incremental construction without loading everything into memory first.

### Methods
#### `Qrels.add(...)`
Adds data for a single query.

#### `Qrels.add_multi(...)`
Adds data for multiple queries.

#### `Qrels.add_score(...)`
Updates or adds a single score for a query-document pair.

#### `Run.add(...)`
Adds data for a single query.

#### `Run.add_multi(...)`
Adds data for multiple queries.

### Request Example
```python
from ranx import Qrels, Run

# Incrementally build a Qrels
qrels = Qrels(name="streaming_qrels")
qrels.add(q_id="q_1", doc_ids=["d_1", "d_2"], scores=[2, 1])
qrels.add_multi(
    q_ids=["q_2", "q_3"],
    doc_ids=[["d_3", "d_4"], ["d_5"]],
    scores=[[3, 1], [2]],
)
qrels.add_score(q_id="q_1", doc_id="d_3", score=3)  # update/add single score

# Incrementally build a Run
run = Run(name="my_model")
run.add(q_id="q_1", doc_ids=["d_1", "d_2", "d_3"], scores=[0.9, 0.7, 0.5])
run.add_multi(
    q_ids=["q_2", "q_3"],
    doc_ids=[["d_4", "d_5"], ["d_6", "d_7"]],
    scores=[[0.8, 0.6], [0.95, 0.4]],
)

# Convert back to dict or DataFrame for downstream use
qrels_dict = qrels.to_dict()
run_df = run.to_dataframe()
```
```

--------------------------------

### Import Qrels from JSON file

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Load Qrels data from a JSON file using the Qrels.from_file method. The file format is automatically detected.

```python
# Import from JSON file
qrels = Qrels.from_file("notebooks/data/small_qrels.json")

print(qrels)
```

--------------------------------

### Download Test Data

Source: https://github.com/amenra/ranx/blob/master/notebooks/3_evaluation.ipynb

Download the 'qrels' and 'results' test files required for evaluation. These files are saved in the 'notebooks/data' directory.

```python
import os
import requests

for file in ["qrels", "results"]:
    os.makedirs("notebooks/data", exist_ok=True)

    with open(f"notebooks/data/{file}.test", "w") as f:
        master = f"https://raw.githubusercontent.com/AmenRa/ranx/master/notebooks/data/{file}.test"
        f.write(requests.get(master).text)
```

--------------------------------

### Import Qrels from TREC-Style file

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Load Qrels data from a TREC-style file using the Qrels.from_file method. The file format is automatically detected.

```python
# Import from TREC-Style file
qrels = Qrels.from_file("notebooks/data/small_qrels.trec")

print(qrels)
```

--------------------------------

### Load Run from Dictionary

Source: https://context7.com/amenra/ranx/llms.txt

Create a Run object from a Python dictionary. Keys are query IDs, and values are dictionaries mapping document IDs to relevance scores.

```python
from ranx import Run
import pandas as pd

# --- From a Python dictionary ---
run_dict = {
    "q_1": {"d_1": 1.5, "d_2": 2.6},
    "q_2": {"d_3": 2.8, "d_2": 1.2, "d_5": 3.1},
}
run = Run(run_dict, name="bm25")
```

--------------------------------

### Fusion Algorithms Requiring Optimization

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Demonstrates the initial step for fusion algorithms that require an optimization phase, such as RRF. It prints the baseline scores before proceeding to optimization.

```python
from ranx import fuse, evaluate, optimize_fusion

print(run_4.name, evaluate(qrels, run_4, "ndcg@100"))
print(run_5.name, evaluate(qrels, run_5, "ndcg@100"))


```

--------------------------------

### Load Qrels from ir-datasets

Source: https://github.com/amenra/ranx/blob/master/docs/qrels.md

Load Qrels directly from the ir-datasets library. A comprehensive list of available datasets can be found on the ir-datasets website.

```python
qrels = Qrels.from_ir_datasets("msmarco-document/dev")
```

--------------------------------

### Load Run from File

Source: https://context7.com/amenra/ranx/llms.txt

Load Run objects from various file formats including TREC, JSON, LZ4 (ranx native), and gzipped TREC. The `kind` parameter can override the inferred file type.

```python
# --- From files ---
run = Run.from_file("path/to/run.trec")          # TREC-style
run = Run.from_file("path/to/run.json")          # JSON
run = Run.from_file("path/to/run.lz4")           # LZ4 (ranx native)
run = Run.from_file("path/to/run.gz")            # Gzipped TREC
run = Run.from_file("path/to/run.custom", kind="json")
```

--------------------------------

### Load Qrels from Files

Source: https://github.com/amenra/ranx/blob/master/docs/qrels.md

Qrels can be loaded from various file formats including JSON, TREC qrels, and gzipped TREC qrels. The file format is inferred from the extension, but can be overridden with the `kind` argument.

```python
qrels = Qrels.from_file("path/to/qrels.json")  # JSON file
```

```python
qrels = Qrels.from_file("path/to/qrels.trec")  # TREC-Style file
```

```python
qrels = Qrels.from_file("path/to/qrels.txt")   # TREC-Style file with txt extension
```

```python
qrels = Qrels.from_file("path/to/qrels.gz")    # Gzipped TREC-Style file
```

```python
qrels = Qrels.from_file("path/to/qrels.custom", kind="json")  # Loaded as JSON file
```

--------------------------------

### Load Qrels from File

Source: https://context7.com/amenra/ranx/llms.txt

Load Qrels from various file formats including TREC, JSON, gzipped TREC, and Parquet. The `kind` parameter can be used to override the inferred file type.

```python
# --- From a file (JSON, TREC, gzipped TREC, or Parquet) ---
qrels = Qrels.from_file("path/to/qrels.trec")         # TREC-style
qrels = Qrels.from_file("path/to/qrels.json")         # JSON
qrels = Qrels.from_file("path/to/qrels.gz")           # Gzipped TREC
qrels = Qrels.from_file("path/to/qrels.custom", kind="json")  # Override
```

--------------------------------

### Load Qrels from Parquet Files

Source: https://github.com/amenra/ranx/blob/master/docs/qrels.md

Load Qrels from Parquet files, supporting both local and remote sources. Additional arguments for pandas.read_parquet can be passed via `pd_kwargs`.

```python
qrels = Qrels.from_parquet(
    path="/path/to/parquet/file",
    q_id_col="q_id",
    doc_id_col="doc_id",
    score_col="score",
    pd_kwargs=None,
)
```

--------------------------------

### Load Run from Files

Source: https://github.com/amenra/ranx/blob/master/docs/run.md

Load a Run object from various file formats including JSON, TREC, gzipped TREC, and LZ4. The file extension typically infers the format, but the 'kind' argument can override this. A 'name' can be assigned to the run.

```python
run = Run.from_file("path/to/run.json")  # JSON file
```

```python
run = Run.from_file("path/to/run.trec")  # TREC-Style file
```

```python
run = Run.from_file("path/to/run.txt")   # TREC-Style file with txt extension
```

```python
run = Run.from_file("path/to/run.gz")    # Gzipped TREC-Style file
```

```python
run = Run.from_file("path/to/run.lz4")   # lz4 file produced by saving a ranx.Run as lz4
```

```python
run = Run.from_file("path/to/run.custom", kind="json")  # Loaded as JSON file
```

--------------------------------

### Import Ranx Components

Source: https://github.com/amenra/ranx/blob/master/notebooks/4_comparison_and_report.ipynb

Imports the necessary classes and functions from the ranx library for working with qrels, runs, and comparisons.

```python
from ranx import Qrels, Run, compare
```

--------------------------------

### Create Run from Dictionary

Source: https://github.com/amenra/ranx/blob/master/docs/run.md

Instantiate a Run object by converting a Python dictionary. The dictionary structure should map query IDs to document IDs and their corresponding relevance scores. Scores can be zero or negative.

```python
from ranx import Run

run_dict = {
    "q_1": {
        "d_1": 1.5,
        "d_2": 2.6,
    },
    "q_2": {
        "d_3": 2.8,
        "d_2": 1.2,
        "d_5": 3.1,
    },
}

run = Run(run_dict, name="bm25")
```

--------------------------------

### compare — Multi-System Comparison with Statistical Tests

Source: https://context7.com/amenra/ranx/llms.txt

Evaluates multiple runs against the same qrels and performs pairwise statistical significance testing. Returns a Report object with superscript annotations indicating significant differences.

```APIDOC
## compare — Multi-System Comparison with Statistical Tests

`compare` evaluates multiple runs against the same qrels and performs pairwise statistical significance testing. It returns a `Report` object with superscript annotations indicating which systems are significantly better. Supports Paired Student's t-Test (default), Fisher's Randomization Test, and Tukey's HSD Test.

```python
from ranx import Qrels, Run, compare

qrels = Qrels.from_file("qrels.trec")
run_1 = Run.from_file("run1.trec", name="BM25")
run_2 = Run.from_file("run2.trec", name="TF-IDF")
run_3 = Run.from_file("run3.trec", name="BERT")
run_4 = Run.from_file("run4.trec", name="ColBERT")
run_5 = Run.from_file("run5.trec", name="DPR")

# Compare with default Student's t-Test
report = compare(
    qrels=qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,           # p-value threshold for significance
    stat_test="student",  # "student" (default), "fisher", or "tukey"
)

print(report)
# #    Model    MAP@100    MRR@100    NDCG@10
# ---  -------  ---------  ---------  ---------
# a    BM25     0.320ᵇ     0.320ᵇ     0.368ᵇᶜ
# b    TF-IDF   0.233      0.234      0.239
# c    BERT     0.308ᵇ     0.309ᵇ     0.330ᵇ
# d    ColBERT  0.366ᵃᵇᶜ   0.367ᵃᵇᶜ   0.408ᵃᵇᶜ
# e    DPR      0.405ᵃᵇᶜᵈ  0.406ᵃᵇᶜᵈ  0.451ᵃᵇᶜᵈ

# Export to LaTeX for publications
latex_code = report.to_latex()
print(latex_code)

# Control display precision
report.rounding_digits = 4
report.show_percentages = True
print(report)
```
```

--------------------------------

### Evaluate Ranking Metrics

Source: https://github.com/amenra/ranx/blob/master/README.md

Compute scores for single or multiple ranking metrics. Ensure qrels and run objects are properly loaded.

```python
from ranx import evaluate

# Compute score for a single metric
evaluate(qrels, run, "ndcg@5")
>>> 0.7861
```

```python
# Compute scores for multiple metrics at once
evaluate(qrels, run, ["map@5", "mrr"])
>>> {"map@5": 0.6416, "mrr": 0.75}
```

--------------------------------

### Unsupervised Fusion with Different Methods

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Applies various unsupervised fusion algorithms (CombMIN, CombMAX, CombMED, CombSUM, CombANZ, CombMNZ) to combine search results using min-max normalization.

```python
from ranx import fuse, evaluate

print(run_4.name, evaluate(qrels, run_4, "ndcg@100"))
print(run_5.name, evaluate(qrels, run_5, "ndcg@100"))

for method in [
    "min",  # Alias for CombMIN
    "max",  # Alias for CombMAX
    "med",  # Alias for CombMED
    "sum",  # Alias for CombSUM
    "anz",  # Alias for CombANZ
    "mnz",  # Alias for CombMNZ
]:
    combined_run = fuse(
        runs=[run_4, run_5],
        norm="min-max",  # Default normalization strategy
        method=method,
    )

    print(combined_run.name, evaluate(qrels, combined_run, "ndcg@100"))
```

--------------------------------

### Import Ranx Modules

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Imports necessary classes and functions from the Ranx library for creating Qrels, Runs, and performing evaluations.

```python
from ranx import Qrels, Run, evaluate, compare
```

--------------------------------

### Create Qrels from Dictionary

Source: https://github.com/amenra/ranx/blob/master/docs/qrels.md

The preferred method for creating a Qrels instance is by converting a Python dictionary. Ensure the dictionary structure matches the expected format for queries and their associated document relevance scores.

```python
from ranx import Qrels

qrels_dict = {
    "q_1": {
        "d_1": 1,
        "d_2": 2,
    },
    "q_2": {
        "d_3": 2,
        "d_2": 1,
        "d_5": 3,
    },
}

qrels = Qrels(qrels_dict, name="MSMARCO")
```

--------------------------------

### Load Qrels from Dictionary

Source: https://context7.com/amenra/ranx/llms.txt

Construct a Qrels object from a Python dictionary. The dictionary keys are query IDs and values are dictionaries mapping document IDs to relevance scores.

```python
from ranx import Qrels
import pandas as pd

# --- From a Python dictionary ---
qrels_dict = {
    "q_1": {"d_1": 1, "d_2": 2},
    "q_2": {"d_3": 2, "d_2": 1, "d_5": 3},
}
qrels = Qrels(qrels_dict, name="my_dataset")
```

--------------------------------

### Download TREC data

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Downloads necessary qrels and run files from a GitHub repository to the local 'notebooks/data' directory.

```python
import os
import requests

for file in ["qrels", "run_4", "run_5"]:
    os.makedirs("notebooks/data", exist_ok=True)

    with open(f"notebooks/data/{file}.trec", "w") as f:
        master = f"https://raw.githubusercontent.com/AmenRa/ranx/master/notebooks/data/{file}.trec"
        f.write(requests.get(master).text)
```

--------------------------------

### Compare with Formatting Options

Source: https://github.com/amenra/ranx/blob/master/notebooks/4_comparison_and_report.ipynb

Performs a comparison and sets report formatting options (rounding digits and percentages) directly during the `compare` function call.

```python
# `rounding_digits` and `show_percentages` can be passed directly when
# calling `compare`
report = compare(
    qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,  # P-value threshold
    rounding_digits=3,
    show_percentages=True,
)

report
```

--------------------------------

### Compare Runs with Statistical Tests

Source: https://github.com/amenra/ranx/blob/master/notebooks/4_comparison_and_report.ipynb

Compares multiple retrieval runs against qrels using specified metrics and performs statistical significance testing (e.g., Student's t-test).

```python
# Compares different runs and performs statistical tests (Fisher's Randomization test)
report = compare(
    qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,  # P-value threshold
    # Use `student` for Two-sided Paired Student's t-Test (fast / default),
    # `fisher` for Fisher's Randomization Test (slow),
    # or `tukey` for Tukey's HSD test (fast)
    stat_test="student",
)

# The comparison results are saved in a Report instance,
# which provides handy functionalities such as tabular formatting
# (superscripts denote statistical significance differences)
report
```

--------------------------------

### Load Run from RanxHub

Source: https://context7.com/amenra/ranx/llms.txt

Load a Run object from the ranxhub repository, which hosts pre-computed runs.

```python
# --- From ranxhub (pre-computed runs repository) ---
run = Run.from_ranxhub("amenra/bm25-msmarco-passage-dev")
```

--------------------------------

### Optimize and Fuse Ranking Runs

Source: https://github.com/amenra/ranx/blob/master/README.md

Optimize fusion parameters using training data and then fuse test runs. Supports different normalization methods and fusion algorithms like Weighted Sum (wsum).

```python
from ranx import fuse, optimize_fusion

best_params = optimize_fusion(
    qrels=train_qrels,
    runs=[train_run_1, train_run_2, train_run_3],
    norm="min-max",     # The norm. to apply before fusion
    method="wsum",      # The fusion algorithm to use (Weighted Sum)
    metric="ndcg@100",  # The metric to maximize
)

combined_test_run = fuse(
    runs=[test_run_1, test_run_2, test_run_3],  
    norm="min-max",       
    method="wsum",        
    params=best_params,
)
```

--------------------------------

### Save a Run in LZ4 Format

Source: https://github.com/amenra/ranx/blob/master/notebooks/6_ranxhub.ipynb

Load a run from a file and save it in LZ4 compressed format to reduce memory footprint. The '.lz4' extension is automatically inferred by ranx.

```python
from ranx import Run

run = Run.from_file("path-to-run")
run.save("save/to/run.lz4")
```

--------------------------------

### Compare Runs and Perform Statistical Tests

Source: https://github.com/amenra/ranx/blob/master/notebooks/1_overview.ipynb

Compares multiple retrieval runs using specified metrics and performs Fisher's Randomization test to identify statistically significant differences. The results are presented in a Report object.

```python
# Compares different runs and performs statistical tests (Fisher's Randomization test)
report = compare(
    qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,  # P-value threshold
)

# The comparison results are saved in a Report instance,
# which provides handy functionalities such as tabular formatting
# (superscripts denote statistical significance differences)
report
```

--------------------------------

### Load Qrels from Pandas DataFrame

Source: https://github.com/amenra/ranx/blob/master/docs/qrels.md

Convert a Pandas DataFrame into a Qrels instance. Specify the column names for query IDs, document IDs, and relevance scores.

```python
from pandas import DataFrame

qrels_df = DataFrame.from_dict({
    "q_id":   [ "q_1",  "q_1",  "q_2",  "q_2"  ],
    "doc_id": [ "d_12", "d_25", "d_11", "d_22" ],
    "score":  [  5,      3,      6,      1     ],
})

qrels = Qrels.from_df(
    df=qrels_df,
    q_id_col="q_id",
    doc_id_col="doc_id",
    score_col="score",
)
```

--------------------------------

### Compare Multiple Runs with Statistical Tests

Source: https://context7.com/amenra/ranx/llms.txt

Use `compare` to evaluate multiple runs against the same Qrels and perform pairwise statistical significance testing. It returns a `Report` object with annotations. Supported tests include Student's t-Test, Fisher's Randomization Test, and Tukey's HSD Test. The `max_p` parameter sets the p-value threshold for significance.

```python
from ranx import Qrels, Run, compare

qrels = Qrels.from_file("qrels.trec")
run_1 = Run.from_file("run1.trec", name="BM25")
run_2 = Run.from_file("run2.trec", name="TF-IDF")
run_3 = Run.from_file("run3.trec", name="BERT")
run_4 = Run.from_file("run4.trec", name="ColBERT")
run_5 = Run.from_file("run5.trec", name="DPR")

# Compare with default Student's t-Test
report = compare(
    qrels=qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,           # p-value threshold for significance
    stat_test="student",  # "student" (default), "fisher", or "tukey"
)

print(report)
# #    Model    MAP@100    MRR@100    NDCG@10
# ---  -------  ---------  ---------  ---------
# a    BM25     0.320ᵇ     0.320ᵇ     0.368ᵇᶜ
# b    TF-IDF   0.233      0.234      0.239
# c    BERT     0.308ᵇ     0.309ᵇ     0.330ᵇ
# d    ColBERT  0.366ᵃᵇᶜ   0.367ᵃᵇᶜ   0.408ᵃᵇᶜ
# e    DPR      0.405ᵃᵇᶜᵈ  0.406ᵃᵇᶜᵈ  0.451ᵃᵇᶜᵈ

# Export to LaTeX for publications
latex_code = report.to_latex()
print(latex_code)

# Control display precision
report.rounding_digits = 4
report.show_percentages = True
print(report)
```

--------------------------------

### Compare and Plot Retrieval Model Performance

Source: https://context7.com/amenra/ranx/llms.txt

Compares multiple retrieval runs across specified metrics and generates a visualization. The `compare` function computes the metrics, and `plot` visualizes the results, optionally saving to a file.

```python
from ranx import Qrels, Run, compare, plot

qrels = Qrels.from_file("qrels.trec")
runs = [Run.from_file(f"run{i}.trec", name=f"model_{i}") for i in range(1, 6)]

report = compare(
    qrels=qrels,
    runs=runs,
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,
)

# Plot and save the comparison
plot(
    report,
    metrics=["map@100", "ndcg@10"],
    save_path="comparison.pdf",  # Optional: saves to file
)
```

--------------------------------

### Load Run from Parquet

Source: https://context7.com/amenra/ranx/llms.txt

Load a Run object from a Parquet file, specifying the file path and optionally a name for the run.

```python
# --- From Parquet ---
run = Run.from_parquet("/path/to/run.parquet", name="dense_model")
```

--------------------------------

### Optimize and Fuse Runs with Weighted Sum

Source: https://github.com/amenra/ranx/blob/master/notebooks/5_fusion.ipynb

Optimize fusion parameters for a weighted sum method and then fuse the runs. This is useful for finding the best combination of weights for merging results.

```python
from ranx import fuse, evaluate, optimize_fusion

print(run_4.name, evaluate(qrels, run_4, "ndcg@100"))
print(run_5.name, evaluate(qrels, run_5, "ndcg@100"))

# Optimize a given fusion method
best_params = optimize_fusion(
    qrels=qrels,
    runs=[run_4, run_5],
    norm="min-max",  # Default value
    method="wsum",  # Alias for Weighted Sum
    metric="ndcg@100",  # Metric we want to maximize
    step=0.01,
)

combined_run = fuse(
    runs=[run_4, run_5],
    norm="min-max",  # Default value
    method="wsum",  # Alias for Weighted Sum
    params=best_params,
)

print(combined_run.name, evaluate(qrels, combined_run, "ndcg@100"))
```

--------------------------------

### Display Report as Percentages

Source: https://github.com/amenra/ranx/blob/master/notebooks/4_comparison_and_report.ipynb

Configures the report to display metric values as percentages. The precision is controlled by `rounding_digits`.

```python
# You can show percentages instead of digits
# Note that the number of shown digits is based on
# the `rounding_digits` attribute, try changing it
report.rounding_digits = 3
report.show_percentages = True
report
```

--------------------------------

### Compute Ranking Metrics with evaluate

Source: https://context7.com/amenra/ranx/llms.txt

Use `evaluate` to compute one or more ranking metrics for a Qrels and Run pair. Metrics can include cutoffs (e.g., `@5`) and persistence parameters (e.g., `.8`). The `return_mean` parameter controls whether to return a single mean score or per-query scores. Relevance level can be filtered using `rel_lvl`.

```python
from ranx import Qrels, Run, evaluate

qrels = Qrels({
    "q_1": {"d_12": 5, "d_25": 3},
    "q_2": {"d_11": 6, "d_22": 1},
})
run = Run({
    "q_1": {"d_12": 0.9, "d_23": 0.8, "d_25": 0.7, "d_36": 0.6, "d_32": 0.5, "d_35": 0.4},
    "q_2": {"d_12": 0.9, "d_11": 0.8, "d_25": 0.7, "d_36": 0.6, "d_22": 0.5, "d_35": 0.4},
}, name="bm25")

# Single metric
score = evaluate(qrels, run, "ndcg@5")
# >>> 0.7861

# Multiple metrics at once
scores = evaluate(qrels, run, ["map@5", "mrr", "precision@10", "recall@100"])
# >>> {"map@5": 0.6416, "mrr": 0.75, "precision@10": 0.1, "recall@100": 1.0}

# Rank-biased Precision with persistence parameter
score = evaluate(qrels, run, "rbp.8")  # p=0.8

# Per-query scores (returns dict of {query_id: score})
per_query = evaluate(qrels, run, "ndcg@10", return_mean=False)
# >>> {"q_1": 0.9203, "q_2": 0.6520}

# Evaluate with relevance level threshold
score = evaluate(qrels, run, "ndcg@10", rel_lvl=2)  # only docs with score >= 2 are relevant
```

--------------------------------

### Compare Runs and Generate Report

Source: https://github.com/amenra/ranx/blob/master/docs/report.md

Compares multiple runs against ground truth data and generates a report. This is useful for evaluating and comparing the performance of different models or algorithms across specified metrics. The report can be printed directly or exported.

```python
from ranx import compare

# Compare different runs and perform statistical tests
report = compare(
    qrels=qrels,
    runs=[run_1, run_2, run_3, run_4, run_5],
    metrics=["map@100", "mrr@100", "ndcg@10"],
    max_p=0.01,  # P-value threshold
)

print(report)
```

--------------------------------

### Evaluate Multiple Metrics with Cutoffs

Source: https://github.com/amenra/ranx/blob/master/notebooks/3_evaluation.ipynb

Evaluate multiple metrics, each with its own specified cutoff. This allows for flexible analysis of performance at different ranking depths.

```python
# Multiple metrics with cutoffs (you can use different cutoffs for each metric)
evaluate(qrels, run, ["map@100", "mrr@10", "ndcg@10"])
```

--------------------------------

### Import Qrels from Pandas DataFrame

Source: https://github.com/amenra/ranx/blob/master/notebooks/2_qrels_and_run.ipynb

Load Qrels data into a Qrels object from a Pandas DataFrame. Specify the column names for query ID, document ID, and score.

```python
from pandas import DataFrame

qrels_df = DataFrame.from_dict(
    {
        "q_id": ["q_1", "q_1", "q_2", "q_2"],
        "doc_id": ["d_12", "d_25", "d_11", "d_22"],
        "score": [5, 3, 6, 1],
    }
)

qrels = Qrels.from_df(
    df=qrels_df,
    q_id_col="q_id",
    doc_id_col="doc_id",
    score_col="score",
)

print(qrels)
```

--------------------------------

### Load Runs from Ranxhub

Source: https://github.com/amenra/ranx/blob/master/README.md

Load search result runs from the Ranxhub repository. Provide a valid run ID.

```python
run = Run.from_ranxhub("run-id")
```

--------------------------------

### Load Qrels from Parquet

Source: https://context7.com/amenra/ranx/llms.txt

Load Qrels from a Parquet file, which can be local or remote. Specify the column names for query ID, document ID, and score.

```python
# --- From a Parquet file (local or remote) ---
qrels = Qrels.from_parquet(
    path="/path/to/qrels.parquet",
    q_id_col="q_id",
    doc_id_col="doc_id",
    score_col="score",
)
```

--------------------------------

### Load Run from Pandas DataFrame

Source: https://context7.com/amenra/ranx/llms.txt

Create a Run object from a Pandas DataFrame. Specify the column names for query ID, document ID, and score.

```python
# --- From a Pandas DataFrame ---
run_df = pd.DataFrame({
    "q_id":   ["q_1", "q_1", "q_2", "q_2"],
    "doc_id": ["d_12", "d_25", "d_11", "d_22"],
    "score":  [0.9, 0.7, 0.8, 0.5],
})
run = Run.from_df(run_df, q_id_col="q_id", doc_id_col="doc_id", score_col="score", name="bm25")
```

--------------------------------

### Load Run from Parquet Files

Source: https://github.com/amenra/ranx/blob/master/docs/run.md

Load a Run object from Parquet files, supporting remote sources. Additional arguments for pandas.read_parquet can be passed via 'pd_kwargs'. Specify column names and an optional run name.

```python
run = Run.from_parquet(
    path="/path/to/parquet/file",
    q_id_col="q_id",
    doc_id_col="doc_id",
    score_col="score",
    pd_kwargs=None,
    name="my_run",
)
```