### Development Installation with uv

Source: https://castorini.github.io/rank_llm

Sets up a local virtual environment for development using uv. It clones the repository, creates a Python 3.11 environment, and installs development dependencies.

```bash
git clone https://github.com/castorini/rank_llm.git
cd rank_llm
uv python install 3.11
uv venv --python 3.11
source .venv/bin/activate
uv sync --group dev

```

--------------------------------

### Install Multiple Optional Extras with pip

Source: https://castorini.github.io/rank_llm

Installs multiple optional stacks for rank-llm simultaneously, such as 'openai' and 'api', by combining them in a single pip install command.

```bash
pip install -e ".[openai,api]"

```

--------------------------------

### Install Gemini Provider Extra

Source: https://castorini.github.io/rank_llm

Install the necessary Gemini provider extra for using Gemini models. This can be done using `uv sync` with the `dev` group or via `pip install`.

```bash
uv sync --group dev --extra genai
```

```bash
pip install -e ".[genai]"
```

--------------------------------

### Install flashinfer and flash-attn

Source: https://castorini.github.io/rank_llm

Installs flashinfer and flash-attn, which are required for specific backends like 'sglang' and optimized training workflows. Note the specific index URL for flashinfer.

```bash
pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/
pip install flash-attn --no-build-isolation

```

--------------------------------

### Install PyPI Package with uv

Source: https://castorini.github.io/rank_llm

Installs the published rank-llm package using uv into an isolated virtual environment. Ensures a clean environment for using the package.

```bash
uv venv --python 3.11
source .venv/bin/activate
uv pip install rank-llm

```

--------------------------------

### Install Llama Index and RankLLM Packages

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Install the necessary packages for Llama Index core, HuggingFace embeddings, RankLLM postprocessor, and the rank_llm library.

```bash
pip install llama-index-core llama-index-embeddings-huggingface llama-index-postprocessor-rank-llm rank_llm transformers requests

```

--------------------------------

### Install RankLLM and Dependencies

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Install necessary packages for using RankLLM with LangChain, FAISS, and HuggingFace embeddings.

```bash
pip install langchain-community faiss-gpu torch transformers sentence-transformers huggingface-hub rank_llm

```

--------------------------------

### Fallback conda/pip Installation

Source: https://castorini.github.io/rank_llm

Installs rank-llm using conda for environment management and pip for package installation. This serves as a fallback if uv is not preferred.

```bash
conda create -n rankllm python=3.11 -c conda-forge -y
conda activate rankllm
pip install -e .

```

--------------------------------

### Install Optional Extras with pip

Source: https://castorini.github.io/rank_llm

Installs specific optional stacks for rank-llm using pip, allowing for modular installation of features like 'openai' or 'api'.

```bash
pip install -e "
.[<extra>]
"

```

--------------------------------

### Install uv with Astral's Official Installer

Source: https://castorini.github.io/rank_llm

Installs the uv package manager using a curl script and adds it to the PATH. This is the recommended workflow contributor tool.

```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"

```

--------------------------------

### Install Rerankers with RankLLM

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Install the rerankers package with the rankllm extra. This command installs the necessary dependencies for using RankLLM functionality.

```bash
pip install "rerankers[rankllm]"
```

--------------------------------

### Rank Documents with Reranker

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Demonstrates how to initialize the Reranker with the 'rankllm' model type and use its rank method. This snippet shows a basic example of providing a query and a list of documents to be ranked.

```python
from rerankers import Reranker

ranker = Reranker('rank_zephyr', model_type="rankllm")

results = ranker.rank(query="I love you", docs=["I hate you", "I really like you"], doc_ids=[0,1])
print(results)
```

--------------------------------

### RankLLM Python Workflow: Retrieval, Reranking, Evaluation, and Saving

Source: https://castorini.github.io/rank_llm

This Python script demonstrates a full workflow using rank-llm. It performs document retrieval, reranking using a specified model, evaluates the results using standard metrics, analyzes invocation history, and saves the outputs in various formats. Ensure necessary libraries are installed and models are accessible.

```python
from pathlib import Path

from rank_llm.analysis.response_analysis import ResponseAnalyzer
from rank_llm.data import DataWriter
from rank_llm.evaluation.trec_eval import EvalFunction
from rank_llm.rerank import Reranker, get_openai_api_key
from rank_llm.rerank.listwise import (
    SafeOpenai,
    VicunaReranker,
    ZephyrReranker,
)
from rank_llm.retrieve.retriever import RetrievalMethod, Retriever
from rank_llm.retrieve.topics_dict import TOPICS

# -------- Retrieval --------

# By default BM25 is used for retrieval of top 100 candidates.
dataset_name = "dl19"
retrieved_results = Retriever.from_dataset_with_prebuilt_index(dataset_name)

# Users can specify other retrieval methods and number of retrieved candidates.
# retrieved_results = Retriever.from_dataset_with_prebuilt_index(
#     dataset_name, RetrievalMethod.SPLADE_P_P_ENSEMBLE_DISTIL, k=50
# )
# ---------------------------
```

```python
# --------- Rerank ----------

# Rank Zephyr model
reranker = ZephyrReranker()

# Rank Vicuna model
# reranker = VicunaReranker()

# RankGPT
# model_coordinator = SafeOpenai("gpt-4o-mini", 4096, keys=get_openai_api_key())
# reranker = Reranker(model_coordinator)

kwargs = {"populate_invocations_history": True}
rerank_results = reranker.rerank_batch(requests=retrieved_results, **kwargs)
# ---------------------------
```

```python
# ------- Evaluation --------

# Evaluate retrieved results.
topics = TOPICS[dataset_name]
ndcg_10_retrieved = EvalFunction.from_results(retrieved_results, topics)
print(ndcg_10_retrieved)

# Evaluate rerank results.
ndcg_10_rerank = EvalFunction.from_results(rerank_results, topics)
print(ndcg_10_rerank)

# By default ndcg@10 is the eval metric, other value can be specified:
# eval_args = ["-c", "-m", "map_cut.100", "-l2"]
# map_100_rerank = EvalFunction.from_results(rerank_results, topics, eval_args)
# print(map_100_rerank)

# eval_args = ["-c", "-m", "recall.20"]
# recall_20_rerank = EvalFunction.from_results(rerank_results, topics, eval_args)
# print(recall_20_rerank)

# ---------------------------
```

```python
# --- Analyze invocations ---
analyzer = ResponseAnalyzer.from_inline_results(rerank_results)
error_counts = analyzer.count_errors(verbose=True)
print(error_counts)
# ---------------------------
```

```python
# ------ Save results -------
writer = DataWriter(rerank_results)
Path(f"demo_outputs/").mkdir(parents=True, exist_ok=True)
writer.write_in_jsonl_format(f"demo_outputs/rerank_results.jsonl")
writer.write_in_trec_eval_format(f"demo_outputs/rerank_results.txt")
writer.write_inference_invocations_history(
    f"demo_outputs/inference_invocations_history.json"
)
# ---------------------------
```

--------------------------------

### Perform Retrieval Without Reranking

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Invoke the retriever with a query to get relevant documents before reranking.

```python
query = "What was done to Russia?"
docs = retriever.invoke(query)
pretty_print_docs(docs)


```

--------------------------------

### Set Up Vector Store Retriever with LangChain

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Load documents, split them, create embeddings using HuggingFace, and set up a FAISS vector store retriever. Ensure 'cuda' is available for GPU acceleration.

```python
from langchain_community.document_loaders import TextLoader
from langchain_community.vectorstores import FAISS
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
import torch
import os

device = "cuda"

documents = TextLoader("state_of_the_union.txt").load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=100)
texts = text_splitter.split_documents(documents)
for idx, text in enumerate(texts):
    text.metadata["id"] = idx

embedding = HuggingFaceEmbeddings(
    model_name="BAAI/bge-small-en", # or any model of your choice
    model_kwargs={'device': 'cuda'},
    encode_kwargs={'normalize_embeddings': True}
)

retriever = FAISS.from_documents(texts, embedding).as_retriever(search_kwargs={"k": 20})


```

--------------------------------

### Load Data and Build Index with Llama Index

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Loads Wikipedia content, sets up a HuggingFace embedder, and builds a VectorStoreIndex from the loaded documents. Ensure the 'data_wiki' directory is created.

```python
import os
import requests
from pathlib import Path

from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
from llama_index.core.retrievers import VectorIndexRetriever
from llama_index.core import QueryBundle
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.postprocessor.rankllm_rerank import RankLLMRerank

# Load Wikipedia content
wiki_titles = ["Vincent van Gogh"]
data_path = Path("data_wiki")
data_path.mkdir(exist_ok=True)

for title in wiki_titles:
    response = requests.get(
        "https://en.wikipedia.org/w/api.php",
        params={
            "action": "query",
            "format": "json",
            "titles": title,
            "prop": "extracts",
            "explaintext": True,
        },
    ).json()
    page = next(iter(response["query"]["pages"].values()))
    wiki_text = page["extract"]
    with open(data_path / f"{title}.txt", "w") as fp:
        fp.write(wiki_text)

# Set HuggingFace embedder
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.chunk_size = 512

# Load and index documents
documents = SimpleDirectoryReader("data_wiki").load_data()
index = VectorStoreIndex.from_documents(documents)


```

--------------------------------

### Run LiT5-Distill Original Model

Source: https://castorini.github.io/rank_llm

This command executes the original LiT5-Distill model, which operates with a window size of 20. Configuration includes model path, dataset, retrieval method, prompt template, context size, batch size, and enabling variable passages.

```bash
python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/LiT5-Distill-large --top_k_candidates=100 --dataset=dl19 \
    --retrieval_method=bm25 --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_fid_template.yaml  --context_size=150 --batch_size=32 \
    --variable_passages
```

--------------------------------

### Run DuoT5 Model

Source: https://castorini.github.io/rank_llm

Run the DuoT5 model, specifically the #B variant trained for 10K steps. Due to its O(n^2) complexity, reranking the top 50 candidates is recommended.

```python
python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/duot5-3b-msmarco-10k --top_k_candidates=50 --dataset=dl19 \
    --retrieval_method=bm25 --prompt_template_path=src/rank_llm/rerank/prompt_templates/duot5_template.yaml
```

--------------------------------

### Run RankGPT4-o Model

Source: https://castorini.github.io/rank_llm

Execute the RankGPT4-o model for reranking. The `--prompt_template_path` can be adjusted to use different prompt templates, such as 'rank_gpt_apeer' or 'rank_GPT'.

```bash
rank-llm rerank --model-path gpt-4o --top-k-candidates 100 --dataset dl20 \
  --retrieval-method bm25 --prompt-template-path src/rank_llm/rerank/prompt_templates/rank_gpt_apeer_template.yaml --context-size 4096 --use-azure-openai

```

--------------------------------

### Run MonoT5 3B Model

Source: https://castorini.github.io/rank_llm

This command runs the 3B variant of MonoT5 trained for 10K steps. It typically reranks 1K candidates and requires specifying the model path, dataset, retrieval method, prompt template, and context size.

```bash
python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/monot5-3b-msmarco-10k --top_k_candidates=1000 --dataset=dl19 \
    --retrieval_method=bm25 --prompt_template_path=src/rank_llm/rerank/prompt_templates/monot5_template.yaml --context_size=512
```

--------------------------------

### RankLLMRerank Field Arguments and Defaults

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Lists all available arguments and their default values for the RankLLMRerank class, useful for configuring reranking behavior.

```python
model: str = Field(
  description="Model name.",
  default="rank_zephyr"
)
top_n: Optional[int] = Field(
  description="Number of nodes to return sorted by reranking score."
)
window_size: int = Field(
  description="Reranking window size. Applicable only for listwise and pairwise models.",
  default=20
)
batch_size: Optional[int] = Field(
  description="Rereranking batch size. Applicable only for pointwise models."
)
context_size: int = Field(
  description="Maximum number of tokens for the context window.",
  default=4096
)
prompt_template_path: str = Field(
  description="Yaml file for the prompt template, please refer to the official RankLLM repo for examples and usage.",
)
num_gpus: int = Field(
  description="Number of GPUs to use for inference if applicable.",
  default=1
)
num_few_shot_examples: int = Field(
  description="Number of few-shot examples to include in the prompt.",
  default=0
)
few_shot_file: Optional[str] = Field(
  description="Path to a file containing few-shot examples, used if few-shot prompting is enabled.",
  default=None
)
use_logits: bool = Field(
  description="Whether to use raw logits for reranking scores instead of probabilities.",
  default=False
)
use_alpha: bool = Field(
  description="Whether to apply an alpha scaling factor in the reranking score calculation.",
  default=False
)
variable_passages: bool = Field(
  description="Whether to allow passages of variable lengths instead of fixed-size chunks.",
  default=False
)
stride: int = Field(
  description="Stride to use when sliding over long documents for reranking.",
  default=10
)
use_azure_openai: bool = Field(
  description="Whether to use Azure OpenAI instead of the standard OpenAI API.",
  default=False
)


```

--------------------------------

### Run Gemini 3 Flash Preview Model

Source: https://castorini.github.io/rank_llm

Execute the Gemini 3 Flash Preview model for reranking. This command specifies a context size of 4096 and uses a specific retrieval method and prompt template.

```python
python src/rank_llm/scripts/run_rank_llm.py  --model_path=gemini-3-flash-preview --top_k_candidates=100 --dataset=dl20 \
    --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_gpt_apeer_template.yaml  --context_size=4096
```

--------------------------------

### Run MonoELECTRA Model

Source: https://castorini.github.io/rank_llm

Execute the MonoELECTRA model for reranking. Ensure the model path is correctly specified, either as a short name or a full Hugging Face path.

```python
python src/rank_llm/scripts/run_rank_llm.py --model_path=monoelectra --top_k_candidates=1000 --dataset=dl19 \
    --retrieval_method=bm25 --context_size=512
```

```python
python src/rank_llm/scripts/run_rank_llm.py --model_path=castorini/monoelectra-base --top_k_candidates=1000 --dataset=dl19 \
    --retrieval_method=bm25 --context_size=512
```

--------------------------------

### Run LiT5-Distill V2 Model

Source: https://castorini.github.io/rank_llm

Use this command to run the LiT5-Distill V2 model, which can rerank 100 documents in a single pass. It requires specifying the model path, dataset, retrieval method, prompt template, context size, batch size, and enabling variable passages with a window size.

```bash
python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/LiT5-Distill-large-v2 --top_k_candidates=100 --dataset=dl19 \
        --retrieval_method=bm25 --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_fid_template.yaml  --context_size=150 --batch_size=4 \
    --variable_passages --window_size=100
```

--------------------------------

### Run LiT5-Score Model

Source: https://castorini.github.io/rank_llm

Command to run the LiT5-Score model. It requires specifying the model path, dataset, retrieval method, prompt template, context size, batch size, window size, and enabling variable passages.

```bash
python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/LiT5-Score-large --top_k_candidates=100 --dataset=dl19 \
    --retrieval_method=bm25 --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_fid_score_template.yaml --context_size=150 --batch_size=8 \
    --window_size=100 --variable_passages
```

--------------------------------

### Run FirstMistral Model

Source: https://castorini.github.io/rank_llm

Execute the FirstMistral model for reranking using only first-token logits. Omit the `--use_logits` flag for traditional listwise reranking. This command also specifies a context size of 4096 and uses multiple GPUs.

```python
python src/rank_llm/scripts/run_rank_llm.py  --model_path=castorini/first_mistral --top_k_candidates=100 --dataset=dl20 --retrieval_method=SPLADE++_EnsembleDistil_ONNX --prompt_template_path=src/rank_llm/rerank/prompt_templates/rank_zephyr_template.yaml  --context_size=4096 --variable_passages --use_logits --use_alpha --num_gpus 1
```

--------------------------------

### Test Retrieval With RankLLM

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Executes a query using RankLLM for reranking, specifying the model and window size, then visualizes all reranked results.

```python
# With RankLLM
new_nodes = get_retrieved_nodes(
    "Which date did Paul Gauguin arrive in Arles?",
    vector_top_k=50,
    reranker_top_n=3,
    with_reranker=True,
    model="rank_zephyr",
    window_size=15,
)
visualize_retrieved_nodes(new_nodes)


```

--------------------------------

### RankLLM CLI Commands

Source: https://castorini.github.io/rank_llm

Execute common rank-llm operations using the CLI. Use 'rerank' for reranking, 'prompt list' to see available prompts, 'view' to inspect results, 'evaluate' for model performance, and 'serve' for model deployment.

```bash
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --dataset dl20 \
  --retrieval-method bm25 --top-k-candidates 100
```

```bash
rank-llm prompt list
```

```bash
rank-llm view demo_outputs/rerank_results.jsonl
```

```bash
rank-llm evaluate --model-name castorini/rank_zephyr_7b_v1_full
```

```bash
rank-llm serve http --model-path castorini/rank_zephyr_7b_v1_full --port 8082
```

```bash
rank-llm serve mcp --transport stdio
```

--------------------------------

### Run RankZephyr Model

Source: https://castorini.github.io/rank_llm

Execute the RankZephyr model for reranking. Use `--sglang_batched` for batched mode with SGLang. The `--num_passes` flag can be used for multiple model passes.

```bash
rank-llm rerank --model-path castorini/rank_zephyr_7b_v1_full --top-k-candidates 100 --dataset dl20 \
--retrieval-method SPLADE++_EnsembleDistil_ONNX --prompt-template-path src/rank_llm/rerank/prompt_templates/rank_zephyr_template.yaml --context-size 4096 --variable-passages

```

--------------------------------

### RankLLMRerank Field Arguments

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Defines the available arguments for the RankLLMRerank class, including model path, top_n, and context size.

```python
model_path: str = Field(default="rank_zephyr")
top_n: int = Field(default=3)
window_size: int = Field(default=20)
context_size: int = Field(default=4096)
prompt_template_path: str
num_gpus: int = Field(default=1)
num_few_shot_examples: int = Field(default=0)
few_shot_file: Optional[str] = Field(default=None)
use_logits: bool = Field(default=False)
use_alpha: bool = Field(default=False)
variable_passages: bool = Field(default=False)
stride: int = Field(default=10)
use_azure_openai: bool = Field(default=False)
model_coordinator: Any = Field(default=None, exclude=True)


```

--------------------------------

### Retrieval With Reranking Using RankLLM

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Integrates RankLLMRerank with a base retriever for contextual compression. Clears CUDA cache before reranking.

```python
torch.cuda.empty_cache()

compressor = RankLLMRerank(top_n=3, model_path="rank_zephyr")
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=retriever
)

del compressor

compressed_docs = compression_retriever.invoke(query)
pretty_print_docs(compressed_docs)


```

--------------------------------

### Test Retrieval Without RankLLM

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Executes a query to retrieve nodes without using the RankLLM reranker, then visualizes the top 3 results.

```python
# Without RankLLM
new_nodes = get_retrieved_nodes(
    "Which date did Paul Gauguin arrive in Arles?",
    vector_top_k=50,
    with_reranker=False,
)
visualize_retrieved_nodes(new_nodes[:3])


```

--------------------------------

### Reranker Arguments for RankLLM

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Lists all field arguments and their default values for the Reranker class when model_type is set to 'rankllm'. These parameters control the behavior and configuration of the reranking process.

```python
model: str = "rank_zephyr",
window_size: int = 20,
context_size: int = 4096,
prompt_template_path: str,
num_few_shot_examples: int = 0,
few_shot_file: Optional[str] = None,
num_gpus: int = 1,
variable_passages: bool = False,
use_logits: bool = False,
use_alpha: bool = False,
stride: int = 10,
use_azure_openai: bool = False,
```

--------------------------------

### Retrieval with Optional RankLLM Reranking

Source: https://castorini.github.io/rank_llm/docs/external-integrations.html

Defines a function to retrieve nodes from an index, with an option to apply RankLLM reranking. Includes cleanup for the reranker model to free GPU memory.

```python
import torch
from IPython.display import display, HTML
import pandas as pd

def get_retrieved_nodes(
    query_str,
    vector_top_k=10,
    reranker_top_n=3,
    with_reranker=False,
    model="rank_zephyr",
    window_size=None,
):
    query_bundle = QueryBundle(query_str)
    # configure retriever
    retriever = VectorIndexRetriever(
        index=index,
        similarity_top_k=vector_top_k,
    )
    retrieved_nodes = retriever.retrieve(query_bundle)
    retrieved_nodes.reverse()

    if with_reranker:
        # configure reranker
        reranker = RankLLMRerank(
            model=model, top_n=reranker_top_n, window_size=window_size
        )
        retrieved_nodes = reranker.postprocess_nodes(
            retrieved_nodes, query_bundle
        )

        # clear cache, rank_zephyr uses 16GB of GPU VRAM
        del reranker
        torch.cuda.empty_cache()

    return retrieved_nodes


def pretty_print(df):
    return display(HTML(df.to_html().replace("\n", "<br>")))


def visualize_retrieved_nodes(nodes) -> None:
    result_dicts = []
    for node in nodes:
        result_dict = {"Score": node.score, "Text": node.node.get_text()}
        result_dicts.append(result_dict)

    pretty_print(pd.DataFrame(result_dicts))


```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.