### Install BEIR from Source

Source: https://github.com/beir-cellar/beir/blob/main/README.md

Installs the BEIR library from its source code. This method is useful for developers who want to contribute to the project or test unreleased features. It requires cloning the repository and installing it in editable mode.

```bash
git clone https://github.com/beir-cellar/beir.git
cd beir
pip install -e .
```

--------------------------------

### Configure Logging for BEIR Examples

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet configures the Python logging module to display informational messages with timestamps. It uses BEIR's LoggingHandler to format the output, making it easier to follow the execution flow of the retrieval examples. This setup is common across various BEIR examples.

```python
import logging
from beir.util import LoggingHandler

logging.basicConfig(format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    level=logging.INFO,
                    handlers=[LoggingHandler()])
```

--------------------------------

### Build and Develop BEIR from Source

Source: https://github.com/beir-cellar/beir/wiki/Installing-beir

Clones the BEIR repository from GitHub and installs it in editable mode. This method is suitable for developers who want to contribute to BEIR or use the latest development version. Requires git and pip.

```bash
$ git clone https://github.com/beir-cellar/beir.git
$ cd beir
$ pip install -e .
```

--------------------------------

### Install FAISS for HuggingFace Retrieval

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This is a conditional installation instruction. If you plan to use the `encode_and_retrieve()` method with HuggingFace models, you need to ensure the 'faiss-cpu' library is installed. This is a prerequisite for efficient similarity search operations.

```shell
# pip install faiss-cpu
```

--------------------------------

### BEIR Setup and Logging Configuration

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code sets up basic logging for BEIR, printing debug information to standard output. It configures the format, date format, logging level, and handlers for the logger.

```python
import logging
from beir import LoggingHandler

logging.basicConfig(format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    level=logging.INFO,
                    handlers=[LoggingHandler()])
```

--------------------------------

### Install BEIR with Optional Packages

Source: https://github.com/beir-cellar/beir/wiki/Installing-beir

Installs the BEIR Python package along with optional dependencies for specific functionalities like FAISS or PEFT. Use this if you need advanced features. Requires pip to be installed.

```bash
pip install beir[faiss]
pip install beir[peft]
```

--------------------------------

### Install PEFT, Accelerate, and vLLM Libraries

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet provides the pip commands to install the necessary libraries for using vLLM with LoRA models, including PEFT (Parameter-Efficient Fine-Tuning), Accelerate, and vLLM itself. These libraries are essential for advanced model loading and inference with large language models.

```shell
pip install peft
pip install accelerate
pip install vllm
```

--------------------------------

### Quick Example with Sentence-BERT

Source: https://github.com/beir-cellar/beir/blob/main/README.md

Demonstrates a quick example of using BEIR with a Sentence-BERT model for retrieval. It includes setting up logging, importing necessary modules, and initializing a dense retrieval model.

```python
from beir import util, LoggingHandler
from beir.retrieval import models
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

import logging
import pathlib, os

#### Just some code to print debug information to stdout
logging.basicConfig(format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S',
                    level=logging.INFO,
                    handlers=[LoggingHandler()])
#### /print debug information to stdout

```

--------------------------------

### Install BEIR using pip

Source: https://github.com/beir-cellar/beir/wiki/Installing-beir

Installs the stable version of the BEIR Python package. This is the recommended method for most users. Requires pip to be installed.

```bash
pip install beir
```

--------------------------------

### Initialize SBERT Model and DenseRetrievalExactSearch

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet initializes a retrieval model using SentenceBERT from the 'beir.retrieval.models' module and wraps it with DenseRetrievalExactSearch (DRES) for exact search. It specifies the SBERT model name and batch size. This setup is used for embedding queries and documents for retrieval.

```python
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

model = DRES(models.SentenceBERT("Alibaba-NLP/gte-modernbert-base"), batch_size=16)
```

--------------------------------

### Scale Dense Retrieval Across Multiple GPUs (Python)

Source: https://context7.com/beir-cellar/beir/llms.txt

Demonstrates how to leverage multiple GPUs for efficient encoding of large corpora in dense retrieval tasks. It involves initializing a SentenceBERT model, starting a multi-process pool targeting specific GPU devices, and then using the `encode_corpus_parallel` method for distributed encoding.

```python
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

# Initialize model with multi-GPU support
model = models.SentenceBERT("sentence-transformers/all-mpnet-base-v2")

# Start multi-process pool for parallel encoding
target_devices = ["cuda:0", "cuda:1", "cuda:2", "cuda:3"]
pool = model.start_multi_process_pool(target_devices=target_devices)

# Encode corpus in parallel across GPUs
corspus_embeddings = model.encode_corpus_parallel(
    corpus=corpus,
    pool=pool,
    batch_size=128,
    chunk_id=0
)

```

--------------------------------

### Create Conda Environment for beir-ColBERT

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Creates a new Conda virtual environment using the provided 'conda_env.yml' file. This environment includes the necessary dependencies, including the beir repository installed via pip. It also activates the 'colbert-v0.2' environment.

```bash
# https://github.com/NThakur20/beir-ColBERT#installation

conda env create -f conda_env.yml
conda activate colbert-v0.2
```

--------------------------------

### Neural Reranking with Cross-Encoder

Source: https://context7.com/beir-cellar/beir/llms.txt

Enables two-stage retrieval by reranking initial results using a cross-encoder model. This model jointly encodes query-document pairs for more accurate relevance scoring. The example shows initial retrieval with BM25 followed by reranking.

```python
from beir.reranking import Rerank
from beir.reranking.models import CrossEncoder
from beir.retrieval.search.lexical import BM25Search as BM25
from beir.retrieval.evaluation import EvaluateRetrieval

# Stage 1: Initial retrieval with BM25
bm25_model = BM25(index_name="trec-covid", hostname="localhost", initialize=True)
retriever = EvaluateRetrieval(bm25_model)
initial_results = retriever.retrieve(corpus, queries)

# Stage 2: Rerank top-100 with Cross-Encoder
cross_encoder_model = CrossEncoder("cross-encoder/ms-marco-electra-base")
reranker = Rerank(cross_encoder_model, batch_size=128)

# Rerank top-k results from initial retrieval
rerank_results = reranker.rerank(
    corpus=corpus,
    queries=queries,
    results=initial_results,
    top_k=100
)

# Evaluate reranked results
ndcg, _map, recall, precision = EvaluateRetrieval.evaluate(
    qrels, rerank_results, retriever.k_values
)
```

--------------------------------

### Download and Unzip Dataset with BEIR Utilities

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet shows how to download a specified dataset (e.g., scifact) from a URL and unzip it into a designated directory using BEIR's utility functions. It requires the 'beir' library and standard Python modules like 'os' and 'pathlib'. The output is the path to the unzipped dataset.

```python
import os
import pathlib
from beir import util

dataset = "scifact"
url = f"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{dataset}.zip"
out_dir = os.path.join(pathlib.Path(__file__).parent.absolute(), "datasets")
data_path = util.download_and_unzip(url, out_dir)
```

--------------------------------

### Load Dataset with BEIR GenericDataLoader

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code demonstrates loading a dataset (corpus, queries, and qrels) from a specified data path using BEIR's GenericDataLoader. It's typically used after downloading and unzipping the dataset.

```python
from beir.datasets.data_loader import GenericDataLoader

corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")
```

--------------------------------

### Load Dataset for Retrieval with GenericDataLoader

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code snippet demonstrates how to load a dataset (corpus, queries, and question-answer pairs) for retrieval tasks using BEIR's GenericDataLoader. It takes the data path obtained from the download step and specifies the split (e.g., 'test'). The function returns the corpus, queries, and qrels (query-relevance judgments).

```python
from beir.datasets.data_loader import GenericDataLoader

# Assuming data_path is already defined from the download step
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")
```

--------------------------------

### Download and Load BEIR Dataset (Python)

Source: https://github.com/beir-cellar/beir/wiki/Datasets-available

This snippet demonstrates how to download a specified BEIR dataset using a provided URL and then load it into corpus, queries, and qrels variables. It utilizes the `beir.util` and `beir.datasets.data_loader` modules. The input is a dataset name (e.g., 'scifact'), and the output is the loaded dataset components.

```python
from beir import util
from beir.datasets.data_loader import GenericDataLoader

dataset = "scifact"
url = "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)
data_path = util.download_and_unzip(url, "datasets")
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")
```

--------------------------------

### Load Dense Retrieval Model (SBERT, ANCE, USE-QA, DPR) (Python)

Source: https://github.com/beir-cellar/beir/wiki/Models-available

Initializes a dense retrieval model using SentenceBERT and sets up an evaluation retriever. Supports various dense models and score functions like cosine similarity or dot product.

```python
from beir.retrieval import models
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

model = DRES(models.SentenceBERT("msmarco-distilbert-base-v3"), batch_size=16)
retriever = EvaluateRetrieval(model, score_function="cos_sim") # or "dot" for dot-product
```

--------------------------------

### Initialize HuggingFace Model and Retriever in BEIR

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet initializes a dense retrieval model using HuggingFace's `e5-mistral-7b-instruct` model via BEIR's `DRES` class. It configures model parameters like `max_length`, `pooling`, and attention implementation, then sets up an `EvaluateRetrieval` object for cosine similarity.

```python
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval
from beir import util

# Assuming data_path, corpus, queries, qrels are already loaded
query_prompt = "Instruct: Given a question, retrieve relevant documents that best answer the question\nQuery: "

model = DRES(
    util.HuggingFace(
        model_path="intfloat/e5-mistral-7b-instruct",
        max_length=512,
        pooling="eos",
        append_eos_token=True,
        normalize=True,
        prompts={"query": query_prompt, "passage": ""},
        attn_implementation="flash_attention_2",
        torch_dtype="bfloat16"
    ),
    batch_size=128,
)

retriever = EvaluateRetrieval(model, score_function="cos_sim")
```

--------------------------------

### Load SPARTA Sparse Retrieval Model (Python)

Source: https://github.com/beir-cellar/beir/wiki/Models-available

Loads the SPARTA sparse retrieval model using the BEIR library. Requires specifying the model path and batch size for efficient processing.

```python
from beir.retrieval.search.sparse import SparseSearch
from beir.retrieval import models

model_path = "BeIR/sparta-msmarco-distilbert-base-v1"
sparse_model = SparseSearch(models.SPARTA(model_path), batch_size=128)
```

--------------------------------

### Load and Evaluate on Custom Datasets (Python)

Source: https://context7.com/beir-cellar/beir/llms.txt

Shows how to load and evaluate retrieval models on custom datasets stored locally. The GenericDataLoader class supports loading corpus, queries, and relevance judgments (qrels) from specified files or directories in the BEIR format. It then integrates with EvaluateRetrieval for model assessment.

```python
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

# Load custom dataset from local files
# Expected format:
# - corpus.jsonl: {"_id": "doc1", "title": "...", "text": "..."}
# - queries.jsonl: {"_id": "q1", "text": "query text"}
# - qrels/test.tsv: query-id\tcorpus-id\tscore

loader = GenericDataLoader(
    data_folder="./my_custom_dataset",
    corpus_file="corpus.jsonl",
    query_file="queries.jsonl",
    qrels_folder="qrels"
)

corpus, queries, qrels = loader.load(split="test")

# Or load with custom qrels file directly
loader = GenericDataLoader(
    corpus_file="./data/corpus.jsonl",
    query_file="./data/queries.jsonl",
    qrels_file="./data/test_qrels.tsv"
)
corspus, queries, qrels = loader.load_custom()

# Evaluate with your model
model = DRES(models.SentenceBERT("all-MiniLM-L6-v2"), batch_size=64)
retriever = EvaluateRetrieval(model, score_function="cos_sim")
results = retriever.retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
```

--------------------------------

### Load Custom Dataset using GenericDataLoader in Python

Source: https://github.com/beir-cellar/beir/wiki/Load-your-custom-dataset

Loads a custom preprocessed dataset from specified file paths using the GenericDataLoader class. Requires corpus, query, and qrels files in JSONL and TSV formats respectively. Outputs corpus, queries, and qrels data structures.

```python
from beir.datasets.data_loader import GenericDataLoader

corpus_path = "your_corpus_file.jsonl"
query_path = "your_query_file.jsonl"
qrels_path = "your_qrels_file.tsv"

corpus, queries, qrels = GenericDataLoader(
    corpus_file=corpus_path, 
    query_file=query_path, 
    qrels_file=qrels_path).load_custom()
```

--------------------------------

### Download and Load BEIR Dataset

Source: https://context7.com/beir-cellar/beir/llms.txt

Downloads a specified BEIR dataset from a URL, unzips it, and loads the corpus, queries, and relevance judgments into dictionaries. This is useful for local evaluation of retrieval models.

```python
from beir import util
from beir.datasets.data_loader import GenericDataLoader
import os

# Download and unzip a BEIR dataset
dataset = "scifact"
url = f"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{dataset}.zip"
out_dir = "./datasets"
data_path = util.download_and_unzip(url, out_dir)

# Load corpus, queries, and relevance judgments
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

# corpus: {"doc_id": {"title": "...", "text": "..."}, ...}
# queries: {"query_id": "query text", ...}
# qrels: {"query_id": {"doc_id": relevance_score}, ...}

print(f"Loaded {len(corpus)} documents, {len(queries)} queries")
# Output: Loaded 5183 documents, 300 queries
```

--------------------------------

### Initialize HuggingFace Model for Dense Retrieval

Source: https://context7.com/beir-cellar/beir/llms.txt

Initializes a HuggingFace model for dense retrieval with custom configurations like pooling strategy, attention implementation, and data type. It then sets up a DRES retriever and evaluates its performance.

```python
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval.models import HuggingFace

query_prompt = "Instruct: Given a question, retrieve relevant documents\nQuery: "

model = HuggingFace(
    model_path="intfloat/e5-mistral-7b-instruct",
    max_length=512,
    pooling="eos",                    # Options: "cls", "mean", "eos"
    append_eos_token=True,
    normalize=True,
    prompts={"query": query_prompt, "passage": ""},
    attn_implementation="flash_attention_2",
    torch_dtype="bfloat16"
)

retriever_model = DRES(model, batch_size=32)
retriever = EvaluateRetrieval(retriever_model, score_function="cos_sim")

# Encode, save embeddings, and retrieve from saved files
results = retriever.encode_and_retrieve(
    corpus, queries,
    encode_output_path="./embeddings/",
    overwrite=False
)

ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
```

--------------------------------

### Save Retrieval Results and Runfile with BEIR

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code demonstrates how to save the retrieval results and evaluation metrics to files. It creates a results directory if it doesn't exist and saves the runfile in TREC format and the evaluation metrics in JSON format.

```python
import os
import pathlib

# Assuming dataset, results, ndcg, _map, recall, precision, mrr are available
results_dir = os.path.join(pathlib.Path(__file__).parent.absolute(), "results")
os.makedirs(results_dir, exist_ok=True)

util.save_runfile(os.path.join(results_dir, f"{dataset}.run.trec"), results)
util.save_results(os.path.join(results_dir, f"{dataset}.json"), ndcg, _map, recall, precision, mrr)
```

--------------------------------

### Save Retrieval Results and Runfile

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code demonstrates how to save the retrieval results and evaluation metrics to files. It first creates a directory for results if it doesn't exist. Then, it saves the runfile in TREC format and the evaluation metrics (NDCG, MAP, Recall, Precision, MRR) in a JSON file. This is useful for later analysis or reranking.

```python
import os
import pathlib
from beir import util

# Assuming dataset, results, ndcg, _map, recall, precision, mrr are defined
results_dir = os.path.join(pathlib.Path(__file__).parent.absolute(), "results")
os.makedirs(results_dir, exist_ok=True)

util.save_runfile(os.path.join(results_dir, f"{dataset}.run.trec"), results)
util.save_results(os.path.join(results_dir, f"{dataset}.json"), ndcg, _map, recall, precision, mrr)
```

--------------------------------

### Load BM25 Lexical Retrieval Model (Python)

Source: https://github.com/beir-cellar/beir/wiki/Models-available

Initializes and loads the BM25 lexical retrieval model from Elasticsearch. Requires specifying hostname and index name. The `initialize` parameter controls whether to reindex documents.

```python
from beir.retrieval.search.lexical import BM25Search as BM25

hostname = "your-hostname" #localhost
index_name = "your-index-name" # scifact
initialize = True # True, will delete existing index with same name and reindex all documents
model = BM25(index_name=index_name, hostname=hostname, initialize=initialize)
```

--------------------------------

### Load vLLM Embed Model with LoRA and DenseRetrieval

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code initializes a retrieval model using vLLM for embedding, incorporating a LoRA adapter. It specifies the base model path, LoRA adapter name, maximum sequence length, LoRA rank, pooling strategy, and prompt templates. The model is then wrapped in DenseRetrievalExactSearch (DRES) with a specified batch size.

```python
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

model = DRES(
    models.VLLMEmbed(
        model_path="Qwen/Qwen2.5-7B",
        lora_name_or_path="rlhn/Qwen2.5-7B-rlhn-400K",
        max_length=512,
        lora_r=16,
        pooling="eos",
        append_eos_token=True,
        normalize=True,
        prompts={"query": "query: ", "passage": "passage: "},
        convert_to_numpy=True
    ),
    batch_size=128,
)
```

--------------------------------

### Provide Custom Dataset Directly in Python

Source: https://github.com/beir-cellar/beir/wiki/Load-your-custom-dataset

Allows direct provision of corpus, queries, and relevance data as Python dictionaries, bypassing file loading. This method is useful when data is already in memory. The format for each data structure is specified.

```python
corpus = {
    "doc1" : {
        "title": "Albert Einstein", 
        "text": "Albert Einstein was a German-born theoretical physicist. who developed the theory of relativity, \n                 one of the two pillars of modern physics (alongside quantum mechanics). His work is also known for \n                 its influence on the philosophy of science. He is best known to the general public for his mass–energy \n                 equivalence formula E = mc2, which has been dubbed 'the world's most famous equation'. He received the 1921 \n                 Nobel Prize in Physics 'for his services to theoretical physics, and especially for his discovery of the law \n                 of the photoelectric effect', a pivotal step in the development of quantum theory."
        },
    "doc2" : {
        "title": "", # Keep title an empty string if not present
        "text": "Wheat beer is a top-fermented beer which is brewed with a large proportion of wheat relative to the amount of \n                 malted barley. The two main varieties are German Weißbier and Belgian witbier; other types include Lambic (made\n                 with wild yeast), Berliner Weisse (a cloudy, sour beer), and Gose (a sour, salty beer)."
    },
}

queries = {
    "q1" : "Who developed the mass-energy equivalence formula?",
    "q2" : "Which beer is brewed with a large proportion of wheat?"
}

qrels = {
    "q1" : {"doc1": 1},
    "q2" : {"doc2": 1},
}
```

--------------------------------

### Lexical Retrieval with BM25/Elasticsearch

Source: https://context7.com/beir-cellar/beir/llms.txt

Provides lexical retrieval using Elasticsearch's BM25 algorithm. This requires a running Elasticsearch instance. The code demonstrates loading a dataset, initializing the BM25 model, and evaluating retrieval performance.

```python
from beir.retrieval.search.lexical import BM25Search as BM25
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.datasets.data_loader import GenericDataLoader
from beir import util

# Load dataset
data_path = util.download_and_unzip(
    "https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/scifact.zip",
    "./datasets"
)
corpus, queries, qrels = GenericDataLoader(data_path).load(split="test")

# Initialize BM25 with Elasticsearch
model = BM25(
    index_name="scifact",
    hostname="localhost",
    initialize=True,          # True: recreate index, False: use existing
    number_of_shards=1,       # Use 1 for small datasets (<5k docs)
    language="english"
)

retriever = EvaluateRetrieval(model)
results = retriever.retrieve(corpus, queries)

# Evaluate retrieval performance
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
# Output: NDCG@10: 0.6789, MAP@10: 0.6123
```

--------------------------------

### BEIR Data Preprocessing for ColBERT

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Preprocesses BEIR datasets into a ColBERT-friendly format, converting JSONL to TSV. Requires the dataset name, split, and paths for collection and queries.

```bash
python -m colbert.data_prep \
  --dataset ${dataset} \
  --split "test" \
  --collection $COLLECTION \
  --queries $QUERIES \

```

--------------------------------

### Sparse Retrieval with SPLADE

Source: https://context7.com/beir-cellar/beir/llms.txt

Supports learned sparse retrieval models like SPLADE, which produce sparse term-weighted representations. The code initializes the SPLADE model, creates a sparse searcher, and retrieves results.

```python
from beir.retrieval.search.sparse import SparseSearch
from beir.retrieval.models import SPLADE
from beir.retrieval.evaluation import EvaluateRetrieval

# Initialize SPLADE model for sparse retrieval
sparse_model = SPLADE(
    model_path="naver/splade-cocondenser-ensembledistil",
    max_length=256
)

# Create sparse searcher
searcher = SparseSearch(sparse_model, batch_size=16)
retriever = EvaluateRetrieval(searcher, score_function="dot")

# Retrieve using sparse representations
results = retriever.retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
```

--------------------------------

### Initialize Cohere API Retriever in BEIR

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet shows how to initialize a dense retrieval model using Cohere's embedding API through BEIR's `DRES` class. It requires a Cohere API key and specifies the model path and data type. An `EvaluateRetrieval` object is then set up for cosine similarity.

```python
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval import apis
import os

# Assuming data_path, corpus, queries, qrels are already loaded
cohere_api_key = os.getenv("COHERE_API_KEY")

model = DRES(
    apis.CohereEmbedAPI(
        api_key=cohere_api_key, 
        model_path="embed-v4.0", 
        normalize=True, 
        torch_dtype="float32"
    ),
    batch_size=96,
)

retriever = EvaluateRetrieval(model, score_function="cos_sim")
```

--------------------------------

### Encode and Retrieve Documents with BEIR

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code uses the initialized retriever to encode the corpus and queries, then performs retrieval. It saves the embeddings to a specified path and returns the retrieval results.

```python
# Assuming retriever is initialized and corpus, queries are loaded
results = retriever.encode_and_retrieve(corpus, queries, encode_output_path="./embeddings/")
```

--------------------------------

### Generate Synthetic Queries for Data Augmentation (Python)

Source: https://context7.com/beir-cellar/beir/llms.txt

Utilizes the QueryGenerator class with seq2seq models (e.g., T5) to create synthetic queries from document passages. This is beneficial for domain adaptation and augmenting datasets for retrieval tasks. The generated queries and relevance judgments are saved to specified directories.

```python
from beir.generation import QueryGenerator
from beir.generation.models import QGenModel

# Initialize query generation model
qgen_model = QGenModel("BeIR/query-gen-msmarco-t5-base-v1")

# Create query generator
generator = QueryGenerator(model=qgen_model)

# Generate synthetic queries from corpus
generator.generate(
    corpus=corpus,
    output_dir="./generated_data",
    ques_per_passage=3,        # Generate 3 queries per passage
    max_length=64,             # Max query length
    top_p=0.95,               # Nucleus sampling
    top_k=25,
    batch_size=32,
    prefix="gen"              # Output file prefix
)

# Output files:
# - ./generated_data/gen-queries.jsonl
# - ./generated_data/gen-qrels/train.tsv
```

--------------------------------

### Encode and Retrieve with vLLM Embeddings

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet uses the initialized vLLM retriever to encode the corpus and queries and then perform retrieval. It specifies an output path for the encoded embeddings, which can be useful for debugging or further processing. This method is part of the EvaluateRetrieval class.

```python
# Assuming retriever, corpus, and queries are already defined
results = retriever.encode_and_retrieve(corpus, queries, encode_output_path="./qwen_embeddings/")
```

--------------------------------

### Cite BEIR Leaderboard Publication (BibTeX)

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This BibTeX entry is for citing the publication related to the BEIR leaderboard, which provides resources for reproducible reference models and statistical analyses for the BEIR benchmark.

```bibtex
@inproceedings{kamalloo:2024,
    author = {Kamalloo, Ehsan and Thakur, Nandan and Lassance, Carlos and Ma, Xueguang and Yang, Jheng-Hong and Lin, Jimmy},
    title = {Resources for Brewing BEIR: Reproducible Reference Models and Statistical Analyses},
    year = {2024},
    isbn = {9798400704314},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    url = {https://doi.org/10.1145/3626772.3657862},
    doi = {10.1145/3626772.3657862},
    abstract = {BEIR is a benchmark dataset originally designed for zero-shot evaluation of retrieval models across 18 different domain/task combinations. In recent years, we have witnessed the growing popularity of models based on representation learning, which naturally begs the question: How effective are these models when presented with queries and documents that differ from the training data? While BEIR was designed to answer this question, our work addresses two shortcomings that prevent the benchmark from achieving its full potential: First, the sophistication of modern neural methods and the complexity of current software infrastructure create barriers to entry for newcomers. To this end, we provide reproducible reference implementations that cover learned dense and sparse models. Second, comparisons on BEIR are performed by reducing scores from heterogeneous datasets into a single average that is difficult to interpret. To remedy this, we present meta-analyses focusing on effect sizes across datasets that are able to accurately quantify model differences. By addressing both shortcomings, our work facilitates future explorations in a range of interesting research questions.},
    booktitle = {Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval},
    pages = {1431–1440},
    numpages = {10},
    keywords = {domain generalization, evaluation, reproducibility},
    location = {Washington DC, USA},
    series = {SIGIR '24}
}
```

--------------------------------

### Evaluate Retrieval Performance with Standard and Custom Metrics (Python)

Source: https://context7.com/beir-cellar/beir/llms.txt

Demonstrates how to use the EvaluateRetrieval class to compute standard Information Retrieval metrics like NDCG, MAP, Recall, and Precision, as well as custom metrics such as MRR, Recall Cap, Top-K Accuracy, and Hole. It requires a model, corpus, queries, and ground truth relevance judgments (qrels).

```python
from beir.retrieval.evaluation import EvaluateRetrieval

# Standard evaluation: NDCG, MAP, Recall, Precision at k=[1,3,5,10,100,1000]
retriever = EvaluateRetrieval(model, k_values=[1, 3, 5, 10, 100, 1000])

# Run retrieval
results = retriever.retrieve(corpus, queries)

# Evaluate with standard metrics
ndcg, _map, recall, precision = retriever.evaluate(
    qrels, results, retriever.k_values,
    ignore_identical_ids=True  # Ignore self-retrieval
)

# Custom metrics
mrr = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="mrr")
recall_cap = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="recall_cap")
accuracy = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="top_k_accuracy")
hole = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="hole")

# Results format:
# ndcg = {"NDCG@1": 0.45, "NDCG@10": 0.67, ...}
# mrr = {"MRR@1": 0.45, "MRR@10": 0.72, ...}
print(f"NDCG@10: {ndcg['NDCG@10']:.4f}, MRR@10: {mrr['MRR@10']:.4f}")
```

--------------------------------

### Load BEIR Dataset from HuggingFace

Source: https://context7.com/beir-cellar/beir/llms.txt

Loads BEIR datasets directly from the HuggingFace Hub using HFDataLoader. It supports optional streaming for large datasets and can keep data in memory. The loaded data can be accessed as HuggingFace datasets or converted to dictionaries.

```python
from beir.datasets.data_loader_hf import HFDataLoader

# Load dataset from HuggingFace repository
loader = HFDataLoader(
    hf_repo="BeIR/scifact",
    hf_repo_qrels="BeIR/scifact-qrels",
    streaming=False,
    keep_in_memory=True
)

corpus, queries, qrels = loader.load(split="test")

# Access as HuggingFace datasets
print(f"Corpus columns: {corpus.column_names}")
# Output: Corpus columns: ['id', 'title', 'text']

# Convert to dictionary format if needed
corpus_dict = {row["id"]: {"title": row["title"], "text": row["text"]} for row in corpus}
```

--------------------------------

### FAISS IVFPQ Index Creation

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Stores and trains an IVFPQ FAISS index for end-to-end retrieval. Requires specifying the number of partitions, which should be chosen per dataset.

```bash
python -m colbert.index_faiss \
  --index_root $INDEX_ROOT \
  --index_name $INDEX_NAME \
  --partitions $NUM_PARTITIONS \
  --sample 0.3 \
  --root $OUTPUT_DIR \
  --experiment ${dataset}

```

--------------------------------

### Dense Retrieval with HuggingFace Models

Source: https://context7.com/beir-cellar/beir/llms.txt

Utilizes any HuggingFace Transformers model for dense retrieval. This wrapper supports various pooling strategies (cls, mean, eos) for LLM-based embedding models, enabling flexible dense vector generation for semantic search.

```python
from beir.retrieval import models
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval
```

--------------------------------

### Dense Retrieval with SentenceBERT

Source: https://context7.com/beir-cellar/beir/llms.txt

Implements dense retrieval using Sentence-Transformers models. It encodes queries and documents into dense vectors for semantic search and evaluates retrieval performance using standard metrics like NDCG, MAP, Recall, and Precision.

```python
from beir import util, LoggingHandler
from beir.retrieval import models
from beir.datasets.data_loader import GenericDataLoader
from beir.retrieval.evaluation import EvaluateRetrieval
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
import logging

logging.basicConfig(format='%(asctime)s - %(message)s', level=logging.INFO, handlers=[LoggingHandler()])

# Load dataset
dataset = "scifact"
url = f"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{dataset}.zip"
data_path = util.download_and_unzip(url, "./datasets")
corpus, queries, qrels = GenericDataLoader(data_folder=data_path).load(split="test")

# Initialize SentenceBERT model with custom prompts
model = models.SentenceBERT(
    "Alibaba-NLP/gte-modernbert-base",
    max_length=512,
    prompts={"query": "query: ", "passage": "passage: "}
)

# Create dense retrieval searcher
retriever_model = DRES(model, batch_size=128, corpus_chunk_size=50000)
retriever = EvaluateRetrieval(retriever_model, score_function="cos_sim")

# Retrieve and evaluate
results = retriever.retrieve(corpus, queries)
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)

# Output: NDCG@10: 0.6721, MAP@10: 0.6234, Recall@10: 0.8945
```

--------------------------------

### API-based Dense Retrieval with Cohere

Source: https://context7.com/beir-cellar/beir/llms.txt

Enables dense retrieval using Cohere's embedding API, suitable for production without local GPUs. It initializes the Cohere model, sets up a DRES retriever, and performs encoding and retrieval.

```python
import os
from beir.retrieval import apis
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES
from beir.retrieval.evaluation import EvaluateRetrieval

# Set API key (or use environment variable COHERE_API_KEY)
os.environ["COHERE_API_KEY"] = "your-api-key"

# Initialize Cohere embedding API model
model = apis.CohereEmbedAPI(
    api_key=os.getenv("COHERE_API_KEY"),
    model_path="embed-v4.0",
    normalize=True,
    torch_dtype="float32"
)

retriever_model = DRES(model, batch_size=96)
retriever = EvaluateRetrieval(retriever_model, score_function="cos_sim")

# Encode and retrieve
results = retriever.encode_and_retrieve(
    corpus, queries,
    encode_output_path="./cohere/embeddings/"
)

ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
```

--------------------------------

### Perform Retrieval with EvaluateRetrieval and Cosine Similarity

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This code initializes an EvaluateRetrieval object with a specified model and score function (e.g., 'cos_sim' for cosine similarity). It then uses the retriever to find relevant documents for given queries based on the corpus. The results are stored in a dictionary. Dependencies include the BEIR library.

```python
from beir.retrieval.evaluation import EvaluateRetrieval

# Assuming model, corpus, and queries are already defined
retriever = EvaluateRetrieval(model, score_function="cos_sim") # or "dot" for dot product
results = retriever.retrieve(corpus, queries)
```

--------------------------------

### Rerank Results using Cross-Encoder Model (Python)

Source: https://github.com/beir-cellar/beir/wiki/Models-available

Loads a Cross-Encoder model for reranking search results and initializes a Rerank object. This is typically used to refine top-k results from an initial retrieval step like BM25.

```python
from beir.reranking.models import CrossEncoder
from beir.reranking import Rerank

cross_encoder_model = CrossEncoder('cross-encoder/ms-marco-electra-base')
reranker = Rerank(cross_encoder_model, batch_size=128)

# Rerank top-100 results retrieved by BM25
# rerank_results = reranker.rerank(corpus, queries, bm25_results, top_k=100)
```

--------------------------------

### Clone beir-ColBERT Repository

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Clones the modified beir-ColBERT repository from GitHub, which is essential for evaluating ColBERT models on the BEIR benchmark.

```bash
git clone https://github.com/NThakur20/beir-ColBERT.git
```

--------------------------------

### Cite BEIR Benchmark Publication (BibTeX)

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This BibTeX entry is for citing the primary publication of the BEIR benchmark, which focuses on a heterogeneous benchmark for zero-shot evaluation of information retrieval models.

```bibtex
@inproceedings{
    thakur2021beir,
    title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
    author={Nandan Thakur and Nils Reimers and Andreas R{"u}ckl{'e} and Abhishek Srivastava and Iryna Gurevych},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
    year={2021},
    url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}
```

--------------------------------

### Evaluate Retrieval Results using Standard Metrics

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet evaluates the retrieval results using standard metrics like NDCG, MAP, Recall, and Precision at various k values (k=[1,3,5,10,100,1000]). It also calculates Mean Reciprocal Rank (MRR) using a custom evaluation function. This requires the 'qrels' (ground truth) and 'results' from the retrieval step.

```python
# Assuming retriever, qrels, and results are already defined
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
mrr = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="mrr")
```

--------------------------------

### Save and Load Retrieval Results and Evaluation Metrics (Python)

Source: https://context7.com/beir-cellar/beir/llms.txt

Provides utility functions to save retrieval results in the standard TREC run file format and evaluation metrics (NDCG, MAP, Recall, Precision, MRR) to JSON files. It also demonstrates how to load a previously saved run file.

```python
from beir import util
import os

# Save retrieval results in TREC run format
util.save_runfile(
    output_file="./results/scifact.run.trec",
    results=results,
    run_name="beir-sbert",
    top_k=1000
)

# Save evaluation metrics to JSON
util.save_results(
    output_file="./results/scifact.json",
    ndcg=ndcg,
    _map=_map,
    recall=recall,
    precision=precision,
    mrr=mrr
)

# Load existing run file
loaded_results = util.load_runfile("./results/scifact.run.trec")
# Returns: {"query_id": {"doc_id": score, ...}, ...}
```

--------------------------------

### Evaluate Retrieval Model Performance in BEIR

Source: https://github.com/beir-cellar/beir/blob/main/README.md

This snippet evaluates the retrieval results using standard metrics like NDCG, MAP, Recall, and Precision at various k-values. It also calculates Mean Reciprocal Rank (MRR) using a custom evaluation function.

```python
# Assuming retriever, qrels, and results are available
ndcg, _map, recall, precision = retriever.evaluate(qrels, results, retriever.k_values)
mrr = retriever.evaluate_custom(qrels, results, retriever.k_values, metric="mrr")
```

--------------------------------

### Stop Multi-Process Pool in BEIR

Source: https://context7.com/beir-cellar/beir/llms.txt

This snippet demonstrates how to stop a multi-process pool used within the BEIR framework. It's essential for releasing resources after model processing is complete.

```python
model.stop_multi_process_pool(pool)
```

--------------------------------

### ColBERT Indexing for Fast Retrieval

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Indexes precomputed ColBERT representations of passages for efficient retrieval. Requires a trained ColBERT model checkpoint and specifies parameters like document max length, batch size, and AMP usage.

```bash
python -m torch.distributed.launch \
  --nproc_per_node=2 -m colbert.index \
  --root $OUTPUT_DIR \
  --doc_maxlen 300 \
  --mask-punctuation \
  --bsize 128 \
  --amp \
  --checkpoint $CHECKPOINT \
  --index_root $INDEX_ROOT \
  --index_name $INDEX_NAME \
  --collection $COLLECTION \
  --experiment ${dataset}

```

--------------------------------

### Query Retrieval using ColBERT and FAISS

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Retrieves top-k documents for each query using the indexed ColBERT representations and FAISS. Outputs a `ranking.tsv` file with integer document IDs.

```bash
python -m colbert.retrieve \
  --amp \
  --doc_maxlen 300 \
  --mask-punctuation \
  --bsize 256 \
  --queries $QUERIES \
  --nprobe 32 \
  --partitions $NUM_PARTITIONS \
  --faiss_depth 100 \
  --depth 100 \
  --index_root $INDEX_ROOT \
  --index_name $INDEX_NAME \
  --checkpoint $CHECKPOINT \
  --root $OUTPUT_DIR \
  --experiment ${dataset} \
  --ranking_dir $RANKING_DIR

```

--------------------------------

### Implement Custom Cross-Encoder Model for BEIR

Source: https://github.com/beir-cellar/beir/wiki/Evaluate-your-custom-model

Defines a custom cross-encoder model class with a `predict` method for generating similarity scores between query-document pairs. This integrates with BEIR's re-ranking capabilities. Load your model within the `__init__` method.

```python
from beir.reranking import Rerank

class YourCustomCEModel:
    def __init__(self, model_path=None, **kwargs)
        self.model = None # ---> HERE Load your custom model
    
    # Write your own score function, which takes in query-document text pairs and returns the similarity scores
    def predict(self, sentences: List[Tuple[str,str]], batch_size: int, **kwags) -> List[float]:
        pass # return only the list of float scores

reranker = Rerank(YourCustomCEModel(model_path="your-custom-model-path"), batch_size=128)
```

--------------------------------

### Implement Custom Dense-Retriever Model for BEIR

Source: https://github.com/beir-cellar/beir/wiki/Evaluate-your-custom-model

Defines a custom dual-encoder model class with `encode_queries` and `encode_corpus` methods for generating embeddings. This allows integration with BEIR's retrieval functionalities. Ensure your model loading logic is within the `__init__` method.

```python
from beir.retrieval.search.dense import DenseRetrievalExactSearch as DRES

class YourCustomDEModel:
    def __init__(self, model_path=None, **kwargs)
        self.model = None # ---> HERE Load your custom model
    
    # Write your own encoding query function (Returns: Query embeddings as numpy array)
    def encode_queries(self, queries: List[str], batch_size: int, **kwargs) -> np.ndarray:
        pass
    
    # Write your own encoding corpus function (Returns: Document embeddings as numpy array)  
    def encode_corpus(self, corpus: List[Dict[str, str]], batch_size: int, **kwargs) -> np.ndarray:
        pass

custom_model = DRES(YourCustomDEModel(model_path="your-custom-model-path"))
```

--------------------------------

### BEIR Evaluation Script

Source: https://github.com/beir-cellar/beir/blob/main/examples/retrieval/evaluation/late-interaction/README.md

Evaluates the generated `ranking.tsv` file using the official BEIR evaluation script for any specified dataset and split.

```bash
python -m colbert.beir_eval \
  --dataset ${dataset} \
  --split "test" \
  --collection $COLLECTION \
  --rankings "${RANKING_DIR}/ranking.tsv"

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.