### Install and Connect to DuckDB

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vector-stores.md

Installs the DuckDB library and establishes an in-memory connection to the database. This is the initial setup for using DuckDB.

```bash
pip install duckdb
```

```python
import duckdb
conn = duckdb.connect(":memory:")
```

--------------------------------

### Install Weaviate Client

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vector-stores.md

Install the Weaviate Python client library. Use the '-U' flag to ensure the latest version is installed.

```bash
pip install -U weaviate-client
```

--------------------------------

### Linux Installation: Install PyTorch CUDA Version (Example)

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Example command to install a CUDA-enabled PyTorch version for Linux. Replace 'cu126' with your specific CUDA version found on the PyTorch website.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu126
```

--------------------------------

### Install ChromaDB

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vector-stores.md

Install the ChromaDB Python client. This is the first step to using ChromaDB.

```bash
pip install chromadb
```

--------------------------------

### Mac Installation: Install Latest Pyserini with Optional Dependencies

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs the latest Pyserini along with optional dependencies, after core components are already installed.

```bash
pip install 'pyserini==latest'
```

--------------------------------

### Install Hugging Face Hub

Source: https://github.com/castorini/pyserini/blob/master/docs/conceptual-framework3.md

Installs the huggingface_hub library, which is required for authenticating with Hugging Face.

```bash
pip install huggingface_hub

```

--------------------------------

### Start Local Web Server for Visualization

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-nfcorpus-vis.md

Starts a local HTTP server in the 'nfcorpus-atlas' directory to view the generated visualization in a web browser. Navigate to http://localhost:8000 after starting the server.

```bash
cd nfcorpus-atlas
python -m http.server 8000
```

--------------------------------

### Linux Installation: Install Java and Maven

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs OpenJDK 21 and Maven within the 'pyserini' Conda environment on Linux.

```bash
conda install -c conda-forge openjdk=21 maven -y
```

--------------------------------

### Mac Installation: Install Java and Maven

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs OpenJDK 21 and Maven within the 'pyserini' Conda environment on a Mac.

```bash
conda install -c anaconda wget -y
conda install -c conda-forge openjdk=21 maven -y
```

--------------------------------

### Set Up Environment for KILT with Pyserini

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-kilt.md

Installs Pyserini, downloads KILT data, and sets up necessary NLTK dependencies. Ensure you have Python 3.7 and conda installed.

```bash
# Create a virtual env
conda create -n kilt37 -y python=3.7 && conda activate kilt37

# Get the development installation of pyserini
git clone https://github.com/castorini/pyserini.git
pip install pyserini

# Get KILT scripts, input and gold data, and install the package
git clone https://github.com/facebookresearch/KILT.git
cd KILT

# go back to an older version
git reset 2130aafaaee0671bdbd03d781b1fa57ee02650d2
pip install -r requirements.txt
pip install .
mkdir data
python scripts/download_all_kilt_data.py
python scripts/get_triviaqa_input.py
cd ..

# Get NLTK dependencies
python -m nltk.downloader punkt
python -m nltk.downloader stopwords

# Get the KILT knowledge source / wikipedia dump (34.76GiB)
cd pyserini/collections/
wget http://dl.fbaipublicfiles.com/KILT/kilt_knowledgesource.json

# We'll split it in multiple files to make processing faster
mkdir kilt_knowledge_split
cd kilt_knowledge_split
split -l500000 ../kilt_knowledgesource.json kilt_ks.
cd ../../..

# Feel free to delete the kilt_knowledgesource.json file now if you need more disk space.
```

--------------------------------

### Start Pyserini REST Server with Config

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Launches the Pyserini REST server using a specified YAML configuration file. This is the primary command for starting the server with custom settings.

```bash
python -m pyserini.server.rest --config /path/to/server.yaml
```

--------------------------------

### Install Pyserini

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-cc.md

Install the latest version of pyserini from the root directory of the project. This command should be run after cloning the repository.

```bash
pip install .
```

--------------------------------

### Extended Entity Linking Script Example

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-entity-linking.md

An example of running the entity linking script with specific paths and configurations, assuming execution from the scripts directory.

```bash
REL_DATA_PATH=/home/$USER/REL/data
INPUT_JSONL_FILE=../collections/msmarco-passage/collection_jsonl/docs00.json
mkdir ../collections/msmarco-passage/collection_jsonl_with_entities/
OUTPUT_JSONL_FILE=../collections/msmarco-passage/msmarco-passage/collection_jsonl_with_entities/docs00.json
BASE_URL=$REL_DATA_PATH
ED_MODEL=$REL_DATA_PATH/ed-wiki-2019/model
WIKI_VERSION=wiki_2019
WIKIMAPPER_INDEX=$REL_DATA_PATH/index_enwiki-20190420.db

python entity_linking.py --input_path $INPUT_JSONL_FILE \
--rel_base_url $BASE_URL --rel_ed_model_path $ED_MODEL \
--rel_wiki_version $WIKI_VERSION --wikimapper_index $WIKIMAPPER_INDEX \
--spacy_model en_core_web_sm --output_path $OUTPUT_JSONL_FILE
```

--------------------------------

### Install spaCy and Download Model

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-spacy.md

Install the spaCy library and download a small English model for NLP tasks. Choose the best model for your application from the available options.

```bash
pip install spacy
python -m spacy download en_core_web_sm
```

--------------------------------

### Install Pyserini with Optional Dependencies

Source: https://github.com/castorini/pyserini/blob/master/README.md

Install Pyserini with optional features, such as multimodal support, by specifying the optional package.

```bash
pip install 'pyserini[optional]'
```

--------------------------------

### Start Server with Config and No Prebuilt Indexes

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Starts the Pyserini REST server with a configuration file and disables the use of prebuilt indexes. This enforces the use of indexes defined explicitly in the configuration.

```bash
python -m pyserini.server.rest --config /path/to/server.yaml --no-prebuilt-indexes
```

--------------------------------

### Initialize Faiss Searcher from Prebuilt Index

Source: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

Initialize a FaissSearcher from a prebuilt index. The 'msmarco-v1-passage.bge-base-en-v1.5' index is used as an example.

```python
searcher = FaissSearcher.from_prebuilt_index('msmarco-v1-passage.bge-base-en-v1.5', None)
```

--------------------------------

### Install Dependencies

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-nfcorpus-vis.md

Installs necessary Python packages for data preparation and visualization.

```bash
pip install uv datasets pandas
```

--------------------------------

### Mac Installation: Install Latest Pyserini

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs the latest version of Pyserini from PyPI. It's recommended to specify the version.

```bash
pip install pyserini==latest
```

--------------------------------

### Linux Installation: Install PyTorch CPU Version

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs the CPU-only version of PyTorch, torchvision, and torchaudio for Linux. Use the specified index URL for CPU wheels.

```bash
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
```

--------------------------------

### Start Pyserini MCP HTTP Server

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-mcp.md

Start the Pyserini MCP server on a specified port using HTTP transport. This is the initial step for remote access.

```bash
python -m pyserini.server.mcp --transport http --port 8000
```

--------------------------------

### Install Pyserini via PyPI

Source: https://github.com/castorini/pyserini/blob/master/README.md

Use this command to install the Pyserini library from the Python Package Index. This will automatically include major dependencies like PyTorch, 🤗 Transformers, and ONNX Runtime.

```bash
pip install pyserini
```

--------------------------------

### Install ACL Anthology Dependencies

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-acl-anthology.md

Navigate to the 'acl-anthology' directory and install the required Python dependencies using pip.

```bash
cd acl-anthology
pip install -r bin/requirements.txt
```

--------------------------------

### Start Pyserini REST Server

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Launches the Pyserini REST API server with default settings. The server defaults to host 0.0.0.0 and port 8081.

```bash
python -m pyserini.server.rest
```

--------------------------------

### Install vlm2vec-for-pyserini

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vlm2vec.md

Install the necessary package for Pyserini integration with VLM2Vec. Ensure compatible versions of torch, flash-attn, and transformers are handled.

```bash
pip install vlm2vec-for-pyserini
```

--------------------------------

### Run MCP Client Web UI

Source: https://github.com/castorini/pyserini/blob/master/pyserini/server/mcp/mcp-client/README.md

Start the MCP client and interact with it through a web interface. Navigate to http://127.0.0.1:7932 in your browser to access the UI.

```bash
clai web --agent rag_agent:agent
```

--------------------------------

### Start Pyserini REST API Server

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-v2.1.md

Launch the Anserini REST API server on a specified port. This server allows programmatic access to search functionalities.

```bash
python -m pyserini.server.AnseriniApplication --server.port=8082
```

--------------------------------

### Initialize Lucene Searcher from Prebuilt Index

Source: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

Initialize a LuceneSearcher from a prebuilt index. The index 'robust04' is used as an example. The downloaded index will be stored in ~/.cache/pyserini/indexes/.

```python
searcher = LuceneSearcher.from_prebuilt_index('robust04')
```

--------------------------------

### Setup ElasticSearch with Docker

Source: https://github.com/castorini/pyserini/blob/master/scripts/es/README.md

Use these Docker commands to pull the ElasticSearch image, create a network, and run an ElasticSearch instance. Ensure to use the correct image tag as specified.

```bash
docker pull docker.elastic.co/elasticsearch/elasticsearch:8.2.3
```

```bash
docker network create elastic
```

```bash
docker run --name es01 --net elastic -d -p 9200:9200 -p 9300:9300 \
           -e "discovery.type=single-node" \
           -e "xpack.security.enabled=false" \
           -it docker.elastic.co/elasticsearch/elasticsearch:8.2.2
```

--------------------------------

### Setup Kibana with Docker

Source: https://github.com/castorini/pyserini/blob/master/scripts/es/README.md

Optional step to set up Kibana for visualizing and managing your ElasticSearch data. Pull the Kibana image and run it in a detached mode.

```bash
docker pull docker.elastic.co/kibana/kibana:8.2.2
```

```bash
docker run -d --name kib-01 --net elastic -p 5601:5601 docker.elastic.co/kibana/kibana:8.2.2
```

--------------------------------

### Mac Installation: Create Conda Environment

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Sets up a new Conda environment named 'pyserini' with Python 3.12, essential for Pyserini on Mac.

```bash
conda create -n pyserini python=3.12 -y
conda activate pyserini
```

--------------------------------

### Sample Output JSON for Entity Linking

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-entity-linking.md

An example of a JSON object in the output JSONL file after entity linking has been performed.

```json
{
  "id": "doc1",
  "contents": "The Manhattan Project and its atomic bomb helped bring an end to World War II. Its legacy of peaceful uses of atomic energy continues to have an impact on history and science.",
  "entities": [
    {"start_pos": 0, "end_pos": 21, "ent_text": "The Manhattan Project", "wikipedia_id": "Manhattan_Project", "wikidata_id": "Q127050", "ent_type": "ORG"},
    {"start_pos": 65, "end_pos": 77, "ent_text": "World War II", "wikipedia_id": "World_War_II", "wikidata_id": "Q362", "ent_type": "EVENT"}
  ]
}
```

--------------------------------

### Start Pyserini REST Server with Custom Host and Port

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Launches the Pyserini REST API server with a specified host and port. This allows you to control the network interface and port the server listens on.

```bash
python -m pyserini.server.rest --host 127.0.0.1 --port 8080
```

--------------------------------

### Custom M-BEIR Query Instructions Configuration

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-m-beir-uniir.md

Example YAML configuration for custom M-BEIR query instructions. Specify the instruction file, candidate modality, dataset ID, and randomization behavior.

```yaml
instruction_file: collections/m-beir/query_instructions.tsv
candidate_modality: image
dataset_id: 8 # the id for CIRR is 8
randomize_instructions: False # When False, always gets the first available instruction for each query. Set it to true if you want to use instructions at the random indexes.
```

--------------------------------

### Download BRIGHT Corpus Data

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-qwen3-bright.md

Use wget to download the Pony dataset from the BRIGHT corpus. Ensure the path is correct for your setup.

```bash
wget https://huggingface.co/datasets/castorini/collections-bright/resolve/main/corpus/pony/pony.jsonl.gz
```

--------------------------------

### List Available Indexes by Type

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-mcp.md

Query the Pyserini MCP server to list available prebuilt index names for a specified index type. This example requests indexes of type 'tf'.

```json
{
  "index_type": "tf"
}
```

--------------------------------

### Enable JSONL Request Logging

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Starts the Pyserini REST server and directs unified JSONL request logs to a specified file. This provides detailed, machine-readable logs for each request.

```bash
python -m pyserini.server.rest \
  --config /path/to/server.yaml \
  --no-prebuilt-indexes \
  --log-file logs/rest.requests.jsonl
```

--------------------------------

### Run Pyserini Unit Tests

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Execute all unit tests for Pyserini to verify the development installation. This ensures that the codebase is functioning correctly after setup.

```bash
python -m unittest discover -s tests
```

--------------------------------

### Sample Retrieval Command

Source: https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/faiss.mrtydi-v1.1.20220413.aa1c0e9.mdpr-tied-pft-msmarco.README.md

Example command for performing retrieval on the Mr.TyDi v1.1 Arabic test set using the generated Faiss index and a specified encoder.

```bash
set_name=test
python -m pyserini.dsearch \
  --encoder castorini/mdpr-tied-pft-msmarco \
  --topics mrtydi-v1.1-${lang}-${set_name} \
  --index ${index_dir} \
  --output runs/run.mrtydi-v1.1-$lang.${set_name}.txt \
  --batch-size 36 \
  --threads 12 \
  --encoder-class 'auto'

```

--------------------------------

### Get Document with Pyserini MCP

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-mcp.md

Retrieve a specific document by its ID from a given index using the Pyserini MCP 'get_document' tool. This example fetches document '7157707' from 'msmarco-v1-passage'.

```json
{
  "docid": "7157707",
  "index": "msmarco-v1-passage"
}
```

--------------------------------

### Faster Retrieval with Anserini's SearchCollection (Python)

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-atomic-bm25.md

Utilize Anserini's SearchCollection class in a Python shell for potentially faster retrieval, especially with large indexes. This example shows argument setup for both Text-to-Image and Image-to-Text tasks.

```python
from pyserini.pyclass import autoclass
SearchCollection = autoclass("io.anserini.search.SearchCollection")
# Text to Image
t2i_search_args = [
    "-index", "lucene-index.atomic.image.{setting}",
    "-topics", "validation.text.search.jsonl",
    "-topicreader", "JsonString",
    "-topicfield", "title",
    "-output", "runs/run.validation.bm25-anserini-default.t2i.{setting}.trec",
    "-bm25", "-hits", "1000", "-parallelism", "64", "-threads", "64"

]
# Image to Text
i2t_search_args = [
    "-index", "lucene-index.atomic.text.{setting}",
    "-topics", "validation.image-caption.search.jsonl",
    "-topicreader", "JsonString",
    "-topicfield", "title",
    "-output", "runs/run.validation.bm25-anserini-default.i2t.{setting}.trec",
    "-bm25", "-hits", "1000", "-parallelism", "64", "-threads", "64"
]
SearchCollection.main(search_args)
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/castorini/pyserini/blob/master/pyserini/server/mcp/mcp-client/README.md

Set up a Python 3.12 virtual environment named 'agent' using conda and activate it. This is a prerequisite for installing client dependencies.

```bash
conda create -n agent python=3.12 -y
conda activate agent
```

--------------------------------

### Stream Graph Data to Gephi

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-cord19.md

Stream graph data from Neo4j to Gephi using the APOC plugin. This query retrieves the top 10 cited articles and their paths, then streams them to Gephi. Ensure APOC is installed in Neo4j and Graph Streaming is started in Gephi.

```cypher
MATCH (a)<-[r:BIB_REF]-(b) WITH a, count(r) as num_cites
WITH a ORDER BY num_cites DESC LIMIT 10
MATCH path = ()-->(a)
call apoc.gephi.add(null,'workspace1', path, 'weightproperty', ['CordUID','Title','DOI']) yield nodes, relationships
return nodes, relationships
```

--------------------------------

### Generate Run with GME-Qwen2-VL-2B-Instruct on MMLongBench

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/mmeb.html

Use this command to generate search results with the GME-Qwen2-VL-2B-Instruct model on the MMLongBench dataset. Parameters are similar to the VLM2Vec example, with the encoder and index names updated.

```bash
python -m pyserini.search.faiss \
  --encoder-class mmeb \
  --encoder Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
  --index mmeb-visdoc-MMLongBench-page.gme-Qwen2-VL-2B-Instruct \
  --topics-format raw_jsonl \
  --topics mmeb-visdoc-MMLongBench-page-test \
  --output-format trec \
  --output run.mmeb-visdoc-MMLongBench-page.gme-qwen2-vl-2b-instruct.txt \
  --fp16 --pooling eos --l2-norm \
  --hits 1000 --threads 16 --batch-size 512 --device cuda:0
```

--------------------------------

### Download and Extract M-BEIR Query Instructions

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-m-beir-uniir.md

Download the default M-BEIR query instructions and extract them. These are used for search when not providing a custom configuration.

```bash
wget -O collections/m-beir/mbeir_query_images_and_instructions.tar.gz \
"https://huggingface.co/datasets/castorini/prebuilt-indexes-m-beir/blob/main/mbeir_query_images_and_instructions.tar.gz"
tar -xzvf collections/m-beir/mbeir_query_images_and_instructions.tar.gz
```

--------------------------------

### Mac Installation: Install Faiss CPU Variant

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs the CPU version of Faiss within the 'pyserini' Conda environment on Mac. This is optional.

```bash
conda install -c pytorch faiss-cpu -y
```

--------------------------------

### Mac Installation: Install PyTorch

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Installs PyTorch, torchvision, and torchaudio for Mac. Refer to the official PyTorch website for specific version compatibility.

```bash
pip install torch torchvision torchaudio
```

--------------------------------

### Example Lucene Index Statistics Output

Source: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

This is an example of the output from calling the stats() method on a LuceneIndexReader initialized from a prebuilt index.

```json
{"total_terms": 174540872, "documents": 528030, "non_empty_documents": 528030, "unique_terms": 923436}
```

--------------------------------

### List Available Pre-built Indexes

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-interactive-search.md

Shows the command to list all available pre-built indexes that can be loaded by `LuceneSearcher`.

```shell
LuceneSearcher.list_prebuilt_indexes()
```

--------------------------------

### JSON Document Format Example

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-index.md

Example of a JSON document with multiple fields in its 'contents'. Fields are separated by newline characters by default.

```json
{
  "id": "doc1",
  "contents": "www.url.com\ntitle\nthis is the contents.\ndocument expansion"
}
```

```json
{
  "id": "CACM-2636",
  "contents": "Generation of Random Correlated Normal ... \n"
}
```

--------------------------------

### Install Client Dependencies

Source: https://github.com/castorini/pyserini/blob/master/pyserini/server/mcp/mcp-client/README.md

Install the necessary Python packages, pydantic-ai and clai, for the MCP client. This should be done within the activated conda environment.

```bash
pip install pydantic-ai clai
```

--------------------------------

### Install Git LFS

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-wiki-corpora.md

Ensure Git Large File Storage (LFS) is installed before cloning the repository. This is a prerequisite for handling large files.

```bash
git lfs install
```

--------------------------------

### Download AToMiC Data and Qrels

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-atomic-ViT-L-14.laion2b_s32b_b82k.md

Use these commands to download the topic directories and qrels files required for reproducing the AToMiC baselines. Ensure you are in the 'scripts/atomic' directory before running.

```bash
cd scripts/atomic
mkdir topics
wget https://huggingface.co/datasets/TREC-AToMiC/AToMiC-Baselines/resolve/main/topics/ViT-L-14.laion2b_s32b_b82k.image.validation.tar.gz -P topics
tar -xzvf topics/ViT-L-14.laion2b_s32b_b82k.image.validation.tar.gz -C topics
wget https://huggingface.co/datasets/TREC-AToMiC/AToMiC-Baselines/resolve/main/topics/ViT-L-14.laion2b_s32b_b82k.text.validation.tar.gz -P topics
tar -xzvf topics/ViT-L-14.laion2b_s32b_b82k.text.validation.tar.gz -C topics

mkdir qrels
wget https://huggingface.co/spaces/dlrudwo1269/AToMiC_bm25_files/resolve/main/qrels/qrels.atomic.validation.i2t.trec -P qrels
wget https://huggingface.co/spaces/dlrudwo1269/AToMiC_bm25_files/resolve/main/qrels/qrels.atomic.validation.t2i.trec -P qrels
```

--------------------------------

### Install Pyserini in Editable Mode

Source: https://github.com/castorini/pyserini/blob/master/docs/installation.md

Install Pyserini using pip in editable mode from the cloned repository. This allows for development and direct modification of the Pyserini code.

```bash
pip install -e .
```

--------------------------------

### List Available Prebuilt Faiss Indexes

Source: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

Use this snippet to list all prebuilt Faiss indexes available in Pyserini. This requires importing the FaissSearcher class.

```python
from pyserini.search.faiss import FaissSearcher
FaissSearcher.list_prebuilt_indexes()
```

--------------------------------

### Download Pre-built Index and PCA Model

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-dpr-compression.md

Download the pre-built index and PCA model for the DPR-PCA128-PQ2 experiment. These are required for subsequent compression and retrieval steps.

```bash
wget https://www.dropbox.com/s/nq62qhodd237p9t/dindex-dpr-multi-pca128.tar.gz
tar -xvf dindex-dpr-multi-pca128.tar.gz
rm dindex-dpr-multi-pca128.tar.gz
wget https://www.dropbox.com/s/d3osk3cjrgiawdp/dpr-multi-pca128
```

--------------------------------

### Download AToMiC Dataset Files

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-atomic-bm25.md

Download the required topic and qrels files for the AToMiC dataset. Ensure the DATA_DIR environment variable is set correctly.

```bash
export DATA_DIR="https://raw.githubusercontent.com/castorini/anserini-tools/master/topics-and-qrels"

mkdir topics
wget ${DATA_DIR}/topics.atomic.validation.text.jsonl -P topics
wget ${DATA_DIR}/topics.atomic.validation.image-caption.jsonl -P topics

mkdir qrels
wget ${DATA_DIR}/qrels.atomic.validation.i2t.trec -P qrels
wget ${DATA_DIR}/qrels.atomic.validation.t2i.trec -P qrels
```

--------------------------------

### Build Wiki-All Lucene Index

Source: https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/lucene-index-wiki-all-6-3-tamber-20230111-40277a.README.md

Command to build the Lucene index for the wiki-all-6-3-tamber dataset using Pyserini. Includes options for collection type, input path, output index path, document generator, threads, optimization, and storing raw documents.

```bash
python -m pyserini.index.lucene \
  --collection MrTyDiCollection \
  --input odqa-wiki-corpora/wiki-all-6-3-tamber \
  --index indexes/index-wiki-all-6-3-tamber-20230111-40277a \
  --generator DefaultLuceneDocumentGenerator \
  --threads 12 \
  --optimize \
  --storeRaw
```

--------------------------------

### Connect to Weaviate Cloud and Initialize Client

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vector-stores.md

Connect to a Weaviate Cloud instance using environment variables for URL and API key. This code snippet also includes checking if the client is ready.

```python
import weaviate, os
import weaviate.classes as wvc
import json

# Set these environment variables
URL = os.getenv("WEAVIATE_URL")
APIKEY = os.getenv("WEAVIATE_API_KEY")

# Connect to Weaviate Cloud
client = weaviate.connect_to_weaviate_cloud(
    cluster_url=URL,
    auth_credentials=wvc.init.Auth.api_key(APIKEY),
)

# Check connection
client.is_ready()

import weaviate.classes as wvc
```

--------------------------------

### Initialize and Search with SimpleSearcher

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-robust04.md

Load the pre-built 'robust04' index and perform a search query. The results include document IDs and scores.

```python
from pyserini.search.lucene import LuceneSearcher

searcher = LuceneSearcher.from_prebuilt_index('robust04')
hits = searcher.search('hubble space telescope')

# Print the first 10 hits:
for i in range(0, 10):
    print(f'{i+1:2} {hits[i].docid:15} {hits[i].score:.5f}')
```

--------------------------------

### REST API Search Results Example

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-v2.1.md

An example of the JSON output returned by the Pyserini REST API for a search query. It includes the query details and a list of candidate documents with their scores and metadata.

```json
{
  "query": {
    "text": "How does the process of digestion and metabolism of carbohydrates start",
    "qid": ""
  },
  "candidates": [
    {
      "docid": "msmarco_v2.1_doc_15_390497775",
      "score": 14.3364,
      "doc": {
        "url": "https://diabetestalk.net/blood-sugar/conversion-of-carbohydrates-to-glucose",
        "title": "Conversion Of Carbohydrates To Glucose | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    {
      "docid": "msmarco_v2.1_doc_15_416962410",
      "score": 14.2271,
      "doc": {
        "url": "https://diabetestalk.net/insulin/how-is-starch-converted-to-glucose-in-the-body",
        "title": "How Is Starch Converted To Glucose In The Body? | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    ...
  ]
}
```

--------------------------------

### Run Reason-Embed with Qwen3 on Bright Dataset

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-qwen3-bright.md

Executes the Reason-Embed model using Qwen3 for retrieval on a specified Bright dataset. Ensure the model and index paths are correct. The `--query-prefix` is dynamically constructed based on the dataset.

```bash
declare -A INSTRUCTIONS
INSTRUCTIONS["biology"]="Given a Biology post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["earth_science"]="Given an Earth Science post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["economics"]="Given an Economics post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["psychology"]="Given a Psychology post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["robotics"]="Given a Robotics post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["stackoverflow"]="Given a Stack Overflow post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["sustainable_living"]="Given a Sustainable Living post, retrieve relevant passages that help answer the post."
INSTRUCTIONS["leetcode"]="Given a Coding problem, retrieve relevant examples that help answer the problem."
INSTRUCTIONS["pony"]="Given a Pony question, retrieve relevant passages that help answer the question."
INSTRUCTIONS["aops"]="Given a Math problem, retrieve relevant examples that help answer the problem."
INSTRUCTIONS["theoremqa_questions"]="Given a Math problem, retrieve relevant examples that help answer the problem."
INSTRUCTIONS["theoremqa_theorems"]="Given a Math problem, retrieve relevant theorems that help answer the problem."

dataset_name="pony"
model="hanhainebula/reason-embed-qwen3-4b-0928"
model_name="reason-embed-qwen3-4b-0928"
instruction="${INSTRUCTIONS[$dataset_name]}"
python -m pyserini.search.faiss \
        --encoder $model \
        --encoder-class qwen3 \
        --index ./indexes/bright/faiss-flat.bright-${dataset_name}.${model_name} \
        --query-prefix "Instruct: ${instruction}"$'\n'"Query: " \
        --topics bright-${dataset_name}-original \
        --output ./runs/bright/run.bright-${dataset_name}.${model_name}.txt \
        --hits 1000 \
        --remove-query \
        --l2-norm \
        --fp16 \
        --topics-format raw_jsonl \
        --max-length 8192 \
        --device cuda:0
```

--------------------------------

### Example JSONL Request Log Line

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

An example of a single request log entry in JSONL format. This format includes details like authentication status, client IP, request method, path, status code, and latency.

```json
{"auth":"authenticated","client":"127.0.0.1","event":"request","key_id":"7f83b1657ff1","latency_ms":14.217,"method":"GET","path":"/v1/cacm/search","query":"query=information+retrieval&hits=1","query_truncated":false,"request_id":"8dd7f6fa4b7a4a04a029a70c7cf4ec75","status":200,"ts":"2026-05-31T16:42:03.123Z"}
```

--------------------------------

### Initialize Searcher from Pre-built Index

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-interactive-search.md

Shows how to initialize `LuceneSearcher` using a pre-built index name. This is a convenient way to load common indexes.

```python
from pyserini.search.lucene import LuceneSearcher

searcher = LuceneSearcher.from_prebuilt_index('robust04')
```

--------------------------------

### Download and Prepare CORD-19 Data

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-cord19.md

Downloads the CORD-19 dataset distribution and extracts the document parses. Ensure you have wget and tar installed.

```bash
DATE=2020-05-26
DATA_DIR=./collections/cord19-"${DATE}"
mkdir "${DATA_DIR}"

wget https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/"${DATE}"/document_parses.tar.gz -P "${DATA_DIR}"
wget https://ai2-semanticscholar-cord-19.s3-us-west-2.amazonaws.com/"${DATE}"/metadata.csv -P "${DATA_DIR}"

ls "${DATA_DIR}"/document_parses.tar.gz | xargs -I {} tar -zxvf {} -C "${DATA_DIR}"
rm "${DATA_DIR}"/document_parses.tar.gz
```

--------------------------------

### List All Experimental Conditions

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/ciral.html

Use this command to see all available experimental conditions that can be run.

```bash
python -m pyserini.2cr.ciral --list-conditions
```

--------------------------------

### Generate Search Run Command

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/ciral.html

Use this command to generate a search run for the Hausa language. Ensure pyserini is installed and configured.

```bash
python -m pyserini.search.lucene \
  --language ha \

```

--------------------------------

### Downloading and Evaluating Microsoft's Baseline

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-doc.md

Downloads the baseline run file provided by Microsoft and evaluates it using trec_eval to compare against Pyserini's results.

```bash
wget https://msmarco.blob.core.windows.net/msmarcoranking/msmarco-docdev-top100.gz -P runs
gunzip runs/msmarco-docdev-top100.gz

```

```bash
$ tools/eval/trec_eval.9.0.4/trec_eval -c -mrecall.100 -mmap \
   tools/topics-and-qrels/qrels.msmarco-doc.dev.txt runs/msmarco-docdev-top100

map                   	all	0.2219
recall_100            	all	0.7564

```

--------------------------------

### Sample Input JSON for Entity Linking

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-entity-linking.md

An example of a single JSON object in the input JSONL file for the entity linking script.

```json
{"id": "doc1", "contents": "The Manhattan Project and its atomic bomb helped bring an end to World War II. Its legacy of peaceful uses of atomic energy continues to have an impact on history and science."}
```

--------------------------------

### Train LTR Model from Scratch

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-ltr-msmarco-passage-training.md

Initiate the training process for an LTR model using a specified index. The training script will output results and save the trained model to the 'runs/' directory.

```bash
python scripts/ltr_msmarco/train_ltr_model.py  \
 --index ~/.cache/pyserini/indexes/index-msmarco-passage-ltr-20210519-e25e33f.a5de642c268ac1ed5892c069bdc29ae3 
```

--------------------------------

### Download MS MARCO Passage Dataset

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-msmarco-passage.md

Downloads the MS MARCO passage dataset collection and queries tarball. Ensure you have wget installed.

```bash
mkdir collections/msmarco-passage

wget https://msmarco.z22.web.core.windows.net/msmarcoranking/collectionandqueries.tar.gz -P collections/msmarco-passage

# Alternative mirror:
# wget https://www.dropbox.com/s/9f54jg2f71ray3b/collectionandqueries.tar.gz -P collections/msmarco-passage

tar xvfz collections/msmarco-passage/collectionandqueries.tar.gz -C collections/msmarco-passage
```

--------------------------------

### Download Visual Document Datasets

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-vlm2vec.md

Download the visual document datasets required for the MMEB benchmark. Customize the LOCAL_DIR to your preferred storage location.

```bash
LOCAL_DIR="./MMEB-V2" # <--- change this to the desired local directory
PYSERINI_DATA_DIR="$LOCAL_DIR/visdoc-tasks/pyserini"
python -m vlm2vec_for_pyserini.pyserini_integration.download_visdoc --local-dir $LOCAL_DIR
```

--------------------------------

### Initialize Lucene Index Reader from Prebuilt Index

Source: https://github.com/castorini/pyserini/blob/master/docs/prebuilt-indexes.md

Initialize a LuceneIndexReader from a prebuilt index and retrieve its statistics. The 'robust04' index is used as an example.

```python
index_reader = LuceneIndexReader.from_prebuilt_index('robust04')
index_reader.stats()
```

--------------------------------

### Get document by id

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-rest.md

Retrieves a specific document from the index using its unique document ID. Supports optional parsing and length limiting.

```APIDOC
## GET /v1/{index}/doc/{docid}

### Description
Retrieves a specific document from the index using its unique document ID. Supports optional parsing and length limiting.

### Method
GET

### Endpoint
/v1/{index}/doc/{docid}

### Parameters
#### Path Parameters
- **index** (string) - Required - The name of the index to query.
- **docid** (string) - Required - The unique identifier of the document to retrieve.

#### Query Parameters
- **parse** (boolean) - Optional - Defaults to `true`. Whether to parse the document content.
- **max_doc_length** (integer) - Optional - Maximum characters to return for the parsed document. If omitted, the full document is returned. Requires `parse=true`.

### Request Example
```bash
curl "http://localhost:8081/v1/msmarco-v1-passage/doc/7157707"
```

```bash
curl "http://localhost:8081/v1/msmarco-v1-passage/doc/7157707?max_doc_length=500"
```

```bash
curl -H "X-API-Key: {api-key}" \ 
  "http://localhost:8081/v1/msmarco-v1-passage/doc/7157707"
```

### Response
#### Success Response (200)
- **api** (string) - The API version.
- **index** (string) - The name of the index.
- **docid** (string) - The document ID.
- **doc** (string) - The document content or parsed document object.

#### Response Example
```json
{
  "api": "v1",
  "index": "msmarco-v1-passage",
  "docid": "7157707",
  "doc": "..."
}
```
```

--------------------------------

### Encode Wiki Corpus with DPR

Source: https://github.com/castorini/pyserini/blob/master/pyserini/resources/index-metadata/faiss-flat.wiki-all-6-3.dpr2-multi-retriever.20230103.186fa7.README.md

Use this command to generate embeddings for the wiki corpus using the DPR model. Ensure Tevatron is installed and configured.

```bash
python -m tevatron.driver.jax_encode \
  --output_dir=temp \
  --model_name_or_path wiki-all-6-3-multi-dpr2-passage-encoder  \
  --per_device_eval_batch_size 1248 \
  --dataset_name wiki_all_6_3.jsonl \
  --encoded_save_path corpus_emb.pkl \
  --p_max_len 256
```

--------------------------------

### Dense Retrieval with Faiss

Source: https://github.com/castorini/pyserini/blob/master/project-description.md

Utilize FaissSearcher for dense retrieval with Faiss indexes. Requires 'faiss' to be installed. Ensure the correct encoder and index are specified.

```python
from pyserini.encode import TctColBertQueryEncoder
from pyserini.search.faiss import FaissSearcher

encoder = TctColBertQueryEncoder('castorini/tct_colbert-v2-hnp-msmarco')
faiss_searcher = FaissSearcher.from_prebuilt_index(
    'msmarco-v1-passage.tct_colbert-v2-hnp',
    encoder
)
hits = faiss_searcher.search('what is a lobster roll')

for i in range(0, 10):
    print(f'{i+1:2} {hits[i].docid:7} {hits[i].score:.5f}')
```

--------------------------------

### Initialize Pyserini Searcher and Fetch Document

Source: https://github.com/castorini/pyserini/blob/master/docs/working-with-spacy.md

Initialize a SimpleSearcher from a pre-built index and fetch the raw text of a document using its docid. The raw document content is then parsed to extract the actual text.

```python
import json
from pyserini.search import SimpleSearcher

# Initialize a searcher from a pre-built index
searcher = SimpleSearcher.from_prebuilt_index('msmarco-passage')

# Fetch raw text of a document given its docid
raw = searcher.doc('1').raw()
# Get actual content from raw
content = json.loads(raw)['contents']
print(content)
```

--------------------------------

### Perform Dense Retrieval with FaissSearcher

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-search.md

Use FaissSearcher for dense retrieval with pre-built indices and query encoders. Ensure 'faiss' is installed as an optional dependency.

```python
from pyserini.encode import TctColBertQueryEncoder
from pyserini.search.faiss import FaissSearcher

encoder = TctColBertQueryEncoder('castorini/tct_colbert-v2-hnp-msmarco')
faiss_searcher = FaissSearcher.from_prebuilt_index(
    'msmarco-v1-passage.tct_colbert-v2-hnp',
    encoder)
hits = faiss_searcher.search('what is a lobster roll')

for i in range(0, 10):
    print(f'{i+1:2} {hits[i].docid:7} {hits[i].score:.5f}')
```

--------------------------------

### Run Retrieval with Pre-computed Weights

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-unicoil-tilde-expansion.md

Use this command to run retrieval with pre-tokenized queries and pre-computed weights, which are already included in Pyserini. This method is faster as it avoids neural inference over queries. Ensure impact scoring is enabled with the `--impact` flag.

```bash
python -m pyserini.search.lucene \
  --index indexes/lucene-index.msmarco-passage-unicoil-tilde-expansion \
  --topics msmarco-passage-dev-subset-unicoil-tilde \
  --output runs/run.msmarco-passage-unicoil-tilde-expansion.tsv \
  --output-format msmarco \
  --batch 36 --threads 12 \
  --hits 1000 \
  --impact
```

--------------------------------

### Download and Prepare CACM Collection

Source: https://github.com/castorini/pyserini/blob/master/docs/usage-collection.md

This bash script downloads the CACM collection, extracts it, and cleans up the archive file. Ensure you have wget and tar installed.

```bash
wget -O cacm.tar.gz https://github.com/castorini/anserini/blob/master/src/main/resources/cacm/cacm.tar.gz?raw=true
mkdir collections/cacm
tar xvfz cacm.tar.gz -C collections/cacm
rm cacm.tar.gz
```

--------------------------------

### Build Database with DrQA Preprocessing (Text Only)

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-wiki-corpora.md

Builds a database using DrQA's preprocessing script on extracted Wikipedia text. Requires the path to the extracted text and the desired database name.

```bash
python build_db.py \
  ../wiki_extractor_out/wiki-text/ \
  wiki-text.db \
  --preprocess prep_wikipedia.py
```

--------------------------------

### Run MMEB Search with Qwen2-VL-2B-Instruct

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/mmeb.html

Command to generate search results using the MMEB encoder with Alibaba-NLP/gme-Qwen2-VL-2B-Instruct. Ensure the index and topics are correctly specified.

```bash
python -m pyserini.search.faiss \
  --encoder-class mmeb \
  --encoder Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
  --index mmeb-visdoc-ViDoRe_shiftproject.gme-Qwen2-VL-2B-Instruct \
  --topics-format raw_jsonl \
  --topics mmeb-visdoc-ViDoRe_shiftproject-test \
  --output-format trec \
  --output run.mmeb-visdoc-ViDoRe_shiftproject.gme-qwen2-vl-2b-instruct.txt \
  --fp16 --pooling eos --l2-norm \
  --hits 1000 --threads 16 --batch-size 512 --device cuda:0
```

--------------------------------

### Download Pretrained IBM LTR Models

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-ltr-msmarco-passage-training.md

Download and extract pre-trained IBM models for LTR tasks. These models can be used as a starting point for reranking.

```bash
wget https://rgw.cs.uwaterloo.ca/JIMMYLIN-bucket0/pyserini-models/model-ltr-ibm.tar.gz -P collections/msmarco-ltr-passage/
tar -xzvf collections/msmarco-ltr-passage/model-ltr-ibm.tar.gz -C collections/msmarco-ltr-passage/
```

--------------------------------

### Generate MMEB Search Run with Qwen2-VL-2B-Instruct

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/mmeb.html

Command to generate a search run for the mmeb-visdoc-ViDoRe_docvqa dataset using the Alibaba-NLP/gme-Qwen2-VL-2B-Instruct encoder. Ensure the index and topics are correctly specified.

```bash
python -m pyserini.search.faiss \
  --encoder-class mmeb \
  --encoder Alibaba-NLP/gme-Qwen2-VL-2B-Instruct \
  --index mmeb-visdoc-ViDoRe_docvqa.gme-Qwen2-VL-2B-Instruct \
  --topics-format raw_jsonl \
  --topics mmeb-visdoc-ViDoRe_docvqa-test \
  --output-format trec \
  --output run.mmeb-visdoc-ViDoRe_docvqa.gme-qwen2-vl-2b-instruct.txt \
  --fp16 --pooling eos --l2-norm \
  --hits 1000 --threads 16 --batch-size 512 --device cuda:0
```

--------------------------------

### Run All Languages for All Conditions and Display Commands

Source: https://github.com/castorini/pyserini/blob/master/docs/2cr/mrtydi.html

Executes experiments for all languages across all conditions and displays the generated commands.

```bash
python -m pyserini.2cr.mrtydi --all --display-commands
```

--------------------------------

### Run Retrieval with On-the-Fly Query Encoding

Source: https://github.com/castorini/pyserini/blob/master/docs/experiments-unicoil-tilde-expansion.md

Use this command to run retrieval with a transformer model encoding queries on the fly using the CPU. Ensure impact scoring is enabled with the `--impact` flag. Retrieval may take around 25 minutes.

```bash
python -m pyserini.search.lucene \
  --index indexes/lucene-index.msmarco-passage-unicoil-tilde-expansion \
  --topics msmarco-passage-dev-subset \
  --encoder ielab/unicoil-tilde200-msmarco-passage \
  --output runs/run.msmarco-passage-unicoil-tilde-expansion.tsv \
  --output-format msmarco \
  --batch 36 --threads 12 \
  --hits 1000 \
  --impact
```