### Start Solr Instance

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-cord19-extras.md

Start a Solr instance with a specified memory allocation.

```bash
solrini/bin/solr start -c -m 16G
```

--------------------------------

### Unpack and Start Kibana

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-cord19-extras.md

Unpacks Kibana and starts the Kibana service. Assumes Kibana distribution is in the 'anserini/' directory.

```bash
tar -zxvf kibana*.tar.gz -C elastirini --strip-components=1
elastirini/bin/kibana
```

--------------------------------

### Start Elasticsearch

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-cord19-extras.md

Starts the Elasticsearch service from the 'elastirini' directory.

```bash
elastirini/bin/elasticsearch
```

--------------------------------

### Clone Anserini Repository and Initialize Submodules

Source: https://github.com/castorini/anserini/blob/master/docs/start-here.md

Clone the Anserini repository and update its submodules. This is the initial setup step.

```bash
git clone https://github.com/castorini/anserini.git
cd anserini
git submodule update --init --recursive
```

--------------------------------

### Run Complete Regression (Download and Index)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl23-passage.splade-pp-ed.onnx.md

Downloads the required corpus, indexes it, and then performs the complete regression test suite (verification and searching). This command is suitable for initial setup or when the corpus is not available locally.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --download --index --verify --search --config dl23-passage.splade-pp-ed.onnx
```

--------------------------------

### Example Conda Environment for ECIR 2019 Experiments

Source: https://github.com/castorini/anserini/blob/master/docs/runbook-ecir2019-ccrf.md

An example output of `conda list` showing the packages and their versions in the Conda environment used for the ECIR 2019 experiments. This can be used as a reference for setting up a compatible environment.

```bash
$ conda list
# packages in environment at /anaconda3/envs/python36:
#
# Name                    Version                   Build  Channel
blas                      1.0                         mkl  
bzip2                     1.0.6                         1    conda-forge
ca-certificates           2018.11.29           ha4d7672_0    conda-forge
certifi                   2018.11.29            py36_1000    conda-forge
clangdev                  4.0.0                 default_0    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
intel-openmp              2019.1                      144  
libcxx                    4.0.1                hcfea43d_1  
libcxxabi                 4.0.1                hcfea43d_1  
libedit                   3.1.20170329         hb402a30_2  
libffi                    3.2.1                h475c297_4  
libgfortran               3.0.1                h93005f0_2  
libiconv                  1.15                 h470a237_3    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
lightgbm                  2.2.1            py36hfc679d8_0    conda-forge
llvmdev                   4.0.0                 default_0    conda-forge
mkl                       2019.1                      144  
mkl_fft                   1.0.10                   py36_0    conda-forge
mkl_random                1.0.2                    py36_0    conda-forge
ncurses                   6.1                  h0a44026_1  
numpy                     1.15.4           py36hacdab7b_0  
numpy-base                1.15.4           py36h6575580_0  
openmp                    4.0.0                         1    conda-forge
openssl                   1.0.2p               h470a237_1    conda-forge
pip                       18.1                     py36_0  
python                    3.6.6                h5001a0f_0    conda-forge
readline                  7.0                  h1de35cc_5  
scikit-learn              0.20.1           py36h27c97d8_0  
scipy                     1.1.0            py36h1410ff5_2  
setuptools                40.6.2                   py36_0  
sqlite                    3.25.3               ha441bb4_0  
tk                        8.6.8                ha441bb4_0  
wheel                     0.32.3                   py36_0  
xz                        5.2.4                h1de35cc_4  
zlib                      1.2.11               h1de35cc_3  

```

--------------------------------

### Example Indexing Log Output

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-cord19.md

This is an example of the expected log output after successfully indexing documents. It provides a summary of indexed, unindexed, empty, skipped, and error documents, along with the total indexing time.

```log
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:874) - Indexing Complete! 192,459 documents indexed
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:875) - ============ Final Counter Values ============ 
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:876) - indexed:          192,459
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:877) - unindexable:            0
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:878) - empty:                 44
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:879) - skipped:                6
2020-07-22 08:15:07,372 INFO  [main] index.IndexCollection (IndexCollection.java:880) - errors:                 0
2020-07-22 08:15:07,378 INFO  [main] index.IndexCollection (IndexCollection.java:883) - Total 192,459 documents indexed in 00:02:46
```

```log
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:874) - Indexing Complete! 192,460 documents indexed
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:875) - ============ Final Counter Values ============ 
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:876) - indexed:          192,460
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:877) - unindexable:            0
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:878) - empty:                 43
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:879) - skipped:                6
2020-07-22 08:23:04,801 INFO  [main] index.IndexCollection (IndexCollection.java:880) - errors:                 0
2020-07-22 08:23:04,806 INFO  [main] index.IndexCollection (IndexCollection.java:883) - Total 192,460 documents indexed in 00:07:56
```

--------------------------------

### Example Index Location Schema

Source: https://github.com/castorini/anserini/blob/master/docs/prebuilt-indexes.md

This shows the expected directory structure for a manually managed index within the default cache location. It includes the index name, version, and checksum.

```text
~/.cache/pyserini/indexes/lucene-inverted.msmarco-v1-passage.20221004.252b5e.678876e8c99a89933d553609a0fd8793
```

--------------------------------

### Start Anserini REST API

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.38.0.md

Command to start the Anserini REST API server.

```APIDOC
## Start Anserini REST API

To start the REST API, run the following command:

```bash
java -cp $ANSERINI_JAR io.anserini.server.Application --server.port=8081
```

Once started, you can access the webapp at `http://localhost:8081/`.
```

--------------------------------

### Run Complete Regression (download corpus)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl19-passage.bm25-b8.md

Download the corpus (as quantized BM25 weights) and perform the complete regression, including indexing, verification, and search, from any machine.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --download --index --verify --search --config dl19-passage.bm25-b8

```

--------------------------------

### Install Python Dependencies

Source: https://github.com/castorini/anserini/blob/master/docs/runbook-ecir2019-axiomatic.md

Installs the necessary Python packages using pip. Ensure you have Python 2.6+ or 3.5+.

```bash
pip install -r src/main/python/requirements.txt
```

--------------------------------

### Download and View Statistics of a Prebuilt Index

Source: https://github.com/castorini/anserini/blob/master/docs/prebuilt-indexes.md

Use this command to download a prebuilt index and display its statistics. Anserini automatically handles the download and caching of the specified index.

```bash
bin/run.sh io.anserini.index.IndexReaderUtils -index cacm -stats
```

--------------------------------

### Reproduce BRIGHT Core Regressions

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.6.0.md

Use this command to reproduce the core regressions for the BRIGHT benchmark. It will download necessary indexes and compute results. Be aware of the significant disk space required.

```bash
java -cp $ANSERINI_JAR io.anserini.reproduce.RunRegressionsFromPrebuiltIndexes -printCommands -computeIndexSize -regression bright.core
```

--------------------------------

### Start Anserini REST API

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.37.0.md

Command to start the Anserini REST API server. The server will be accessible at http://localhost:8081.

```APIDOC
## Start Anserini REST API

### Description
Starts the Anserini REST API server on the specified port.

### Command
```bash
java -cp $ANSERINI_JAR io.anserini.server.Application --server.port=8081
```
```

--------------------------------

### Reproduce Full Pipeline (Index, Verify, Search)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/msmarco-v1-passage.bge-base-en-v1.5.parquet.hnsw.onnx.md

Executes the complete reproduction pipeline, including indexing, verification, and searching, using a pre-defined configuration.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --index --verify --search --config msmarco-v1-passage.bge-base-en-v1.5.parquet.hnsw.onnx \
  --corpus-path collections/msmarco-passage-bge-base-en-v1.5.parquet
```

--------------------------------

### Install ONNX Conversion Dependencies

Source: https://github.com/castorini/anserini/blob/master/docs/onnx-conversion.md

Installs necessary Python packages for ONNX conversion and optimization. Versions tested are also listed.

```bash
pip install torch transformers onnx onnxruntime onnxoptimizer onnxscript
```

--------------------------------

### Start Anserini REST API

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.36.1.md

Starts the Anserini built-in webapp and REST API on port 8081. Access the web interface at http://localhost:8081/.

```bash
java -cp $ANSERINI_JAR io.anserini.server.Application --server.port=8081
```

--------------------------------

### Reproduce BRIGHT Core Runs

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.7.0.md

Executes reproduction commands for the core BRIGHT benchmark configurations. This command will download necessary indexes.

```bash
java $JAVA_OPTS io.anserini.reproduce.ReproduceFromPrebuiltIndexes --print-commands --compute-index-size --config bright.core
```

--------------------------------

### Quick Build Script

Source: https://github.com/castorini/anserini/blob/master/CLAUDE.md

Use this script for a quick build of the Anserini project.

```bash
bin/qbuild.sh
```

--------------------------------

### Example JSONL output structure

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.2.0.md

This is an example of the JSONL output generated by `GenerateRerankerRequests`. It includes query information and a list of candidate documents with their scores and content.

```json
{
  "query": {
    "qid": "2024-105741",
    "text": "is it dangerous to have wbc over 15,000 without treatment?"
  },
  "candidates": [
    {
      "docid": "msmarco_v2.1_doc_16_287012450#4_490828734",
      "score": 15.8199,
      "doc": {
        "url": "https://emedicine.medscape.com/article/961169-treatment",
        "title": "Bacteremia Treatment & Management: Medical Care",
        "headings": "Bacteremia Treatment & Management\nBacteremia Treatment & Management\nMedical Care\nHow well do low-risk criteria work?\nEmpiric antibiotics: How well do they work?\nTreatment algorithms\n",
        "segment": "band-to-neutrophil ratio\n< 0.2\n< 20,000/μL\n5-15,000/μL; ABC < 1,000\n5-15,000/μL; ABC < 1,000\nUrine assessment\n< 10 WBCs per HPF; Negative for bacteria\n< 10 WBCs per HPF; Leukocyte esterase negative\n< 10 WBCs per HPF\n< 5 WBCs per HPF\nCSF assessment\n< 8 WBCs per HPF; Negative for bacteria\n< 10 WBCs per HPF\n< 10-20 WBCs per HPF\n…\nChest radiography\nNo infiltrate\nWithin reference range, if obtained\nWithin reference range, if obtained\n…\nStool culture\n< 5 WBCs per HPF\n…\n< 5 WBCs per HPF\n…\n* Acute illness observation score\nHow well do low-risk criteria work? The above guidelines are presented to define a group of febrile young infants who can be treated without antibiotics. Statistically, this translates into a high NPV (ie, a very high proportion of true negative cultures is observed in patients deemed to be at low risk). The NPV of various low-risk criteria for serious bacterial infection and occult bacteremia are as follows [ 10, 14, 16, 19, 74, 75, 76] : Philadelphia NPV - 95-100%\nBoston NPV - 95-98%\nRochester NPV - 98.3-99%\nAAP 1993 - 99-99.8%\nIn basic terms, even by the most stringent criteria, somewhere between 1 in 100 and 1 in 500 low-risk, but bacteremic, febrile infants are missed.",
        "start_char": 2846,
        "end_char": 4049
      }
    },
    {
      "docid": "msmarco_v2.1_doc_16_287012450#3_490827079",
      "score": 15.231,
      "doc": {
        "url": "https://emedicine.medscape.com/article/961169-treatment",
        "title": "Bacteremia Treatment & Management: Medical Care",
        "headings": "Bacteremia Treatment & Management\nBacteremia Treatment & Management\nMedical Care\nHow well do low-risk criteria work?\nEmpiric antibiotics: How well do they work?\nTreatment algorithms\n",
        "segment": "73] Since then, numerous studies have evaluated combinations of age, temperature, history, examination findings, and laboratory results to determine which young infants are at a low risk for bacterial infection. [ 10, 66, 74, 75, 76]\nThe following are the low-risk criteria established by groups from Philadelphia, Boston, and Rochester and the 1993 American Academy of Pediatrics (AAP) guideline. Table 11. Low-Risk Criteria for Infants Younger than 3 Months [ 10, 74, 75, 76] (Open Table in a new window)\nCriterion\nPhiladelphia\nBoston\nRochester\nAAP 1993\nAge\n1-2 mo\n1-2 mo\n0-3 mo\n1-3 mo\nTemperature\n38.2°C\n≥38°C\n≥38°C\n≥38°C\nAppearance\nAIOS * < 15\nWell\nAny\nWell\nHistory\nImmune\nNo antibiotics in the last 24 h; No immunizations in the last 48 h\nPreviously healthy\nPreviously healthy\nExamination\nNonfocal\nNonfocal\nNonfocal\nNonfocal\nWBC count\n< 15,000/μL; band-to-neutrophil ratio\n< 0.2\n< 20,000/μL\n5-15,000/μL; ABC < 1,000\n5-15,000/μL; ABC < 1,000\nUrine assessment\n< 10 WBCs per HPF; Negative for bacteria\n< 10 WBCs per HPF; Leukocyte esterase negative\n< 10 WBCs per HPF\n< 5 WBCs per HPF\nCSF assessment\n< 8 WBCs per HPF;",
        "start_char": 1993,
        "end_char": 3111
      }
    },
    ...
  ]
}
```

--------------------------------

### Start Anserini REST API

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.39.0.md

This command starts the Anserini REST API server on port 8081. It requires the Anserini fatjar to be downloaded and the ANSERINI_JAR environment variable to be set.

```APIDOC
## Start Anserini REST API

### Description
Starts the Anserini REST API server.

### Command
```bash
java -cp $ANSERINI_JAR io.anserini.server.Application --server.port=8081
```
```

--------------------------------

### Example JSONL Output for Reranker Requests

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.36.0.md

An example of the JSONL output format generated for reranker requests. Each entry contains query information and a list of candidate documents with their scores.

```json
{
  "query": {
    "text": "How does the process of digestion and metabolism of carbohydrates start",
    "qid": 2000138
  },
  "candidates": [
    {
      "docid": "msmarco_v2.1_doc_15_390497775",
      "score": 14.3364,
      "doc": {
        "url": "https://diabetestalk.net/blood-sugar/conversion-of-carbohydrates-to-glucose",
        "title": "Conversion Of Carbohydrates To Glucose | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    {
      "docid": "msmarco_v2.1_doc_15_416962410",
      "score": 14.2271,
      "doc": {
        "url": "https://diabetestalk.net/insulin/how-is-starch-converted-to-glucose-in-the-body",
        "title": "How Is Starch Converted To Glucose In The Body? | DiabetesTalk.Net",
        "headings": "...",
        "body": "..."
      }
    },
    ...
  ]
}
```

--------------------------------

### Sample Indexing Command

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/beir-v1.0.0-cqadupstack-english.splade-v3.onnx.md

Index the BEIR v1.0.0 CQADupStack-english dataset using SPLADE-v3 with specific indexing options for impact scoring and pre-tokenized input.

```bash
bin/run.sh io.anserini.index.IndexCollection \
  -threads 16 \
  -collection JsonVectorCollection \
  -input /path/to/beir-v1.0.0-cqadupstack-english.splade-v3 \
  -generator DefaultLuceneDocumentGenerator \
  -index indexes/lucene-inverted.beir-v1.0.0-cqadupstack-english.splade-v3/ \
  -impact -pretokenized \
  >& logs/log.beir-v1.0.0-cqadupstack-english.splade-v3 &
```

--------------------------------

### Sample Indexing Command

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/beir-v1.0.0-cqadupstack-physics.splade-v3.onnx.md

Index the BEIR v1.0.0 CQADupStack-physics dataset using SPLADE-v3 with specific indexing options. Use `-impact -pretokenized` for efficient indexing without default BM25 doclengths and to prevent additional tokenization.

```bash
bin/run.sh io.anserini.index.IndexCollection \
  -threads 16 \
  -collection JsonVectorCollection \
  -input /path/to/beir-v1.0.0-cqadupstack-physics.splade-v3 \
  -generator DefaultLuceneDocumentGenerator \
  -index indexes/lucene-inverted.beir-v1.0.0-cqadupstack-physics.splade-v3/ \
  -impact -pretokenized \
  >& logs/log.beir-v1.0.0-cqadupstack-physics.splade-v3 &
```

--------------------------------

### Reproduce BRIGHT Optional Regressions

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.6.0.md

Use this command to reproduce the optional regressions for the BRIGHT benchmark. This command also downloads indexes and computes results. Note the substantial disk space requirement.

```bash
java -cp $ANSERINI_JAR io.anserini.reproduce.RunRegressionsFromPrebuiltIndexes -printCommands -computeIndexSize -regression bright.optional
```

--------------------------------

### Sample Indexing Command

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/beir-v1.0.0-webis-touche2020.splade-v3.onnx.md

Indexes the BEIR v1.0.0 Webis-Touche2020 dataset using the SPLADE-v3 model with specific indexing options for impact and pre-tokenization.

```bash
bin/run.sh io.anserini.index.IndexCollection \
  -threads 16 \
  -collection JsonVectorCollection \
  -input /path/to/beir-v1.0.0-webis-touche2020.splade-v3 \
  -generator DefaultLuceneDocumentGenerator \
  -index indexes/lucene-inverted.beir-v1.0.0-webis-touche2020.splade-v3/ \
  -impact -pretokenized \
  >& logs/log.beir-v1.0.0-webis-touche2020.splade-v3 &
```

--------------------------------

### Perform Retrieval with Tuned BM25, Rocchio, and Negative Examples

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl20-passage.md

Executes retrieval using tuned BM25 parameters with Rocchio query expansion, including negative examples and a rerank cutoff. This command enables negative feedback in Rocchio expansion with optimized BM25.

```bash
bin/run.sh io.anserini.search.SearchCollection \
  -index indexes/lucene-inverted.msmarco-v1-passage/ \
  -topics tools/topics-and-qrels/topics.dl20.txt \
  -topicReader TsvInt \
  -output runs/run.msmarco-passage.bm25-tuned+rocchio-neg.txt \
  -bm25 -bm25.k1 0.82 -bm25.b 0.68 -rocchio -rocchio.useNegative -rerankCutoff 1000 &
```

--------------------------------

### Reproduce BRIGHT Optional Runs

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.7.0.md

Executes reproduction commands for optional BRIGHT benchmark configurations. This command will download necessary indexes.

```bash
java $JAVA_OPTS io.anserini.reproduce.ReproduceFromPrebuiltIndexes --print-commands --compute-index-size --config bright.optional
```

--------------------------------

### Search API Endpoint

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.38.0.md

Example of using the REST API to perform a search query.

```APIDOC
## Search API Endpoint

### Description
This endpoint allows you to search within a specified index using a given query.

### Method
GET

### Endpoint
`/api/v1.0/indexes/{index_name}/search`

### Parameters
#### Query Parameters
- **query** (string) - Required - The search query string.
- **hits** (integer) - Optional - The number of search hits to return. Defaults to a system-defined value.

### Request Example
```bash
curl -X GET "http://localhost:8081/api/v1.0/indexes/msmarco-v2.1-doc/search?query=How%20does%20the%20process%20of%20digestion%20and%20metabolism%20of%20carbohydrates%20start&hits=1000"
```

### Response
#### Success Response (200)
The response contains JSON results similar to the output of the `-outputRerankerRequests` option in `SearchCollection`.

#### Response Example
(Response structure is the same as `SearchCollection -outputRerankerRequests` output, which is not detailed here.)

**Note:** To query segmented documents, switch the index name in the route to `msmarco-v2.1-doc-segmented`.
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.3.0.md

Fetches the Anserini fatjar for version 1.3.0 from Maven Central. Ensure you have wget installed.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/1.3.0/anserini-1.3.0-fatjar.jar
```

--------------------------------

### Reproduce Full Pipeline with Document Collection

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl20-passage.cos-dpr-distil.parquet.hnsw.onnx.md

Executes the complete pipeline from downloading to searching, using a configuration file to manage the process. This command performs indexing, verification, and searching.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --index --verify --search --config dl20-passage.cos-dpr-distil.parquet.hnsw.onnx \
  --corpus-path collections/msmarco-passage-cos-dpr-distil.parquet
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.2.2.md

Fetches the Anserini fatjar JAR file from Maven Central. Ensure you have wget installed.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/1.2.2/anserini-1.2.2-fatjar.jar
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.39.0.md

Fetches the Anserini fatjar JAR file from Maven Central. Ensure you have wget installed.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/0.39.0/anserini-0.39.0-fatjar.jar
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.36.1.md

Fetches the Anserini fatjar for version 0.36.1 using wget. Prebuilt indexes are downloaded to ~/.cache/pyserini/indexes/ by default.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/0.36.1/anserini-0.36.1-fatjar.jar
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.37.0.md

Fetches the Anserini fatjar JAR file using wget. Ensure you have wget installed.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/0.37.0/anserini-0.37.0-fatjar.jar
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.38.0.md

Fetches the Anserini fatjar for version 0.38.0 using wget. Prebuilt indexes are downloaded to ~/.cache/pyserini/indexes/ by default.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/0.38.0/anserini-0.38.0-fatjar.jar
```

--------------------------------

### Pretty-print Reranker Request JSONL

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.38.0.md

Example of how to view the head of the generated JSONL file using `jq` for pretty-printing.

```bash
$ head -n 1 $OUTPUT_DIR/results.msmarco-v2.1-doc.bm25.rag24.test.jsonl | jq
```

--------------------------------

### Run Full Regression (Download and Local)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/rag24-doc-segmented-raggy-dev.splade-v3.onnx.md

Downloads the necessary corpus and then executes the complete regression test suite for the specified configuration. Suitable for running end-to-end on any machine.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --download --index --verify --search --config rag24-doc-segmented-raggy-dev.splade-v3.onnx
```

--------------------------------

### Run Complete Regression (download corpus first)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl21-passage.splade-pp-sd.onnx.md

Download the corpus and then perform the complete regression, including indexing, verification, and search. This command can be run from any machine.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --download --index --verify --search --config dl21-passage.splade-pp-sd.onnx
```

--------------------------------

### Run Complete Regression (Existing Infrastructure)

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/dl20-passage.bge-base-en-v1.5.parquet.hnsw.onnx.md

Executes the full regression test suite, including indexing, verification, and searching, on a pre-configured environment. Assumes the necessary data and model are already available.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --index --verify --search --config dl20-passage.bge-base-en-v1.5.parquet.hnsw.onnx
```

--------------------------------

### Download Anserini Fatjar

Source: https://github.com/castorini/anserini/blob/master/docs/installation-fatjar.md

Fetches the latest Anserini fatjar (v2.2.0) using curl. Ensure Java 21 is installed.

```bash
curl -fL -o anserini-2.2.0-fatjar.jar https://repo1.maven.org/maven2/io/anserini/anserini/2.2.0/anserini-2.2.0-fatjar.jar
```

--------------------------------

### Download Prebuilt MS MARCO Document Indexes

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-msmarco-doc-leaderboard.md

Use Pyserini to automatically download prebuilt indexes for MS MARCO document search. Indexes are downloaded to `~/.cache/pyserini/indexes/`.

```python
from pyserini.search import SimpleSearcher; SimpleSearcher.from_prebuilt_index('msmarco-doc')
```

```python
from pyserini.search import SimpleSearcher; SimpleSearcher.from_prebuilt_index('msmarco-doc-per-passage')
```

--------------------------------

### Download Anserini Fatjar v0.36.0

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v0.36.0.md

Fetches the Anserini fatjar for version 0.36.0 using wget. Ensure you have wget installed.

```bash
wget https://repo1.maven.org/maven2/io/anserini/anserini/0.36.0/anserini-0.36.0-fatjar.jar
```

--------------------------------

### Example Evaluation Results

Source: https://github.com/castorini/anserini/blob/master/docs/experiments-doc2query.md

Displays the expected evaluation results for MAP and reciprocal rank after running the retrieval and evaluation commands.

```text
map                   	all	0.1807
recip_rank            	all	0.2750
```

--------------------------------

### Dry Run Reproductions with Prebuilt Indexes

Source: https://github.com/castorini/anserini/blob/master/docs/release-notes/fatjar-reproduction-notes-v2.1.0.md

Use these commands to preview available reproduction configurations for various datasets without executing the actual runs. Ensure the Anserini fatjar is in the classpath.

```bash
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config beir.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config beir.optional

java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config bright.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config bright.optional

java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v1-passage.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v1-passage.optional
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v1-doc.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v1-doc.optional

java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2-passage.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2-passage.optional
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2-doc.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2-doc.optional

java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2.1-doc-segmented.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2.1-doc-segmented.optional
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2.1-doc.core
java -cp `ls *-fatjar.jar` io.anserini.reproduce.ReproduceFromPrebuiltIndexes --dry-run --config msmarco-v2.1-doc.optional
```

--------------------------------

### Conda Environment Packages

Source: https://github.com/castorini/anserini/blob/master/docs/runbook-trec2018-h2oloo.md

Lists packages installed in a specific conda environment, useful for reproducing the environment for running the evaluation commands.

```bash
$ conda list
# packages in environment at /anaconda3/envs/python36:
#

```

--------------------------------

### Reproduce MS MARCO V2 Document Indexes (Optional)

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.7.1.md

Use this command to reproduce runs for MS MARCO v2 document optional indexes. It prints commands and computes index size. Beware of large download sizes.

```bash
java $JAVA_OPTS io.anserini.reproduce.ReproduceFromPrebuiltIndexes --print-commands --compute-index-size --config msmarco-v2-doc.optional
```

--------------------------------

### Query Anserini REST API

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.0.0.md

Issues a GET request to the Anserini REST API to search the msmarco-v2.1-doc-segmented index with a specific query.

```bash
curl -X GET "http://localhost:8081/api/v1.0/indexes/msmarco-v2.1-doc-segmented/search?query=How%20does%20the%20process%20of%20digestion%20and%20metabolism%20of%20carbohydrates%20start"
```

--------------------------------

### Sample Indexing Command

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/beir-v1.0.0-climate-fever.splade-v3.onnx.md

Index the Climate-FEVER dataset using the SPLADE-v3 model with ONNX. Key options include `-impact` and `-pretokenized` for specific indexing behaviors.

```bash
bin/run.sh io.anserini.index.IndexCollection \
  -threads 16 \
  -collection JsonVectorCollection \
  -input /path/to/beir-v1.0.0-climate-fever.splade-v3 \
  -generator DefaultLuceneDocumentGenerator \
  -index indexes/lucene-inverted.beir-v1.0.0-climate-fever.splade-v3/ \
  -impact -pretokenized \
  >& logs/log.beir-v1.0.0-climate-fever.splade-v3 &
```

--------------------------------

### Convert Shapefile to GeoJSON with GDAL

Source: https://github.com/castorini/anserini/blob/master/docs/elk-hydrosheds.md

Use the ogr2ogr command-line tool to convert shapefile data to GeoJSON format. Ensure GDAL is installed in your environment.

```bash
conda install -c conda-forge gdal
ogr2ogr -f GeoJSON output.geojson HydroRIVERS_v10_gr.shp
```

--------------------------------

### Reproduce MS MARCO V2 Document Indexes (Core)

Source: https://github.com/castorini/anserini/blob/master/docs/fatjar-regressions/fatjar-regressions-v1.7.1.md

Use this command to reproduce runs for MS MARCO v2 document core indexes. It prints commands and computes index size. Beware of large download sizes.

```bash
java $JAVA_OPTS io.anserini.reproduce.ReproduceFromPrebuiltIndexes --print-commands --compute-index-size --config msmarco-v2-doc.core
```

--------------------------------

### Run Full Regression

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/bright-leetcode.bm25qs.md

Executes the complete regression pipeline, including indexing, verification, and search. Ensure the configuration file 'bright-leetcode.bm25qs' is correctly set up.

```bash
bin/run.sh io.anserini.reproduce.ReproduceFromDocumentCollection --index --verify --search --config bright-leetcode.bm25qs
```

--------------------------------

### Example Download URL for msmarco-v1-passage

Source: https://github.com/castorini/anserini/blob/master/docs/prebuilt-indexes.md

This is the direct URL to download the compressed tarball for the msmarco-v1-passage index. Verify the MD5 checksum provided to ensure data integrity.

```text
https://huggingface.co/datasets/castorini/prebuilt-indexes-msmarco-v1/resolve/main/passage/original/lucene-inverted/tf/lucene-inverted.msmarco-v1-passage.20221004.252b5e.tar.gz
```

--------------------------------

### Sample Indexing Command

Source: https://github.com/castorini/anserini/blob/master/docs/reproduce/from-document-collection/beir-v1.0.0-cqadupstack-webmasters.splade-v3.onnx.md

Index the BEIR v1.0.0 CQADupStack-webmasters collection using SPLADE-v3 with specific indexing options. The -impact and -pretokenized flags are important for this configuration.

```bash
bin/run.sh io.anserini.index.IndexCollection \
  -threads 16 \
  -collection JsonVectorCollection \
  -input /path/to/beir-v1.0.0-cqadupstack-webmasters.splade-v3 \
  -generator DefaultLuceneDocumentGenerator \
  -index indexes/lucene-inverted.beir-v1.0.0-cqadupstack-webmasters.splade-v3/ \
  -impact -pretokenized \
  >& logs/log.beir-v1.0.0-cqadupstack-webmasters.splade-v3 &
```