### Environment Setup Scripts (Bash)

Source: https://context7.com/orionw/promptriever/llms.txt

Provides bash scripts to set up the development environment. This includes installing Conda, installing project requirements, and configuring Git and Hugging Face credentials.

```bash
# Install Conda if not already installed:
bash scripts/setup/install_conda.sh

# Install all requirements:
bash scripts/setup/install_req.sh

# Manual installation steps:
pip install deepspeed accelerate
pip install transformers datasets peft
pip install faiss-cpu
pip install -r requirements.txt
pip install -e .

# Install custom Tevatron fork:
pip install git+https://github.com/orionw/tevatron

# Configure git credentials for Hugging Face:
git config --global credential.helper store
huggingface-cli login --token $HF_TOKEN --add-to-git-credential
```

--------------------------------

### Filtering Instruction-Negative Pairs using FollowIR Model

Source: https://context7.com/orionw/promptriever/llms.txt

This code snippet filters query-document pairs using the FollowIR model to identify and score relevance of instruction-negative examples during dataset creation. It utilizes Hugging Face Transformers for model loading and inference. Dependencies include PyTorch and Transformers.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

TEMPLATE = """<s> [INST] You are an expert Google searcher, whose job is to determine if the following document is relevant to the query (true/false). Answer using only one word, one of those two choices.

Query: {query}
Document: {text}
Relevant (only output one word, either "true" or "false"): [/INST] """

def load_followir(model_name="jhu-clsp/FollowIR-7B"):
    model = AutoModelForCausalLM.from_pretrained(
        model_name, torch_dtype=torch.bfloat16
    ).cuda()
    tokenizer = AutoTokenizer.from_pretrained(
        model_name, padding_side="left", fast=True
    )
    tokenizer.pad_token = tokenizer.eos_token
    token_false_id = tokenizer.get_vocab()["false"]
    token_true_id = tokenizer.get_vocab()["true"]
    return model, tokenizer, token_true_id, token_false_id

def rank_batch_followir(queries, passages, tokenizer, model, true_token_id, false_token_id):
    prompts = [TEMPLATE.format(query=q, text=p) for q, p in zip(queries, passages)]
    inputs = tokenizer(prompts, padding=True, truncation=True, return_tensors="pt")

    with torch.no_grad():
        inputs = {k: v.cuda() for k, v in inputs.items()}
        batch_scores = model(**inputs).logits[:, -1, :]
        true_vector = batch_scores[:, true_token_id]
        false_vector = batch_scores[:, false_token_id]
        batch_scores = torch.stack([false_vector, true_vector], dim=1)
        batch_scores = torch.nn.functional.log_softmax(batch_scores, dim=1)
        scores = batch_scores[:, 1].exp().tolist()
    return scores
```

```bash
# Filter batch of query-document pairs:
python scripts/filtering/filter_query_doc_pairs_from_batch_gpt.py \
  -i batch_outputs/batch_instances.jsonl \
  -o batch_outputs/followir_scores.tsv \
  -b 8

# Input JSONL format:
# {"query": "...", "instruction": "...", "passage": {"title": "...", "text": "..."}, "joint_id": "..."}

# Output TSV format:
# doc_id<TAB>score
# query1_neg    0.234
# query1_real   0.891
```

--------------------------------

### Direct Search and Evaluation for Single Dataset with Prompt

Source: https://context7.com/orionw/promptriever/llms.txt

This section outlines the process for performing direct search and evaluation on a single dataset using a specified prompt. It involves using Tevatron for search and Pyserini for evaluation, with outputs saved in TREC format. Dependencies include Tevatron and Pyserini libraries.

```bash
python -m tevatron.retriever.driver.search \
  --query_reps ./promptriever-eval/fiqa/fiqa_queries_emb_abc123.pkl \
  --passage_reps "./promptriever-eval/fiqa/corpus_emb.*.pkl" \
  --batch_size 64 \
  --depth 1000 \
  --save_text \
  --save_ranking_to ./promptriever-eval/fiqa/rank.fiqa_abc123.txt

python -m tevatron.utils.format.convert_result_to_trec \
  --input ./promptriever-eval/fiqa/rank.fiqa_abc123.txt \
  --output ./promptriever-eval/fiqa/rank.fiqa_abc123.trec \
  --remove_query

python -m pyserini.eval.trec_eval -c -mrecall.100 -mndcg_cut.10 \
  beir-v1.0.0-fiqa-test \
  ./promptriever-eval/fiqa/rank.fiqa_abc123.trec \
  > ./promptriever-eval/fiqa/rank.fiqa_abc123.eval
```

--------------------------------

### Train Promptriever Models with LoRA

Source: https://context7.com/orionw/promptriever/llms.txt

Trains a Promptriever bi-encoder model using LoRA adapters and DeepSpeed distributed training. Supports various base models and instruction-augmented datasets. Requires specifying run name, dataset, GPU IDs, and port suffix.

```bash
bash scripts/training/train_instruct.sh my-run samaya-ai/msmarco-w-instructions "0,1,2,3" 1
```

```bash
deepspeed --include localhost:0,1,2,3 --master_port 60001 --module tevatron.retriever.driver.train \
  --deepspeed deepspeed/ds_zero3_config.json \
  --output_dir retriever-llama2-instruct-my-run \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --lora \
  --lora_r 32 \
  --lora_target_modules q_proj,k_proj,v_proj,o_proj,down_proj,up_proj,gate_proj \
  --save_steps 500 \
  --dataset_name samaya-ai/msmarco-w-instructions \
  --query_prefix "query: " \
  --passage_prefix "passage: " \
  --bf16 \
  --pooling eos \
  --append_eos_token \
  --normalize \
  --temperature 0.01 \
  --per_device_train_batch_size 8 \
  --gradient_checkpointing \
  --train_group_size 16 \
  --learning_rate 1e-4 \
  --query_max_len 304 \
  --passage_max_len 196 \
  --num_train_epochs 1 \
  --logging_steps 10 \
  --overwrite_output_dir \
  --warmup_steps 100 \
  --gradient_accumulation_steps 4 \
  --negatives_first_n 3
```

```bash
bash scripts/training/train_instruct_llama3.sh my-run samaya-ai/msmarco-w-instructions "0,1,2,3" 2
```

```bash
bash scripts/training/train_instruct_mistral_v1.sh my-run samaya-ai/msmarco-w-instructions "4,5,6,7" 3
```

--------------------------------

### Upload Folder to Hugging Face Hub (Python)

Source: https://context7.com/orionw/promptriever/llms.txt

Uploads a local folder to a Hugging Face Hub repository. It can optionally create the repository if it doesn't exist. This is useful for sharing trained models or datasets.

```python
import huggingface_hub

def upload_folder(folder_path, repo_name, skip_create=False):
    api = huggingface_hub.HfApi()

    if not skip_create:
        api.create_repo(
            repo_name,
            repo_type="model",
            exist_ok=False,
            private=True
        )

    api.upload_folder(
        folder_path=folder_path,
        repo_id=repo_name,
        repo_type="model",
        multi_commits=True,
        multi_commits_verbose=True,
    )
```

```bash
# Upload trained model to Hugging Face:
python scripts/utils/upload_to_hf.py \
  -f retriever-llama2-instruct-myrun/final \
  -r myorg/promptriever-custom-v1

# Skip repo creation if it already exists:
python scripts/utils/upload_to_hf.py \
  -f retriever-llama2-instruct-myrun/final \
  -r myorg/promptriever-custom-v1 \
  --skip_create
```

--------------------------------

### Running Batch Evaluation with Multiple Prompts

Source: https://context7.com/orionw/promptriever/llms.txt

Executes batch evaluation across all BEIR datasets using multiple prompts defined in a CSV file. This facilitates comprehensive testing of different prompt strategies for retrieval.

```bash
bash scripts/beir/run_all_prompts.sh \
  samaya-ai/promptriever-llama2-7b-v1 \
  promptriever-eval \
  meta-llama/Llama-2-7b-hf
```

```bash
bash scripts/beir/search_all_prompts.sh promptriever-eval
```

--------------------------------

### BM25 Baseline Search using BM25S Library

Source: https://context7.com/orionw/promptriever/llms.txt

This functionality demonstrates how to run BM25 baseline experiments using the BM25S library. It loads a pre-built index from Hugging Face Hub, processes queries with an optional prompt, and outputs results in TREC format. Dependencies include bm25s, datasets, and Stemmer libraries.

```python
import bm25s.hf
from datasets import load_dataset
import Stemmer

# Load BM25 index from Hugging Face Hub
retriever = bm25s.hf.BM25HF.load_from_hub(
    "xhluca/bm25s-fiqa-index",
    load_corpus=True,
    mmap=True
)

# Load queries
dataset = load_dataset("orionweller/beir", "fiqa", trust_remote_code=True)("test")
queries = {row['query_id']: row['query'] for row in dataset}

# Initialize stemmer
stemmer = Stemmer.Stemmer("english")

# Search with optional prompt appended
prompt = "Find financial documents that answer:"
for query_id, query in queries.items():
    full_query = f"{query} {prompt}"
    query_tokenized = bm25s.tokenize([full_query], stemmer=stemmer)
    results, scores = retriever.retrieve(query_tokenized, k=1000)

    # Write TREC format results
    for rank, (doc, score) in enumerate(zip(results[0], scores[0])):
        print(f"{query_id} Q0 {doc['id']} {rank+1} {score} bm25s")
```

```bash
# Command line usage:
python scripts/beir/bm25/run_bm25s.py \
  --dataset_name fiqa \
  --prompt "Find financial documents that answer:" \
  --prompt_hash abc123 \
  --top_k 1000 \
  --output_dir ./bm25_results

# Run BM25 on all BEIR datasets with prompts:
bash scripts/beir/bm25/prompt_all_bm25.sh

# Search all BM25 results:
bash scripts/beir/bm25/search_all_bm25.sh
```

--------------------------------

### Encoding Queries with Prompts

Source: https://context7.com/orionw/promptriever/llms.txt

Encodes BEIR queries, optionally using natural language prompts to influence retrieval behavior. Each prompt generates a separate embedding file identified by an MD5 hash. This allows for controlled retrieval experiments.

```bash
bash scripts/beir/encode_beir_queries.sh \
  meta-llama/Llama-2-7b-hf \
  ./output/fiqa \
  samaya-ai/promptriever-llama2-7b-v1 \
  fiqa \
  0 \
  "Find financial advice documents that answer the question"
```

```python
CUDA_VISIBLE_DEVICES=0 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --lora_name_or_path samaya-ai/promptriever-llama2-7b-v1 \
  --lora \
  --query_prefix "query: " \
  --passage_prefix "passage: " \
  --bf16 \
  --pooling eos \
  --append_eos_token \
  --normalize \
  --encode_is_query \
  --per_device_eval_batch_size 16 \
  --query_max_len 512 \
  --passage_max_len 512 \
  --dataset_name orionweller/beir \
  --dataset_config "fiqa" \
  --dataset_split test \
  --encode_output_path ./output/fiqa/fiqa_queries_emb_abc123.pkl \
  --prompt "Find financial advice documents that answer the question"
```

```bash
bash scripts/beir/encode_beir_queries.sh \
  meta-llama/Llama-2-7b-hf \
  ./output/fiqa \
  samaya-ai/promptriever-llama2-7b-v1 \
  fiqa \
  0
```

--------------------------------

### Running Dense Retrieval Search and Evaluation

Source: https://context7.com/orionw/promptriever/llms.txt

Performs dense retrieval search using pre-encoded query and corpus embeddings. It includes steps for running the search, converting results to TREC format, and evaluating performance using standard metrics like nDCG@10 and Recall@100.

```bash
bash scripts/msmarco/search.sh ./embeddings/msmarco
```

```python
python -m tevatron.retriever.driver.search \
  --query_reps ./embeddings/msmarco/dl19_queries_emb.pkl \
  --passage_reps "./embeddings/msmarco/corpus_emb.*.pkl" \
  --depth 1000 \
  --batch_size 64 \
  --save_text \
  --save_ranking_to ./embeddings/msmarco/rank.dl19.txt
```

```python
python -m tevatron.utils.format.convert_result_to_trec \
  --input ./embeddings/msmarco/rank.dl19.txt \
  --output ./embeddings/msmarco/rank.dl19.trec \
  --remove_query
```

```python
python -m pyserini.eval.trec_eval -c -mrecall.100 -mndcg_cut.10 \
  dl19-passage ./embeddings/msmarco/rank.dl19.trec
```

```python
python -m pyserini.eval.trec_eval -c -M 100 -m recip_rank \
  msmarco-passage-dev-subset ./embeddings/msmarco/rank.dev.trec
```

--------------------------------

### Gather and Aggregate Evaluation Results (Python)

Source: https://context7.com/orionw/promptriever/llms.txt

Collects evaluation results from .eval files within specified directories and aggregates them into a CSV format. It extracts recall@100, NDCG@10, and MRR scores using regular expressions.

```python
# scripts/tables/gather_results.py
import os
import re
import csv

def extract_scores(file_path):
    with open(file_path, 'r') as file:
        content = file.read()
        scores = {}

        patterns = {
            'recall@100': r'recall_100\s+all\s+([\d.]+)',
            'ndcg@10': r'ndcg_cut_10\s+all\s+([\d.]+)',
            'mrr': r'recip_rank\s+all\s+([\d.]+)'
        }

        for metric, pattern in patterns.items():
            match = re.search(pattern, content)
            if match:
                scores[metric] = float(match.group(1)) * 100

        return scores

def process_directory(directory):
    results = []
    for root, _, files in os.walk(directory):
        for file in files:
            if file.endswith('.eval'):
                file_path = os.path.join(root, file)
                scores = extract_scores(file_path)
                results.append({
                    'dataset': file.split("_")[0].replace("rank.", ""),
                    'prompt_hash': file.split('_')[-1].split('.')[0],
                    **scores
                })
    return results

# Process multiple model directories:
directories = ['joint-full', 'reproduced-v2', 'llama3.1', 'mistral-v0.1']
for directory in directories:
    results = process_directory(directory)
    # Write to CSV...
```

```bash
# Run the results gathering script:
python scripts/tables/gather_results.py

# Output: results/{model_name}_results.csv
# CSV columns: dataset, prompt_hash, filename, recall@100, ndcg@10, mrr
```

--------------------------------

### Encode MS MARCO Queries

Source: https://context7.com/orionw/promptriever/llms.txt

Encodes queries from MS MARCO evaluation sets (DL19, DL20, Dev) for retrieval. This script generates dense embeddings for queries using a specified Promptriever model. Requires output path, model name, and optionally the base model.

```bash
bash scripts/msmarco/encode_queries.sh ./embeddings/msmarco samaya-ai/promptriever-llama2-7b-v1
```

```python
CUDA_VISIBLE_DEVICES=0 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --lora_name_or_path samaya-ai/promptriever-llama2-7b-v1 \
  --lora \
  --query_prefix "query: " \
  --passage_prefix "passage: " \
  --bf16 \
  --pooling eos \
  --append_eos_token \
  --normalize \
  --encode_is_query \
  --per_device_eval_batch_size 128 \
  --query_max_len 32 \
  --passage_max_len 156 \
  --dataset_name Tevatron/msmarco-passage \
  --dataset_split dl19 \
  --encode_output_path ./embeddings/msmarco/dl19_queries_emb.pkl
```

--------------------------------

### Encoding BEIR Benchmark Corpora

Source: https://context7.com/orionw/promptriever/llms.txt

Encodes document corpora from BEIR benchmark datasets using specified models. This process generates embeddings for cross-domain evaluation. It supports encoding all BEIR corpora or individual datasets.

```bash
bash scripts/beir/run_all.sh samaya-ai/promptriever-llama2-7b-v1 promptriever-eval
```

```bash
bash scripts/beir/run_all.sh samaya-ai/promptriever-mistral-v0.1-7b-v1 mistral-eval mistralai/Mistral-7B-v0.1
```

```bash
bash scripts/beir/encode_beir_corpus.sh ./output/fiqa samaya-ai/promptriever-llama2-7b-v1 fiqa meta-llama/Llama-2-7b-hf
```

```python
CUDA_VISIBLE_DEVICES=0 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --lora_name_or_path samaya-ai/promptriever-llama2-7b-v1 \
  --lora \
  --query_prefix "query: " \
  --passage_prefix "passage: " \
  --bf16 \
  --pooling eos \
  --append_eos_token \
  --normalize \
  --per_device_eval_batch_size 32 \
  --query_max_len 512 \
  --passage_max_len 512 \
  --dataset_name "orionweller/beir-corpus" \
  --dataset_config "fiqa" \
  --dataset_split "train" \
  --dataset_number_of_shards 8 \
  --dataset_shard_index 0 \
  --encode_output_path ./output/fiqa/corpus_emb.0.pkl
```

--------------------------------

### Encode MS MARCO Document Corpus

Source: https://context7.com/orionw/promptriever/llms.txt

Encodes the MS MARCO passage corpus into dense embeddings using a trained Promptriever model. This process is parallelized across multiple GPUs and supports dataset sharding. Requires specifying output path, model name, and optionally the base model type.

```bash
bash scripts/msmarco/encode_corpus.sh ./embeddings/msmarco samaya-ai/promptriever-llama2-7b-v1
```

```bash
bash scripts/msmarco/encode_corpus.sh ./embeddings/msmarco samaya-ai/promptriever-llama2-7b-v1 meta-llama/Llama-2-7b-hf
```

```python
CUDA_VISIBLE_DEVICES=0 python -m tevatron.retriever.driver.encode \
  --output_dir=temp \
  --model_name_or_path meta-llama/Llama-2-7b-hf \
  --lora_name_or_path samaya-ai/promptriever-llama2-7b-v1 \
  --lora \
  --query_prefix "query: " \
  --passage_prefix "passage: " \
  --bf16 \
  --pooling eos \
  --append_eos_token \
  --normalize \
  --per_device_eval_batch_size 128 \
  --query_max_len 32 \
  --passage_max_len 156 \
  --dataset_name Tevatron/msmarco-passage-corpus \
  --dataset_number_of_shards 8 \
  --dataset_shard_index 0 \
  --encode_output_path ./embeddings/msmarco/corpus_emb.0.pkl
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.