### Run Project Quickstart

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/_docs/home.md

Basic usage example for running the project. Ensure 'my_project' is installed.

```python
import my_project

my_project.run()
```

--------------------------------

### Install LLM2Vec and Flash Attention

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Install the llm2vec package from PyPI and flash-attention. For the latest version, clone the repository and install in editable mode.

```bash
pip install llm2vec
pip install flash-attn --no-build-isolation
```

```bash
pip install -e .
pip install flash-attn --no-build-isolation
```

--------------------------------

### Word Task Configuration Example

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Example JSON configuration for training word-level tasks, specifying model, PEFT adapter, and dataset details.

```json
{
    "model_name_or_path": "meta-llama/Llama-2-7b-chat-hf",
    "peft_addr": "McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp", // or any local directory containing `adapter_model` files.
    "model_class": "custom",
    "bidirectional": true,
    "classifier_dropout": 0.1,
    "merge_subwords": true,
    "retroactive_labels": "next_token",
    "output_dir": "output/word-task/pos_tags/Llama2/bi-mntp",
    "dataset_name": "conll2003",
    "task": "pos_tags", // or ner_tags, or chunk_tags
    // ....
}
```

--------------------------------

### Install Dependencies for MNTP Training

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Installs the necessary libraries for Masked Next Token Prediction (MNTP) training. `flash-attn` is recommended for performance.

```bash
pip install llm2vec
pip install flash-attn --no-build-isolation
```

--------------------------------

### LLM2Vec Training Configuration Example

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Example of key fields in a training configuration JSON file for LLM2Vec, specifying model paths, training parameters, and optimization settings.

```json
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    "bidirectional": true,
    "pooling_mode": "mean",
    "dataset_name": "E5",
    "dataset_file_path": "cache/echo-data",
    "learning_rate": 2e-4,
    "num_train_epochs": 3,
    "per_device_train_batch_size": 64,
    "lora_r": 16,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2"
}
```

--------------------------------

### Install LLM2Vec Evaluation Dependencies

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Command to install the necessary dependencies for MTEB evaluation, including the mteb library.

```bash
pip install llm2vec[evaluation]  # installs mteb>=1.14.12
```

--------------------------------

### Install LLM2Vec for Evaluation

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Install the llm2vec library with evaluation dependencies using pip. Ensure you have mteb>=1.12.60.

```bash
pip install llm2vec[evaluation]
```

--------------------------------

### Supervised Contrastive Training Configuration (Meta-Llama-3-8B)

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Example configuration for supervised contrastive training with the Meta-Llama-3-8B model. This JSON specifies hyperparameters and the path to the 'echo-data' directory.

```json
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    "bidirectional": true,
    "pooling_mode": "mean",
    "dataset_name": "E5",
    "dataset_file_path": "cache/echo-data",
    "learning_rate": 2e-4,
    "num_train_epochs": 3,
    "warmup_steps": 300,
    "per_device_train_batch_size": 64,
    "lora_r": 16,
    "gradient_checkpointing": true,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2"
    "// ....": ""
}
```

--------------------------------

### SimCSE Training Configuration (Meta-Llama-3-8B)

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Example configuration for unsupervised contrastive training using SimCSE with the Meta-Llama-3-8B model. This JSON defines hyperparameters and dataset paths.

```json
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    "simcse_dropout": 0.3,
    "bidirectional": true,
    "pooling_mode": "mean",
    "dataset_name": "Wiki1M",
    "dataset_file_path": "cache/wiki1m_for_simcse.txt",
    "learning_rate": 3e-5,
    "loss_scale": 20,
    "per_device_train_batch_size": 128,
    "max_seq_length": 128,
    "stop_after_n_steps": 1000,
    "lora_r": 16,
    "gradient_checkpointing": true,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2",
    "// ....": ""
}
```

--------------------------------

### Train LLM2Vec Model (Single/Multi-GPU)

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Commands to train the LLM2Vec model using a specified configuration file. Supports both single and multi-GPU setups.

```bash
python experiments/run_supervised.py train_configs/supervised/MetaLlama3.json
```

```bash
torchrun --nproc_per_node=8 experiments/run_supervised.py train_configs/supervised/MetaLlama3.json
```

--------------------------------

### Run SimCSE Training

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Executes SimCSE unsupervised contrastive training. This process requires MNTP-trained LoRA weights as a starting point and uses a specified configuration file.

```bash
# Run SimCSE training on top of MNTP checkpoint
python experiments/run_simcse.py train_configs/simcse/MetaLlama3.json
```

--------------------------------

### Use Bidirectional Llama Model

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Demonstrates direct use of a bidirectional Llama model. This is typically handled by `LLM2Vec.from_pretrained`. Ensure `flash_attention_2` is installed for optimal performance.

```python
import torch
from llm2vec.models import LlamaBiModel, MistralBiModel, GemmaBiModel, Qwen2BiModel
from transformers import AutoTokenizer

# Direct use of the bidirectional model (normally done via LLM2Vec.from_pretrained)
model = LlamaBiModel.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "left"

inputs = tokenizer(
    ["The quick brown fox jumps over the lazy dog"],
    return_tensors="pt",
    padding=True,
).to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# outputs.last_hidden_state: [batch, seq_len, hidden_dim]
print(outputs.last_hidden_state.shape)  # torch.Size([1, 10, 4096])

# Verify bidirectionality: all attention layers have is_causal=False
for layer in model.layers:
    assert layer.self_attn.is_causal == False, "Should be bidirectional"
print("All attention layers are bidirectional.")
```

--------------------------------

### Calculate STS with LLM2Vec

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Loads a pre-trained LLM2Vec model and evaluates its performance on the STS17 dataset by calculating the Spearman correlation between predicted and gold similarity scores. Ensure you have the necessary libraries (torch, numpy, datasets, scikit-learn, scipy, llm2vec) installed.

```python
import torch
import numpy as np
import datasets
from sklearn.metrics.pairwise import paired_cosine_distances
from scipy.stats import spearmanr
from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

dataset     = datasets.load_dataset("mteb/sts17-crosslingual-sts", "en-en")
instruction = "Retrieve semantically similar text:"
normalize   = lambda x: (x - 0) / (5 - 0)
gold_scores = list(map(normalize, dataset["test"]["score"]))

pairs1 = [[instruction, s, 0] for s in dataset["test"]["sentence1"]]
pairs2 = [[instruction, s, 0] for s in dataset["test"]["sentence2"]]

emb1 = np.asarray(model.encode(pairs1, batch_size=8))
emb2 = np.asarray(model.encode(pairs2, batch_size=8))

cos_scores = 1 - paired_cosine_distances(emb1, emb2)
spearman_corr, _ = spearmanr(gold_scores, cos_scores)
print(f"Spearman correlation: {spearman_corr:.4f}")
# Spearman correlation: 0.9022
```

--------------------------------

### Run Jekyll Server Locally

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/README.md

Navigate to the docs directory and execute this command to serve the documentation locally using Jekyll.

```bash
cd docs/
bundle exec jekyll serve
```

--------------------------------

### Run Supervised Contrastive Training

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Initiate supervised contrastive training using the 'run_supervised.py' script. The number of GPUs can be adjusted via the '--nproc_per_node' argument. Ensure the dataset is placed in the 'cache' directory.

```bash
torchrun --nproc_per_node=8 experiments/run_supervised.py train_configs/supervised/MetaLlama3.json
```

--------------------------------

### Download SimCSE Dataset

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Use this command to download the Wiki1M dataset for SimCSE training. Ensure the downloaded file is placed in the 'cache' directory.

```bash
wget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt
```

--------------------------------

### Clone Project Template with Git

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/README.md

Use this command to clone the project template into your existing project directory. Ensure you are in the root of your project before executing.

```bash
cd ..  # Go to parent directory
git clone https://github.com/McGill-NLP/project-page-template
cp -r project-page-template/docs my-project/
cp project-page-template/README.md my-project/docs/
cd my-project/
```

--------------------------------

### Initialize LLM2Vec with Supervised Weights

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Initialize an LLM2Vec model using a base model identifier and supervised-trained PEFT weights. Supports specifying device mapping and data type.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)
```

--------------------------------

### Run SimCSE Training

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Execute the SimCSE training script with the specified configuration file for Meta-Llama-3-8B. Adjust dataset_file_path in the config if the dataset is not in the 'cache' directory.

```bash
python experiments/run_simcse.py train_configs/simcse/MetaLlama3.json
```

--------------------------------

### Initialize LLM2Vec with Unsupervised Weights

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Initialize an LLM2Vec model using a base model identifier and unsupervised-trained PEFT weights. Supports specifying device mapping and data type.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)
```

--------------------------------

### Load Wiki1M Dataset

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Loads the Wiki1M dataset for training. Ensure the dataset file is downloaded and placed in the specified cache directory.

```python
wiki_dataset = load_dataset(
    dataset_name="Wiki1M",
    split="train",
    file_path="cache/wiki1m_for_simcse.txt",
)

print(f"Total samples: {len(wiki_dataset)}")  # ~1,000,000

sample = wiki_dataset[0]
print(type(sample))         # <class 'llm2vec.dataset.dataset.TrainSample'>
print(sample.texts[0])      # original Wikipedia sentence
print(sample.texts[1])      # same sentence (SimCSE positive pair)
print(sample.label)         # 1.0

# Direct instantiation
wiki = Wiki1M(
    dataset_name="Wiki1M",
    split="train",
    file_path="cache/wiki1m_for_simcse.txt",
)
```

--------------------------------

### Load Data with Pandas

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/_docs/training.md

Use pandas to load your project data from a CSV file.

```python
import pandas as pd

df = pd.read_csv('project_data.csv')
# ...
```

--------------------------------

### Download Wiki1M Corpus for SimCSE

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Downloads the Wiki1M corpus required for SimCSE training. The corpus will be saved to the `cache/` directory.

```bash
# Download the Wiki1M corpus
wget https://huggingface.co/datasets/princeton-nlp/datasets-for-simcse/resolve/main/wiki1m_for_simcse.txt -P cache/
```

--------------------------------

### Evaluate LLM2Vec Model on MTEB Benchmark

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Commands to evaluate an LLM2Vec model on MTEB benchmark tasks. Specify the model, task, instruction mapping, and output directory.

```bash
python experiments/mteb_eval.py \
    --model_name McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised \
    --task_name STS16 \
    --task_to_instructions_fp test_configs/mteb/task_to_instructions.json \
    --output_dir results
```

```bash
python experiments/mteb_eval.py \
    --model_name McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised \
    --task_name NFCorpus \
    --task_to_instructions_fp test_configs/mteb/task_to_instructions.json \
    --output_dir results
```

--------------------------------

### Load Pre-trained Model with Transformers

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/_docs/training.md

Load a pre-trained model using the transformers library.

```python
import transformers

model = transformers.AutoModel.from_pretrained(...)
# ...
```

--------------------------------

### Wiki1M Dataset for Unsupervised SimCSE

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

A PyTorch Dataset for loading the 1-million Wikipedia sentence corpus for unsupervised SimCSE training. Each sentence is used as both the query and positive sample.

```python
from llm2vec.dataset.utils import load_dataset
from llm2vec.dataset.Wiki1M import Wiki1M

# Requires cache/wiki1m_for_simcse.txt downloaded from:
```

--------------------------------

### SimCSE Training Configuration

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Key fields for SimCSE unsupervised contrastive training configuration. Includes model paths, dropout settings, pooling mode, dataset details, batch size, and training steps.

```json
// train_configs/simcse/MetaLlama3.json (key fields)
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "peft_model_name_or_path": "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    "simcse_dropout": 0.3,
    "bidirectional": true,
    "pooling_mode": "mean",
    "dataset_name": "Wiki1M",
    "dataset_file_path": "cache/wiki1m_for_simcse.txt",
    "per_device_train_batch_size": 128,
    "stop_after_n_steps": 1000,
    "lora_r": 16,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2"
}
```

--------------------------------

### Train Word-Level Tasks

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Use this command to train a classifier on top of LLM models for word-level tasks. Configuration is managed via JSON files.

```bash
python experiments/run_word_task.py train_configs/word-task/Llama2-bi-mntp.json
```

```bash
python experiments/test_word_task.py --config_file test_configs/word-task/Llama2-bi-mntp.json
```

--------------------------------

### Run MNTP Training

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Command to initiate Masked Next Token Prediction (MNTP) training for a specified model configuration. This command uses the `run_mntp.py` script and points to the relevant training configuration file.

```bash
python experiments/run_mntp.py train_configs/mntp/MetaLlama3.json
```

--------------------------------

### MNTP Training Configuration for Meta-Llama-3-8B

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

This JSON configuration is used for training the Meta-Llama-3-8B model with Masked Next Token Prediction (MNTP). It specifies model, dataset, masking strategy, and training hyperparameters. Ensure the `experiments/run_mntp.py` script is used for training.

```json
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "dataset_name": "wikitext",
    "dataset_config_name": "wikitext-103-raw-v1",
    "mask_token_type": "blank",
    "data_collator_type": "default",
    "mlm_probability": 0.2,
    "lora_r": 16,
    "gradient_checkpointing": true,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2"
    // ....
}
```

--------------------------------

### E5Data for Supervised Contrastive Training

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

A PyTorch Dataset for loading the E5 echo-embeddings training corpus. It provides (query, positive, negative) triplets with task-specific instructions and batches samples by dataset source.

```python
from llm2vec.dataset.utils import load_dataset
from llm2vec.dataset.E5Data import E5Data

# Requires cache/echo-data/ directory with .jsonl files downloaded from
# https://github.com/jakespringer/echo-embeddings#training
train_dataset = load_dataset(
    dataset_name="E5",
    split="train",
    file_path="cache/echo-data",
    effective_batch_size=64,            # groups same-task samples into batches
    shuffle_individual_datasets=True,
)

print(f"Total training samples: {len(train_dataset)}")

# Each item is a TrainSample with .texts = [query, positive, negative]
sample = train_dataset[0]
print(sample.texts[0][:80])  # query with instruction prefix
print(sample.texts[1][:80])  # positive document
print(sample.texts[2][:80])  # hard negative document
print(sample.label)          # 1.0

# Direct instantiation with custom parameters
e5 = E5Data(
    dataset_name="E5",
    split="train",
    file_path="cache/echo-data",
    effective_batch_size=32,
    shuffle_individual_datasets=False,
    separator="!@#$%^&*()",
)
```

--------------------------------

### MNTP Training Configuration

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Key fields for MNTP training configuration. Specifies model, dataset, masking strategy, LoRA parameters, and hardware acceleration settings.

```json
// train_configs/mntp/MetaLlama3.json (key fields)
{
    "model_name_or_path": "meta-llama/Meta-Llama-3-8B-Instruct",
    "dataset_name": "wikitext",
    "dataset_config_name": "wikitext-103-raw-v1",
    "mask_token_type": "blank",
    "mlm_probability": 0.2,
    "lora_r": 16,
    "gradient_checkpointing": true,
    "torch_dtype": "bfloat16",
    "attn_implementation": "flash_attention_2"
}
```

--------------------------------

### Evaluate Model on MTEB Benchmark

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Run the MTEB evaluation script for a specific model and task. This command evaluates the supervised trained Meta-Llama-3-8B model on the STS16 task.

```bash
python experiments/mteb_eval.py --model_name McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised \
--task_name STS16 \
--task_to_instructions_fp test_configs/mteb/task_to_instructions.json \
--output_dir results
```

--------------------------------

### Theme Toggling JavaScript

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/docs/_includes/head/custom.html

Use this JavaScript to toggle between light and dark themes. It updates session storage and dynamically changes stylesheet links.

```javascript
const updateNodesRel = theme => {
    const node_light = document.getElementById('theme-css');
    const node_dark = document.getElementById('theme-css-dark');
    if (theme === "dark") {
        node_light.setAttribute('rel', 'stylesheet alternate');
        node_dark.setAttribute('rel', 'stylesheet');
    } else if (theme === "light") {
        node_light.setAttribute('rel', 'stylesheet');
        node_dark.setAttribute('rel', 'stylesheet alternate');
    }
}

const changeTheme = () => {
    let theme = sessionStorage.getItem('theme');
    // Change the theme to the other option
    if (theme === "light") {
        theme = "dark";
    } else {
        theme = "light";
    }
    // Update the stored session and the nodes' rel attribute
    sessionStorage.setItem('theme', theme);
    updateNodesRel(theme);
    return false;
}

if (sessionStorage.getItem('theme') === null) {
    sessionStorage.setItem('theme', "light");
}

const theme = sessionStorage.getItem('theme');
updateNodesRel(theme);
```

--------------------------------

### LLM2Vec.save

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Persists the LLM2Vec model, tokenizer, and configuration to disk. Supports merging PEFT adapters into the base model before saving for a self-contained checkpoint.

```APIDOC
## LLM2Vec.save — Persist model and tokenizer to disk

Saves the underlying HuggingFace model, tokenizer, and a `llm2vec_config.json` file (pooling mode, max lengths, skip_instruction flag) to the given output directory. If the model still has an active PEFT adapter and `merge_before_save=True` is set, the LoRA weights are merged into the base model before saving, producing a self-contained checkpoint that can be reloaded with `from_pretrained` without a separate adapter path.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)

# Save with LoRA adapter kept separately
l2v.save("output/llm2vec-llama3-supervised", merge_before_save=False, save_config=True)

# Save with adapter merged into base weights (single standalone checkpoint)
l2v.save("output/llm2vec-llama3-merged", merge_before_save=True, save_config=True)

# Reload the merged model — no peft_model_name_or_path needed
l2v_reloaded = LLM2Vec.from_pretrained(
    "output/llm2vec-llama3-merged",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)
reps = l2v_reloaded.encode(["test sentence"])
print(reps.shape)  # torch.Size([1, 4096])
```
```

--------------------------------

### LLM2Vec.tokenize

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Performs instruction-aware tokenization, creating an embed_mask to identify text tokens. Pads from the left and truncates to max_length.

```APIDOC
## LLM2Vec.tokenize — Instruction-aware tokenization with embed mask

Tokenizes a list of pre-formatted strings (instruction + separator + text) and builds an `embed_mask` that marks exactly which tokens belong to the text portion (after the `!@#$%^&*()` separator) rather than the instruction prefix. This mask is later used by `get_pooling` to restrict pooling to text tokens when `skip_instruction=True`. The tokenizer always pads from the left and truncates to `max_length`.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)

# prepare_for_tokenization wraps text in model-specific chat tokens
formatted = l2v.prepare_for_tokenization(
    "Retrieve semantically similar text: !@#$%^&*()The quick brown fox"
)
print(formatted)
```
```

--------------------------------

### LLM2Vec.from_pretrained

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Loads a base decoder-only model with bidirectional attention enabled and optionally merges PEFT/LoRA adapter weights. It forwards all HuggingFace `from_pretrained` keyword arguments and accepts encoder-specific parameters.

```APIDOC
## LLM2Vec.from_pretrained — Load a bidirectional LLM encoder from HuggingFace

Class method that loads a base decoder-only model (Llama, Mistral, Gemma, or Qwen2) with bidirectional attention enabled and optionally merges PEFT/LoRA adapter weights on top. All HuggingFace `from_pretrained` keyword arguments (e.g., `device_map`, `torch_dtype`, `attn_implementation`) are forwarded to the underlying model loader. Encoder-specific parameters (`pooling_mode`, `max_length`, `doc_max_length`, `skip_instruction`) can be passed directly or are read from a `llm2vec_config.json` file saved alongside the PEFT adapter. Setting `merge_peft=True` merges the LoRA weights into the base model before returning.

```python
import torch
from llm2vec import LLM2Vec

# Load with unsupervised SimCSE LoRA weights (no labeled data required)
l2v_unsup = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

# Load with supervised LoRA weights (state-of-the-art on MTEB public data)
l2v_sup = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
    pooling_mode="mean",   # options: "mean", "weighted_mean", "eos_token", "bos_token"
    max_length=512,
)

# Load without bidirectional attention (standard causal LLM behavior)
l2v_causal = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    enable_bidirectional=False,
    device_map="cpu",
    torch_dtype=torch.float32,
)
```
```

--------------------------------

### Load Bidirectional LLM Encoder with LLM2Vec.from_pretrained

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Loads a base decoder-only model with bidirectional attention enabled. Optionally merges PEFT/LoRA adapter weights. Supports various HuggingFace arguments and encoder-specific parameters. Set `enable_bidirectional=False` for standard causal LLM behavior.

```python
import torch
from llm2vec import LLM2Vec

# Load with unsupervised SimCSE LoRA weights (no labeled data required)
l2v_unsup = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-unsup-simcse",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

# Load with supervised LoRA weights (state-of-the-art on MTEB public data)
l2v_sup = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
    pooling_mode="mean",   # options: "mean", "weighted_mean", "eos_token", "bos_token"
    max_length=512,
)

# Load without bidirectional attention (standard causal LLM behavior)
l2v_causal = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    enable_bidirectional=False,
    device_map="cpu",
    torch_dtype=torch.float32,
)
```

--------------------------------

### Save and Reload LLM2Vec Model

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Saves the LLM2Vec model and tokenizer to disk. Supports merging PEFT adapters into the base model before saving for a self-contained checkpoint. Reloaded models can be used directly for encoding.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)

# Save with LoRA adapter kept separately
l2v.save("output/llm2vec-llama3-supervised", merge_before_save=False, save_config=True)

# Save with adapter merged into base weights (single standalone checkpoint)
l2v.save("output/llm2vec-llama3-merged", merge_before_save=True, save_config=True)

# Reload the merged model — no peft_model_name_or_path needed
l2v_reloaded = LLM2Vec.from_pretrained(
    "output/llm2vec-llama3-merged",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)
reps = l2v_reloaded.encode(["test sentence"])
print(reps.shape)  # torch.Size([1, 4096])
```

--------------------------------

### Text Classification with LLM2Vec Embeddings

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Python code demonstrating text classification using LLM2Vec embeddings. Loads a pre-trained model, encodes a dataset, and trains a Logistic Regression classifier.

```python
import torch
import numpy as np
import datasets
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

dataset = datasets.load_dataset("mteb/amazon_counterfactual", "en")
instruction = "Classify a given Amazon customer review text as either counterfactual or not counterfactual:"

def to_pairs(instruction, sentences):
    return [[instruction, s, 0] for s in sentences]

X_train = np.asarray(model.encode(to_pairs(instruction, dataset["train"]["text"]), batch_size=8))
X_test  = np.asarray(model.encode(to_pairs(instruction, dataset["test"]["text"]),  batch_size=8))

clf = LogisticRegression(random_state=42, max_iter=100)
clf.fit(X_train, dataset["train"]["label"])
y_pred = clf.predict(X_test)

print({"accuracy": accuracy_score(dataset["test"]["label"], y_pred),
       "f1":       f1_score(dataset["test"]["label"], y_pred, average="macro")})
# {'accuracy': 0.891, 'f1': 0.828}
```

--------------------------------

### Encode Queries and Documents for Similarity

Source: https://github.com/mcgill-nlp/llm2vec/blob/main/README.md

Use this snippet to encode text inputs (queries and documents) into embeddings. Instructions are required for queries in symmetric tasks but not for documents. Ensure PyTorch is imported for tensor operations.

```python
# Encoding queries using instructions
instruction = (
    "Given a web search query, retrieve relevant passages that answer the query:"
)
queries = [
    [instruction, "how much protein should a female eat"],
    [instruction, "summit define"],
]
q_reps = l2v.encode(queries)

# Encoding documents. Instruction are not required for documents
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day. But, as you can see from this chart, you'll need to increase that if you're expecting or training for a marathon. Check out the chart below to see how much protein you should be eating each day.",
    "Definition of summit for English Language Learners. : 1  the highest point of a mountain : the top of a mountain. : 2  the highest level. : 3  a meeting or series of meetings between the leaders of two or more governments.",
]
d_reps = l2v.encode(documents)

# Compute cosine similarity
q_reps_norm = torch.nn.functional.normalize(q_reps, p=2, dim=1)
d_reps_norm = torch.nn.functional.normalize(d_reps, p=2, dim=1)
cos_sim = torch.mm(q_reps_norm, d_reps_norm.transpose(0, 1))

print(cos_sim)
```

```text
tensor([[0.6470, 0.1619],
        [0.0786, 0.5844]])
```

--------------------------------

### Text Clustering with LLM2Vec Embeddings

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Python code for text clustering using LLM2Vec embeddings. It loads a model, encodes a dataset, applies MiniBatchKMeans for clustering, and calculates the V-measure.

```python
import torch
import numpy as np
import datasets
import sklearn.cluster, sklearn.metrics
from llm2vec import LLM2Vec

model = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

dataset    = datasets.load_dataset("mteb/twentynewsgroups-clustering")
instruction = "Identify the topic or theme of the given news articles:"
v_measures = []

for cluster_set in dataset["test"]:
    sentences = cluster_set["sentences"]
    labels    = cluster_set["labels"]
    pairs     = [[instruction, s, 0] for s in sentences]
    embeddings = np.asarray(model.encode(pairs, batch_size=32))

    km = sklearn.cluster.MiniBatchKMeans(n_clusters=len(set(labels)), batch_size=500)
    km.fit(embeddings)
    v_measures.append(sklearn.metrics.v_measure_score(labels, km.labels_))

print(f"V-measure: {np.mean(v_measures):.4f} ± {np.std(v_measures):.4f}")
# V-measure: 0.5137 ± ...
```

--------------------------------

### Tokenize Text for llm2vec

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Prepares text for tokenization by llm2vec, including adding special tokens and formatting. The embed_mask is generated to identify text tokens after separators.

```python
batch = [formatted, l2v.prepare_for_tokenization("!@#$%^&*()Another sentence")]
features = l2v.tokenize(batch)

print(features.keys())          # dict_keys(['input_ids', 'attention_mask', 'embed_mask'])
print(features["input_ids"].shape)    # torch.Size([2, <padded_length>])
print(features["embed_mask"].shape)   # torch.Size([2, <padded_length>])
# embed_mask is 1 only for tokens that belong to the text part (after the separator)
print(features["embed_mask"].sum(dim=1))  # number of text tokens per sample
```

--------------------------------

### HardNegativeNLLLoss for Supervised Training

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Implements a contrastive loss for supervised embedding training. It computes a scaled cosine similarity matrix and optimizes cross-entropy. Supports multi-GPU distributed training.

```python
import torch
from llm2vec.loss.HardNegativeNLLLoss import HardNegativeNLLLoss

loss_fn = HardNegativeNLLLoss(scale=20.0)  # scale temperature parameter

# Simulated embeddings (e.g., from l2v.forward)
batch_size, hidden_dim = 4, 4096
q_reps     = torch.randn(batch_size, hidden_dim)
d_reps_pos = torch.randn(batch_size, hidden_dim)
d_reps_neg = torch.randn(batch_size, hidden_dim)  # optional hard negatives

# Without hard negatives (in-batch negatives only)
loss_no_neg = loss_fn(q_reps, d_reps_pos)
print(f"Loss (no hard neg): {loss_no_neg.item():.4f}")

# With explicit hard negatives concatenated to the document pool
loss_with_neg = loss_fn(q_reps, d_reps_pos, d_reps_neg)
print(f"Loss (with hard neg): {loss_with_neg.item():.4f}")

# Integration in a training loop
from llm2vec import LLM2Vec
from llm2vec.loss.utils import load_loss

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    device_map="cpu", torch_dtype=torch.bfloat16,
)
train_loss = load_loss("HardNegativeNLLLoss", scale=50.0)

query_features   = l2v.tokenize(["!@#$%^&*()query text"])
pos_features     = l2v.tokenize(["!@#$%^&*()positive document"])
q_emb = l2v.forward(query_features)
d_emb = l2v.forward(pos_features)
loss  = train_loss(q_emb, d_emb)
loss.backward()
```

--------------------------------

### LLM2Vec.encode

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Encodes a list of sentences or `[instruction, text]` pairs into L2-normalizable dense vectors. It handles batching, distribution across GPUs, and accepts various input formats.

```APIDOC
## LLM2Vec.encode — Encode sentences into dense embedding vectors

Encodes a list of sentences or `[instruction, text]` pairs into L2-normalizable dense vectors. Input can be plain strings, `[instruction, text]` lists, or `[instruction, text, 0]` triples (the trailing integer is ignored and exists for MTEB compatibility). Internally, sentences are sorted by length for efficient batching, then processed in batches using `_encode`. On systems with multiple CUDA GPUs, encoding is automatically distributed across all available devices via multiprocessing. Returns a float32 CPU tensor by default; set `convert_to_numpy=True` for a NumPy array.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
)

# --- Asymmetric retrieval: instruction for queries, no instruction for docs ---
instruction = "Given a web search query, retrieve relevant passages that answer the query:"
queries = [
    [instruction, "how much protein should a female eat"],
    [instruction, "what is the summit of a mountain"],
]
documents = [
    "As a general guideline, the CDC's average requirement of protein for women ages 19 to 70 is 46 grams per day.",
    "Definition of summit: the highest point of a mountain; the top of a mountain.",
]

q_reps = l2v.encode(queries, batch_size=16, show_progress_bar=True)
d_reps = l2v.encode(documents, batch_size=16, show_progress_bar=False)

# Cosine similarity matrix
q_norm = torch.nn.functional.normalize(q_reps, p=2, dim=1)
d_norm = torch.nn.functional.normalize(d_reps, p=2, dim=1)
cos_sim = torch.mm(q_norm, d_norm.transpose(0, 1))
print(cos_sim)
```
```

--------------------------------

### LLM2Vec Tokenization with Embed Mask

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Tokenizes pre-formatted strings and creates an embed_mask to identify text tokens after a separator. This mask is used by `get_pooling` to focus on text content. The tokenizer pads from the left and truncates to `max_length`.

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cpu",
    torch_dtype=torch.bfloat16,
)

# prepare_for_tokenization wraps text in model-specific chat tokens
formatted = l2v.prepare_for_tokenization(
    "Retrieve semantically similar text: !@#$%^&*()The quick brown fox"
)
print(formatted)
```

--------------------------------

### LLM2Vec.get_pooling

Source: https://context7.com/mcgill-nlp/llm2vec/llms.txt

Reduces token hidden states to a single sentence embedding using the configured pooling mode. Supports 'mean', 'weighted_mean', 'eos_token', and 'last_token'. Handles instruction-aware pooling by default.

```APIDOC
## LLM2Vec.get_pooling — Extract sentence embeddings from hidden states

Internal method that reduces a sequence of token hidden states into a single sentence embedding using the configured `pooling_mode`. Operates on left-padded sequences (the tokenizer always uses `padding_side="left"`). When `skip_instruction=True` (default), only the token positions corresponding to the actual text (not the instruction prefix) are included in pooling, using an `embed_mask` populated by `tokenize`. Supported modes: `"mean"` (average over non-padding tokens), `"weighted_mean"` (linearly increasing positional weights), `"eos_token"` / `"last_token"` (final token), `"bos_token"` (first special token).

```python
import torch
from llm2vec import LLM2Vec

l2v = LLM2Vec.from_pretrained(
    "McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp",
    peft_model_name_or_path="McGill-NLP/LLM2Vec-Mistral-7B-Instruct-v2-mntp-supervised",
    device_map="cuda" if torch.cuda.is_available() else "cpu",
    torch_dtype=torch.bfloat16,
    pooling_mode="mean",          # default
)

# Compare pooling modes by re-instantiating with different settings
for mode in ["mean", "weighted_mean", "eos_token"]:
    l2v.pooling_mode = mode
    reps = l2v.encode(["The quick brown fox"], show_progress_bar=False)
    print(f"{mode}: shape={reps.shape}, norm={reps.norm().item():.4f}")

# Direct forward pass (used internally by encode)
l2v.eval()
texts = ["Retrieve semantically similar text: !@#$%^&*()Hello world"]
features = l2v.tokenize([l2v.prepare_for_tokenization(t) for t in texts])
device = "cuda" if torch.cuda.is_available() else "cpu"
features = {k: v.to(device) for k, v in features.items() if isinstance(v, torch.Tensor)}
l2v.to(device)
with torch.no_grad():
    embeddings = l2v.forward(features)
print(embeddings.shape)  # torch.Size([1, 4096])
```
```