### Install Accelerate Library

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Installs the 'accelerate' library from Hugging Face, which simplifies running PyTorch training across any distributed setup.

```python
! pip install accelerate
```

--------------------------------

### Install bitsandbytes Package

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Install the 'bitsandbytes' library, which is often required for efficient model loading and quantization, especially for large language models.

```bash
!pip install bitsandbytes
```

--------------------------------

### Install bitsandbytes Library

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Installs the 'bitsandbytes' library, which provides optimized CUDA-accelerated primitives for deep learning.

```python
! pip install bitsandbytes
```

--------------------------------

### Install ChromaDB

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Install the ChromaDB package using pip. ChromaDB is a vector database that can be used for efficient similarity search.

```python
!pip install chromadb
```

--------------------------------

### Example Query

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Represents a sample query string used for document retrieval.

```python
query
```

--------------------------------

### Install pypdf Package

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Installs the 'pypdf' library, which is required for reading and processing PDF files. This is a prerequisite for extracting text from documents.

```bash
!pip install pypdf
```

--------------------------------

### Install langchain_community Package

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Installs the 'langchain_community' package, providing access to various community-contributed tools and integrations for Langchain. This is crucial for building hybrid search capabilities.

```bash
!pip install langchain_community
```

--------------------------------

### RetrievalQA Chain Setup with Quantized LLM

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Sets up a RetrievalQA chain using a 4-bit quantized HuggingFace LLM. Demonstrates standard (semantic only) and hybrid retrieval chain configurations.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline
from langchain import HuggingFacePipeline
from langchain.chains import RetrievalQA

model_name = "HuggingFaceH4/zephyr-7b-beta"

# Load 4-bit quantized model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype=torch.bfloat16, quantization_config=bnb_config
)
tokenizer = AutoTokenizer.from_pretrained(model_name, return_token_type_ids=False)
tokenizer.bos_token_id = 1

# Build HuggingFace pipeline
pipe = pipeline(
    "text-generation", model=model, tokenizer=tokenizer,
    use_cache=True, device_map="auto", max_length=2048,
    do_sample=True, top_k=5, num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.pad_token_id,
)
llm = HuggingFacePipeline(pipeline=pipe)

# Standard chain (semantic retriever only)
normal_chain = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=vectorstore_retriever
)

# Hybrid chain (BM25 + semantic ensemble)
hybrid_chain = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=retriever  # EnsembleRetriever
)

question = "What is Abstractive Question Answering?"

response_normal = normal_chain.invoke(question)
response_hybrid = hybrid_chain.invoke(question)

print("=== Standard (Semantic) ===")
print(response_normal["result"])

print("\n=== Hybrid (BM25 + Semantic) ===")
print(response_hybrid["result"])

# Hybrid response typically retrieves more contextually diverse chunks,

```

--------------------------------

### Install RankBM25 Package

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Installs the `rank_bm25` library, which is required for keyword-based retrieval using the BM25 algorithm. This is an alternative to TF-IDF for keyword matching.

```bash
#for keyword search ,we require bm25,which is updated version of tfidf
!pip install rank_bm25
```

--------------------------------

### Create Text Generation Pipeline

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Sets up a text generation pipeline using Hugging Face's `pipeline` function. It configures the model, tokenizer, and generation parameters. Ensure `transformers` and `torch` are installed.

```python
from transformers import pipeline

pipeline = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    use_cache=True,
    device_map="auto",
    max_length=2048,
    do_sample=True,
    top_k=5,
    num_return_sequences=1,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
)
```

--------------------------------

### Question Answering with Hybrid Search

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Example of performing question answering using a loaded model. This demonstrates how to format the input and generate a response.

```python
def answer_question(question):
    prompt = f"<s>[INST] {question} [/INST]"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(**inputs, max_new_tokens=500)
    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return response

print(answer_question(question='What is a large language model?'))
```

--------------------------------

### Import HuggingFace Inference Embeddings

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Import the HuggingFaceInferenceAPIEmbeddings class for creating embeddings. Ensure you have the langchain library installed.

```python
from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings
```

--------------------------------

### Invoke RetrievalQA chain and print result

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Invokes the RetrievalQA chain with a query and prints the 'result' from the response. This demonstrates how to get the answer from the chain's output.

```python
response1 = normal_chain.invoke("What is Abstractive Question Answering?")
```

```python
print(response1.get("result"))
```

--------------------------------

### Initialize Tokenizer

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Initializes a Hugging Face tokenizer for a given model. It sets the beginning of sentence token ID. Ensure `transformers` is installed. This is a prerequisite for text generation tasks.

```python
from transformers import AutoTokenizer

def initialize_tokenizer(model_name: str):
    """
    model_name: Name or path of the model for tokenizer initialization.
    return: Initialized tokenizer.
    """
    tokenizer = AutoTokenizer.from_pretrained(model_name, return_token_type_ids=False)
    tokenizer.bos_token_id = 1  # Set beginning of sentence token id
    return tokenizer
```

--------------------------------

### Import Core Libraries for Hybrid Search

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Imports essential libraries for natural language processing and machine learning tasks, including PyTorch, Hugging Face Transformers, and LangChain. Ensure these libraries are installed in your environment.

```python
import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    pipeline,
)
from langchain import HuggingFacePipeline
```

--------------------------------

### Load 4-bit Quantized Model

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Loads a Hugging Face causal language model with 4-bit quantization. Ensure `bitsandbytes` and `transformers` are installed. This is useful for reducing memory footprint when working with large models.

```python
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
import torch

def load_quantized_model(model_name: str):
    """
    model_name: Name or path of the model to be loaded.
    return: Loaded quantized model.
    """
    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.bfloat16,
        quantization_config=bnb_config,
    )
    return model
```

--------------------------------

### Initialize RetrievalQA with 'stuff' chain type

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Sets up a RetrievalQA chain using the 'stuff' chain type with a standard retriever. Use this for small, concise, and relevant documents.

```python
normal_chain = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=vectorstore_retriever
)
```

--------------------------------

### Initialize Hybrid RetrievalQA with 'stuff' chain type

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Sets up a RetrievalQA chain for hybrid search using the 'stuff' chain type with a hybrid retriever. This is suitable when all documents are relevant and fit within the LLM's context.

```python
hybrid_chain = RetrievalQA.from_chain_type(
    llm=llm, chain_type="stuff", retriever=retriever
)
```

--------------------------------

### Import Chroma Vector Store

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Imports the Chroma vector store class from the Langchain library. This is the first step to creating a vector database.

```python
from langchain.vectorstores import Chroma
```

--------------------------------

### Initialize Ensemble Retriever

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Configure an EnsembleRetriever by specifying the individual retrievers and their respective weights. This allows for a weighted combination of results from different retrieval strategies.

```python
retriever = EnsembleRetriever(retrievers=[keyword_retriever, vectorstore_retriever],weights = [0.7,0.3])
```

--------------------------------

### Define Sample Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

A list of strings representing the documents to be indexed and searched.

```python
documents = [
    "This is a list which containig sample documents.",
    "Keywords are important for keyword-based search.",
    "Document analysis involves extracting keywords.",
    "Keyword-based search relies on sparse embeddings."
]
```

--------------------------------

### Create BM25 Keyword Retriever

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Initializes a BM25Retriever from a list of document chunks. This sets up the keyword-based search index.

```python
#CREATING SPARSE EMBEDDINGS
keyword_retriever = BM25Retriever.from_documents(chunks)
```

--------------------------------

### Import RetrievalQA

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Imports the `RetrievalQA` class from LangChain, which is commonly used for building question-answering systems over documents.

```python
from langchain.chains import RetrievalQA
```

--------------------------------

### Instantiate Tokenizer

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Instantiates the tokenizer using the `initialize_tokenizer` function. This code assumes `model_name` is already defined.

```python
tokenizer = initialize_tokenizer(model_name)
```

--------------------------------

### Create Chroma Vector Store

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Initializes a Chroma vector store from a list of document chunks and embeddings. This creates the vector index for semantic search.

```python
vectorstore = Chroma.from_documents(chunks, embeddings)
```

--------------------------------

### Display Vector Store Retriever

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Displays the configuration of the created vector store retriever, showing its type and search parameters.

```python
vectorstore_retriever
```

--------------------------------

### Initialize Ranks Array

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Creates an empty NumPy array to store the ranks of documents, matching the shape of the ranked indices.

```python
ranks = np.empty_like(ranked_indices)
ranks[ranked_indices] = np.arange(len(similarities))

print("Similarities:", similarities)
print("Ranked Indices:", ranked_indices)
print("Ranks:", ranks)
```

--------------------------------

### Load PDF Document with PyPDFLoader

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Use PyPDFLoader to load a PDF document from a specified path. Ensure the 'doc_path' variable is correctly set before execution.

```python
from langchain_community.document_loaders import PyPDFLoader
```

```python
loader = PyPDFLoader(doc_path)
```

--------------------------------

### Import BM25 and Ensemble Retrievers

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Imports the necessary retriever classes from Langchain for implementing both keyword-based (BM25Retriever) and hybrid search (EnsembleRetriever).

```python
from langchain.retrievers import BM25Retriever, EnsembleRetriever
```

--------------------------------

### Initialize HuggingFace Embeddings

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Initialize the HuggingFaceInferenceAPIEmbeddings with your API key and the desired model name. This object will be used to generate embeddings for text.

```python
embeddings = HuggingFaceInferenceAPIEmbeddings(api_key=HF_TOKEN, model_name="BAAI/bge-base-en-v1.5")
```

--------------------------------

### Wrap Pipeline with LangChain

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Wraps the Hugging Face pipeline with LangChain's `HuggingFacePipeline` for easier integration into LangChain workflows. Note the deprecation warning and the recommendation to use `langchain-huggingface`.

```python
from langchain_community.llms.huggingface_pipeline import HuggingFacePipeline

llm = HuggingFacePipeline(pipeline=pipeline)
```

--------------------------------

### Initialize TfidfVectorizer

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Creates an instance of TfidfVectorizer to convert text into TF-IDF features.

```python
vector = TfidfVectorizer()
```

--------------------------------

### Document Loading and Chunking with LangChain

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Loads a PDF document using `PyPDFLoader` and splits it into overlapping chunks using `RecursiveCharacterTextSplitter`. These chunks form the retrieval corpus for both the BM25 keyword retriever and the Chroma vectorstore.

```python
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter

# Load PDF
loader = PyPDFLoader("/content/2005.11401v4.pdf")
docs = loader.load()

# Split into chunks of 500 characters with 50-character overlap
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = splitter.split_documents(docs)

print(f"Total chunks: {len(chunks)}")
```

--------------------------------

### Load Quantized Model Instance

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Loads the quantized model using the `load_quantized_model` function. This code assumes `model_name` is already defined.

```python
model = load_quantized_model(model_name)
```

--------------------------------

### Print First 200 Characters of a Chunk

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Prints the first 200 characters of the page content for a LangChain Document chunk.

```python
print(chunks[0].page_content[:200])
```

--------------------------------

### Invoke Hybrid Chain for Question Answering

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

This snippet demonstrates how to invoke a hybrid chain to answer a question. It's useful for performing abstractive question answering using the RAG model.

```python
response2 = hybrid_chain.invoke("What is Abstractive Question Answering?")
```

--------------------------------

### Re-ranking Documents by Similarity (Alternative)

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

An alternative method to rank documents by similarity, ensuring the similarities array is flattened before sorting.

```python
import numpy as np

similarities = np.array([[0.        ],
                         [0.50551777],
                         [0.        ],
                         [0.48693426]])

# Flatten similarities if it's a 2D column vector
similarities = similarities.flatten()

# Get the ranked indices in descending order
ranked_indices = np.argsort(similarities)[::-1]
```

--------------------------------

### Display Response from Hybrid Chain

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

This snippet shows how to display the response obtained from the hybrid chain invocation. It's typically used to view the generated answer to a query.

```python
response2
```

--------------------------------

### Preprocess Query

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Applies the preprocessing function to the search query.

```python
preprocessesd_query = preprocess_text(query)
```

--------------------------------

### TF-IDF Vector for First Document

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Shows the TF-IDF vector for the first document in the corpus.

```python
X.toarray()[0]
```

--------------------------------

### Load Document Content

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Call the load() method on the PyPDFLoader instance to retrieve the document content. This will return a list of Document objects.

```python
docs = loader.load()
```

--------------------------------

### Preprocess Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Applies the preprocessing function to all documents in the corpus.

```python
preprocess_documents = [preprocess_text(doc) for doc in documents]
```

--------------------------------

### Display Ranked Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Iterates through the ranked documents and prints each document along with its rank.

```python
for i,doc in enumerate(ranked_documents):
    print(f"Rank {i+1}: {doc}")
```

--------------------------------

### Import Libraries for TF-IDF and Cosine Similarity

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Imports necessary libraries for text vectorization, similarity calculations, and data manipulation.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd
import numpy as np
```

--------------------------------

### Create Vector Store Retriever

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Converts the Chroma vector store into a retriever object. This allows the vector store to be used within Langchain's retrieval chains, specifying the number of results (k) to return.

```python
vectorstore_retriever = vectorstore.as_retriever(search_kwargs = {"k": 3})
```

--------------------------------

### Display Text Chunks

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Display the generated text chunks. Each chunk is a Document object containing page content and metadata.

```python
chunks
```

--------------------------------

### Chroma Vectorstore Retriever (Dense/Semantic)

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Embeds document chunks using HuggingFace Inference API and stores them in Chroma. Use this for semantic similarity searches.

```python
from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings
from langchain.vectorstores import Chroma

HF_TOKEN = "hf_..."  # HuggingFace Inference API token

# Create embeddings model
embeddings = HuggingFaceInferenceAPIEmbeddings(
    api_key=HF_TOKEN,
    model_name="BAAI/bge-base-en-v1.5"
)

# Build vectorstore from document chunks
vectorstore = Chroma.from_documents(chunks, embeddings)

# Create a retriever that returns top-3 results
vectorstore_retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

# Retrieve relevant chunks for a query
results = vectorstore_retriever.get_relevant_documents("What is Abstractive QA?")
for doc in results:
    print(doc.page_content[:150])
```

--------------------------------

### BM25 Keyword Retriever (Sparse)

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Creates a BM25-based keyword retriever from document chunks for efficient term frequency matching. No embedding computation is required.

```python
# pip install rank_bm25
from langchain.retrievers import BM25Retriever

# Build BM25 index from chunks (in-memory)
keyword_retriever = BM25Retriever.from_documents(chunks)
keyword_retriever.k = 3  # return top-3 results

# Retrieve by keyword
results = keyword_retriever.get_relevant_documents("Abstractive Question Answering")
for doc in results:
    print(doc.page_content[:150])
```

--------------------------------

### Define Search Query

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

A string representing the user's search query.

```python
query ="keyword-based search"
```

--------------------------------

### Define Document Path

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Specifies the local path to the PDF document that will be processed. Ensure this path is correct for your system.

```python
doc_path = r"/content/2005.11401v4.pdf"
```

--------------------------------

### Display Ranked Documents with Indices

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Iterates through the ranked indices and prints each document along with its rank.

```python
for i,idx in enumerate(ranked_indices):
    print(f"Rank {i+1}: {documents[idx]}")
```

--------------------------------

### Fit and Transform Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Fits the TfidfVectorizer to the preprocessed documents and transforms them into a TF-IDF matrix.

```python
X=vector.fit_transform(preprocess_documents)
```

--------------------------------

### Retrieve Ranked Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Constructs a list of documents in their ranked order using the computed ranked indices.

```python
ranked_documents = [documents[i] for i in ranked_indices]
```

--------------------------------

### Rank Documents by Similarity

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Sorts the documents based on their similarity scores in descending order and returns the indices of the ranked documents.

```python
ranked_indices = np.argsort(similarities,axis=0)[::-1].flatten()
```

--------------------------------

### Rank Documents by Similarity

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Sorts the documents based on their similarity scores in descending order to find the most relevant ones.

```python
ranked_indices = np.argsort(similarities, axis=0)[::-1].flatten()
```

--------------------------------

### Load 4-bit Quantized Model

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Function for loading a 4-bit quantized model. This is useful for efficient memory usage and faster inference in question answering tasks.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.1",
    quantization_config=quantization_config,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
```

--------------------------------

### Split Documents into Chunks with RecursiveCharacterTextSplitter

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Initialize RecursiveCharacterTextSplitter to split the loaded documents into smaller chunks. Configure chunk_size and chunk_overlap for optimal segmentation. This is crucial for efficient retrieval and processing.

```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
```

```python
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
```

--------------------------------

### Split Documents

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Apply the configured text splitter to the loaded documents to generate text chunks. The 'chunks' variable will hold a list of Document objects, each representing a chunk of text.

```python
chunks = splitter.split_documents(docs)
```

--------------------------------

### Ranked Document Indices

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Shows the indices of the documents sorted by their relevance to the query.

```python
ranked_indices
```

--------------------------------

### Set BM25 Retriever k Value

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Configures the number of documents (k) to retrieve using the BM25Retriever. This specifies the top results to be considered from the keyword search.

```python
keyword_retriever.k = 3
```

--------------------------------

### Text Preprocessing Function

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

A function to clean text by removing punctuation and converting to lowercase. This is crucial for consistent vectorization.

```python
import re
def preprocess_text(text):
    # Remove punctuation and special characters
    text = re.sub(r'[^\w\s]', '', text)
    # Convert to lowercase
    text = text.lower()
    return text
```

--------------------------------

### Transform Query into TF-IDF Vector

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Transforms the preprocessed query into a TF-IDF vector using the fitted TfidfVectorizer. This vector can then be compared to document vectors.

```python
query_embedding=vector.transform([preprocessesd_query])
```

--------------------------------

### EnsembleRetriever — Hybrid Search (BM25 + Semantic)

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Combines BM25 and Chroma semantic retrievers using EnsembleRetriever for hybrid search. Results are merged with weighted Reciprocal Rank Fusion (RRF).

```python
from langchain.retrievers import BM25Retriever, EnsembleRetriever
from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceInferenceAPIEmbeddings

# Build both retrievers (assumes `chunks` already created)
keyword_retriever = BM25Retriever.from_documents(chunks)
keyword_retriever.k = 3

vectorstore_retriever = Chroma.from_documents(
    chunks,
    HuggingFaceInferenceAPIEmbeddings(api_key=HF_TOKEN, model_name="BAAI/bge-base-en-v1.5")
).as_retriever(search_kwargs={"k": 3})

# Combine: 70% weight on BM25, 30% on semantic
retriever = EnsembleRetriever(
    retrievers=[keyword_retriever, vectorstore_retriever],
    weights=[0.7, 0.3]
)

results = retriever.get_relevant_documents("What is Abstractive Question Answering?")
for doc in results:
    print(doc.page_content[:150])
```

--------------------------------

### TF-IDF Sparse Retrieval with Cosine Similarity

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Transforms documents into TF-IDF sparse vectors and ranks them against a query using cosine similarity. Best for exact-keyword matching where speed and interpretability matter. Uses scikit-learn's `TfidfVectorizer` and `cosine_similarity`.

```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
import re

# Sample corpus
documents = [
    "This is a list which containig sample documents.",
    "Keywords are important for keyword-based search.",
    "Document analysis involves extracting keywords.",
    "Keyword-based search relies on sparse embeddings."
]
query = "keyword-based search"

# Preprocessing: remove punctuation, lowercase
def preprocess_text(text):
    text = re.sub(r'[^\w\s]', '', text)
    return text.lower()

preprocessed_docs = [preprocess_text(doc) for doc in documents]
preprocessed_query = preprocess_text(query)
# -> 'keywordbased search'

# Fit TF-IDF on corpus, transform query
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(preprocessed_docs)          # shape: (4, vocab_size)
query_vec = vectorizer.transform([preprocessed_query])   # shape: (1, vocab_size)

# Rank documents by cosine similarity
similarities = cosine_similarity(X, query_vec).flatten()
ranked_indices = np.argsort(similarities)[::-1]

for rank, idx in enumerate(ranked_indices):
    print(f"Rank {rank+1} (score={similarities[idx]:.4f}): {documents[idx]}")

# Expected output:
# Rank 1 (score=0.5055): Keywords are important for keyword-based search.
# Rank 2 (score=0.4869): Keyword-based search relies on sparse embeddings.
# Rank 3 (score=0.0000): Document analysis involves extracting keywords.
# Rank 4 (score=0.0000): This is a list which containig sample documents.
```

--------------------------------

### HuggingFace API Token

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Define your HuggingFace API token. This token is required for authentication when using the HuggingFace Inference API.

```python
HF_TOKEN = "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"
```

--------------------------------

### Dense Vector Retrieval with Cosine Similarity

Source: https://context7.com/jhaayush01/hybrid-retrieval-systems/llms.txt

Ranks documents using pre-computed dense embedding vectors. Captures semantic meaning so documents with related concepts rank highly even without exact keyword overlap.

```python
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Simulated dense embeddings (e.g., from SentenceTransformers or HuggingFace)
document_embeddings = np.array([
    [0.634, 0.234, 0.867, 0.042, 0.249],
    [0.123, 0.456, 0.789, 0.321, 0.654],
    [0.987, 0.654, 0.321, 0.123, 0.456]
])
query_embedding = np.array([[0.789, 0.321, 0.654, 0.987, 0.123]])

similarities = cosine_similarity(document_embeddings, query_embedding).flatten()
ranked_indices = np.argsort(similarities)[::-1]

documents = [
    "This is a list which containig sample documents.",
    "Keywords are important for keyword-based search.",
    "Document analysis involves extracting keywords."
]

for rank, idx in enumerate(ranked_indices):
    print(f"Rank {rank+1} (score={similarities[idx]:.4f}): {documents[idx]}")

# Expected output:
# Rank 1 (score=0.7356): This is a list which containig sample documents.
# Rank 2 (score=0.7152): Document analysis involves extracting keywords.
# Rank 3 (score=0.6736): Keywords are important for keyword-based search.
```

--------------------------------

### TF-IDF Matrix Representation

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Displays the TF-IDF matrix as a dense NumPy array. Each row represents a document, and each column represents a term.

```python
X.toarray()
```

--------------------------------

### Query TF-IDF Vector Representation

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Displays the TF-IDF vector for the query as a dense NumPy array.

```python
query_embedding.toarray()
```

--------------------------------

### Cosine Similarity Scores

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Displays the calculated cosine similarity scores between each document and the query.

```python
similarities
```

--------------------------------

### Define Model Name

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/Hybrid_search.ipynb

Specify the name of the pre-trained model to be used for retrieval or other natural language processing tasks.

```python
model_name =  "HuggingFaceH4/zephyr-7b-beta"
```

--------------------------------

### Calculate Cosine Similarity

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Computes the cosine similarity between document embeddings and a query embedding.

```python
# Calculate cosine similarity between query and documents
similarities = cosine_similarity(document_embeddings, query_embedding)
```

--------------------------------

### Calculate Cosine Similarities

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Computes the cosine similarity between the TF-IDF matrix of documents and the TF-IDF vector of the query.

```python
similarities = cosine_similarity(X, query_embedding)
```

--------------------------------

### Document Embeddings

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Defines a NumPy array representing the embeddings for multiple documents.

```python
document_embeddings = np.array([
    [0.634, 0.234, 0.867, 0.042, 0.249],
    [0.123, 0.456, 0.789, 0.321, 0.654],
    [0.987, 0.654, 0.321, 0.123, 0.456]
])
```

--------------------------------

### Query Embedding

Source: https://github.com/jhaayush01/hybrid-retrieval-systems/blob/main/TfidfVectorizer.ipynb

Defines a NumPy array representing the embedding for a query.

```python
query_embedding = np.array([[0.789, 0.321, 0.654, 0.987, 0.123]])
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.