### Setup and Run Python Example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-document-agentic-rag/README.md

Installs dependencies, sets up a virtual environment, and runs the main Python script for the Multi-Document Agentic RAG example.

```bash
python3 -m venv env
source env/bin/activate
pip3 install -r requirements.txt
cd examples/multi-document-agentic-rag
python3 main.py
```

--------------------------------

### Install LangChain and Autogen

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/LangChain_and_Autogen.ipynb

Install the necessary libraries for LangChain and Autogen. This is a prerequisite for using the subsequent code examples.

```bash
pip install langchain autogen
```

--------------------------------

### Install and Run Development Server

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/multimodal-search/README.md

Use npm, yarn, or pnpm to install dependencies and start the Next.js development server. Ensure your Roboflow API key is configured in .env.local.

```bash
npm run dev
```

```bash
yarn dev
```

```bash
pnpm dev
```

--------------------------------

### Install Dependencies and Run arXiv Recommender

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/arxiv-recommender/README.md

Install the required packages using pip and then run the main Python script to start the arXiv recommender.

```bash
pip install -r requirements.txt

python main.py
```

--------------------------------

### Install LangChain and Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.ipynb

Installs the necessary LangChain libraries and OpenAI integration. This is a prerequisite for running the Q&A bot example.

```python
! pip install -U langchain langchain-openai langchain-community
```

--------------------------------

### Autogen Agent Setup with LanceDB Integration

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen/main.ipynb

This example shows the basic setup for an Autogen agent that can interact with LanceDB. It involves defining agent configurations and initializing the LanceDB client.

```python
import autogen
from autogen.agentchat.contrib.lancedb_agent import LanceDBAgent

config_list = [
    {
        "model": "gpt-4",
        "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    }
]

llm_config = {"config_list": config_list, "cache_seed": 42}

user_proxy = autogen.UserProxyAgent("user_proxy",
                                  llm_config=llm_config,
                                  human_input_mode="ALWAYS",
                                  code_execution_config=False)

vector_db_agent = LanceDBAgent("vector_db_agent",
                                llm_config=llm_config,
                                db_path="/tmp/my_lancedb",
                                table_name="my_table")

user_proxy.initiate_chat(vector_db_agent, message="Hello Vector DB Agent!")
```

--------------------------------

### Initialize LanceDB and HoneyHive

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

This snippet shows the initial setup for both LanceDB and HoneyHive, including necessary imports and client initializations. Ensure you have the required libraries installed.

```python
import os

from lance.vector import VectorIndex
from lancedb.embeddings import EmbeddingFunction, SentenceTransformerEmbeddings
from lancedb.pydantic import pydanticModel, BaseModel
from lancedb.query import LanceDBQuery
from lancedb.table import LanceDBTable
from lancedb.vector import LanceDBVector

from honeyhive.api import API
from honeyhive.trace import Trace
from honeyhive.trace.utils import TraceConfig

# Initialize HoneyHive API
api = API(api_key=os.environ.get("HONEYHIVE_API_KEY"))

# Initialize TraceConfig
trace_config = TraceConfig(api=api)

# Initialize Trace
trace = Trace(trace_config=trace_config)

# Initialize LanceDB
vector_db = LanceDBVector()

# Initialize Embedding Function
embedding_function = SentenceTransformerEmbeddings(model_name="all-MiniLM-L6-v2")
```

--------------------------------

### Basic RAG Setup with LanceDB

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-document-agentic-rag/main.ipynb

This snippet demonstrates the fundamental setup for a RAG system, including loading documents and initializing the LanceDB vector store. It's a starting point for more complex agentic workflows.

```python
from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated
import operator
from langchain_core.documents import Document
from langchain_core.runnables import RunnablePassthrough
from langchain_core.runnables.base import Runnable
from langchain_community.vectorstores import LanceDB
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate


class AgentState(TypedDict):
    """State for the agent."""

    question: str
    documents: list[Document]
    intermediate_steps: Annotated[list[tuple[str, str]], operator.add]
    answer: str


class Agent: # noqa
    def __init__(self, vector_store: LanceDB):
        self.vector_store = vector_store
        self.retriever = vector_store.as_retriever()
        self.llm = ChatOpenAI(model="gpt-4o")
        self.prompt = ChatPromptTemplate.from_template(
            """You are a helpful assistant. Use the following pieces of context to answer the question at the end.

            Context:
            {context}

            Question:
            {question}

            Only use the positive and helpful information from the context to answer the question. If you don't know the answer, just say that you don't know, don't try to make up an answer.

            Helpful Answer:"""
        )
        self.chain = (self.prompt | self.llm | operator.itemgetter("content"))

    def retrieve(self, state):
        """Retrieve documents from the vector store."""
        question = state.get("question")
        documents = self.retriever.invoke(question)
        return {"documents": documents, "question": question}

    def generate(self, state):
        """Generate an answer using the retrieved documents."""
        documents = state.get("documents")
        question = state.get("question")
        answer = self.chain.invoke({"context": documents, "question": question})
        return {"answer": answer}


def create_agent_graph(agent: Agent):
    """Create the agent graph."""
    workflow = StateGraph(AgentState)
    workflow.add_node("retrieve", agent.retrieve)
    workflow.add_node("generate", agent.generate)
    workflow.add_edge("retrieve", "generate")
    workflow.set_entry_point("retrieve")
    return workflow.compile()


# Example Usage:
# embeddings = OpenAIEmbeddings()
# vector_store = LanceDB.from_documents(
#     [Document(page_content="This is a test document.")],
#     embeddings,
#     mode="hybrid",
#     # LanceDB specific args
#     # For example, to specify a data directory:
#     # data_dir=".lancedb",
# )
# agent = Agent(vector_store)
# graph = create_agent_graph(agent)
# result = graph.invoke({"question": "What is this document about?"})
# print(result["answer"])

```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Installs the required libraries for fine-tuning, including `transformers`, `datasets`, `peft`, `bitsandbytes`, and `accelerate`. Ensure these are installed before proceeding.

```bash
pip install -U transformers datasets peft bitsandbytes accelerate
```

--------------------------------

### Install LlamaIndex

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/RAG-with_MatryoshkaEmbed-Llamaindex/RAG_with_MatryoshkaEmbedding_and_Llamaindex.ipynb

Install the LlamaIndex library to get started with RAG and data indexing.

```shell
pip install llama-index
```

--------------------------------

### Download and Navigate Project Files

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/multimodal-recipe-agent/README.md

Download the tutorial files from GitHub and navigate to the project directory. This is the first step for local setup.

```bash
# Download the tutorial files from GitHub
# Extract all files to a folder named 'multimodal-recipe-agent'
# Navigate to the folder
cd multimodal-recipe-agent
```

--------------------------------

### Navigate to Project Directory

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/ts_example/quickstart/README.md

Change the current directory to the project's quickstart folder.

```bash
cd ts_example/quickstart
```

--------------------------------

### Partial Query Example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/ColPali-vision-retriever/colpali.ipynb

Initiates a search query with ColPali, but the code snippet is incomplete, only showing the start of the query setup.

```python
import time

t1 = time.time()
user_query = "How did the fish become men, women, and children?"

```

--------------------------------

### Install Dependencies and Run App

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/imagebind_demo/README.md

Install the necessary Python dependencies and then run the Imagebind demo application. Ensure you are in the correct directory after cloning.

```bash
pip install -r requirements.txt
python3 app.py
```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

Install the required libraries for HoneyHive and LanceDB. This is a prerequisite for running the subsequent code examples.

```python
import os

import pandas as pd
import pyarrow as pa

from lancedb.embeddings import EmbeddingFunction, SentenceTransformerEmbeddings
from lancedb.pydantic import pydantic_schema
from lancedb.vectorstore import LanceDB

from honeyhive.api import Api
from honeyhive.trace import Trace
from honeyhive.trace.utils import TraceConfig

# Set up environment variables for HoneyHive
os.environ["HONEYHIVE_API_KEY"] = "YOUR_API_KEY"
os.environ["HONEYHIVE_PROJECT_ID"] = "YOUR_PROJECT_ID"

# Initialize HoneyHive API and TraceConfig
api = Api()
trace_config = TraceConfig(api=api)

# Initialize LanceDB vector store
vector_store = LanceDB(uri="./lancedb")

```

--------------------------------

### Start the Full Project

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/node/hierarchical-multi-agent/README.md

Execute the main start command for the entire project, launching all components. This is used to run the complete application.

```bash
npm start
```

--------------------------------

### Start Application

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Deepseek_R1_VS_GPT_4o/README.md

Command to launch the main application script after setting up the model.

```bash
python app.py
```

--------------------------------

### Install LanceDB and Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/Umap_on_LanceDB_Cloud.ipynb

Install the LanceDB Python client and the UMAP library. This is a prerequisite for running the subsequent code examples.

```python
pip install lancedb umap-learn
```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/customer_support_agent_langgraph/LangGraph_LanceDB.ipynb

Installs required Python packages for LangGraph, LanceDB, and related AI/ML libraries. This is a prerequisite for running the examples.

```python
%%capture --no-stderr
%pip install -U colorama langgraph langchain-community langchain-openai langchain-anthropic tavily-python pandas openai lancedb sentence-transformers
```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/time-travel-rag/README.md

Install the required dependencies for the tutorial using pip or uv.

```bash
pip install -r requirements.txt
```

```bash
# OR, If you're using uv
uv pip install -r requirements.txt
```

--------------------------------

### Navigate to Example Directory

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/ts_example/hybrid-search/README.md

Navigate to the hybrid-search directory within the ts_example folder.

```bash
cd ts_example/hybrid-search
```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/NER-powered-Semantic-Search/NER_powered_Semantic_Search_with_LanceDB.ipynb

Installs necessary Python packages for sentence transformers, datasets, and LanceDB. Use this at the beginning of your environment setup.

```python
!pip install sentence_transformers datasets lancedb -qU
```

--------------------------------

### Run Jupyter Notebook Tutorial

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/multimodal-recipe-agent/README.md

Start the Jupyter Notebook server to access the interactive tutorial. This allows for step-by-step learning.

```bash
uv run jupyter notebook multimodal-recipe-agent.ipynb
```

--------------------------------

### Configure and start training

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Sets up the `TrainingArguments` for the fine-tuning process, including output directories, learning rate, batch size, and optimization parameters. Then, it initializes and starts the `SFTTrainer`.

```python
from transformers import TrainingArguments
from trl import SFTTrainer

# Define training arguments
training_arguments = TrainingArguments(
    output_dir="./results",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=2,
    optim="paged_adamw_32bit",
    learning_rate=0.0002,
    fp16=True,
    logging_steps=10,
    save_strategy="epoch",
    report_to="none",
)

# Initialize the SFTrainer
sft_trainer = SFTTrainer(
    model=model,
    train_dataset=tokenized_dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_arguments,
    data_collator=data_collator,
)

# Start training
sft_trainer.train()

```

--------------------------------

### Installing PyAutogen

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen/main.ipynb

Installs the pyautogen library, which is a prerequisite for running the Autogen-related functionalities in this example. Ensure you have Python version 3.8 or higher.

```bash
!pip install pyautogen
```

--------------------------------

### Setting up Environment for QLoRA Fine-tuning

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

This snippet installs the necessary libraries for QLoRA fine-tuning, including `transformers`, `peft`, `bitsandbytes`, and `accelerate`. Ensure these are installed before proceeding with fine-tuning.

```python
import os
import sys
import datasets
import transformers
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer

# Set environment variables for Hugging Face
os.environ["HF_HOME"] = os.getcwd() + "/huggingface"
os.environ["TRANSFORMERS_CACHE"] = os.getcwd() + "/huggingface/transformers"
os.environ["PEFT_CACHE"] = os.getcwd() + "/huggingface/peft"
os.environ["BITSANDBYTES_CACHE"] = os.getcwd() + "/huggingface/bitsandbytes"

# Check PyTorch version
print(f"PyTorch version: {torch.__version__}")

# Check CUDA availability
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"CUDA version: {torch.version.cuda}")
print(f"Number of GPUs: {torch.cuda.device_count()}")
print(f"GPU Name: {torch.cuda.get_device_name(0)}")

# Set random seed for reproducibility
torch.manual_seed(42)

# Define model and dataset paths
model_name = "meta-llama/Llama-2-7b-hf"
dataset_name = "timdettmers/openassistant-guanaco"
output_dir = "./results"

```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Advance-RAG-with-HyDE/main.ipynb

Installs necessary Python libraries for Langchain, OpenAI, LanceDB, and PDF processing. This is a prerequisite for running the HyDE examples.

```python
%%capture
!pip install langchain openai tiktoken langchain-community
!pip install lancedb
!pip install pypdf
!pip install -q langchain-openai
```

--------------------------------

### Setup and Initialization

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-document-agentic-rag/main.ipynb

Initializes the environment and loads necessary components for the multi-document agentic RAG system. This includes setting up the LangChain environment and defining the path to the source documents.

```python
from langgraph.graph import StateGraph, END
from typing import TypedDict, Annotated
import operator
from langchain_core.tools import tool
from langchain_core.messages import BaseMessage, HumanMessage, SystemMessage, AIMessage
from langchain_community.document_loaders import PyPDFLoader
from langchain_community.vectorstores import LanceDB
from langchain_community.embeddings import OpenAIEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.llms import OpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.schema import format_document
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain.chains import create_retrieval_chain
import os
import lancedb

# Set environment variables
os.environ["OPENAI_API_KEY"] = "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

# Define the path to the documents
docs_path = "/lancedb/vectordb-recipes/"

```

--------------------------------

### Install LanceDB and Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/Umap_on_LanceDB_Cloud.ipynb

Install the LanceDB Python client and necessary libraries for data manipulation and embedding generation. This is a prerequisite for running the subsequent examples.

```python
pip install lancedb[notebook] umap-learn
```

--------------------------------

### Start Training

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Starts the fine-tuning process using the configured trainer.

```python
# Train the model
trainer.train()
```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/RAG-On-PDF/main.ipynb

Installs necessary Python packages including lancedb, tantivy, langchain-openai, langchain-community, pypdf, and their dependencies. This is the initial setup step for the project.

```python
pip install lancedb
pip install tantivy
pip install langchain-openai
pip install langchain-community
pip install pypdf
```

--------------------------------

### Install Dependencies and Run Notebook Locally

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/llm-pretraining-dataloading/README.md

Install required packages, including ipython for interactive execution, and then run the training notebook locally.

```shell
pip install -r requirements.txt
pip install -q ipython

ipython train.ipynb
```

--------------------------------

### Install LlamaIndex CLI and Download Dataset

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

Installs the LlamaIndex CLI tool and downloads a sample dataset from LlamaIndex. This is a prerequisite for running the RAG pipeline example.

```bash
!pip install llama-index-cli -q
!llamaindex-cli download-llamadataset PaulGrahamEssayDataset --download-dir ./data
```

--------------------------------

### Basic Autogen Agent Setup

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/LangChain_and_Autogen.ipynb

Demonstrates the fundamental setup of an Autogen agent, including defining its role and initiating a conversation.

```python
import autogen

config_list = [
    {
        "model": "gpt-4",
        "api_key": "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
    }
]

llm_config = {
    "config_list": config_list,
    "temperature": 0.7,
}

user_proxy = autogen.UserProxyAgent(
    name="User_Proxy",
    is_termination_msg=lambda x: x.get("content", "").rstrip().endswith("TERMINATE"),
    human_input_mode="NEVER",
    code_execution_config=False,
)

agent1 = autogen.AssistantAgent(
    name="Assistant_Agent_1",
    llm_config=llm_config,
    system_message="You are a helpful assistant.",
)

# Initiate a chat between user_proxy and agent1
chat_result = user_proxy.initiate_chat(
    agent1,
    message="Hello, Assistant Agent 1! How can you help me today?"
)

print(chat_result)
```

--------------------------------

### Install LanceDB, HoneyHive, and Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

Install the necessary Python packages for LanceDB, HoneyHive tracing, sentence transformers, and OpenAI. This is a prerequisite for running the RAG examples.

```python
!pip install lancedb honeyhive sentence-transformers openai pandas
```

--------------------------------

### Install Dependencies

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Accelerate-Vector-Search-Applications-Using-OpenVINO/clip_text_image_search.ipynb

Installs necessary Python packages including transformers, torch, datasets, gdcm, pydicom, lancedb, and onnx. This is a prerequisite for running the subsequent code examples.

```python
! pip install transformers torch datasets gdcm pydicom lancedb onnx
```

--------------------------------

### Install Dependencies and Run Data Preparation

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/llm-pretraining-dataloading/README.md

Install required packages from requirements.txt and then execute the data preparation script. This should be run once before training.

```shell
pip install -r requirements.txt

python data_prep.py
```

--------------------------------

### Install Required Libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Late_Chunking/Late_Chunking_(Chunked_Pooling).ipynb

Installs the necessary Python packages for advanced RAG, including transformers, datasets, einops, lancedb, and sentence-transformers. This is a prerequisite for running the late chunking examples.

```python
!pip install transformers datasets einops lancedb sentence-transformers -qq
```

--------------------------------

### Start Development Server

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/node/lanchain_writing_assistant/README.md

Launches the application in development mode. Access the app at http://localhost:3000.

```bash
npm run dev
```

--------------------------------

### Text-to-Speech Conversion

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Multi_Speaker_Context_Aware_Podcast_Generation/Multi_Speaker_Context_Aware_Podcast_Generation.ipynb

Converts generated text into speech using a TTS model. This example uses the 'Coqui TTS' library. Ensure it's installed (`pip install TTS`).

```python
from TTS.api import TTS

# Get device
device = "cuda" if torch.cuda.is_available() else "cpu"

# List available 🐸TTS models
# print(TTS().list_models()) # Uncomment to see available models

# Init TTS
tts = TTS("tts_models/en/ljspeech/tacotron2-DDC", gpu=torch.cuda.is_available())

# Run TTS
speaker_text = "This is a sample sentence to be converted to speech."
tts.tts_to_file(text=speaker_text, file_path="output.wav")
```

--------------------------------

### Initialize SFTTrainer and Start Training

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/social-media-caption-generation-with-llama3.2/social_media_caption_generation_llama3_2_11B.ipynb

Sets up the SFTTrainer with the configured model, dataset, LoRA, and training arguments, then initiates the fine-tuning process. This step requires significant computational resources.

```python
# Initialize SFTTrainer
trainer = SFTTrainer(
    model=model_name,
    train_dataset=dataset,
    peft_config=lora_config,
    dataset_text_field="text",
    max_seq_length=512,
    tokenizer=tokenizer,
    args=training_arguments,
    packing=True,
    generation_config=generation_config,
    quantization_config=quantization_config,
)

# Start training
trainer.train()

# Save the fine-tuned model and tokenizer
trainer.save_model(output_dir)
trainer.tokenizer.save_pretrained(output_dir)

```

--------------------------------

### Context-Aware Generation Setup

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Multi_Speaker_Context_Aware_Podcast_Generation/Multi_Speaker_Context_Aware_Podcast_Generation.ipynb

Demonstrates setting up a pipeline for context-aware generation. This involves loading a model and tokenizer.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer for context-aware generation
# Replace 'your-model-name' with an appropriate model
model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)

# Ensure the tokenizer has a padding token if it doesn't
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token
```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Install the required libraries for fine-tuning LLMs with PEFT and QLoRA. This includes `transformers`, `peft`, `bitsandbytes`, `accelerate`, and `datasets`.

```bash
pip install -q transformers peft bitsandbytes accelerate datasets
```

--------------------------------

### Numpy ufunc signature example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.ipynb

Shows how to get the signature of a Numpy universal function (ufunc).

```python
numpy_docs/reference/generated/numpy.ufunc.signature.html
```

--------------------------------

### Install Necessary Libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Install the required libraries for fine-tuning LLMs with PEFT and QLoRA. This includes `transformers`, `peft`, `bitsandbytes`, `accelerate`, and `datasets`.

```bash
pip install transformers peft bitsandbytes accelerate datasets
```

--------------------------------

### Example Data Generation and Loading in LanceDB

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Late_Chunking/Late_Chunking_(Chunked_Pooling).ipynb

Demonstrates how to generate and load example data into LanceDB, a vector database. This setup is crucial for testing and implementing retrieval strategies like Late Chunking.

```python
from lancedb.embeddings import EmbeddingFunction, SentenceTransformerEmbeddings
from lancedb.pydantic import pydantic_to_schema, LanceModel
from pydantic import Field
import pandas as pd
import numpy as np
import lancedb

class TextChunk(LanceModel):
    text: str
    vector: list[float] = Field(default=None, ndim=384)
    doc_id: str
    chunk_id: int


class TextChunkEmbedder(EmbeddingFunction):
    def __init__(self, model_name="sentence-transformers/all-MiniLM-L6-v2"):
        self.model = SentenceTransformerEmbeddings(model_name)

    def __call__(self, texts):
        return self.model(texts)


def create_dummy_data(num_chunks=100):
    data = []
    for i in range(num_chunks):
        data.append({
            "text": f"This is chunk number {i} of the document.",
            "doc_id": "doc1",
            "chunk_id": i,
        })
    return pd.DataFrame(data)

db = lancedb.connect("~/lancedb")

# Create a table with a schema that includes text, vector, doc_id, and chunk_id
table = db.create_table("late_chunking_test", schema=TextChunk, exist_ok=True)

# Generate dummy data and embed it
df = create_dummy_data()
embedder = TextChunkEmbedder()

# Embed the text column and add it to the dataframe
df["vector"] = embedder(df["text"].tolist())

# Convert the dataframe to the LanceDB schema and add it to the table
table.add(df)

print(f"Table '{table.name}' created with {table.count()} records.")
```

--------------------------------

### Configure and start training

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Sets up the TrainingArguments and starts the fine-tuning process using the Trainer API. This includes defining hyperparameters and output directories.

```python
from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling

# Define training arguments
training_args = TrainingArguments(
    output_dir="./results",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    num_train_epochs=1,
    logging_steps=10,
    save_steps=100,
    fp16=True,
    push_to_hub=False,
)

# Define data collator
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)

# Initialize Trainer
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_datasets["train"],
    data_collator=data_collator,
)
```

--------------------------------

### Initialize Clients and Setup

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

Initializes HoneyHive tracer, OpenAI client, and configures logging. Ensure HONEYHIVE_API_KEY, HONEYHIVE_PROJECT, and OPENAI_API_KEY environment variables are set.

```python
import os
import sys
import logging
import pandas as pd
import lancedb
from lancedb.pydantic import LanceModel, Vector
from lancedb.embeddings import get_registry
import openai
from honeyhive import HoneyHiveTracer, trace
from typing import List, Dict, Any

# Configure logging
logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s - %(name)s - %(levelname)s - %(message)s",
    handlers=[
        logging.FileHandler("rag_pipeline.log"),
        logging.StreamHandler(sys.stdout),
    ],
)
logger = logging.getLogger("lancedb_rag")

# Initialize HoneyHive tracer
HONEYHIVE_API_KEY = os.environ.get("HONEYHIVE_API_KEY", "your honeyhive api key")
HONEYHIVE_PROJECT = os.environ.get("HONEYHIVE_PROJECT", "your honeyhive project name")

HoneyHiveTracer.init(
    api_key=HONEYHIVE_API_KEY,
    project=HONEYHIVE_PROJECT,
    source="dev",
    session_name="lancedb_rag_session",
)

# Set OpenAI API key
OPENAI_API_KEY = os.environ.get("OPENAI_API_KEY", "your openai api key")
openai.api_key = OPENAI_API_KEY
```

--------------------------------

### Numpy polynomial Laguerre basis example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Code-Documentation-QA-Bot/main.ipynb

Shows how to get the basis polynomials for Numpy's Laguerre polynomials.

```python
numpy_docs/reference/generated/numpy.polynomial.laguerre.Laguerre.basis.html
```

--------------------------------

### Get LanceDB Cloud URI

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/Umap_on_LanceDB_Cloud.ipynb

Obtain the database URI from the LanceDB Cloud dashboard. The URI starts with `db://`.

```python
db_uri = "db://test-sfifxz"
```

--------------------------------

### Initialize LanceDB and HoneyHive

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/HoneyHive_x_LanceDB/HoneyHive_x_LanceDB.ipynb

This snippet shows the initial setup for both LanceDB and HoneyHive, including importing necessary libraries and initializing the LanceDB client and HoneyHive tracer.

```python
import os
import pandas as pd
from lancedb import LanceDBClient
from honeyhive.tracer import Tracer

# Initialize LanceDB client
client = LanceDBClient(uri=os.environ.get("LOCAL_DATA_PATH", "data"))

# Initialize HoneyHive tracer
tracer = Tracer(project_id=os.environ.get("PROJECT_ID"), api_key=os.environ.get("API_KEY"))
```

--------------------------------

### Query the System

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/SuperAgent_Autogen/main.ipynb

Example of how to query the food Q&A system with a specific question. The `answer_food_question` function is used to get the response.

```python
# query
question = "what is good food"
answer_food_question(question)
```

--------------------------------

### Install Necessary Libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Install the required libraries for QLoRA fine-tuning, including `transformers`, `peft`, `bitsandbytes`, `accelerate`, and `datasets`.

```python
import os
import bitsandbytes as bnb
import torch
import torch.nn as nn
import transformers
from datasets import load_dataset
from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, logging

# Suppress logging
logging.set_verbosity_error()
```

--------------------------------

### Dependency Conflict Example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Graphrag/main.ipynb

This output shows a dependency conflict between cudf-cu12 and pandas, indicating potential issues with installed package versions.

```text
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cudf-cu12 24.10.1 requires pandas<2.2.3dev0,>=2.0, but you have pandas 2.2.3 which is incompatible.
```

--------------------------------

### Install Kaggle CLI

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/product-recommender/main.ipynb

Installs the Kaggle CLI and sets up the necessary directory and permissions for using Kaggle datasets.

```bash
! pip install kaggle
! mkdir ~/.kaggle
! cp kaggle.json ~/.kaggle/
! chmod 600 ~/.kaggle/kaggle.json
```

--------------------------------

### Get Speaker Segments

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Speaker_Mapped_Transcription/Speaker_Mapping.ipynb

Retrieves a list of segments (start and end times) for a given speaker ID. This can be used to analyze speaking intervals.

```python
speaker_id_for_segments = 'speaker_1'
speaker_segments = mapper.get_speaker_segments(speaker_id=speaker_id_for_segments)
print(speaker_segments)
```

--------------------------------

### Basic RAG Pipeline Setup

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Evaluating_RAG_with_RAGAs/Evaluating_RAG_with_RAGAs.ipynb

Demonstrates a minimal setup for a RAG pipeline, including loading a dataset, defining a generator, and a question-answerer. This serves as a foundation for evaluation.

```python
from datasets import load_dataset
from ragas import evaluate
from ragas.llms.prompt import Prompt
from ragas.metrics import answer_relevancy, faithfulness, context_precision, context_recall

# Load a dataset (e.g., from Hugging Face)
dataset = load_dataset("explodinggradients/quora_duplicate_questions", "test")

# Define your generator and question-answerer (replace with your actual models)
def generator(batch):
    # Replace with your actual generator logic
    return ["" for _ in batch["question"]]

def question_answerer(batch):
    # Replace with your actual question-answerer logic
    return ["" for _ in batch["question"]]

# Define the RAG pipeline
pipeline = {
    "question": dataset["test"]["question"],
    "ground_truth": dataset["test"]["is_duplicate"],
    "answer": question_answerer({"question": dataset["test"]["question"]} ),
    "contexts": generator({"question": dataset["test"]["question"]} )
}

# Evaluate the pipeline
result = evaluate(
    pipeline,
    metrics=[
        answer_relevancy,
        faithfulness,
        context_precision,
        context_recall,
    ],
    # Add your LLM configuration here if not using default
    # llm=LangchainLLM(model=ChatOpenAI(model_name="gpt-4o", temperature=0))
)

# Print the results
print(result)
```

--------------------------------

### Gradio Interface Setup

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/Agentic_RAG/main.ipynb

Sets up a Gradio interface for the Agentic RAG application, defining the function, inputs, outputs, title, description, and example questions.

```python
# Define example questions to guide the user
example_questions = [
    "explain me in short what is PM Gati Shakti National Master Plan (NMP)?"
]

# Create a Gradio interface
iface = gr.Interface(
    fn=process_message,
    inputs="text",
    outputs="text",
    title="Agentic RAG ",
    description="Enter a message to query related to export import .",
    examples=example_questions,
)
```

--------------------------------

### Get Database URI

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/saas_examples/python_notebook/LanceDB_Cloud_quickstart.ipynb

The database URI is required to connect to your LanceDB Cloud instance. It starts with 'db://' and can be found on your project page in the dashboard.

```text
db uri starts with `db://`, which can be obtained from the project page on the dashboard. In the following example, `db uri` is `db://test-sfifxz`.
```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Installs the required libraries for fine-tuning LLMs with QLoRA. Ensure you have a compatible environment.

```bash
# Install dependencies
!pip install -q -U bitsandbytes
!pip install -q -U peft
!pip install -q -U transformers
!pip install -q -U accelerate
!pip install -q -U trl
!pip install -q -U peft4rec
!pip install -q -U datasets
!pip install -q -U pandas
!pip install -q -U scikit-learn
!pip install -q -U torch
!pip install -q -U sentencepiece
!pip install -q -U protobuf
!pip install -q -U accelerate
!pip install -q -U huggingface_hub
!pip install -q -U ipywidgets
!pip install -q -U tensorboard
!pip install -q -U matplotlib
!pip install -q -U seaborn
!pip install -q -U scikit-learn
!pip install -q -U nltk
!pip install -q -U transformers
!pip install -q -U datasets
!pip install -q -U peft
!pip install -q -U trl
!pip install -q -U bitsandbytes
!pip install -q -U accelerate
!pip install -q -U huggingface_hub
!pip install -q -U sentencepiece
!pip install -q -U protobuf
!pip install -q -U ipywidgets
!pip install -q -U tensorboard
!pip install -q -U matplotlib
!pip install -q -U seaborn
!pip install -q -U scikit-learn
!pip install -q -U nltk
!pip install -q -U torch
!pip install -q -U peft4rec
```

--------------------------------

### Load and Query Multi-Document Data

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/multi-document-agentic-rag/main.ipynb

This example demonstrates loading multiple documents and performing a RAG query. It assumes a setup where documents are processed and indexed into LanceDB.

```python
import os
import lance
import numpy as np
from lancedb.embeddings import EmbeddingFunction
from lancedb.pydantic import pydantic_to_schema
from pydantic import BaseModel

# Assume EmbeddingFunction and BaseModel are defined elsewhere
# For demonstration, we'll use a placeholder
class MyEmbeddingFunction(EmbeddingFunction):
    def __init__(self, model_name="placeholder-model"):
        self.model_name = model_name
        self.vector_size = 128 # Example size

    def __call__(self, texts):
        # In a real scenario, this would call an embedding model
        return np.random.rand(len(texts), self.vector_size).astype("float32")

class Document(BaseModel):
    text: str
    id: int

def main():
    # Initialize LanceDB and embedding function
    db = lance.LanceDB("~/lancedb", mode="w+")
    embedder = MyEmbeddingFunction()

    # Create table with schema derived from Pydantic model
    schema = pydantic_to_schema(Document, embedder)
    table = db.create_table("multi_doc_table", schema=schema, mode="w")

    # Sample documents
    docs = [
        Document(text="This is the first document.", id=1),
        Document(text="This document is about RAG.", id=2),
        Document(text="LanceDB is a vector database.", id=3),
    ]

    # Process and embed documents
    texts = [doc.text for doc in docs]
    ids = [doc.id for doc in docs]
    vectors = embedder(texts)

    # Insert into table
    table.add(vectors=vectors, text=texts, id=ids)

    # Query
    query_text = "What is LanceDB?"
    query_vector = embedder([query_text])[0]
    results = table.lance.vector_search(query_vector, k=1)

    print(f"Query: {query_text}")
    print(f"Results: {results}")

if __name__ == "__main__":
    main()
```

--------------------------------

### Configure Environment Variables

Source: https://github.com/lancedb/vectordb-recipes/blob/main/applications/node/mutimodal_meme_finder/README.md

Set up the .env file with necessary API keys and database configurations.

```bash
OPENAI_API_KEY = ''
LANCEDB_URI = 'database'
LANCEDB_TABLE_NAME = 'table'
LANCEDB_TABLE_NAME_TEXT = 'table_text'
RF_API_KEY = ''
```

--------------------------------

### Setup and Imports for CrewAI RAG

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/AI-Trends-with-CrewAI/CrewAI_AI_Trends.ipynb

Initializes the environment by importing necessary libraries and setting up the API key for OpenAI. Ensure you have the 'crewai' and 'openai' libraries installed.

```python
import os
import openai

from crewai
from crewai.tools import tool

from dotenv import load_dotenv

load_dotenv()

os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
openai.api_key = os.getenv("OPENAI_API_KEY")
```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/search-within-images-with-sam-and-clip/main.ipynb

Install the required libraries for the project, including LanceDB, CLIP, and SAM.

```bash
pip install "lancedb[all]" transformers torch torchvision torchaudio Pillow segmentation-models-pytorch albumentations --upgrade
```

--------------------------------

### RAG Context Enrichment Example

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Context_Enrichment_Window/Advanced_RAG_Context_Enrichment_Window.ipynb

This snippet demonstrates a RAG pipeline with context enrichment. It requires the 'lancedb' library and assumes necessary setup for vector database interaction.

```python
import lancedb
from lancedb.embeddings import EmbeddingFunction
from lancedb.pydantic import pydantic_to_schema
from lancedb.vectorstore import VectorStore
from pydantic import BaseModel

class MySchema(BaseModel):
    """My schema for the LanceDB table."""
    vector: list[float]
    text: str
    id: int


def main():
    """Main function to demonstrate RAG context enrichment."""
    db = lancedb.connect("~/lancedb")
    schema = pydantic_to_schema(MySchema)
    table = db.create_table("my_table", schema=schema)

    # Add data to the table
    table.add([
        {"vector": [1.0, 2.0], "text": "This is the first document.", "id": 1},
        {"vector": [3.0, 4.0], "text": "This is the second document.", "id": 2},
        {"vector": [5.0, 6.0], "text": "This is the third document.", "id": 3},
    ])

    # Perform a search
    results = table.search(query=[1.5, 2.5]).limit(1).to_pydantic(MySchema)
    print(results)

    # Example of context enrichment (conceptual)
    # In a real scenario, this would involve more complex logic
    # to retrieve and augment context based on the query.
    enriched_context = ""
    for item in results:
        enriched_context += item.text + " "

    print(f"Enriched Context: {enriched_context}")

if __name__ == "__main__":
    main()

```

--------------------------------

### Set up Training Arguments

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Defines the training arguments for the QLoRA fine-tuning process, including output directory and learning rate.

```python
from transformers import TrainingArguments

output_dir = "./results"
training_arguments = TrainingArguments(
    output_dir=output_dir,
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    optim="paged_adamw_32bit",
    learning_rate=0.0002,
    logging_steps=100,
    num_train_epochs=1,
    report_to="tensorboard",
    save_strategy="steps",
    save_steps=500,
    fp16=True,
    warmup_ratio=0.05,
    lr_scheduler_type="cosine",
)

```

--------------------------------

### Install Dependencies and Run Training Script

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/llm-pretraining-dataloading/README.md

Install required packages and then execute the training script. This script performs the GPT model pre-training.

```shell
pip install -r requirements.txt

python train.py
```

--------------------------------

### Example Usage of Chunking Function

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Chunking_Analysis/Chunking_Analysis_Latest.ipynb

Demonstrates how to use the initialized model and sentence splitting function with sample Spanish text. This serves as a starting point for applying the chunking logic.

```python
# Assuming initialize_model_and_nltk and split_into_sentences are defined elsewhere

model = initialize_model_and_nltk()  # you can chose to remove progress bar
text = spanish_text  # change your text here

sentences = split_into_sentences(text)
```

--------------------------------

### Implementing Chunked Pooling with LanceDB

Source: https://github.com/lancedb/vectordb-recipes/blob/main/examples/Advanced_RAG_Late_Chunking/Late_Chunking_(Chunked_Pooling).ipynb

This example demonstrates how to integrate the ChunkedPooling class with LanceDB for creating and querying a vector index. It shows the setup for embedding generation and data insertion.

```python
import lance
import pyarrow as pa
from lance.embeddings import Pooling
from lance.embeddings import EmbeddingFunction

class ChunkedPooling(EmbeddingFunction):
    def __init__(self, embedding_function: EmbeddingFunction, pool_size: int = 10):
        self.embedding_function = embedding_function
        self.pool_size = pool_size

    def __call__(self, texts: list[str]) -> list[list[float]]:
        pooled_embeddings = []
        for i in range(0, len(texts), self.pool_size):
            chunk_batch = texts[i : i + self.pool_size]
            embeddings = self.embedding_function(chunk_batch)
            pooled_embedding = Pooling.mean(embeddings)
            pooled_embeddings.append(pooled_embedding)
        return pooled_embeddings

async def main():
    # Dummy embedding function for demonstration
    class DummyEmbeddingFunction(EmbeddingFunction):
        def __call__(self, texts: list[str]) -> list[list[float]]:
            return [[float(ord(c)) for c in text[:5]] for text in texts]

    embedding_fn = DummyEmbeddingFunction()
    chunked_pooling_fn = ChunkedPooling(embedding_fn, pool_size=2)

    # Sample data
    data = {
        "text": [
            "This is the first sentence.",
            "This is the second sentence.",
            "This is the third sentence.",
            "This is the fourth sentence.",
            "This is the fifth sentence.",
        ],
        "id": [1, 2, 3, 4, 5]
    }
    table = pa.Table.from_pydict(data)

    # Create LanceDB table with chunked pooling embeddings
    uri = "my_chunked_pooling_table"
    await lance.write_table(table, uri, embedding_functions=chunked_pooling_fn)

    # Load the table and perform a similarity search
    db = await lance.connect(uri)
    tbl = await db.open_table("my_chunked_pooling_table")

    query_text = "Search for something related to the first two sentences."
    query_embedding = chunked_pooling_fn([query_text])[0]

    results = await tbl.lance_table.to_table( 
        nearest_to=query_embedding, 
        k=2, 
        # Use the same embedding function for search
        embedding_function=chunked_pooling_fn
    )
    print(results)

if __name__ == "__main__":
    import asyncio
    asyncio.run(main())
```

--------------------------------

### Initialize LlamaIndex and Load Data

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/RAG-with_MatryoshkaEmbed-Llamaindex/RAG_with_MatryoshkaEmbedding_and_Llamaindex.ipynb

This snippet shows the basic setup for LlamaIndex, including loading data from a specified source. Ensure LlamaIndex is installed and the data path is correct.

```python
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader

# Load documents from a directory
documents = SimpleDirectoryReader("data").load_data()

# Build the index
index = VectorStoreIndex.from_documents(documents)
```

--------------------------------

### Install necessary libraries

Source: https://github.com/lancedb/vectordb-recipes/blob/main/tutorials/fine-tuning_LLM_with_PEFT_QLoRA/main.ipynb

Installs the required libraries for fine-tuning LLMs with PEFT and QLoRA. Ensure you have a compatible environment.

```bash
pip install -q bitsandbytes datasets accelerate peft transformers trl
```