### Query Log Examples (INFO)

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/logging.md

Provides example INFO level log messages for query operations, indicating the start of a query and its successful completion with relevant metrics.

```log
2025-04-03 12:40:12 - dsrag.query - INFO - Starting query - {"kb_id": "kb123", "query_id": "q789", "num_search_queries": 3}
2025-04-03 12:40:15 - dsrag.query - INFO - Query successful - {"kb_id": "kb123", "query_id": "q789", "total_duration_s": 3.2109, "num_final_segments": 4}
```

--------------------------------

### Setup and Imports for dsRAG

Source: https://github.com/d-star-ai/dsrag/blob/main/examples/dsRAG_motivation.ipynb

Imports necessary libraries and sets up API keys for OpenAI and Cohere. Ensure you have the dsrag library installed and the required API keys configured.

```python
import sys
sys.path.append("../")
from dsrag.knowledge_base import KnowledgeBase
from dsrag.dsparse.file_parsing.non_vlm_file_parsing import extract_text_from_pdf
from dsrag.rse import get_best_segments
import cohere
import os
from scipy.stats import beta
import numpy as np
import matplotlib.pyplot as plt

# set API keys
import os
os.environ["OPENAI_API_KEY"] = ""
os.environ["CO_API_KEY"] = ""
```

--------------------------------

### Document Ingestion Log Examples (INFO)

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/logging.md

Shows example INFO level log messages for document ingestion, including start and success events with associated metadata.

```log
2025-04-03 12:34:56 - dsrag.ingestion - INFO - Starting document ingestion - {"kb_id": "kb123", "doc_id": "doc456", "file_path": "example.pdf"}
2025-04-03 12:35:01 - dsrag.ingestion - INFO - Document ingestion successful - {"kb_id": "kb123", "doc_id": "doc456", "total_duration_s": 5.1234}
```

--------------------------------

### Configure Basic Logging

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/logging.md

Sets up basic logging to display INFO level messages and initializes a KnowledgeBase instance. Ensure the `dsrag` library is installed.

```python
import logging

# Configure basic logging to see INFO level messages
logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

# Use dsRAG normally
from dsrag.knowledge_base import KnowledgeBase
kb = KnowledgeBase("example_kb")
```

--------------------------------

### Install Development Dependencies

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/community/contributing.md

Install the project's development dependencies, including the package itself in editable mode.

```bash
pip install -e ".[dev]"
```

--------------------------------

### Example Unit Test Structure

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/community/contributing.md

A basic structure for writing unit tests using Python's `unittest` module. Includes setup, test execution, and teardown.

```python
import unittest
from dsrag import YourModule

class TestYourFeature(unittest.TestCase):
    def setUp(self):
        # Set up any test fixtures
        pass

    def test_your_feature(self):
        # Test your new functionality
        result = YourModule.your_function()
        self.assertEqual(result, expected_value)

    def tearDown(self):
        # Clean up after tests
        pass
```

--------------------------------

### Install dsRAG with Qdrant Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Qdrant vector database support. This command includes Qdrant as a dependency.

```bash
pip install dsrag[qdrant]
```

--------------------------------

### Install dsRAG with All Vector Database Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with support for all available vector databases. This is a comprehensive installation for maximum compatibility.

```bash
pip install dsrag[all-vector-dbs]
```

--------------------------------

### Example Chat Thread Parameters

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/api/chat.md

Illustrates how to construct the `ChatThreadParams` dictionary for creating a new chat thread. This includes specifying knowledge bases, LLM model, and other conversational settings.

```python
# Create chat thread parameters
params: ChatThreadParams = {
    "kb_ids": ["kb1"],
    "model": "gpt-4o-mini",
    "temperature": 0.2,
    "system_message": "You are a helpful assistant",
    "auto_query_model": "gpt-4o-mini",
    "auto_query_guidance": "",
    "target_output_length": "medium",
    "max_chat_history_tokens": 8000,
    "supp_id": ""
}
```

--------------------------------

### Install dsRAG and Optional Dependencies

Source: https://context7.com/d-star-ai/dsrag/llms.txt

Install the base dsRAG package or with specific vector database support. VLM file parsing requires poppler, and API keys for services like OpenAI, Cohere, and Gemini must be set as environment variables.

```bash
pip install dsrag

```

```bash
pip install dsrag[faiss]

```

```bash
pip install dsrag[chroma]

```

```bash
pip install dsrag[weaviate]

```

```bash
pip install dsrag[qdrant]

```

```bash
pip install dsrag[milvus]

```

```bash
pip install dsrag[pinecone]

```

```bash
pip install dsrag[all]

```

```bash
brew install poppler

```

```bash
export OPENAI_API_KEY="sk-..."

```

```bash
export CO_API_KEY="..."

```

```bash
export GEMINI_API_KEY="..."

```

--------------------------------

### Install dsRAG with All Optional Dependencies

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with all optional dependencies, including all vector database support and any other extras. This provides the full dsRAG feature set.

```bash
pip install dsrag[all]
```

--------------------------------

### Install dsRAG Package

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/getting-started/installation.md

Use pip to install the dsRAG package. This is the primary method for getting the library into your Python environment.

```bash
pip install dsrag
```

--------------------------------

### Serve Documentation Locally

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/community/contributing.md

Build and serve the project's documentation locally to preview changes. Requires MkDocs.

```bash
mkdocs serve
```

--------------------------------

### Set Up Virtual Environment

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/community/contributing.md

Create and activate a Python virtual environment for development. Use `venv\Scripts\activate` on Windows.

```bash
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate
```

--------------------------------

### Create a Basic Knowledge Base

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/knowledge-bases.md

Initialize a knowledge base with a unique ID, title, and description. This is the simplest way to set up a new knowledge base.

```python
from dsrag.knowledge_base import KnowledgeBase

# Create a basic knowledge base
kb = KnowledgeBase(
    kb_id="my_kb",
    title="Product Documentation",
    description="Technical documentation for XYZ product"
)
```

--------------------------------

### Initialize KnowledgeBase with Default VLM Client

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Instantiate a KnowledgeBase with a default VLM client. This client will be used for VLM parsing unless overridden.

```python
from dsrag.knowledge_base import KnowledgeBase
from dsrag.dsparse.file_parsing.vlm_clients import GeminiVLM

kb = KnowledgeBase(
    kb_id="my_kb",
    vlm_client=GeminiVLM(model="gemini-2.0-flash"),  # used by default for VLM parsing
)
```

--------------------------------

### Initialize Custom KnowledgeBase

Source: https://context7.com/d-star-ai/dsrag/llms.txt

Create a KnowledgeBase with fully customized components for LLM, reranker, embedding model, vector database, and chunk database. All component choices are saved to JSON and reloaded automatically.

```python
from dsrag.knowledge_base import KnowledgeBase
from dsrag.llm import OpenAIChatAPI, AnthropicChatAPI
from dsrag.reranker import CohereReranker, NoReranker
from dsrag.embedding import OpenAIEmbedding, CohereEmbedding
from dsrag.database.vector.chroma_db import ChromaDB
from dsrag.database.chunk.sqlite_db import SQLiteDB

# Fully customised
kb = KnowledgeBase(
    kb_id="finance_docs_custom",
    title="Financial Reports KB",
    description="10-Ks and 10-Qs for S&P 500 companies",
    storage_directory="./kb_storage",          # where JSON config + local data are saved
    embedding_model=OpenAIEmbedding(model="text-embedding-3-large"),
    reranker=CohereReranker(model="rerank-english-v3.0"),
    auto_context_model=AnthropicChatAPI(model="claude-3-5-haiku-latest"),
    vector_db=ChromaDB(kb_id="finance_docs_custom", storage_directory="./kb_storage"),
    chunk_db=SQLiteDB(kb_id="finance_docs_custom", storage_directory="./kb_storage"),
)

```

--------------------------------

### Create Knowledge Base and Query

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/index.md

Use this snippet to create a knowledge base from a file and then query it. Ensure the file path is correct and a kb_id is provided.

```python
from dsrag.create_kb import create_kb_from_file

# Create a knowledge base from a file
file_path = "path/to/your/document.pdf"
kb_id = "my_knowledge_base"
kb = create_kb_from_file(kb_id, file_path)

# Query the knowledge base
search_queries = ["What are the main topics covered?"]
results = kb.query(search_queries)
for segment in results:
    print(segment)
```

--------------------------------

### Initialize KnowledgeBase with Defaults

Source: https://context7.com/d-star-ai/dsrag/llms.txt

Create a KnowledgeBase instance using default components for embeddings, reranker, and vector database. The kb_id is used to load or create a persistent knowledge base.

```python
from dsrag.knowledge_base import KnowledgeBase

# Minimal — all defaults (OpenAI embeddings, Cohere reranker, BasicVectorDB)
kb = KnowledgeBase(kb_id="finance_docs")

```

--------------------------------

### Install dsRAG with Pinecone Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Pinecone vector database support. This command includes Pinecone as a dependency.

```bash
pip install dsrag[pinecone]
```

--------------------------------

### Install dsRAG with Milvus Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Milvus vector database support. This command includes Milvus as a dependency.

```bash
pip install dsrag[milvus]
```

--------------------------------

### Initialize KnowledgeBase with Default VLM Client

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Instantiate a KnowledgeBase with a default VLM client. This client will be used for VLM parsing unless overridden per document. Ensure the VLM client is correctly configured.

```python
from dsrag.knowledge_base import KnowledgeBase
from dsrag.dsparse.file_parsing.vlm_clients import GeminiVLM

kb = KnowledgeBase(
    kb_id="my_kb",
    vlm_client=GeminiVLM(model="gemini-2.0-flash"),  # default VLM used for VLM parsing
)
```

--------------------------------

### Create Knowledge Base and Add Document from File

Source: https://context7.com/d-star-ai/dsrag/llms.txt

A convenience function to initialize a KnowledgeBase and ingest a single document from a specified file path in one step. Uses default component settings.

```python
from dsrag.create_kb import create_kb_from_file

kb = create_kb_from_file(
    kb_id="levels_of_agi",
    file_path="papers/levels_of_agi.pdf",
)

# Query immediately
results = kb.query(["What are the levels of AGI?", "What is the highest level of AGI?"])
for segment in results:
    print(segment["content"])
```

--------------------------------

### Create a Knowledge Base with Custom Configuration

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/knowledge-bases.md

Initialize a knowledge base with custom configurations for storage, embedding models, rerankers, and databases. This allows for fine-grained control over the knowledge base's backend.

```python
# Or with custom configuration
kb = KnowledgeBase(
    kb_id="my_kb",
    storage_directory="path/to/storage",  # Where to store KB data
    embedding_model=custom_embedder,      # Custom embedding model
    reranker=custom_reranker,            # Custom reranking model
    vector_db=custom_vector_db,          # Custom vector database
    chunk_db=custom_chunk_db            # Custom chunk database
)
```

--------------------------------

### Install dsRAG with Weaviate Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Weaviate vector database support. This command includes Weaviate as a dependency.

```bash
pip install dsrag[weaviate]
```

--------------------------------

### KnowledgeBase Initialization and Document Addition (Local Ollama)

Source: https://context7.com/d-star-ai/dsrag/llms.txt

Demonstrates how to initialize a KnowledgeBase using Ollama for embeddings and LLM, and add a document for local processing.

```APIDOC
## Fully Local Configuration (Ollama)

Run dsRAG with no external API calls using Ollama for embeddings and LLM.

```python
from dsrag.knowledge_base import KnowledgeBase
from dsrag.llm import OllamaChatAPI
from dsrag.reranker import NoReranker
from dsrag.embedding import OllamaEmbedding

kb = KnowledgeBase(
    kb_id="local_kb",
    embedding_model=OllamaEmbedding(model="nomic-embed-text"),
    reranker=NoReranker(),
    auto_context_model=OllamaChatAPI(model="llama3.1:8b"),
)

kb.add_document(
    doc_id="local_doc",
    file_path="docs/handbook.pdf",
    semantic_sectioning_config={"use_semantic_sectioning": False},  # requires cloud LLM
)

results = kb.query(["What is the onboarding process?"])
```
```

--------------------------------

### Install dsRAG with Chroma Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Chroma vector database support. This command includes Chroma as a dependency.

```bash
pip install dsrag[chroma]
```

--------------------------------

### Configure Knowledge Base with OpenAI-Only Services

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/getting-started/quickstart.md

Set up a KnowledgeBase using only OpenAI services by specifying OpenAIChatAPI for context and NoReranker. This is useful if Cohere is not available.

```python
from dsrag.llm import OpenAIChatAPI
from dsrag.reranker import NoReranker

# Configure components
llm = OpenAIChatAPI(model='gpt-4o-mini')
reranker = NoReranker()

# Create knowledge base with custom configuration
kb = KnowledgeBase(
    kb_id="my_knowledge_base",
    reranker=reranker,
    auto_context_model=llm
)

# Add documents
kb.add_document(
    doc_id="user_manual",  # Use a meaningful ID
    file_path="path/to/your/document.pdf",
    document_title="User Manual",  # Optional but recommended
    metadata={"type": "manual"}    # Optional metadata
)
```

--------------------------------

### Install dsRAG with Faiss Support

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Install dsRAG with optional Faiss vector database support. This command includes Faiss as a dependency.

```bash
pip install dsrag[faiss]
```

--------------------------------

### Create and Add Documents to a Knowledge Base

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/getting-started/quickstart.md

Initialize a KnowledgeBase and add documents using their file paths. The KnowledgeBase persists to disk automatically.

```python
from dsrag.knowledge_base import KnowledgeBase

# Create a knowledge base
kb = KnowledgeBase(kb_id="my_knowledge_base")

# Add documents
kb.add_document(
    doc_id="user_manual",  # Use a meaningful ID if possible
    file_path="path/to/your/document.pdf",
    document_title="User Manual",  # Optional but recommended
    metadata={"type": "manual"}    # Optional metadata
)
```

--------------------------------

### Install Poppler Dependency on MacOS

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/getting-started/installation.md

Install the poppler dependency using Homebrew on MacOS. This is required for VLM file parsing, which converts PDFs to images.

```bash
brew install poppler
```

--------------------------------

### Create KnowledgeBase from File

Source: https://github.com/d-star-ai/dsrag/blob/main/README.md

Create a new KnowledgeBase directly from a PDF file using the `create_kb_from_file` function. KnowledgeBase objects persist to disk automatically.

```python
from dsrag.create_kb import create_kb_from_file

file_path = "dsRAG/tests/data/levels_of_agi.pdf"
kb_id = "levels_of_agi"
kb = create_kb_from_file(kb_id, file_path)
```

--------------------------------

### Metadata Filter Example (In)

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/config.md

Example of a metadata filter using the 'in' operator to select documents where a field's value is present in a list. This allows for selecting multiple specific documents.

```python
metadata_filter = {
    "field": "doc_id",
    "operator": "in",
    "value": ["test_id_1", "test_id_2"]
}
```

--------------------------------

### Metadata Filter Example (Equals)

Source: https://github.com/d-star-ai/dsrag/blob/main/docs/concepts/config.md

Example of a metadata filter using the 'equals' operator to select documents where a specific field matches a given value. Useful for precise document retrieval.

```python
metadata_filter = {
    "field": "doc_id",
    "operator": "equals",
    "value": "test_id_1"
}
```