### Minimal Example

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Demonstrates the basic usage of the Storage API to create an embedded storage, store a document, and retrieve it.

```APIDOC
## Minimal Example

### Description
This example shows how to initialize an embedded storage, create a document with content, store it using an `AccessContext`, and then retrieve it back from the storage.

### Method
`Storage::new_embedded()`
`Storage::store_document()`
`Storage::get_document()`

### Parameters
#### `Storage::new_embedded()`
No parameters.

#### `Storage::store_document()`
- **document** (`&Document`): The document to store.
- **context** (`&AccessContext`): The access context for the operation.

#### `Storage::get_document()`
- **document_id** (`&Uuid`): The ID of the document to retrieve.
- **context** (`&AccessContext`): The access context for the operation.

### Request Example
```rust
use reasonkit_mem::storage::{Storage, AccessContext, AccessLevel};
use reasonkit_mem::{Document, DocumentType, Source, SourceType};
use chrono::Utc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let storage = Storage::new_embedded().await?;
    let context = AccessContext::new(
        "user_123".to_string(),
        AccessLevel::ReadWrite,
        "store_document".to_string(),
    );
    let source = Source {
        source_type: SourceType::Local,
        url: None,
        path: Some("notes.md".to_string()),
        arxiv_id: None,
        github_repo: None,
        retrieved_at: Utc::now(),
        version: None,
    };
    let doc = Document::new(DocumentType::Note, source)
        .with_content("Hello world - this is my first document.".to_string());
    storage.store_document(&doc, &context).await?;
    let retrieved = storage.get_document(&doc.id, &context).await?;
    println!("Retrieved: {:?}", retrieved);
    Ok(())
}
```

### Response
#### Success Response
- **retrieved** (`Document`): The retrieved document object.
```

--------------------------------

### Minimal Storage Example

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Demonstrates creating embedded storage, storing a document, and retrieving it. Requires `tokio` for async operations.

```rust
use reasonkit_mem::storage::{Storage, AccessContext, AccessLevel};
use reasonkit_mem::{Document, DocumentType, Source, SourceType};
use chrono::Utc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create embedded storage (file-based, no external dependencies)
    let storage = Storage::new_embedded().await?;

    // Create access context for operations
    let context = AccessContext::new(
        "user_123".to_string(),
        AccessLevel::ReadWrite,
        "store_document".to_string(),
    );

    // Create a document
    let source = Source {
        source_type: SourceType::Local,
        url: None,
        path: Some("notes.md".to_string()),
        arxiv_id: None,
        github_repo: None,
        retrieved_at: Utc::now(),
        version: None,
    };

    let doc = Document::new(DocumentType::Note, source)
        .with_content("Hello world - this is my first document.".to_string());

    // Store the document
    storage.store_document(&doc, &context).await?;

    // Retrieve it back
    let retrieved = storage.get_document(&doc.id, &context).await?;
    println!("Retrieved: {:?}", retrieved);

    Ok(())
}
```

--------------------------------

### Install ReasonKit Mem with Universal Installer

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/README.md

Installs all 4 ReasonKit projects, including memory, with support for all platforms and shells. Auto-detects shell and configures PATH.

```bash
curl -fsSL https://get.reasonkit.sh | bash -s -- --with-memory
```

--------------------------------

### Manual Compaction Example

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/COLD_MEMORY_DESIGN.md

Example of manually triggering a compaction operation on cold memory. Recommended during maintenance or after large data deletions.

```rust
// Force flush and update stats
cold.compact().await?;
```

--------------------------------

### Initialize DualLayerStorage with Default Configuration

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Creates a new DualLayerStorage instance using default settings. This is the simplest way to get started.

```rust
pub async fn new() -> Result<Self> {
        Self::with_config(DualLayerConfig::default()).await
    }
```

--------------------------------

### Verify ReasonKit Mem Installation

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/README.md

Quickly verify your ReasonKit Mem installation by creating in-memory storage. This snippet requires the `tokio` runtime.

```rust
use reasonkit_mem::storage::Storage;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Quick verification - creates in-memory storage
    let storage = Storage::new_embedded().await?;
    println!("ReasonKit Mem initialized successfully!");
    Ok(())
}
```

--------------------------------

### Storage with Vector Search Example

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Shows how to configure storage with a custom path, store embeddings, and perform vector similarity searches. Requires `tokio` and `uuid`.

```rust
use reasonkit_mem::storage::{Storage, EmbeddedStorageConfig, AccessContext, AccessLevel};
use uuid::Uuid;
use std::path::PathBuf;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Configure storage with custom path
    let config = EmbeddedStorageConfig::file_only(PathBuf::from("./data"));
    let storage = Storage::new_embedded_with_config(config).await?;

    let context = AccessContext::new(
        "system".to_string(),
        AccessLevel::Admin,
        "vector_ops".to_string(),
    );

    // Store an embedding (768-dimensional vector)
    let chunk_id = Uuid::new_v4();
    let embedding: Vec<f32> = (0..768).map(|i| (i as f32 * 0.01).sin()).collect();
    storage.store_embeddings(&chunk_id, &embedding, &context).await?;

    // Search by vector similarity
    let query_vector: Vec<f32> = (0..768).map(|i| (i as f32 * 0.01).cos()).collect();
    let results = storage.search_by_vector(&query_vector, 5, &context).await?;

    for (id, score) in results {
        println!("Match: {} (score: {:.4})", id, score);
    }

    Ok(())
}
```

--------------------------------

### Run Qdrant with Docker

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

Starts a Qdrant instance using Docker, exposing the necessary ports for communication. This is the recommended method for setting up Qdrant for embedded mode.

```bash
# Run Qdrant in Docker
docker run -p 6333:6333 -p 6334:6334 qdrant/qdrant

# Verify it's running
curl http://localhost:6333/readyz
```

--------------------------------

### Start Background Synchronization Task

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Initiates a background task for synchronization operations, such as syncing the WAL to cold storage and performing tiering. This is started if background sync is enabled in the configuration.

```rust
fn start_background_sync(&mut self) {
        let hot = self.hot.clone();
        let cold = self.cold.clone();
        let wal = self.wal.clone();
        let tier_index = self.tier_index.clone();
        let access_tracker = self.access_tracker.clone();
        let config = self.config.clone();

        let handle = tokio::spawn(async move {
            let interval = Duration::from_secs(config.sync.background_sync_interval_secs);
            let mut ticker = tokio::time::interval(interval);

            loop {
                ticker.tick().await;

                // Sync WAL to cold storage
                if let Some(ref wal) = *wal.read().await {
                    let _ = wal.checkpoint().await;
                }

                // Run tiering
                let _ = Self::run_tiering_internal(
                    &hot,

```

--------------------------------

### Basic API Embedding with OpenAI

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDING_PIPELINE_GUIDE.md

Example of creating an OpenAI embedding provider and embedding a single piece of text asynchronously. Requires `tokio` runtime.

```rust
use reasonkit_mem::embedding::{OpenAIEmbedding, EmbeddingProvider};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create provider
    let provider = OpenAIEmbedding::openai()?;

    // Embed single text
    let result = provider.embed("What is machine learning?").await?;
    if let Some(embedding) = result.dense {
        println!("Embedding dimension: {}", embedding.len());
    }

    Ok(())
}
```

--------------------------------

### Cold Memory Configuration using Builder Pattern

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/COLD_MEMORY_DESIGN.md

Example of configuring cold memory using a builder pattern, allowing for fluent and chained setting of various configuration options.

```rust
let cold = ColdMemoryBuilder::new()
    .path(PathBuf::from("./data/cold"))
    .cache_size_mb(256)
    .flush_interval_secs(30)
    .compression(true)
    .parallel_threshold(1000)
    .build()
    .await?;
```

--------------------------------

### Basic Dual Layer Storage Operations in Rust

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Demonstrates creating, storing, retrieving, and managing documents within the dual-layer storage system. Includes examples of moving documents between storage tiers and checking statistics.

```rust
use reasonkit_mem::storage::{
    DualLayerStorage, DualLayerConfig, StorageTier,
    AccessContext, AccessLevel,
};
use reasonkit_mem::{Document, DocumentType, Source, SourceType};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create storage with default configuration
    let storage = DualLayerStorage::new().await?;

    // Or create embedded storage (file-based, no external dependencies)
    let storage = DualLayerStorage::embedded("./data".into()).await?;

    // Create access context
    let context = AccessContext::new(
        "user123".to_string(),
        AccessLevel::ReadWrite,
        "store_document".to_string(),
    );

    // Create a document
    let source = Source {
        source_type: SourceType::Local,
        url: None,
        path: Some("example.md".to_string()),
        arxiv_id: None,
        github_repo: None,
        retrieved_at: chrono::Utc::now(),
        version: None,
    };

    let doc = Document::new(DocumentType::Note, source)
        .with_content("This is an example document.".to_string());

    // Store document (auto-routes to hot tier for small documents)
    storage.store_document(&doc, &context).await?;

    // Retrieve document
    let retrieved = storage.get_document(&doc.id, &context).await?;
    assert!(retrieved.is_some());

    // Check which tier the document is in
    let tier = storage.get_tier(&doc.id).await?;
    println!("Document is in {:?} tier", tier);

    // Force move to cold tier
    storage.move_to_tier(&doc.id, StorageTier::Cold, &context).await?;

    // Get statistics
    let stats = storage.detailed_stats(&context).await?;
    println!("Hot tier: {} docs, Cold tier: {} docs",
        stats.hot.document_count,
        stats.cold.document_count);

    Ok(())
}
```

--------------------------------

### Migrate from FileStorage to Qdrant

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Guides the migration of data from a file-based storage to Qdrant for production environments. This involves opening the file storage, creating Qdrant storage, and then iterating through all documents and their embeddings to store them in Qdrant.

```rust
use reasonkit_mem::storage::{
    Storage,
    FileStorage,
    EmbeddedStorageConfig,
    AccessContext,
    AccessLevel,
};
use std::path::PathBuf;

async fn migrate_to_qdrant(file_path: PathBuf) -> anyhow::Result<Storage> {
    // Open file storage
    let file_storage = Storage::file(file_path.clone()).await?;

    // Create Qdrant storage
    let qdrant_config = EmbeddedStorageConfig::with_qdrant(
        "http://localhost:6333",
        "migrated_collection",
        1536,
    );
    let qdrant_storage = Storage::new_embedded_with_config(qdrant_config).await?;

    let read_ctx = AccessContext::new(
        "migration".to_string(),
        AccessLevel::Admin,
        "read".to_string(),
    );
    let write_ctx = AccessContext::new(
        "migration".to_string(),
        AccessLevel::Admin,
        "write".to_string(),
    );

    // List and migrate all documents
    let doc_ids = file_storage.list_documents(&read_ctx).await?;
    let total = doc_ids.len();

    for (idx, doc_id) in doc_ids.into_iter().enumerate() {
        if let Some(doc) = file_storage.get_document(&doc_id, &read_ctx).await? {
            qdrant_storage.store_document(&doc, &write_ctx).await?;

            // Migrate embeddings for each chunk
            for chunk in &doc.chunks {
                if let Some(embedding) = file_storage
                    .get_embeddings(&chunk.id, &read_ctx)
                    .await?
                {
                    qdrant_storage
                        .store_embeddings(&chunk.id, &embedding, &write_ctx)
                        .await?;
                }
            }
        }

        if (idx + 1) % 100 == 0 {
            tracing::info!("Migrated {}/{} documents", idx + 1, total);
        }
    }

    tracing::info!("Migration complete: {} documents", total);
    Ok(qdrant_storage)
}
```

--------------------------------

### Development Environment Storage Configuration

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/ARCHITECTURE.md

Example configuration for a development environment using file storage and a smaller embedding model. Uses default settings for unspecified fields.

```rust
let config = EmbeddedStorageConfig {
    data_path: PathBuf::from("./data"),
    require_qdrant: false,  // Use file storage
    vector_size: 384,       // Smaller model (E5-small)
    ..Default::default()
};

```

--------------------------------

### Basic Hybrid Search Example

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/HYBRID_SEARCH_GUIDE.md

Perform a hybrid search query, automatically generating embeddings if needed. Allows customization of top_k, alpha (dense/sparse balance), and other retrieval parameters.

```rust
use reasonkit_mem::{
    storage::Storage,
    retrieval::{HybridRetriever, RetrievalConfig},
    indexing::IndexManager,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Initialize storage and index
    let storage = Storage::new_embedded().await?;
    let index = IndexManager::new("./index").await?;

    // Create retriever
    let retriever = HybridRetriever::new(
        storage,
        index,
        RetrievalConfig::default(),
    );

    // Perform hybrid search
    let results = retriever.search_hybrid(
        "What is machine learning?",
        None, // Will generate embedding automatically
        &RetrievalConfig {
            top_k: 10,
            alpha: 0.7, // 70% dense, 30% sparse
            ..Default::default()
        },
    ).await?;

    for result in results {
        println!("Score: {:.3}, Text: {}", result.score, result.text);
    }

    Ok(())
}
```

--------------------------------

### Initialize DualLayerStorage with Qdrant Backend

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Creates a DualLayerStorage instance configured to use Qdrant as the cold storage backend. Requires Qdrant URL and collection name.

```rust
pub async fn with_qdrant(qdrant_url: &str, collection_name: &str) -> Result<Self> {
        let config = DualLayerConfig {
            cold: ColdMemoryConfig {
                backend: ColdBackendType::QdrantLocal {
                    url: qdrant_url.to_string(),
                },
                collection_name: collection_name.to_string(),
                ..Default::default()
            },
            ..Default::default()
        };

        Self::with_config(config).await
    }
```

--------------------------------

### Error Handling: Default Configuration (File Storage)

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

Shows successful storage initialization using the default configuration, which falls back to file storage if Qdrant is not available. This always succeeds.

```rust
let config = EmbeddedStorageConfig::default();
let storage = Storage::new_embedded(config).await?;
// Always succeeds, uses file storage
```

--------------------------------

### Run Qdrant Binary

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

Downloads and runs the Qdrant binary directly on the system. This is an alternative to Docker for setting up a local Qdrant instance.

```bash
# Download Qdrant binary
wget https://github.com/qdrant/qdrant/releases/download/v1.10.0/qdrant-x86_64-unknown-linux-gnu.tar.gz
tar -xzf qdrant-x86_64-unknown-linux-gnu.tar.gz

# Run Qdrant
./qdrant
```

--------------------------------

### Initialize OpenAI Embeddings and Knowledge Base

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/README.md

Demonstrates how to set up an OpenAI embedding provider and a knowledge base for hybrid search. Requires the OPENAI_API_KEY environment variable.

```rust
use reasonkit_mem::embedding::{EmbeddingConfig, EmbeddingPipeline, OpenAIEmbedding};
use reasonkit_mem::retrieval::KnowledgeBase;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create OpenAI embedding provider (requires OPENAI_API_KEY env var)
    let embedding_provider = OpenAIEmbedding::openai()?;
    let pipeline = Arc::new(EmbeddingPipeline::new(Arc::new(embedding_provider)));

    // Create knowledge base with embedding support
    let kb = KnowledgeBase::in_memory()? 
        .with_embedding_pipeline(pipeline);

    // Now hybrid search will use both dense (vector) and sparse (BM25)
    // let results = kb.query("semantic search query", 10).await?;

    Ok(())
}
```

--------------------------------

### Batch Store Embeddings

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Optimizes storage by batching operations. This example shows how to store embeddings in chunks of 100.

```rust
// 1. Use appropriate vector dimensions
//    - 1536 for OpenAI ada-002 (high quality)
//    - 768 for E5-base (balanced)
//    - 384 for E5-small (fast)

// 2. Enable scalar quantization (in Qdrant config)
//    Reduces memory by 4x with minimal accuracy loss

// 3. Tune HNSW parameters for your use case
//    - m: 16 (default) - higher = better recall, more memory
//    - ef_construct: 100 (default) - higher = better index quality
//    - ef: 128 - higher = better search quality, slower

// 4. Batch operations when possible
let embeddings: Vec<(Uuid, Vec<f32>)> = generate_embeddings();
for chunk in embeddings.chunks(100) {
    for (id, emb) in chunk {
        storage.store_embeddings(id, emb, &context).await?;
    }
}
```

--------------------------------

### Storage Backend Selection Logic

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/ARCHITECTURE.md

Illustrates the decision tree for selecting the storage backend based on Qdrant availability and health.

```mermaid
graph TD
                    +------------------+
                    | Storage::new()   |
                    +--------+---------+ 
                             |
               +-------------+-------------+
               |                           |
        +------v------+             +------v------+
        | require_    |             | require_    |
        | qdrant=true |             | qdrant=false|
        +------+------+             +------+------+
               |
        +------v------+             +------v------+
        | Check Qdrant|             | Use File    |
        | Health      |             | Storage     |
        +------+------+             +-------------+ 
               |
       +-------+-------+
       |               |
  +----v----+    +-----v-----+
  | Healthy |    | Unhealthy |
  +---------+    +-----------+
       |
  +----v----+    +-----v-----+
  | Qdrant  |    | Error:    |
  | Storage |    | Required  |
  +---------+    +-----------+

```

--------------------------------

### Get WAL Status

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves the current status of the Write-Ahead Log (WAL), providing insights into its state and readiness.

```APIDOC
## wal_status

### Description
Gets the current status of the Write-Ahead Log (WAL).

### Method
`async fn wal_status(&self) -> Result<WalStatus>`

### Parameters
None

### Response
#### Success Response (200)
- **WalStatus** - The current status of the WAL.
```

--------------------------------

### Clone Repository and Run Tests

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/CONTRIBUTING.md

Clone the repository, navigate into the directory, and run the project's tests and clippy linter.

```bash
git clone https://github.com/reasonkit/reasonkit-mem
cd reasonkit-mem

cargo test


```

```bash
cargo clippy -- -D warnings


```

```bash
cargo fmt
```

--------------------------------

### Get RAPTOR Tree Statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/RAPTOR_TREE_GUIDE.md

Retrieves and prints statistics about the RAPTOR tree, including node counts and depth.

```rust
let stats = tree.stats();
println!("Total nodes: {}", stats.total_nodes);
println!("Leaf nodes: {}", stats.leaf_nodes);
println!("Max depth: {}", stats.max_depth);
println!("Root count: {}", stats.root_count);
```

--------------------------------

### Get an embedding from the cache

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Retrieves an embedding from the cache using its chunk ID. Returns None if the embedding is not found or has expired.

```rust
pub fn get(&mut self, chunk_id: &Uuid) -> Option<Vec<f32>>;

```

--------------------------------

### Get Cold Tier Statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves specific statistics for the cold storage tier. Helps in understanding the usage and cost-effectiveness of archival storage.

```APIDOC
## cold_stats

### Description
Gets statistics specifically for the cold storage tier.

### Method
`async fn cold_stats(&self, context: &AccessContext) -> Result<TierStats>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **context** (*AccessContext*) - Access context for the operation.

### Response
#### Success Response (200)
- **TierStats** - Statistics for the cold tier.
```

--------------------------------

### Get Promotion Candidates

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves a list of document IDs that are candidates for promotion to the hot tier. This helps in managing data access frequency.

```APIDOC
## get_promotion_candidates

### Description
Gets a list of document IDs that are candidates for promotion to the hot tier.

### Method
`async fn get_promotion_candidates(&self, limit: usize) -> Result<Vec<Uuid>>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **limit** (*usize*) - The maximum number of candidates to retrieve.

### Response
#### Success Response (200)
- **Vec<Uuid>** - A vector of document IDs eligible for promotion.
```

--------------------------------

### Storage API: Qdrant With Config Constructor

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Creates Qdrant-backed storage with comprehensive configuration, including connection, cache, and access control settings.

```rust
/// Create Qdrant storage with full configuration
pub async fn qdrant_with_config(
    host: &str,
    port: u16,
    grpc_port: u16,
    collection_name: String,
    vector_size: usize,
    embedded: bool,
    conn_config: QdrantConnectionConfig,
    cache_config: EmbeddingCacheConfig,
    access_config: AccessControlConfig,
) -> Result<Self>;
```

--------------------------------

### Get Document with Tier Information

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves a document along with its current storage tier. Useful for understanding data placement and for subsequent operations.

```APIDOC
## get_document_with_tier

### Description
Retrieves a document and its associated storage tier.

### Method
`async fn get_document_with_tier(&self, id: &Uuid, context: &AccessContext) -> Result<Option<(Document, StorageTier)>>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **id** (*Uuid*) - The unique identifier of the document.
- **context** (*AccessContext*) - Access context for the operation.

### Response
#### Success Response (200)
- **(Document, StorageTier)** - A tuple containing the document and its storage tier, or None if the document is not found.
```

--------------------------------

### Hybrid Search with KnowledgeBase in Rust

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/README.md

Demonstrates creating an in-memory knowledge base, adding a document with content and source information, and performing a sparse search using BM25. Requires tokio runtime.

```rust
use reasonkit_mem::retrieval::KnowledgeBase;
use reasonkit_mem::{Document, DocumentType, Source, SourceType};
use chrono::Utc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create in-memory knowledge base
    let kb = KnowledgeBase::in_memory()?;

    // Create a document
    let source = Source {
        source_type: SourceType::Local,
        url: None,
        path: Some("notes.md".to_string()),
        arxiv_id: None,
        github_repo: None,
        retrieved_at: Utc::now(),
        version: None,
    };

    let doc = Document::new(DocumentType::Note, source)
        .with_content("Machine learning is a subset of artificial intelligence.".to_string());

    // Add document to knowledge base
    kb.add(&doc).await?;

    // Search using sparse retrieval (BM25)
    let results = kb.retriever().search_sparse("machine learning", 5).await?;

    for result in results {
        println!("Score: {:.3}, Text: {}", result.score, result.text);
    }

    Ok(())
}
```

--------------------------------

### Get Hot Tier Statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves specific statistics for the hot storage tier. Useful for monitoring the performance and usage of frequently accessed data.

```APIDOC
## hot_stats

### Description
Gets statistics specifically for the hot storage tier.

### Method
`async fn hot_stats(&self) -> Result<TierStats>`

### Parameters
None

### Response
#### Success Response (200)
- **TierStats** - Statistics for the hot tier.
```

--------------------------------

### Get Current Tier for a Document

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves the current storage tier of a document without fetching the document content itself. Efficient for checking data location.

```APIDOC
## get_tier

### Description
Gets the current storage tier of a document.

### Method
`async fn get_tier(&self, id: &Uuid) -> Result<Option<StorageTier>>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **id** (*Uuid*) - The unique identifier of the document.

### Response
#### Success Response (200)
- **Option<StorageTier>** - The storage tier of the document, or None if the document is not found.
```

--------------------------------

### Initialize DualLayerStorage with Custom Configuration

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Creates a new DualLayerStorage instance with a specific configuration. Allows customization of hot, cold, WAL, and sync settings.

```rust
pub async fn with_config(config: DualLayerConfig) -> Result<Self> {
        // Initialize hot layer
        let hot = Self::create_hot_layer(&config.hot).await?;

        // Initialize cold layer
        let cold = Self::create_cold_layer(&config.cold).await?;

        // Initialize WAL if enabled
        let wal = if config.wal.enabled {
            Some(WriteAheadLog::new(&config.wal).await?)
        } else {
            None
        };

        // Initialize tier index
        let tier_index = TierIndex::new();

        // Initialize access tracker
        let access_tracker = AccessTracker::new(&config.sync.tiering_policy);

        let storage = Self {
            config: config.clone(),
            hot: Arc::new(RwLock::new(hot)),
            cold: Arc::new(cold),
            wal: Arc::new(RwLock::new(wal)),
            tier_index: Arc::new(RwLock::new(tier_index)),
            access_tracker: Arc::new(RwLock::new(access_tracker)),
            sync_handle: None,
        };

        // Start background sync if configured
        if config.sync.background_sync_interval_secs > 0 {
            storage.start_background_sync();
        }

        Ok(storage)
    }
```

--------------------------------

### Get Cold Memory Statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/COLD_MEMORY_DESIGN.md

Retrieves the current statistics for the cold memory database, including entry counts, storage sizes, and performance metrics.

```rust
// Get statistics
async fn stats(&self) -> ColdMemoryStats;

```

--------------------------------

### Initialize Dual Layer Storage Sync

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Initializes the synchronization handle for the dual layer storage. This is typically called during the setup phase of the storage system.

```rust
self.sync_handle = Some(handle);
```

--------------------------------

### High-Performance Qdrant Storage Configuration

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Set up Qdrant storage for production with optimized connection pooling, embedding cache, and access control.

```rust
use reasonkit_mem::storage::{
    QdrantConnectionConfig,
    QdrantSecurityConfig,
    EmbeddingCacheConfig,
    AccessControlConfig,
    Storage,
};

// Connection pool configuration
let conn_config = QdrantConnectionConfig {
    max_connections: 20,              // Increase pool size
    connect_timeout_secs: 10,         // Faster timeout detection
    request_timeout_secs: 30,         // Shorter request timeout
    health_check_interval_secs: 60,   // More frequent health checks
    max_idle_secs: 300,               // 5 minute idle timeout
    security: QdrantSecurityConfig {
        api_key: Some("your-api-key".to_string()),
        tls_enabled: true,
        ..Default::default()
    },
};

// Larger embedding cache
let cache_config = EmbeddingCacheConfig {
    max_size: 50000,        // Cache up to 50k embeddings
    ttl_secs: 7200,         // 2 hour TTL
};

// Access control
let access_config = AccessControlConfig::default();

// Create high-performance storage
let storage = Storage::qdrant_with_config(
    "qdrant.example.com",
    6333,               // REST port
    6334,               // gRPC port
    "production_kb".to_string(),
    1536,
    false,              // Not embedded mode
    conn_config,
    cache_config,
    access_config,
).await?;
```

--------------------------------

### Basic RAPTOR Tree Usage

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/RAPTOR_TREE_GUIDE.md

Demonstrates creating, building, and searching a RAPTOR tree. Requires embedding and summarization functions.

```rust
use reasonkit_mem::{
    Chunk, Document, raptor::RaptorTree,
    storage::Storage,
};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create tree
    let mut tree = RaptorTree::new(3, 5);

    // Build from document chunks
    let chunks = vec![/* your chunks */];
    tree.build_from_chunks(
        &chunks,
        &embed_text,      // Embedding function
        &summarize_text,  // Summarization function
    ).await?;

    // Search the tree
    let query_embedding = embed_text("What is machine learning?")?;
    let results = tree.search(&query_embedding, 10)?;

    for (node_id, score) in results {
        if let Some(node) = tree.get_node(&node_id) {
            println!("Score: {:.3}, Text: {}", score, node.text);
        }
    }

    Ok(())
}
```

--------------------------------

### Basic Cold Memory Usage in Rust

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/COLD_MEMORY_DESIGN.md

Demonstrates the basic usage of cold memory in Rust, including creating an instance, storing an entry with text and embeddings, and searching for similar entries.

```rust
use reasonkit_mem::storage::cold::{ColdMemory, ColdMemoryConfig, ColdMemoryEntry};
use std::path::PathBuf;

// Create cold memory
let config = ColdMemoryConfig::new(PathBuf::from("./data/cold"));
let cold = ColdMemory::new(config).await?;

// Store an entry
let entry = ColdMemoryEntry::new(
    "Machine learning is transforming industries.".to_string(),
    vec![0.1, 0.2, 0.3, 0.4, 0.5],  // 5-dim embedding
);
cold.store(&entry).await?;

// Search for similar
let results = cold.search_similar(&[0.1, 0.2, 0.3, 0.4, 0.5], 10).await?;
for (id, score) in results {
    println!("ID: {}, Score: {:.4}", id, score);
}
```

--------------------------------

### Get Detailed Statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves comprehensive statistics about the storage system, including a breakdown by storage tier. Provides deep insights into storage usage and performance.

```APIDOC
## detailed_stats

### Description
Gets detailed statistics about the storage system, including tier breakdown.

### Method
`async fn detailed_stats(&self, context: &AccessContext) -> Result<DualLayerStats>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **context** (*AccessContext*) - Access context for the operation.

### Response
#### Success Response (200)
- **DualLayerStats** - Detailed statistics object.
```

--------------------------------

### Run Custom Comprehensive Storage Benchmark

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Initiates a comprehensive benchmark using a custom `StorageBenchmarker` instance, allowing for detailed configuration and result export to JSON. Requires `tokio` runtime.

```rust
use reasonkit_mem::storage::{Storage, AccessLevel};
use reasonkit_mem::storage::benchmarks::StorageBenchmarker;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let storage = Storage::new_embedded().await?;

    let benchmarker = StorageBenchmarker::new(
        storage,
        "benchmark_user".to_string(),
        AccessLevel::Admin,
    );

    // Run comprehensive benchmark
    let results = benchmarker
        .run_comprehensive_benchmark(
            500,    // 500 documents
            5000,   // 5000 embeddings
            1536,   // OpenAI dimensions
        )
        .await?;

    // Export results
    let json = serde_json::to_string_pretty(&results)?;
    std::fs::write("benchmark_results.json", json)?;

    Ok(())
}
```

--------------------------------

### Get storage statistics

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Retrieves statistics about the storage, such as document, chunk, and embedding counts, and total size in bytes. Requires Admin access level for QdrantStorage.

```rust
pub async fn stats(
        &self,
        context: &AccessContext,
    ) -> Result<StorageStats>;

```

--------------------------------

### Local ONNX Embedding Usage

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDING_PIPELINE_GUIDE.md

Example of using the LocalONNXEmbedding provider for local inference. Requires the `local-embeddings` feature flag. Runs embeddings without API calls.

```rust
#[cfg(feature = "local-embeddings")]
use reasonkit_mem::embedding::local::LocalONNXEmbedding;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Load BGE-M3 model
    let provider = LocalONNXEmbedding::bge_m3("./models/bge-m3")?;

    // Embed text (no API calls, runs locally)
    let result = provider.embed("Hello, world!").await?;

    println!("Dimension: {}", provider.dimension());
    println!("Model: {}", provider.model_name());

    Ok(())
}
```

--------------------------------

### Error Handling: Qdrant Required but Unavailable

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

Demonstrates the error returned when `require_qdrant` is true and the specified Qdrant URL is unreachable. The storage initialization will fail.

```rust
let config = EmbeddedStorageConfig::with_qdrant(
    "http://localhost:99999", // Non-existent
    "test",
    768,
);
let result = Storage::new_embedded(config).await;
// Returns: Err("Qdrant required but not available at http://localhost:99999: ...")
```

--------------------------------

### Storage API: Qdrant Constructor

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Creates Qdrant-backed storage, specifying connection details, collection name, vector size, and embedded mode.

```rust
/// Create Qdrant-backed storage
pub async fn qdrant(
    host: &str,
    port: u16,
    grpc_port: u16,
    collection_name: String,
    vector_size: usize,
    embedded: bool,
) -> Result<Self>;
```

--------------------------------

### Get Multiple Documents by IDs (Bulk)

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves multiple documents based on their unique identifiers in a single batch request. Improves performance when fetching several documents.

```APIDOC
## get_documents_bulk

### Description
Retrieves multiple documents by their IDs in a batch operation.

### Method
`async fn get_documents_bulk(&self, ids: &[Uuid], context: &AccessContext) -> Result<Vec<(Uuid, Option<Document>)>>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **ids** (*&[Uuid]*) - A slice of document IDs to retrieve.
- **context** (*AccessContext*) - Access context for the operation.

### Response
#### Success Response (200)
- **Vec<(Uuid, Option<Document>)>** - A vector of tuples, where each tuple contains a document ID and the corresponding document (or None if not found).
```

--------------------------------

### Initialize Embedding Cache

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDING_PIPELINE_GUIDE.md

Initialize an LRU cache for embeddings with a specified maximum number of entries and a time-to-live in seconds. Operations like putting, getting, and clearing are demonstrated.

```rust
use reasonkit_mem::embedding::EmbeddingCache;

let cache = EmbeddingCache::new(
    10000,    // max_entries
    86400,    // ttl_secs (24 hours)
);

// Cache operations
cache.put("key".to_string(), embedding);
let cached = cache.get("key");
cache.clear();
```

--------------------------------

### Custom Dual Layer Storage Configuration

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Demonstrates how to create a custom configuration for the DualLayerStorage, specifying parameters for hot and cold memory, Write-Ahead Logging (WAL), and data synchronization tiers. This is useful for tailoring the storage behavior to specific application needs.

```rust
use reasonkit_mem::storage::*;
use std::path::PathBuf;

async fn custom_config_example() -> anyhow::Result<()> {
    let config = DualLayerConfig {
        hot: HotMemoryConfig {
            max_capacity_bytes: 512 * 1024 * 1024, // 512MB
            max_entries: 50_000,
            ttl_secs: 7200, // 2 hours
            eviction_policy: EvictionPolicy::Adaptive,
            backend: HotBackendType::InMemory,
            compression_enabled: true,
            compression_level: 5,
        },
        cold: ColdMemoryConfig {
            backend: ColdBackendType::QdrantLocal {
                url: "http://localhost:6333".to_string(),
            },
            collection_name: "my_collection".to_string(),
            vector_size: 768,
            quantization_enabled: true,
            quantization_type: QuantizationType::Int8,
            connection: QdrantConnectionConfig {
                max_connections: 20,
                connect_timeout_secs: 30,
                ..Default::default()
            },
            embedding_cache: EmbeddingCacheConfig {
                max_size: 20000,
                ttl_secs: 7200,
            },
            access_control: AccessControlConfig::default(),
        },
        wal: WalConfig {
            enabled: true,
            path: PathBuf::from("./wal"),
            max_file_size: 128 * 1024 * 1024, // 128MB
            sync_mode: WalSyncMode::Interval,
            sync_interval_ms: 500,
            retention_secs: 172800, // 48 hours
            compression_enabled: true,
            checkpoint_threshold: 5000,
        },
        sync: SyncConfig {
            tiering_policy: TieringPolicy {
                default_tier: StorageTier::Hot,
                access_based: true,
                age_based: true,
                size_based: true,
                size_threshold_bytes: 5 * 1024 * 1024, // 5MB
                age_threshold_secs: 7200, // 2 hours
            },
            background_sync_interval_secs: 30,
            sync_batch_size: 200,
            auto_promote: true,
            promotion_threshold: 5,
            demotion_threshold_secs: 3600,
            parallel_sync: true,
            max_concurrent_syncs: 8,
        },
    };

    let storage = DualLayerStorage::with_config(config).await?;

    Ok(())
}

```

--------------------------------

### Run Quality Gates

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/CONTRIBUTING.md

Execute the required quality gates for contributions: build, lint, format check, test, and benchmark.

```bash
cargo build --release
```

```bash
cargo clippy -- -D warnings
```

```bash
cargo fmt --check
```

```bash
cargo test --all-features
```

```bash
cargo bench
```

--------------------------------

### Transaction Management in Rust

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Illustrates how to perform atomic operations using transactions. This includes starting a transaction, creating savepoints, performing operations, and committing or rolling back changes.

```rust
use reasonkit_mem::storage::{DualLayerStorage, StorageTransaction, IsolationLevel};

async fn transaction_example(storage: Arc<DualLayerStorage>) -> anyhow::Result<()> {
    // Begin transaction
    let mut tx = StorageTransaction::begin(storage.clone()).await?;

    // Create savepoint
    tx.savepoint("before_changes")?;

    // Perform operations
    let doc1 = create_document("doc1");
    tx.store_document(doc1).await?;

    let doc2 = create_document("doc2");
    tx.store_document(doc2).await?;

    // Rollback to savepoint if needed
    // tx.rollback_to_savepoint("before_changes")?;

    // Commit transaction
    let result = tx.commit().await?;
    println!(
        "Transaction {} committed: {} ops in ?",
        result.transaction_id,
        result.operations_succeeded,
        result.duration
    );

    Ok(())
}
```

--------------------------------

### Configure Qdrant Connection Pool (Low-Latency)

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Sets up a Qdrant connection pool optimized for low-latency scenarios, prioritizing faster response times.

```rust
use reasonkit_mem::storage::QdrantConnectionConfig;

// For low-latency scenarios
let config = QdrantConnectionConfig {
    max_connections: 10,
    connect_timeout_secs: 2,
    request_timeout_secs: 10,
    health_check_interval_secs: 10,
    max_idle_secs: 60,
    ..Default::default()
};
```

--------------------------------

### Get Demotion Candidates

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_API_DESIGN.md

Retrieves a list of document IDs that are candidates for demotion to the cold tier. Useful for optimizing storage costs by moving less frequently accessed data.

```APIDOC
## get_demotion_candidates

### Description
Gets a list of document IDs that are candidates for demotion to the cold tier.

### Method
`async fn get_demotion_candidates(&self, limit: usize) -> Result<Vec<Uuid>>`

### Parameters
#### Path Parameters
None

#### Query Parameters
None

#### Request Body
- **limit** (*usize*) - The maximum number of candidates to retrieve.

### Response
#### Success Response (200)
- **Vec<Uuid>** - A vector of document IDs eligible for demotion.
```

--------------------------------

### Basic Embedded Storage with File Fallback

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

Initializes embedded storage using the default configuration, which prioritizes file storage and falls back to Qdrant if available. This is suitable for development environments where Qdrant might not always be running.

```rust
use reasonkit_mem::storage::{Storage, EmbeddedStorageConfig};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Default config uses file storage (require_qdrant=false)
    let config = EmbeddedStorageConfig::default();
    let storage = Storage::new_embedded(config).await?;

    // Use storage...
    Ok(())
}
```

--------------------------------

### Initialize Storage with Custom Configuration in Rust

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/README.md

Initializes embedded storage with custom configurations, such as a specific file path or Qdrant connection details. Requires tokio runtime.

```rust
use reasonkit_mem::storage::{Storage, EmbeddedStorageConfig};
use std::path::PathBuf;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Create storage with custom file path
    let config = EmbeddedStorageConfig::file_only(PathBuf::from("./data"));
    let storage = Storage::new_embedded_with_config(config).await?;

    // Or use Qdrant (requires running server)
    let qdrant_config = EmbeddedStorageConfig::with_qdrant(
        "http://localhost:6333",
        "my_collection",
        1536,
    );
    let qdrant_storage = Storage::new_embedded_with_config(qdrant_config).await?;

    Ok(())
}
```

--------------------------------

### Supported URL Formats for Embedded Mode

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/EMBEDDED_MODE_GUIDE.md

The embedded mode can parse various URL formats for connecting to Qdrant. These examples show different valid ways to specify the host and port.

```rust
// All of these work:
"http://localhost:6333"
"localhost:6333"
"localhost"              // Defaults to port 6333
"127.0.0.1:6334"
"https://qdrant.example.com:6333"
```

--------------------------------

### Vector Backend Options

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/DUAL_LAYER_STORAGE_ARCHITECTURE.md

Enumerates the available vector store backends for the cold storage layer. Choose the backend that best suits your performance and dependency requirements.

```rust
/// Vector store backend options
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
pub enum VectorBackend {
    /// Pure Rust HNSW implementation (portable, no dependencies)
    PureRust,
    /// LanceDB embedded (best performance, requires Arrow)
    LanceDb,
    /// Qdrant embedded (feature-rich, higher memory)
    QdrantEmbedded,
}
```

--------------------------------

### Run Built-in Storage Benchmarks

Source: https://github.com/reasonkit/reasonkit-mem/blob/main/docs/STORAGE_API.md

Executes a quick benchmark of document storage and vector search operations using the built-in benchmarker. Requires `tokio` runtime.

```rust
use reasonkit_mem::storage::{Storage, AccessLevel};
use reasonkit_mem::storage::benchmarks::{StorageBenchmarker, run_storage_benchmarks};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let storage = Storage::new_embedded().await?;

    // Quick benchmark
    let results = run_storage_benchmarks(
        storage,
        100,    // documents
        1000,   // embeddings
        768,    // vector size
    ).await?;

    println!("=== Benchmark Results ===");
    println!("Document Storage:");
    println!("  Avg latency: {:?}", results.document_storage.avg_latency);
    println!("  Ops/sec: {:.2}", results.document_storage.ops_per_second);
    println!("  P95 latency: {:?}", results.document_storage.p95_latency);

    println!("\nVector Search:");
    println!("  Avg latency: {:?}", results.vector_search.avg_latency);
    println!("  Ops/sec: {:.2}", results.vector_search.ops_per_second);

    Ok(())
}
```