### KVault Quick Start Example Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/kvault.md Demonstrates basic KVault initialization, storing different data types (dictionary, NumPy array, raw bytes), and retrieving them. ```python from kohakuvault import KVault import numpy as np kv = KVault("app.db") # auto-pack is on by default kv["config"] = {"timeout": 30, "retries": 3} # auto MessagePack kv["embedding"] = np.random.randn(768).astype(np.float32) kv["image.jpg"] = open("image.jpg", "rb").read() # raw bytes stay raw assert kv["config"]["timeout"] == 30 assert kv["embedding"].shape == (768,) ``` -------------------------------- ### KohakuVault Development Workflow Setup Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/README.md Sets up the development environment by installing KohakuVault in editable mode with development dependencies. This command is typically run once. ```bash pip install -e .[dev] maturin develop ``` -------------------------------- ### Create and Use Single-Column TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Illustrates the default single-column TextVault setup and performing an exact match lookup. ```python tv = TextVault("data.db") # defaults to columns=["content"] tv.insert("Hello world", {"meta": "data"}) value = tv["Hello world"] # exact match lookup ``` -------------------------------- ### Install KohakuVault for Development Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/README.md Installs KohakuVault in editable mode with development dependencies, including tools for building PyO3 extensions. Run 'maturin develop' after changing Rust code. ```bash pip install -e .[dev] # everytime you change the rust code, you should run this command maturin develop ``` -------------------------------- ### Install KohakuVault Package Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/README.md Installs the KohakuVault library using pip. This is the standard command for adding the package to your Python environment. ```bash pip install kohakuvault ``` -------------------------------- ### TextVault Auto-Packing Examples Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Demonstrates storing various Python objects as values in TextVault, leveraging MessagePack and DataPacker. ```python # Store any Python object as values tv.insert("text", {"dict": "value"}) # MessagePack tv.insert("text", [1, 2, 3]) # MessagePack tv.insert("text", np.array([1, 2, 3])) # DataPacker vec:* tv.insert("text", 42) # DataPacker i64 tv.insert("text", b"raw bytes") # Raw (no header) ``` -------------------------------- ### KohakuVault Core Functionality Example Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/README.md Demonstrates the primary usage of KVault for storing raw bytes and auto-packed metadata, TextVault for full-text search, ColumnVault for incremental columnar logging, and VectorKVault for k-NN search within the same SQLite database. ```python from kohakuvault import KVault, TextVault, ColumnVault, VectorKVault, CSBTree import numpy as np DB = "test.db" # Store media + metadata in one file kv = KVault(DB, table="media") kv["video:42"] = b"abcdefg" # raw bytes stay raw kv["video:42:meta"] = {"fps": 60, "tags": ["tutorial", "gpu"]} # auto MessagePack # Full-text search with BM25 ranking (great for RAG) tv = TextVault(DB, table="docs") tv.insert("Machine learning is a subset of AI", {"category": "tech"}) for doc_id, score, meta in tv.search("machine learning", k=5): print(f"Doc {doc_id}: score={score:.4f}") # Incremental columnar logging (no file rewrites) cols = ColumnVault(kv) # or ColumnVault(DB) ids = cols.ensure("ids", "i64") latency = cols.ensure("latency_ms", "f64") embeddings = cols.ensure("embeddings", "vec:f32:384") with cols.cache(cap_bytes=8 << 20): ids.append(len(ids)) latency.append(12.3) embeddings.append(np.random.randn(384).astype(np.float32)) # Ordered metadata via embedded CSBTree index = CSBTree() index.insert(latency[-1], ids[-1]) # log-latency -> run-id # Vector similarity search (sqlite-vec inside the same DB) search = VectorKVault(DB, table="runs", dimensions=384, metric="cosine") for idx, emb in enumerate(embeddings[:10]): search.insert(emb, str(idx).encode()) for rank, (rid, distance, run_id) in enumerate(search.search(embeddings[-1], k=3), 1): print(rank, distance, run_id.decode()) ``` -------------------------------- ### Initialize and Use VectorKVault for Similarity Search Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/vectors.md Set up `VectorKVault` for k-NN search with specified dimensions, metric, and vector type. This example demonstrates inserting a vector, performing a search, and retrieving results. ```python from kohakuvault import VectorKVault import numpy as np vkv = VectorKVault("search.db", table="search", dimensions=384, metric="cosine", vector_type="f32") vector = np.random.randn(384).astype(np.float32) doc_id = vkv.insert(vector, b"document payload") results = vkv.search(vector, k=10) for row_id, distance, payload in results: print(row_id, distance, payload[:32]) closest = vkv.get(vector) vector2, payload2 = vkv.get_by_id(doc_id) vkv.update(doc_id, vector=new_vector) vkv.delete(doc_id) ``` -------------------------------- ### TextVault Safe Query Escaping Examples Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Demonstrates how TextVault automatically escapes special characters in queries for safe literal matching. ```python tv.search("What is this?") # Works with ? tv.search("C++ programming") # Works with + tv.search("test@email.com") # Works with @ tv.search('He said "hello"') # Works with quotes ``` -------------------------------- ### Get Document by Exact Key (Single-Column) Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Retrieves a document from a single-column TextVault using its exact text content as the key. ```python value = tv.get("exact document text") # or dict-style: value = tv["exact document text"] ``` -------------------------------- ### TextVault Utility Methods Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Provides utility methods for TextVault, including getting the document count, checking existence, retrieving information, listing keys, and clearing all documents. ```python len(tv) # Document count tv.count() # Same as len() tv.count_matches("query") # Count matching documents tv.exists(doc_id) # Check if ID exists doc_id in tv # Same as exists() tv.info() # {"table": "...", "columns": [...], "count": N} tv.columns # ["content"] or ["title", "body", "tags"] tv.keys(limit=100, offset=0) # Paginated list of rowids tv.clear() # Delete all documents ``` -------------------------------- ### Create Image Vector Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Example of creating a column to store image data represented as unsigned 8-bit integer vectors with a 28x28 shape. ```python images = vault.create_column("mnist", "vec:u8:28:28") ``` -------------------------------- ### Get Document by ID from TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Retrieves a document from TextVault using its unique ID, returning either text or a dictionary of columns. ```python text, value = tv.get_by_id(doc_id) # Single column: text is string # Multi-column: text is dict {"title": "...", "body": "...", "tags": "..."} ``` -------------------------------- ### Create Arbitrary-Shape Vector Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/vectors.md Use `vec:f32` for vectors where dimensions can vary. This example shows appending vectors with different shapes to a generic column. ```python generic = cv.create_column("generic", "vec:f32") generic.append(np.random.randn(100).astype(np.float32)) generic.append(np.random.randn(10, 20).astype(np.float32)) ``` -------------------------------- ### Create and Extend Fixed-Shape Vector Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/vectors.md Use `vec:f32:768` for storing vectors of a consistent shape, such as embeddings. This example demonstrates creating a column and extending it with random float32 vectors. ```python from kohakuvault import ColumnVault import numpy as np cv = ColumnVault("vectors.db") embeddings = cv.create_column("bert", "vec:f32:768") embeddings.extend(np.random.randn(10_000, 768).astype(np.float32)) ``` -------------------------------- ### Create and Use Columns Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Demonstrates creating a new column, extending it with data, and reusing an existing column. ```python from kohakuvault import ColumnVault import numpy as np vault = ColumnVault("analytics.db") counts = vault.create_column("counts", "i64") counts.extend(range(10)) # Reuse existing column or create it lazily metrics = vault.ensure("metrics", "msgpack") ``` -------------------------------- ### Create and Search Multi-Column TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Shows how to set up a multi-column TextVault for structured documents and search specific columns. ```python tv = TextVault("data.db", columns=["title", "body", "tags"]) tv.insert( {"title": "Introduction to ML", "body": "Machine learning is...", "tags": "ml ai tutorial"}, {"author": "John", "date": "2024-01-01"} ) # Search specific columns results = tv.search("introduction", column="title") results = tv.search("tutorial", column="tags") ``` -------------------------------- ### Create and Search Single-Column TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Demonstrates creating a single-column TextVault, inserting a document, and performing a BM25 ranked search. ```python from kohakuvault import TextVault # Create a single-column text vault tv = TextVault("documents.db") # Insert documents with any Python value doc_id = tv.insert("Machine learning is a subset of AI", {"category": "tech", "importance": 5}) # Search with BM25 ranking results = tv.search("machine learning", k=10) for doc_id, score, value in results: print(f"ID {doc_id}: score={score:.4f}, value={value}") ``` -------------------------------- ### Instantiate DataPacker with Fixed and Variable Size Types Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/datapacker.md Demonstrates creating DataPacker instances for fixed-size and variable-size data types. Fixed-size types have a defined element size, while variable-size types do not. ```python from kohakuvault import DataPacker fixed = DataPacker("str:32:utf8") assert fixed.elem_size == 32 and not fixed.is_varsize var = DataPacker("vec:f32") assert var.elem_size == 0 and var.is_varsize ``` -------------------------------- ### KohakuVault Formatting, Linting, and Testing Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/README.md Runs code formatting with 'black', Rust code formatting and linting with 'cargo fmt' and 'cargo clippy', builds the PyO3 extension in release mode, and executes Python tests with 'pytest'. ```bash black . cargo fmt cargo clippy maturin develop --release pytest ``` -------------------------------- ### Pack and Unpack Data with DataPacker Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/datapacker.md Shows how to pack a dictionary into a byte blob and then unpack it using a DataPacker instance. Offsets can be used to read multiple values from a single blob. ```python packer = DataPacker("msgpack") blob = packer.pack({"user": 1, "tags": ["vip", "beta"]}) restored = packer.unpack(blob, offset=0) ``` -------------------------------- ### ColumnVault and Column Caching Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Demonstrates using the cache context manager for both the entire vault and individual columns. Useful for optimizing ingest operations by controlling cache size and flush thresholds. ```python with vault.cache(cap_bytes=32 << 20, flush_threshold=8 << 20): counts.extend(range(1_000_000)) col = vault["counts"] with col.cache(cap_bytes=4 << 20): col.append(123) ``` -------------------------------- ### KVault Streaming API for Files Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/kvault.md Demonstrates uploading large files using `put_file` and downloading them using `get_to_file` without loading the entire content into memory. This is useful for handling large blobs efficiently. ```python # Upload large blobs without loading them fully in memory with open("movie.mp4", "rb") as reader: kv.put_file("movie:2024", reader, chunk_size=4 << 20) with open("copy.mp4", "wb") as writer: kv.get_to_file("movie:2024", writer, chunk_size=4 << 20) ``` -------------------------------- ### Create and Extend Vector Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Shows how to create a column for fixed-shape float vectors and extend it with random data. ```python embeddings = vault.create_column("embeddings", "vec:f32:768") embeddings.extend(np.random.randn(10_000, 768).astype(np.float32)) ``` -------------------------------- ### RAG Pipeline with Hybrid Search Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Sets up TextVault and VectorKVault for a RAG pipeline, indexing documents and their embeddings, and implementing a hybrid search function combining BM25 and vector similarity. ```python from kohakuvault import TextVault, VectorKVault import numpy as np # Documents with text search docs = TextVault("rag.db", table="documents") # Embeddings with vector search vecs = VectorKVault("rag.db", table="embeddings", dimensions=384, metric="cosine") # Index documents for idx, doc in enumerate(documents): # Store full document with metadata doc_id = docs.insert(doc["text"], {"source": doc["source"], "date": doc["date"]}) # Store embedding linked to doc_id embedding = model.encode(doc["text"]) vecs.insert(embedding.astype(np.float32), str(doc_id).encode()) # Hybrid search: combine BM25 + vector similarity def hybrid_search(query: str, k: int = 10, alpha: float = 0.5): # BM25 search bm25_results = {doc_id: score for doc_id, score, _ in docs.search(query, k=k*2)} # Vector search query_vec = model.encode(query).astype(np.float32) vec_results = {int(doc_id): 1/(1+dist) for _, dist, doc_id in vecs.search(query_vec, k=k*2)} # Combine scores all_ids = set(bm25_results) | set(vec_results) combined = [] for doc_id in all_ids: bm25_score = bm25_results.get(doc_id, 0) vec_score = vec_results.get(doc_id, 0) combined.append((doc_id, alpha * bm25_score + (1-alpha) * vec_score)) return sorted(combined, key=lambda x: x[1], reverse=True)[:k] ``` -------------------------------- ### KVault Optimization with PRAGMA and VACUUM Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md Performs database optimization by running SQLite's PRAGMA optimize and VACUUM commands. This helps in compacting the database file and improving query performance. ```python KVault.optimize() ``` -------------------------------- ### KVault Auto-Pack Configuration Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/kvault.md Shows how to enable or disable the auto-pack feature, which automatically serializes Python objects into compact binary formats. Use `use_pickle=True` to allow Pickle serialization for unsupported types. ```python kv.enable_auto_pack(use_pickle=True) # default kv.disable_auto_pack() # revert to bytes-only kv.auto_pack_enabled() # -> bool ``` -------------------------------- ### Extend and Modify Fixed-Size Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Shows how to extend a fixed-size column with initial values and then modify a slice of it. ```python scores = vault.create_column("scores", "f64") scores.extend([95.5, 87.3, 99.1]) scores[1:3] = [90.0, 91.5] ``` -------------------------------- ### Batch Pack and Unpack Integer and Vector Data Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/datapacker.md Illustrates using the `pack_many` and `unpack_many` methods for efficient bulk processing of data. This is suitable for large lists of fixed-size types or vectors. ```python from kohakuvault import DataPacker import numpy as np pack_i64 = DataPacker("i64") values = list(range(1_000)) buffer = pack_i64.pack_many(values) assert pack_i64.unpack_many(buffer, count=len(values)) == values pack_vec = DataPacker("vec:f32:768") vectors = [np.random.randn(768).astype(np.float32) for _ in range(1_000)] buffer = pack_vec.pack_many(vectors) restored = pack_vec.unpack_many(buffer, count=len(vectors)) ``` -------------------------------- ### Create Variable-Size Column with JSON Schema Validation Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Demonstrates creating a DataPacker with a JSON schema for validating structured data before appending. ```python from kohakuvault import DataPacker schema = {"type": "object", "required": ["id", "name"]} packer = DataPacker.with_json_schema(schema) ``` -------------------------------- ### Append and Extend Variable-Size Column Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/columnvault.md Illustrates appending single structured items and extending with multiple items to a variable-size column. ```python events = vault.ensure("events", "msgpack") events.append({"type": "login", "user": 42}) events.extend({"type": "purchase", "amount": a} for a in amounts) ``` -------------------------------- ### KVault Caching and Concurrency Control Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/kvault.md Illustrates using KVault's write-back cache for buffering writes and managing concurrency. The `cache` context manager allows specifying capacity and flush thresholds, while `lock_cache` provides temporary blocking of background flushes. ```python with kv.cache(cap_bytes=64 << 20, flush_threshold=8 << 20) as buffered: for i in range(200_000): buffered[f"log:{i}"] = b"..." kv.enable_cache(flush_interval=5.0) # background daemon flush with kv.lock_cache(): # block daemon flush temporarily mutate_many_keys() kv.flush_cache() ``` -------------------------------- ### DataPacker with JSON Schema Validation Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/datapacker.md Demonstrates creating a DataPacker instance that validates data against a provided JSON schema. Invalid data will raise a ValueError during packing. ```python schema = { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, }, "required": ["id", "name"], } packer = DataPacker.with_json_schema(schema) packer.pack({"id": 1, "name": "Rin"}) packer.pack({"id": "oops"}) # raises ValueError ``` -------------------------------- ### Insert Document into TextVault (Multi-Column) Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Inserts a structured document with separate searchable fields and metadata into a multi-column TextVault. ```python # Multi-column doc_id = tv.insert( {"title": "Hello", "body": "World", "tags": "greeting"}, {"created": "2024-01-01"} ) ``` -------------------------------- ### Manage Auto-Pack Status Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Check, disable, and re-enable TextVault's auto-pack feature. Auto-packing is enabled by default and can be configured to use pickle for custom objects. ```python # Check status tv.auto_pack_enabled() # True by default # Disable for bytes-only mode tv.disable_auto_pack() # Re-enable tv.enable_auto_pack(use_pickle=True) # use_pickle allows custom objects ``` -------------------------------- ### Insert Document into TextVault (Single-Column) Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Inserts a document with associated metadata into a single-column TextVault. ```python # Single column doc_id = tv.insert("document text", {"key": "value"}) ``` -------------------------------- ### KVault and ColumnVault WAL Checkpointing Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md Initiates a Write-Ahead Logging (WAL) checkpoint operation for KVault or ColumnVault instances. This operation is crucial for maintaining database integrity and performance by merging WAL data into the main database file. ```python KVault.checkpoint() ColumnVault.checkpoint() ``` -------------------------------- ### Storing Documents, Embeddings, and Titles Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/vectors.md This snippet initializes KVault, ColumnVault, and creates columns for embeddings and titles. It then iterates through a dataset, storing document text, titles, and embeddings. Finally, it sets up a VectorKVault for similarity search and inserts the embeddings. ```python from kohakuvault import KVault, ColumnVault, VectorKVault import numpy as np db = "semantic.db" kv = KVault(db, table="documents") cv = ColumnVault(kv) emb = cv.create_column("embeddings", "vec:f32:384") titles = cv.create_column("titles", "str:utf8") for idx, (title, text, embedding) in enumerate(dataset): kv[f"doc:{idx}"] = text.encode() titles.append(title) emb.append(embedding.astype(np.float32)) search = VectorKVault(db, table="search_index", dimensions=384, metric="cosine") for idx, embedding in enumerate(emb[-len(dataset):]): search.insert(embedding, str(idx).encode()) query = model.encode("machine learning").astype(np.float32) for rank, (row_id, dist, doc_idx_bytes) in enumerate(search.search(query, k=5), 1): doc_idx = int(doc_idx_bytes) print(rank, dist, titles[doc_idx], kv[f"doc:{doc_idx}"][:80]) ``` -------------------------------- ### Enabling Raw Byte Headers Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md Controls whether raw byte data stored in KVault should be prefixed with headers. Headers are rarely needed and can be disabled to save space. ```python KVault.enable_headers() ``` -------------------------------- ### Hybrid Layout: Metadata + Blobs Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/structured_columns.md Store binary payloads in KVault and structured metadata in a MessagePack column. This separates large blobs from filterable metadata, improving performance. ```python from kohakuvault import KVault, ColumnVault kv = KVault("media.db") cols = ColumnVault(kv) meta = cols.ensure("media_meta", "msgpack") for asset in assets: kv[f"blob:{asset['id']}"] = asset["bytes"] meta.append({"id": asset["id"], "size": len(asset["bytes"]), "mime": asset["mime"]}) ``` -------------------------------- ### TextVault Search with Snippets Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Fetches search results with highlighted snippets showing context around matches. ```python results = tv.search_with_snippets( "machine learning", k=10, snippet_tokens=15, # words around match highlight_start="", # custom highlight markers highlight_end="" ) for doc_id, score, snippet, value in results: print(f"Snippet: {snippet}") # "...is a machine learning algorithm that..." ``` -------------------------------- ### TextVault Search with Text Content Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Retrieves the indexed text content alongside search results. ```python results = tv.search_with_text("machine learning", k=5) for doc_id, score, text, value in results: print(f"Text: {text}") print(f"Value: {value}") ``` -------------------------------- ### Hybrid Layout: Secondary Indexes with CSBTree Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/structured_columns.md Build in-memory indexes using CSBTree for efficient lookups on column data. This pattern keeps the canonical record in a column while providing fast access via an index. ```python from kohakuvault import ColumnVault, CSBTree cols = ColumnVault("events.db") events = cols.ensure("events", "msgpack") index = CSBTree() row_id = len(events) events.append({"user": 42, "type": "login"}) index.insert(42, row_id) for _, rid in index.range(42, 42): handle(events[rid]) ``` -------------------------------- ### Batch Operations with DataPacker for Vectors Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/vectors.md Utilize `DataPacker` for efficient preprocessing and marshaling of vectors. This snippet shows packing and unpacking a large batch of fixed-shape vectors. ```python from kohakuvault import DataPacker packer = DataPacker("vec:f32:768") vectors = [np.random.randn(768).astype(np.float32) for _ in range(10_000)] buffer = packer.pack_many(vectors) unpacked = packer.unpack_many(buffer, count=len(vectors)) ``` -------------------------------- ### TextVault Raw FTS5 Syntax Search Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Shows how to disable default escaping to use raw FTS5 query operators for advanced searching. ```python tv.search("hello AND world", escape=False) # Both terms required tv.search("python OR java", escape=False) # Either term tv.search("hello NOT goodbye", escape=False) # Exclusion tv.search("mach*", escape=False) # Prefix matching tv.search("NEAR(hello world, 5)", escape=False) # Proximity search ``` -------------------------------- ### Basic TextVault Search Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Performs a basic BM25 ranked search and prints the document ID and score. ```python results = tv.search("hello world", k=10) for doc_id, score, value in results: print(f"Doc {doc_id}: {score:.4f}") ``` -------------------------------- ### Auto-Pack Header Format Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md The 10-byte header used to prefix non-raw auto-packed values in the KVault table. It includes version, encoding, and flags for safe mixing of raw and packed data. ```rust |0x89 0x4B|version|encoding|flags|reserved(3)|0x56 0x4B| ``` -------------------------------- ### Update Document in TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Updates the text content, metadata value, or both for an existing document in TextVault using its ID. ```python # Update text only tv.update(doc_id, texts="new document text") # Update value only tv.update(doc_id, value={"new": "metadata"}) # Update both tv.update(doc_id, texts="new text", value={"new": "value"}) ``` -------------------------------- ### Enforce JSON Schema with DataPacker Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/structured_columns.md Use DataPacker with a JSON schema to validate MessagePack payloads before they are written to SQLite. Failures raise ValueError, allowing for pre-write validation in Python. ```python from kohakuvault import ColumnVault, DataPacker schema = { "type": "object", "properties": { "id": {"type": "integer"}, "name": {"type": "string"}, "tags": {"type": "array", "items": {"type": "string"}}, }, "required": ["id", "name"], } packer = DataPacker.with_json_schema(schema) vault = ColumnVault("users.db") users = vault.ensure("users", "msgpack") users.append({"id": 1, "name": "Rin"}) ``` -------------------------------- ### TextVault Schema Definition Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Defines the SQL schema created by TextVault, consisting of an FTS5 virtual table for indexed text and a separate values table for storing blob values. ```sql -- What TextVault creates: CREATE VIRTUAL TABLE text_vault USING fts5(content, value_ref UNINDEXED); CREATE TABLE text_vault_values (id INTEGER PRIMARY KEY, value BLOB); ``` -------------------------------- ### Retrieve Document by ID Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Retrieves a document by its ID and prints the type of the decoded value. The value can be a dictionary, list, numpy array, or other types. ```python text, value = tv.get_by_id(doc_id) print(type(value)) # , , , etc. ``` -------------------------------- ### Flushing Cached Writes to SQLite Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md Forces any buffered writes held in memory caches for vaults or columns to be immediately written to the underlying SQLite database. This ensures data durability. ```python flush_cache() ``` -------------------------------- ### Disabling Raw Byte Headers Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/architecture.md Disables the automatic addition of headers to raw byte data stored in KVault. This is the default behavior and is useful when minimizing storage footprint. ```python KVault.disable_headers() ``` -------------------------------- ### Delete Document from TextVault Source: https://github.com/kohakublueleaf/kohakuvault/blob/main/docs/textvault.md Removes a document from TextVault using its ID, or by exact key for single-column exact match lookups. ```python tv.delete(doc_id) # or for single-column with exact match: del tv["exact document text"] ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.