### HTTP REST API Server - CoreNN Standalone Service Source: https://context7.com/wilsonzlin/corenn/llms.txt Runs CoreNN as a standalone HTTP service, exposing JSON endpoints for remote vector operations. Supports starting the server with specified database path, port, address, and data type, and provides endpoints for health checks, insertion, and querying. ```bash # Start server corenn serve /path/to/db --port 4224 --addr 127.0.0.1 --dtype F32 # Available dtypes: BF16, F16, F32, F64 # Server provides three endpoints: # GET /healthz - Health check (returns "OK") # POST /insert - Insert vectors # POST /query - Query nearest neighbors ``` ```bash # Health check curl http://localhost:4224/healthz # Response: OK # Insert vectors (parallel processing) curl -X POST http://localhost:4224/insert \ -H "Content-Type: application/json" \ -d '{ "vectors": [ { "key": "doc_001", "vector": [0.1, 0.2, 0.3, 0.4, 0.5] }, { "key": "doc_002", "vector": [0.6, 0.7, 0.8, 0.9, 1.0] } ] }' # Query nearest neighbors curl -X POST http://localhost:4224/query \ -H "Content-Type: application/json" \ -d '{ "vector": [0.5, 0.5, 0.5, 0.5, 0.5], "k": 10 }' # Response: # { # "results": [ # ["doc_002", 0.31622776], # ["doc_001", 0.83666003] # ] # } ``` -------------------------------- ### Create, Insert, and Query Vector Database (Node.js) Source: https://github.com/wilsonzlin/corenn/blob/master/README.md Provides an example of using CoreNN in Node.js to set up a database, insert data with Float32Array vectors, and execute similarity queries. Leverages the @corenn/node package. Queries return an array of objects containing keys and distances. ```typescript import { CoreNN } from "@corenn/node"; const db = CoreNN.create("/path/to/db", { // Specify the dimensionality of your vectors. dim: 3, // All other config options are optional. }); db.insert([ { key: "my_entry_1", vector: new Float32Array([0.5, 4.1, 2.2]), }, { key: "my_entry_2", vector: new Float32Array([3.1, 7.7, 6.4]), }, ]); // Later... const db = CoreNN.open("/path/to/db"); // Array of { key, distance } objects. const results = db.query( new Float32Array([0.0, 1.1, 2.2]), 100, ); ``` -------------------------------- ### CoreNN Vector Data Types and Precision Handling (Rust) Source: https://context7.com/wilsonzlin/corenn/llms.txt Demonstrates the usage of different floating-point data types (BF16, F16, F32, F64) for vectors in Rust using the `libcorenn` library. Shows how to create, get dimensions, convert between types, and access raw bytes for serialization. Dependencies include `libcorenn`, `half`, and `ndarray`. ```rust use libcorenn::vec::VecData; use half::{bf16, f16}; use ndarray::Array1; // Create vectors with different precision levels let vec_bf16 = VecData::BF16(vec![ bf16::from_f32(0.1), bf16::from_f32(0.2), bf16::from_f32(0.3), ]); let vec_f16 = VecData::F16(vec![ f16::from_f32(0.1), f16::from_f32(0.2), f16::from_f32(0.3), ]); let vec_f32 = VecData::F32(vec![0.1, 0.2, 0.3]); let vec_f64 = VecData::F64(vec![0.1, 0.2, 0.3]); // Get dimensionality assert_eq!(vec_f32.dim(), 3); // Convert to f32 for computation (consumes vector) let array_f32: Array1 = vec_f32.into_f32(); // Convert to f32 without consuming (creates copy) let vec_f16_copy = VecData::F16(vec![f16::from_f32(0.5); 128]); let array_f32: Array1 = vec_f16_copy.to_f32(); // Access raw bytes (for serialization) let vec = VecData::F32(vec![1.0, 2.0, 3.0]); let bytes: &[u8] = vec.as_raw_bytes(); assert_eq!(bytes.len(), 3 * 4); // 3 floats × 4 bytes // Memory efficiency comparison: // - bf16: 2 bytes per dimension (~50% of f32) // - f16: 2 bytes per dimension (~50% of f32) // - f32: 4 bytes per dimension (baseline) // - f64: 8 bytes per dimension (2× f32) ``` -------------------------------- ### Database Creation and Initialization with CoreNN Source: https://context7.com/wilsonzlin/corenn/llms.txt Demonstrates how to create a new persistent vector database, open an existing one, or initialize an in-memory database using CoreNN. Configuration options include vector dimensionality, distance metric, compression settings, and graph parameters. The `Cfg` struct allows detailed customization. ```rust use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode}; // Create a new database with full configuration let db = CoreNN::create("/path/to/db", Cfg { dim: 768, // Vector dimensionality (required) metric: StdMetric::Cosine, // Distance metric: L2 or Cosine max_edges: 64, // Maximum graph edges per node max_add_edges: 64, // Maximum additional edges during construction beam_width: 4, // Beam width for graph traversal distance_threshold: 1.1, // Edge pruning threshold compression_mode: CompressionMode::PQ, // PQ or Trunc (Matryoshka) compression_threshold: 10_000_000, // Auto-compress at N vectors compaction_threshold_deletes: 1_000_000, // Compact after N deletions pq_subspaces: 64, // Product Quantization subspaces pq_sample_size: 10_000, // PQ training sample size trunc_dims: 64, // Truncation dimensions for Matryoshka query_search_list_cap: 128, // Query search list capacity update_search_list_cap: 128, // Update search list capacity }); // Open existing database (config is read from disk) let db = CoreNN::open("/path/to/db"); // Create in-memory database for testing let db = CoreNN::new_in_memory(Cfg { dim: 128, ..Default::default() }); // Access configuration let cfg = db.cfg(); println!("Vector dimension: {}", cfg.dim); ``` -------------------------------- ### Database Creation and Initialization Source: https://context7.com/wilsonzlin/corenn/llms.txt Demonstrates how to create a new persistent vector database, open an existing one, or create an in-memory database for testing. Configuration options for dimensionality, distance metric, compression, and graph parameters are shown. ```APIDOC ## Database Creation and Initialization ### Description Create a new persistent vector database or open an existing one with configuration options for dimensionality, distance metric, compression settings, and graph parameters. ### Method `CoreNN::create`, `CoreNN::open`, `CoreNN::new_in_memory` ### Endpoint N/A (Rust library functions) ### Parameters #### Create/New In-Memory Configuration (`Cfg`) - **dim** (usize) - Required - Vector dimensionality. - **metric** (StdMetric) - Optional - Distance metric: `StdMetric::L2` or `StdMetric::Cosine` (defaults to Cosine). - **max_edges** (usize) - Optional - Maximum graph edges per node (defaults to 64). - **max_add_edges** (usize) - Optional - Maximum additional edges during construction (defaults to 64). - **beam_width** (usize) - Optional - Beam width for graph traversal (defaults to 4). - **distance_threshold** (f32) - Optional - Edge pruning threshold (defaults to 1.1). - **compression_mode** (CompressionMode) - Optional - Compression mode: `CompressionMode::PQ` or `CompressionMode::Trunc` (defaults to PQ). - **compression_threshold** (usize) - Optional - Auto-compress at N vectors (defaults to 10,000,000). - **compaction_threshold_deletes** (usize) - Optional - Compact after N deletions (defaults to 1,000,000). - **pq_subspaces** (usize) - Optional - Product Quantization subspaces (defaults to 64). - **pq_sample_size** (usize) - Optional - PQ training sample size (defaults to 10,000). - **trunc_dims** (usize) - Optional - Truncation dimensions for Matryoshka (defaults to 64). - **query_search_list_cap** (usize) - Optional - Query search list capacity (defaults to 128). - **update_search_list_cap** (usize) - Optional - Update search list capacity (defaults to 128). ### Request Example ```rust use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode}; // Create a new database with full configuration let db = CoreNN::create("/path/to/db", Cfg { dim: 768, metric: StdMetric::Cosine, max_edges: 64, max_add_edges: 64, beam_width: 4, distance_threshold: 1.1, compression_mode: CompressionMode::PQ, compression_threshold: 10_000_000, compaction_threshold_deletes: 1_000_000, pq_subspaces: 64, pq_sample_size: 10_000, trunc_dims: 64, query_search_list_cap: 128, update_search_list_cap: 128, }); // Open existing database let db = CoreNN::open("/path/to/db"); // Create in-memory database let db = CoreNN::new_in_memory(Cfg { dim: 128, ..Default::default() }); ``` ### Response #### Success Response (200) - **CoreNN instance** - A handle to the created or opened database. #### Response Example ```rust // Access configuration let cfg = db.cfg(); println!("Vector dimension: {}", cfg.dim); ``` ``` -------------------------------- ### Batch Insert and Query with Python Bindings Source: https://context7.com/wilsonzlin/corenn/llms.txt Demonstrates parallel insertion and querying of vectors using Python bindings for CoreNN. Leverages numpy arrays for data and internally uses Rayon for parallelization. Supports various data types and error handling. ```python from corenn_py import CoreNN import numpy as np # Create database db = CoreNN.create("/path/to/db", { "dim": 768, "metric": "cosine", # "l2" or "cosine" "max_edges": 64, "compression_threshold": 10_000_000, "compression_mode": "pq", # "pq" or "trunc" }) # Batch insert vectors (parallelized internally with Rayon) keys = [f"doc_{i:06d}" for i in range(10000)] vectors = np.random.randn(10000, 768).astype(np.float32) # Normalize for cosine similarity (recommended) vectors = vectors / np.linalg.norm(vectors, axis=1, keepdims=True) db.insert_f32(keys, vectors) # Other insert methods: insert_bf16, insert_f16, insert_f64 # Insert half-precision vectors for memory efficiency vectors_f16 = vectors.astype(np.float16) db.insert_f16(keys, vectors_f16) # Later: open and query db = CoreNN.open("/path/to/db") # Batch query (parallelized) query_vectors = np.random.randn(100, 768).astype(np.float32) query_vectors = query_vectors / np.linalg.norm(query_vectors, axis=1, keepdims=True) k = 50 results = db.query_f32(query_vectors, k) # Results is a list of lists: results[i] contains top-k for query i # Each result is a list of (key, distance) tuples for i, query_results in enumerate(results): print(f"Query {i}:") for rank, (key, distance) in enumerate(query_results[:5]): print(f" {rank + 1}. {key} (distance: {distance:.6f})") # Error handling try: db.insert_f32(["invalid"], np.array([[1.0, 2.0]])) # Wrong dimension except Exception as e: print(f"Error: {e}") ``` -------------------------------- ### Migrate HNSW Index to CoreNN Programmatically (Rust) Source: https://context7.com/wilsonzlin/corenn/llms.txt This Rust function provides a programmatic way to migrate an HNSW index to the CoreNN format. It uses the `hnswlib-rs` and `libcorenn` crates to load an HNSW index and create a new CoreNN database, then transfers the vectors. It requires paths for the HNSW index and output database, vector dimension, and the similarity metric. ```rust // Programmatic HNSW migration use hnswlib_rs::HnswIndex; use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric}; use std::path::Path; fn migrate_hnsw_to_corenn( hnsw_path: impl AsRef, output_path: impl AsRef, dim: usize, metric: StdMetric, ) -> Result<(), Box> { // Load HNSW index let hnsw = HnswIndex::load(hnsw_path)?; // Create CoreNN database let db = CoreNN::create(output_path, Cfg { dim, metric, ..Default::default() }); // Migrate vectors for i in 0..hnsw.len() { let key = format!("vec_{}", i); let vector = hnsw.get_vector(i)?; db.insert(&key, &vector); } Ok(()) } ``` -------------------------------- ### CoreNN Advanced Configuration and Optimization (Rust) Source: https://context7.com/wilsonzlin/corenn/llms.txt This Rust code demonstrates how to configure CoreNN databases for different performance characteristics: high-throughput, memory-optimized, and latency-optimized. It shows how to adjust parameters like `max_edges`, `beam_width`, compression settings, and more to fine-tune the database for specific workloads. It utilizes the `libcorenn` crate. ```rust use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode}; // High-throughput configuration (more edges, wider search) let high_throughput_cfg = Cfg { dim: 768, metric: StdMetric::Cosine, max_edges: 128, // More edges = better recall max_add_edges: 128, beam_width: 8, // Wider beam = better recall, slower search query_search_list_cap: 256, // Larger search list = better recall distance_threshold: 1.2, // Higher threshold = more edges pruned compression_threshold: 5_000_000, // Compress earlier for memory efficiency compression_mode: CompressionMode::PQ, pq_subspaces: 96, // More subspaces = better compression quality ..Default::default() }; // Memory-optimized configuration (fewer edges, aggressive compression) let memory_optimized_cfg = Cfg { dim: 1536, metric: StdMetric::L2, max_edges: 32, // Fewer edges = less memory max_add_edges: 32, beam_width: 2, // Narrower beam = less memory during search query_search_list_cap: 64, // Smaller search list = less memory compression_threshold: 1_000_000, // Compress early compression_mode: CompressionMode::Trunc, trunc_dims: 256, // Aggressive truncation for Matryoshka compaction_threshold_deletes: 100_000, // Compact more frequently ..Default::default() }; // Latency-optimized configuration (fast queries) let latency_optimized_cfg = Cfg { dim: 384, metric: StdMetric::Cosine, max_edges: 48, beam_width: 3, // Balanced beam width query_search_list_cap: 96, distance_threshold: 1.05, // Lower threshold = fewer edges = faster search compression_threshold: 50_000_000, // Delay compression ..Default::default() }; // Create databases with different configurations let high_throughput_db = CoreNN::create("/data/high_throughput", high_throughput_cfg); let memory_optimized_db = CoreNN::create("/data/memory_optimized", memory_optimized_cfg); let latency_optimized_db = CoreNN::create("/data/latency_optimized", latency_optimized_cfg); ``` -------------------------------- ### Create, Insert, and Query Vector Database (Python) Source: https://github.com/wilsonzlin/corenn/blob/master/README.md Shows how to initialize a CoreNN database in Python, insert multiple keys with corresponding float32 vectors, and perform nearest neighbor searches. Uses the corenn_py library and numpy. Supports inserting various float types. ```python from corenn_py import CoreNN db = CoreNN.create("/path/to/db", { # Specify the dimensionality of your vectors. "dim": 3, # All other config options are optional. }) keys = [ "my_entry_1", "my_entry_2", ] vectors = np.array([ [0.3, 0.6, 0.9], [0.4, 1.1, 0.0], ]) # Or insert_bf16, insert_f16, insert_f64 db.insert_f32(keys, vectors) # Later... db = CoreNN.open("/path/to/db") queries = np.array([ [1.0, 1.3, 1.7], [7.3, 2.5, 0.0], ]) # Returns a list of (key, distance) tuples # for each query vector # (so returns a list of lists). k100 = db.query_f32(queries, 100) ``` -------------------------------- ### Migrate HNSW Index to CoreNN Format (Bash) Source: https://context7.com/wilsonzlin/corenn/llms.txt This bash command migrates an existing HNSW index file to the CoreNN database format. It requires the path to the HNSW index, an output directory for the CoreNN database, the vector dimension, and the similarity metric (L2 or Cosine). It preserves the graph structure and vectors. ```bash # Migrate HNSW index to CoreNN corenn migrate-hnsw /path/to/hnsw_index.bin \ --out /path/to/corenn_db \ --dim 768 \ --metric L2 # Supported metrics: L2, Cosine # Reads binary HNSW format and converts to CoreNN database # Preserves graph structure and vectors # After migration, open and use normally corenn serve /path/to/corenn_db --port 4224 ``` -------------------------------- ### Vector Insertion with CoreNN (Rust) Source: https://context7.com/wilsonzlin/corenn/llms.txt Shows how to insert vectors into a CoreNN database. It supports automatic type conversion for common floating-point formats (bf16, f16, f32, f64) and demonstrates thread-safe concurrent insertions using Rust's threading capabilities. Explicit control over vector precision using `VecData` is also illustrated. ```rust use libcorenn::CoreNN; let db = CoreNN::create("/path/to/db", Cfg { dim: 3, ..Default::default() }); // Insert with automatic type conversion (f32 in this case) let key1 = "document_001".to_string(); let vec1 = vec![0.3, 0.6, 0.9]; db.insert(&key1, &vec1); // Insert multiple vectors (concurrent insertions are thread-safe) use std::thread; let db_clone = db.clone(); // CoreNN is Clone + Send + Sync let handles: Vec<_> = (0..100).map(|i| { let db = db.clone(); thread::spawn(move || { let key = format!("doc_{{:04}}", i); let vec = vec![i as f32 * 0.1, i as f32 * 0.2, i as f32 * 0.3]; db.insert(&key, &vec); }) }).collect(); for handle in handles { handle.join().unwrap(); } // Insert with explicit VecData type for precision control use libcorenn::vec::VecData; use half::f16; let vec_f16 = VecData::F16(vec![ f16::from_f32(0.1), f16::from_f32(0.2), f16::from_f32(0.3), ]); db.insert_vec(&"doc_f16".to_string(), vec_f16); ``` -------------------------------- ### CoreNN Query Evaluation and Benchmarking Source: https://context7.com/wilsonzlin/corenn/llms.txt Evaluates query performance against ground truth data and measures recall metrics using the `corenn eval` command. Supports various vector file formats and outputs results for each query, detailing the indices of the nearest neighbors found. ```bash # Run evaluation on test queries corenn eval /path/to/db \ --queries queries.fvecs \ --k 100 \ --results output_results.txt \ --vectors vectors.fvecs \ --dim 768 \ --dtype F32 # File formats supported: # .fvecs - Float vector format (4-byte header + floats) # .bvecs - Byte vector format (4-byte header + bytes) # .ivecs - Integer vector format (4-byte header + ints) # Output format (one line per query): # query_idx: result_1_idx result_2_idx result_3_idx ... result_k_idx # 0: 1234 5678 9012 ... # 1: 3456 7890 1234 ... ``` -------------------------------- ### Node.js Bindings - TypeScript Integration Source: https://context7.com/wilsonzlin/corenn/llms.txt Integrate CoreNN into your Node.js or TypeScript projects for type-safe and performant vector operations using Neon FFI bindings. ```APIDOC ## Node.js Bindings - TypeScript Integration ### Description Type-safe vector operations in Node.js/TypeScript with native performance through Neon FFI bindings. ### Usage ```typescript import { CoreNN, Cfg } from "@corenn/node"; // Configuration for database creation const cfg: Cfg = { dim: 384, metric: "l2", // Type: "l2" | "cosine" maxEdges: 64, compressionThreshold: 5_000_000, compressionMode: "trunc", // Type: "pq" | "trunc" truncDims: 128, // For Matryoshka embeddings }; // Create a new database const db = CoreNN.create("/path/to/db", cfg); // Batch insert vectors with TypedArray for performance const entries = []; for (let i = 0; i < 1000; i++) { const vector = new Float32Array(384); // Initialize vector with random data for (let j = 0; j < 384; j++) { vector[j] = Math.random(); } entries.push({ key: `document_${i.toString().padStart(6, '0')}`, vector: vector, }); } db.insert(entries); // Float64Array is also supported for higher precision const highPrecisionEntry = { key: "high_precision_doc", vector: new Float64Array([0.1, 0.2, 0.3 /* ... */]), }; db.insert([highPrecisionEntry]); // Open an existing database const openedDb = CoreNN.open("/path/to/db"); // Query nearest neighbors const queryVector = new Float32Array(384); // Initialize query vector with random data for (let i = 0; i < 384; i++) { queryVector[i] = Math.random(); } const k = 100; const results = openedDb.query(queryVector, k); // Results format: Array<{ key: string; distance: number }> results.forEach((result, rank) => { console.log(`${rank + 1}. ${result.key} (distance: ${result.distance.toFixed(6)})`); }); // Error handling example try { const wrongDimVector = new Float32Array(128); // Incorrect dimension openedDb.query(wrongDimVector, 10); } catch (error) { console.error("Query failed:", error); } // Async wrapper for non-blocking queries async function performQuery(queryVec: Float32Array) { return new Promise((resolve, reject) => { try { const results = openedDb.query(queryVec, 50); resolve(results); } catch (error) { reject(error); } }); } ``` ``` -------------------------------- ### Create, Insert, and Query Vector Database (Rust) Source: https://github.com/wilsonzlin/corenn/blob/master/README.md Demonstrates how to create a new CoreNN database, insert a key-value pair with a vector, and then query the database for the nearest neighbors. Requires the CoreNN library for Rust. Inserts vectors of a specified dimensionality. ```rust fn main() { let db = CoreNN::create("/path/to/db", Cfg { // Specify the dimensionality of your vectors. dim: 3, // All other config options are optional. ...Default::default() }); let key = "my_entry".to_string(); // This can be bf16, f16, f32, or f64. let vec = vec![0.3, 0.6, 0.9]; db.insert(&key, &vec); // Later... let db = CoreNN::open("/path/to/db"); let query = vec![1.0, 1.3, 1.7]; // Returns Vec of (key, distance) pairs. let k100 = db.query(&query, 100); assert_eq!(k100[0].0.as_str(), "my_entry"); } ``` -------------------------------- ### Query Evaluation and Benchmarking Source: https://context7.com/wilsonzlin/corenn/llms.txt Evaluate query performance against ground truth data and measure recall metrics for your vector datasets. ```APIDOC ## Query Evaluation and Benchmarking ### Description Evaluate query performance with ground truth data and measure recall metrics. ### Run Evaluation ```bash # Run evaluation on test queries, specifying database, query file, k, results file, vectors file, dimension, and data type corenn eval /path/to/db \ --queries queries.fvecs \ --k 100 \ --results output_results.txt \ --vectors vectors.fvecs \ --dim 768 \ --dtype F32 ``` ### Supported File Formats - **.fvecs**: Float vector format (4-byte header + floats). - **.bvecs**: Byte vector format (4-byte header + bytes). - **.ivecs**: Integer vector format (4-byte header + ints). ### Output Format The output file will contain one line per query, listing the indices of the top-k nearest neighbors. ``` query_idx: result_1_idx result_2_idx result_3_idx ... result_k_idx 0: 1234 5678 9012 ... 1: 3456 7890 1234 ... ``` ``` -------------------------------- ### Vector Insertion (Rust) Source: https://context7.com/wilsonzlin/corenn/llms.txt Shows how to insert vectors into the CoreNN database. It supports automatic type conversion for common floating-point types (bf16, f16, f32, f64) and thread-safe concurrent insertions. ```APIDOC ## Vector Insertion (Rust) ### Description Insert vectors with automatic type conversion and thread-safe concurrent access. Supports bf16, f16, f32, and f64 precision formats. ### Method `db.insert`, `db.insert_vec` ### Endpoint N/A (Rust library functions) ### Parameters #### `insert` Method - **key** (&str) - Required - A unique identifier for the vector. - **vector** (&[f32] or &[f16] etc.) - Required - The vector data. Automatic type conversion attempts to match the database's `dim`. #### `insert_vec` Method - **key** (&str) - Required - A unique identifier for the vector. - **vector_data** (VecData) - Required - The vector data, explicitly typed using `VecData` enum for precision control (e.g., `VecData::F32`, `VecData::F16`). ### Request Example ```rust use libcorenn::CoreNN; use libcorenn::vec::VecData; use half::f16; // Assume db is an initialized CoreNN instance // let db = CoreNN::create(...); // Insert with automatic type conversion (f32) let key1 = "document_001".to_string(); let vec1 = vec![0.3, 0.6, 0.9]; db.insert(&key1, &vec1); // Insert multiple vectors concurrently (thread-safe) use std::thread; let db_clone = db.clone(); let handles: Vec<_> = (0..100).map(|i| { let db = db_clone.clone(); thread::spawn(move || { let key = format!("doc_{:04}", i); let vec = vec![i as f32 * 0.1, i as f32 * 0.2, i as f32 * 0.3]; db.insert(&key, &vec); }) }).collect(); for handle in handles { handle.join().unwrap(); } // Insert with explicit VecData type for precision control (f16) let vec_f16 = VecData::F16(vec![ f16::from_f32(0.1), f16::from_f32(0.2), f16::from_f32(0.3), ]); db.insert_vec(&"doc_f16".to_string(), vec_f16); ``` ### Response #### Success Response (200) - **None** - The operation is usually fire-and-forget upon successful insertion. #### Response Example ```rust // No explicit response body for insert operations. // Errors would be handled via Rust's Result type or panics. println!("Vector inserted successfully."); ``` ``` -------------------------------- ### Load Exported Vectors from CoreNN Database (Python) Source: https://context7.com/wilsonzlin/corenn/llms.txt Loads vectors from a specified file path into a Python dictionary. Each line in the file is expected to contain a key followed by floating-point vector components. Dependencies include the numpy library. ```python import numpy as np def load_exported_vectors(filepath): """Load vectors exported from CoreNN database.""" vectors = {} with open(filepath) as f: for line in f: parts = line.strip().split() key = parts[0] vector = np.array([float(x) for x in parts[1:]], dtype=np.float32) vectors[key] = vector return vectors vectors = load_exported_vectors("vectors.txt") print(f"Loaded {len(vectors)} vectors") print(f"Vector dimension: {len(next(iter(vectors.values())))}") ``` -------------------------------- ### HTTP REST API Server Source: https://context7.com/wilsonzlin/corenn/llms.txt Run CoreNN as a standalone HTTP service, exposing vector operations through JSON endpoints for remote access. ```APIDOC ## HTTP REST API Server ### Description Run CoreNN as a standalone HTTP service with JSON endpoints for remote vector operations. ### Start Server ```bash # Start server with specified database path, port, address, and data type corenn serve /path/to/db --port 4224 --addr 127.0.0.1 --dtype F32 # Available dtypes: BF16, F16, F32, F64 ``` ### Endpoints - **GET /healthz**: Health check endpoint. Returns "OK" on success. - **POST /insert**: Inserts vectors into the database. Accepts JSON payload with a list of vectors. - **POST /query**: Queries for nearest neighbors. Accepts JSON payload with a query vector and number of neighbors (k). ### Request Examples #### Health Check ```bash curl http://localhost:4224/healthz # Response: OK ``` #### Insert Vectors ```bash curl -X POST http://localhost:4224/insert \ -H "Content-Type: application/json" \ -d '{ "vectors": [ { "key": "doc_001", "vector": [0.1, 0.2, 0.3, 0.4, 0.5] }, { "key": "doc_002", "vector": [0.6, 0.7, 0.8, 0.9, 1.0] } ] }' ``` #### Query Nearest Neighbors ```bash curl -X POST http://localhost:4224/query \ -H "Content-Type: application/json" \ -d '{ "vector": [0.5, 0.5, 0.5, 0.5, 0.5], "k": 10 }' # Response Example: # { # "results": [ # ["doc_002", 0.31622776], # ["doc_001", 0.83666003] # ] # } ``` ### Python Client Example ```python import requests import numpy as np BASE_URL = "http://localhost:4224" def insert_vectors(vectors_dict): response = requests.post( f"{BASE_URL}/insert", json={"vectors": [ {"key": k, "vector": v.tolist()} for k, v in vectors_dict.items() ]} ) response.raise_for_status() def query_vectors(query_vec, k=10): response = requests.post( f"{BASE_URL}/query", json={"vector": query_vec.tolist(), "k": k} ) response.raise_for_status() return response.json()["results"] # Usage Example vectors = { f"doc_{i}": np.random.randn(768).astype(np.float32) for i in range(100) } insert_vectors(vectors) query = np.random.randn(768).astype(np.float32) results = query_vectors(query, k=20) print(f"Top result: {results[0][0]} at distance {results[0][1]}") ``` ``` -------------------------------- ### Vector Querying in Rust Source: https://context7.com/wilsonzlin/corenn/llms.txt Queries k-nearest neighbors from a CoreNN database using a specified metric for distance calculation. Returns a sorted list of (key, distance) pairs. Supports single and batch queries with parallelization. ```rust use libcorenn::CoreNN; let db = CoreNN::open("/path/to/db"); // Query top-100 nearest neighbors let query = vec![1.0, 1.3, 1.7]; let results = db.query(&query, 100); // Results are sorted by distance (ascending) for (i, (key, distance)) in results.iter().enumerate() { println!("Rank {}: {} (distance: {:.6})", i + 1, key, distance); } // Verify top result assert_eq!(results[0].0.as_str(), "document_001"); assert!(results[0].1 < 1.0); // Distance less than 1.0 // Batch queries using multiple threads use rayon::prelude::*; let queries = vec![ vec![1.0, 2.0, 3.0], vec![4.0, 5.0, 6.0], vec![7.0, 8.0, 9.0], ]; let all_results: Vec> = queries .par_iter() .map(|q| db.query(q, 10)) .collect(); for (query_idx, results) in all_results.iter().enumerate() { println!("Query {}: {} results", query_idx, results.len()); } ``` -------------------------------- ### Export Vectors from CoreNN Database (Bash) Source: https://context7.com/wilsonzlin/corenn/llms.txt This bash command exports all vectors from a CoreNN database to a text file. The output format is a list of key-value pairs, where each line contains a vector key followed by its dimensions. This is useful for backing up data or performing external analysis. ```bash # Export all vectors from database corenn export-vectors /path/to/db --output vectors.txt # Output format (text file): # key1 vector_dim1 vector_dim2 vector_dim3 ... # key2 vector_dim1 vector_dim2 vector_dim3 ... ``` -------------------------------- ### Node.js Bindings - TypeScript Vector Operations Source: https://context7.com/wilsonzlin/corenn/llms.txt Integrates type-safe vector operations into Node.js and TypeScript applications using Neon FFI bindings for native performance. Supports creating, inserting, and querying vector databases with various TypedArray types and includes error handling and an async wrapper. ```typescript import { CoreNN, Cfg } from "@corenn/node"; // Create database with full type safety const cfg: Cfg = { dim: 384, metric: "l2", // Type: "l2" | "cosine" maxEdges: 64, compressionThreshold: 5_000_000, compressionMode: "trunc", // Type: "pq" | "trunc" truncDims: 128, // For Matryoshka embeddings }; const db = CoreNN.create("/path/to/db", cfg); // Batch insert with TypedArray for performance const entries = []; for (let i = 0; i < 1000; i++) { const vector = new Float32Array(384); for (let j = 0; j < 384; j++) { vector[j] = Math.random(); } entries.push({ key: `document_${i.toString().padStart(6, '0')}`, vector: vector, }); } db.insert(entries); // Float64Array also supported const highPrecisionEntry = { key: "high_precision_doc", vector: new Float64Array([0.1, 0.2, 0.3, /* ... */]), }; db.insert([highPrecisionEntry]); // Later: open and query const db = CoreNN.open("/path/to/db"); // Query with TypedArray const queryVector = new Float32Array(384); for (let i = 0; i < 384; i++) { queryVector[i] = Math.random(); } const k = 100; const results = db.query(queryVector, k); // Results type: Array<{ key: string; distance: number }> results.forEach((result, rank) => { console.log(`${rank + 1}. ${result.key} (distance: ${result.distance.toFixed(6)})`); }); // Error handling with try-catch try { const wrongDimVector = new Float32Array(128); // Wrong dimension db.query(wrongDimVector, 10); } catch (error) { console.error("Query failed:", error); } // Async wrapper for non-blocking operations async function performQuery(queryVec: Float32Array) { return new Promise((resolve, reject) => { try { const results = db.query(queryVec, 50); resolve(results); } catch (error) { reject(error); } }); } ``` -------------------------------- ### Python Client for CoreNN HTTP API Source: https://context7.com/wilsonzlin/corenn/llms.txt Provides Python functions to interact with the CoreNN HTTP REST API server for inserting and querying vectors. Uses the `requests` library and `numpy` for vector manipulation, converting numpy arrays to lists for JSON serialization. ```python # Python client example import requests import numpy as np BASE_URL = "http://localhost:4224" # Insert vectors def insert_vectors(vectors_dict): response = requests.post( f"{BASE_URL}/insert", json={"vectors": [ {"key": k, "vector": v.tolist()} for k, v in vectors_dict.items() ]} ) response.raise_for_status() # Query def query_vectors(query_vec, k=10): response = requests.post( f"{BASE_URL}/query", json={"vector": query_vec.tolist(), "k": k} ) response.raise_for_status() return response.json()["results"] # Usage vectors = { f"doc_{i}": np.random.randn(768).astype(np.float32) for i in range(100) } insert_vectors(vectors) query = np.random.randn(768).astype(np.float32) results = query_vectors(query, k=20) print(f"Top result: {results[0][0]} at distance {results[0][1]}") ``` -------------------------------- ### Vector Deletion and Compaction in Rust Source: https://context7.com/wilsonzlin/corenn/llms.txt Deletes vectors from a CoreNN database by their keys and triggers automatic compaction when a deletion threshold is met. Compaction reclaims space and maintains graph quality. Manual access to internal storage is also shown. ```rust use libcorenn::CoreNN; let db = CoreNN::open("/path/to/db"); // Delete single vector db.delete(&"document_001".to_string()); // Bulk deletion let keys_to_delete = vec!["doc_0010", "doc_0020", "doc_0030"]; for key in keys_to_delete { db.delete(&key.to_string()); } // Compaction is triggered automatically when deletions exceed threshold // (default: 1,000,000 deletions) // During compaction: // - Edges to deleted nodes are removed from graph // - Deleted nodes are permanently removed from storage // - Graph quality is maintained by re-pruning edges // - Caches are cleared for freed nodes // Manual access to internal database (advanced usage) use libcorenn::store::Store; let internal_db = db.internal_db(); // Access low-level storage operations ``` -------------------------------- ### Calculate Recall from Evaluation Results (Python) Source: https://context7.com/wilsonzlin/corenn/llms.txt This Python function calculates recall@k for given predicted and ground truth files. It reads data from two files, computes the intersection of predicted items with the top-k ground truth items, and outputs the mean recall for specified k values. It depends on the numpy library. ```python import numpy as np def calculate_recall(predicted_file, ground_truth_file, k_values=[1, 10, 100]): """Calculate recall@k for evaluation results.""" predicted = [] with open(predicted_file) as f: for line in f: parts = line.strip().split(': ') ids = [int(x) for x in parts[1].split()] predicted.append(ids) ground_truth = [] with open(ground_truth_file) as f: for line in f: parts = line.strip().split(': ') ids = [int(x) for x in parts[1].split()] ground_truth.append(ids) recalls = {k: [] for k in k_values} for pred, truth in zip(predicted, ground_truth): pred_set = set(pred) for k in k_values: truth_k = set(truth[:k]) recall = len(pred_set.intersection(truth_k)) / k recalls[k].append(recall) for k, values in recalls.items(): print(f"Recall@{k}: {np.mean(values):.4f}") calculate_recall("output_results.txt", "ground_truth.txt") ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.