CoreNN

https://github.com/wilsonzlin/corenn
Admin
CoreNN is a database designed for querying billions of vectors and embeddings in sublinear time on...
Tokens:10,930
Snippets:28
Trust Score:8.8
Update:11 months ago
Show doc for...
Tokens:
Raw
### HTTP REST API Server - CoreNN Standalone Service

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Runs CoreNN as a standalone HTTP service, exposing JSON endpoints for remote vector operations. Supports starting the server with specified database path, port, address, and data type, and provides endpoints for health checks, insertion, and querying.

```bash
# Start server
corenn serve /path/to/db --port 4224 --addr 127.0.0.1 --dtype F32

# Available dtypes: BF16, F16, F32, F64
# Server provides three endpoints:
#   GET  /healthz - Health check (returns "OK")
#   POST /insert  - Insert vectors
#   POST /query   - Query nearest neighbors

```

```bash
# Health check
curl http://localhost:4224/healthz
# Response: OK

# Insert vectors (parallel processing)
curl -X POST http://localhost:4224/insert \
  -H "Content-Type: application/json" \
  -d '{ 
    "vectors": [
      {
        "key": "doc_001",
        "vector": [0.1, 0.2, 0.3, 0.4, 0.5]
      },
      {
        "key": "doc_002",
        "vector": [0.6, 0.7, 0.8, 0.9, 1.0]
      }
    ]
  }'

# Query nearest neighbors
curl -X POST http://localhost:4224/query \
  -H "Content-Type: application/json" \
  -d '{ 
    "vector": [0.5, 0.5, 0.5, 0.5, 0.5],
    "k": 10
  }'

# Response:
# { 
#   "results": [
#     ["doc_002", 0.31622776],
#     ["doc_001", 0.83666003]
#   ]
# }

```

--------------------------------

### Create, Insert, and Query Vector Database (Node.js)

Source: https://github.com/wilsonzlin/corenn/blob/master/README.md

Provides an example of using CoreNN in Node.js to set up a database, insert data with Float32Array vectors, and execute similarity queries. Leverages the @corenn/node package. Queries return an array of objects containing keys and distances.

```typescript
import { CoreNN } from "@corenn/node";

const db = CoreNN.create("/path/to/db", {
  // Specify the dimensionality of your vectors.
  dim: 3,
  // All other config options are optional.
});
db.insert([
  {
    key: "my_entry_1",
    vector: new Float32Array([0.5, 4.1, 2.2]),
  },
  {
    key: "my_entry_2",
    vector: new Float32Array([3.1, 7.7, 6.4]),
  },
]);

// Later...
const db = CoreNN.open("/path/to/db");
// Array of { key, distance } objects.
const results = db.query(
  new Float32Array([0.0, 1.1, 2.2]),
  100,
);
```

--------------------------------

### CoreNN Vector Data Types and Precision Handling (Rust)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Demonstrates the usage of different floating-point data types (BF16, F16, F32, F64) for vectors in Rust using the `libcorenn` library. Shows how to create, get dimensions, convert between types, and access raw bytes for serialization. Dependencies include `libcorenn`, `half`, and `ndarray`.

```rust
use libcorenn::vec::VecData;
use half::{bf16, f16};
use ndarray::Array1;

// Create vectors with different precision levels
let vec_bf16 = VecData::BF16(vec![
    bf16::from_f32(0.1),
    bf16::from_f32(0.2),
    bf16::from_f32(0.3),
]);

let vec_f16 = VecData::F16(vec![
    f16::from_f32(0.1),
    f16::from_f32(0.2),
    f16::from_f32(0.3),
]);

let vec_f32 = VecData::F32(vec![0.1, 0.2, 0.3]);

let vec_f64 = VecData::F64(vec![0.1, 0.2, 0.3]);

// Get dimensionality
assert_eq!(vec_f32.dim(), 3);

// Convert to f32 for computation (consumes vector)
let array_f32: Array1<f32> = vec_f32.into_f32();

// Convert to f32 without consuming (creates copy)
let vec_f16_copy = VecData::F16(vec![f16::from_f32(0.5); 128]);
let array_f32: Array1<f32> = vec_f16_copy.to_f32();

// Access raw bytes (for serialization)
let vec = VecData::F32(vec![1.0, 2.0, 3.0]);
let bytes: &[u8] = vec.as_raw_bytes();
assert_eq!(bytes.len(), 3 * 4);  // 3 floats × 4 bytes

// Memory efficiency comparison:
// - bf16: 2 bytes per dimension (~50% of f32)
// - f16:  2 bytes per dimension (~50% of f32)
// - f32:  4 bytes per dimension (baseline)
// - f64:  8 bytes per dimension (2× f32)
```

--------------------------------

### Database Creation and Initialization with CoreNN

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Demonstrates how to create a new persistent vector database, open an existing one, or initialize an in-memory database using CoreNN. Configuration options include vector dimensionality, distance metric, compression settings, and graph parameters. The `Cfg` struct allows detailed customization.

```rust
use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode};

// Create a new database with full configuration
let db = CoreNN::create("/path/to/db", Cfg {
    dim: 768,                                    // Vector dimensionality (required)
    metric: StdMetric::Cosine,                   // Distance metric: L2 or Cosine
    max_edges: 64,                               // Maximum graph edges per node
    max_add_edges: 64,                           // Maximum additional edges during construction
    beam_width: 4,                               // Beam width for graph traversal
    distance_threshold: 1.1,                     // Edge pruning threshold
    compression_mode: CompressionMode::PQ,       // PQ or Trunc (Matryoshka)
    compression_threshold: 10_000_000,           // Auto-compress at N vectors
    compaction_threshold_deletes: 1_000_000,     // Compact after N deletions
    pq_subspaces: 64,                            // Product Quantization subspaces
    pq_sample_size: 10_000,                      // PQ training sample size
    trunc_dims: 64,                              // Truncation dimensions for Matryoshka
    query_search_list_cap: 128,                  // Query search list capacity
    update_search_list_cap: 128,                 // Update search list capacity
});

// Open existing database (config is read from disk)
let db = CoreNN::open("/path/to/db");

// Create in-memory database for testing
let db = CoreNN::new_in_memory(Cfg {
    dim: 128,
    ..Default::default()
});

// Access configuration
let cfg = db.cfg();
println!("Vector dimension: {}", cfg.dim);

```

--------------------------------

### Database Creation and Initialization

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Demonstrates how to create a new persistent vector database, open an existing one, or create an in-memory database for testing. Configuration options for dimensionality, distance metric, compression, and graph parameters are shown.

```APIDOC
## Database Creation and Initialization

### Description
Create a new persistent vector database or open an existing one with configuration options for dimensionality, distance metric, compression settings, and graph parameters.

### Method
`CoreNN::create`, `CoreNN::open`, `CoreNN::new_in_memory`

### Endpoint
N/A (Rust library functions)

### Parameters
#### Create/New In-Memory Configuration (`Cfg`)
- **dim** (usize) - Required - Vector dimensionality.
- **metric** (StdMetric) - Optional - Distance metric: `StdMetric::L2` or `StdMetric::Cosine` (defaults to Cosine).
- **max_edges** (usize) - Optional - Maximum graph edges per node (defaults to 64).
- **max_add_edges** (usize) - Optional - Maximum additional edges during construction (defaults to 64).
- **beam_width** (usize) - Optional - Beam width for graph traversal (defaults to 4).
- **distance_threshold** (f32) - Optional - Edge pruning threshold (defaults to 1.1).
- **compression_mode** (CompressionMode) - Optional - Compression mode: `CompressionMode::PQ` or `CompressionMode::Trunc` (defaults to PQ).
- **compression_threshold** (usize) - Optional - Auto-compress at N vectors (defaults to 10,000,000).
- **compaction_threshold_deletes** (usize) - Optional - Compact after N deletions (defaults to 1,000,000).
- **pq_subspaces** (usize) - Optional - Product Quantization subspaces (defaults to 64).
- **pq_sample_size** (usize) - Optional - PQ training sample size (defaults to 10,000).
- **trunc_dims** (usize) - Optional - Truncation dimensions for Matryoshka (defaults to 64).
- **query_search_list_cap** (usize) - Optional - Query search list capacity (defaults to 128).
- **update_search_list_cap** (usize) - Optional - Update search list capacity (defaults to 128).

### Request Example
```rust
use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode};

// Create a new database with full configuration
let db = CoreNN::create("/path/to/db", Cfg {
    dim: 768,
    metric: StdMetric::Cosine,
    max_edges: 64,
    max_add_edges: 64,
    beam_width: 4,
    distance_threshold: 1.1,
    compression_mode: CompressionMode::PQ,
    compression_threshold: 10_000_000,
    compaction_threshold_deletes: 1_000_000,
    pq_subspaces: 64,
    pq_sample_size: 10_000,
    trunc_dims: 64,
    query_search_list_cap: 128,
    update_search_list_cap: 128,
});

// Open existing database
let db = CoreNN::open("/path/to/db");

// Create in-memory database
let db = CoreNN::new_in_memory(Cfg {
    dim: 128,
    ..Default::default()
});
```

### Response
#### Success Response (200)
- **CoreNN instance** - A handle to the created or opened database.

#### Response Example
```rust
// Access configuration
let cfg = db.cfg();
println!("Vector dimension: {}", cfg.dim);
```
```

--------------------------------

### Batch Insert and Query with Python Bindings

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Demonstrates parallel insertion and querying of vectors using Python bindings for CoreNN. Leverages numpy arrays for data and internally uses Rayon for parallelization. Supports various data types and error handling.

```python
from corenn_py import CoreNN
import numpy as np

# Create database
db = CoreNN.create("/path/to/db", {
    "dim": 768,
    "metric": "cosine",  # "l2" or "cosine"
    "max_edges": 64,
    "compression_threshold": 10_000_000,
    "compression_mode": "pq",  # "pq" or "trunc"
})

# Batch insert vectors (parallelized internally with Rayon)
keys = [f"doc_{i:06d}" for i in range(10000)]
vectors = np.random.randn(10000, 768).astype(np.float32)

# Normalize for cosine similarity (recommended)
vectors = vectors / np.linalg.norm(vectors, axis=1, keepdims=True)

db.insert_f32(keys, vectors)

# Other insert methods: insert_bf16, insert_f16, insert_f64
# Insert half-precision vectors for memory efficiency
vectors_f16 = vectors.astype(np.float16)
db.insert_f16(keys, vectors_f16)

# Later: open and query
db = CoreNN.open("/path/to/db")

# Batch query (parallelized)
query_vectors = np.random.randn(100, 768).astype(np.float32)
query_vectors = query_vectors / np.linalg.norm(query_vectors, axis=1, keepdims=True)

k = 50
results = db.query_f32(query_vectors, k)

# Results is a list of lists: results[i] contains top-k for query i
# Each result is a list of (key, distance) tuples
for i, query_results in enumerate(results):
    print(f"Query {i}:")
    for rank, (key, distance) in enumerate(query_results[:5]):
        print(f"  {rank + 1}. {key} (distance: {distance:.6f})")

# Error handling
try:
    db.insert_f32(["invalid"], np.array([[1.0, 2.0]]))  # Wrong dimension
except Exception as e:
    print(f"Error: {e}")
```

--------------------------------

### Migrate HNSW Index to CoreNN Programmatically (Rust)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

This Rust function provides a programmatic way to migrate an HNSW index to the CoreNN format. It uses the `hnswlib-rs` and `libcorenn` crates to load an HNSW index and create a new CoreNN database, then transfers the vectors. It requires paths for the HNSW index and output database, vector dimension, and the similarity metric.

```rust
// Programmatic HNSW migration
use hnswlib_rs::HnswIndex;
use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric};
use std::path::Path;

fn migrate_hnsw_to_corenn(
    hnsw_path: impl AsRef<Path>,
    output_path: impl AsRef<Path>,
    dim: usize,
    metric: StdMetric,
) -> Result<(), Box<dyn std::error::Error>> {
    // Load HNSW index
    let hnsw = HnswIndex::load(hnsw_path)?;

    // Create CoreNN database
    let db = CoreNN::create(output_path, Cfg {
        dim,
        metric,
        ..Default::default()
    });

    // Migrate vectors
    for i in 0..hnsw.len() {
        let key = format!("vec_{}", i);
        let vector = hnsw.get_vector(i)?;
        db.insert(&key, &vector);
    }

    Ok(())
}
```

--------------------------------

### CoreNN Advanced Configuration and Optimization (Rust)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

This Rust code demonstrates how to configure CoreNN databases for different performance characteristics: high-throughput, memory-optimized, and latency-optimized. It shows how to adjust parameters like `max_edges`, `beam_width`, compression settings, and more to fine-tune the database for specific workloads. It utilizes the `libcorenn` crate.

```rust
use libcorenn::{CoreNN, cfg::Cfg, metric::StdMetric, cfg::CompressionMode};

// High-throughput configuration (more edges, wider search)
let high_throughput_cfg = Cfg {
    dim: 768,
    metric: StdMetric::Cosine,
    max_edges: 128,              // More edges = better recall
    max_add_edges: 128,
    beam_width: 8,               // Wider beam = better recall, slower search
    query_search_list_cap: 256,  // Larger search list = better recall
    distance_threshold: 1.2,     // Higher threshold = more edges pruned
    compression_threshold: 5_000_000,  // Compress earlier for memory efficiency
    compression_mode: CompressionMode::PQ,
    pq_subspaces: 96,            // More subspaces = better compression quality
    ..Default::default()
};

// Memory-optimized configuration (fewer edges, aggressive compression)
let memory_optimized_cfg = Cfg {
    dim: 1536,
    metric: StdMetric::L2,
    max_edges: 32,               // Fewer edges = less memory
    max_add_edges: 32,
    beam_width: 2,               // Narrower beam = less memory during search
    query_search_list_cap: 64,   // Smaller search list = less memory
    compression_threshold: 1_000_000,  // Compress early
    compression_mode: CompressionMode::Trunc,
    trunc_dims: 256,             // Aggressive truncation for Matryoshka
    compaction_threshold_deletes: 100_000,  // Compact more frequently
    ..Default::default()
};

// Latency-optimized configuration (fast queries)
let latency_optimized_cfg = Cfg {
    dim: 384,
    metric: StdMetric::Cosine,
    max_edges: 48,
    beam_width: 3,               // Balanced beam width
    query_search_list_cap: 96,
    distance_threshold: 1.05,    // Lower threshold = fewer edges = faster search
    compression_threshold: 50_000_000,  // Delay compression
    ..Default::default()
};

// Create databases with different configurations
let high_throughput_db = CoreNN::create("/data/high_throughput", high_throughput_cfg);
let memory_optimized_db = CoreNN::create("/data/memory_optimized", memory_optimized_cfg);
let latency_optimized_db = CoreNN::create("/data/latency_optimized", latency_optimized_cfg);
```

--------------------------------

### Create, Insert, and Query Vector Database (Python)

Source: https://github.com/wilsonzlin/corenn/blob/master/README.md

Shows how to initialize a CoreNN database in Python, insert multiple keys with corresponding float32 vectors, and perform nearest neighbor searches. Uses the corenn_py library and numpy. Supports inserting various float types.

```python
from corenn_py import CoreNN

db = CoreNN.create("/path/to/db", {
  # Specify the dimensionality of your vectors.
  "dim": 3,
  # All other config options are optional.
})
keys = [
  "my_entry_1",
  "my_entry_2",
]
vectors = np.array([
  [0.3, 0.6, 0.9],
  [0.4, 1.1, 0.0],
])
# Or insert_bf16, insert_f16, insert_f64
db.insert_f32(keys, vectors)

# Later...
db = CoreNN.open("/path/to/db")
queries = np.array([
  [1.0, 1.3, 1.7],
  [7.3, 2.5, 0.0],
])
# Returns a list of (key, distance) tuples
# for each query vector
# (so returns a list of lists).
k100 = db.query_f32(queries, 100)
```

--------------------------------

### Migrate HNSW Index to CoreNN Format (Bash)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

This bash command migrates an existing HNSW index file to the CoreNN database format. It requires the path to the HNSW index, an output directory for the CoreNN database, the vector dimension, and the similarity metric (L2 or Cosine). It preserves the graph structure and vectors.

```bash
# Migrate HNSW index to CoreNN
corenn migrate-hnsw /path/to/hnsw_index.bin \
  --out /path/to/corenn_db \
  --dim 768 \
  --metric L2

# Supported metrics: L2, Cosine
# Reads binary HNSW format and converts to CoreNN database
# Preserves graph structure and vectors

# After migration, open and use normally
corenn serve /path/to/corenn_db --port 4224
```

--------------------------------

### Vector Insertion with CoreNN (Rust)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Shows how to insert vectors into a CoreNN database. It supports automatic type conversion for common floating-point formats (bf16, f16, f32, f64) and demonstrates thread-safe concurrent insertions using Rust's threading capabilities. Explicit control over vector precision using `VecData` is also illustrated.

```rust
use libcorenn::CoreNN;

let db = CoreNN::create("/path/to/db", Cfg {
    dim: 3,
    ..Default::default()
});

// Insert with automatic type conversion (f32 in this case)
let key1 = "document_001".to_string();
let vec1 = vec![0.3, 0.6, 0.9];
db.insert(&key1, &vec1);

// Insert multiple vectors (concurrent insertions are thread-safe)
use std::thread;
let db_clone = db.clone(); // CoreNN is Clone + Send + Sync

let handles: Vec<_> = (0..100).map(|i| {
    let db = db.clone();
    thread::spawn(move || {
        let key = format!("doc_{{:04}}", i);
        let vec = vec![i as f32 * 0.1, i as f32 * 0.2, i as f32 * 0.3];
        db.insert(&key, &vec);
    })
}).collect();

for handle in handles {
    handle.join().unwrap();
}

// Insert with explicit VecData type for precision control
use libcorenn::vec::VecData;
use half::f16;

let vec_f16 = VecData::F16(vec![
    f16::from_f32(0.1),
    f16::from_f32(0.2),
    f16::from_f32(0.3),
]);
db.insert_vec(&"doc_f16".to_string(), vec_f16);

```

--------------------------------

### CoreNN Query Evaluation and Benchmarking

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Evaluates query performance against ground truth data and measures recall metrics using the `corenn eval` command. Supports various vector file formats and outputs results for each query, detailing the indices of the nearest neighbors found.

```bash
# Run evaluation on test queries
corenn eval /path/to/db \
  --queries queries.fvecs \
  --k 100 \
  --results output_results.txt \
  --vectors vectors.fvecs \
  --dim 768 \
  --dtype F32

# File formats supported:
#   .fvecs - Float vector format (4-byte header + floats)
#   .bvecs - Byte vector format (4-byte header + bytes)
#   .ivecs - Integer vector format (4-byte header + ints)

# Output format (one line per query):
# query_idx: result_1_idx result_2_idx result_3_idx ... result_k_idx
# 0: 1234 5678 9012 ...
# 1: 3456 7890 1234 ...

```

--------------------------------

### Node.js Bindings - TypeScript Integration

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Integrate CoreNN into your Node.js or TypeScript projects for type-safe and performant vector operations using Neon FFI bindings.

```APIDOC
## Node.js Bindings - TypeScript Integration

### Description
Type-safe vector operations in Node.js/TypeScript with native performance through Neon FFI bindings.

### Usage
```typescript
import { CoreNN, Cfg } from "@corenn/node";

// Configuration for database creation
const cfg: Cfg = {
    dim: 384,
    metric: "l2",  // Type: "l2" | "cosine"
    maxEdges: 64,
    compressionThreshold: 5_000_000,
    compressionMode: "trunc",  // Type: "pq" | "trunc"
    truncDims: 128,  // For Matryoshka embeddings
};

// Create a new database
const db = CoreNN.create("/path/to/db", cfg);

// Batch insert vectors with TypedArray for performance
const entries = [];
for (let i = 0; i < 1000; i++) {
    const vector = new Float32Array(384);
    // Initialize vector with random data
    for (let j = 0; j < 384; j++) {
        vector[j] = Math.random();
    }
    entries.push({
        key: `document_${i.toString().padStart(6, '0')}`,
        vector: vector,
    });
}
db.insert(entries);

// Float64Array is also supported for higher precision
const highPrecisionEntry = {
    key: "high_precision_doc",
    vector: new Float64Array([0.1, 0.2, 0.3 /* ... */]),
};
db.insert([highPrecisionEntry]);

// Open an existing database
const openedDb = CoreNN.open("/path/to/db");

// Query nearest neighbors
const queryVector = new Float32Array(384);
// Initialize query vector with random data
for (let i = 0; i < 384; i++) {
    queryVector[i] = Math.random();
}

const k = 100;
const results = openedDb.query(queryVector, k);

// Results format: Array<{ key: string; distance: number }>
results.forEach((result, rank) => {
    console.log(`${rank + 1}. ${result.key} (distance: ${result.distance.toFixed(6)})`);
});

// Error handling example
try {
    const wrongDimVector = new Float32Array(128);  // Incorrect dimension
    openedDb.query(wrongDimVector, 10);
} catch (error) {
    console.error("Query failed:", error);
}

// Async wrapper for non-blocking queries
async function performQuery(queryVec: Float32Array) {
    return new Promise((resolve, reject) => {
        try {
            const results = openedDb.query(queryVec, 50);
            resolve(results);
        } catch (error) {
            reject(error);
        }
    });
}
```
```

--------------------------------

### Create, Insert, and Query Vector Database (Rust)

Source: https://github.com/wilsonzlin/corenn/blob/master/README.md

Demonstrates how to create a new CoreNN database, insert a key-value pair with a vector, and then query the database for the nearest neighbors. Requires the CoreNN library for Rust. Inserts vectors of a specified dimensionality.

```rust
fn main() {
  let db = CoreNN::create("/path/to/db", Cfg {
    // Specify the dimensionality of your vectors.
    dim: 3,
    // All other config options are optional.
    ...Default::default()
  });
  let key = "my_entry".to_string();
  // This can be bf16, f16, f32, or f64.
  let vec = vec![0.3, 0.6, 0.9];
  db.insert(&key, &vec);

  // Later...
  let db = CoreNN::open("/path/to/db");
  let query = vec![1.0, 1.3, 1.7];
  // Returns Vec of (key, distance) pairs.
  let k100 = db.query(&query, 100);
  assert_eq!(k100[0].0.as_str(), "my_entry");
}
```

--------------------------------

### Query Evaluation and Benchmarking

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Evaluate query performance against ground truth data and measure recall metrics for your vector datasets.

```APIDOC
## Query Evaluation and Benchmarking

### Description
Evaluate query performance with ground truth data and measure recall metrics.

### Run Evaluation
```bash
# Run evaluation on test queries, specifying database, query file, k, results file, vectors file, dimension, and data type
corenn eval /path/to/db \
  --queries queries.fvecs \
  --k 100 \
  --results output_results.txt \
  --vectors vectors.fvecs \
  --dim 768 \
  --dtype F32
```

### Supported File Formats
- **.fvecs**: Float vector format (4-byte header + floats).
- **.bvecs**: Byte vector format (4-byte header + bytes).
- **.ivecs**: Integer vector format (4-byte header + ints).

### Output Format
The output file will contain one line per query, listing the indices of the top-k nearest neighbors.

```
query_idx: result_1_idx result_2_idx result_3_idx ... result_k_idx
0: 1234 5678 9012 ...
1: 3456 7890 1234 ...
```
```

--------------------------------

### Vector Insertion (Rust)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Shows how to insert vectors into the CoreNN database. It supports automatic type conversion for common floating-point types (bf16, f16, f32, f64) and thread-safe concurrent insertions.

```APIDOC
## Vector Insertion (Rust)

### Description
Insert vectors with automatic type conversion and thread-safe concurrent access. Supports bf16, f16, f32, and f64 precision formats.

### Method
`db.insert`, `db.insert_vec`

### Endpoint
N/A (Rust library functions)

### Parameters
#### `insert` Method
- **key** (&str) - Required - A unique identifier for the vector.
- **vector** (&[f32] or &[f16] etc.) - Required - The vector data. Automatic type conversion attempts to match the database's `dim`.

#### `insert_vec` Method
- **key** (&str) - Required - A unique identifier for the vector.
- **vector_data** (VecData) - Required - The vector data, explicitly typed using `VecData` enum for precision control (e.g., `VecData::F32`, `VecData::F16`).

### Request Example
```rust
use libcorenn::CoreNN;
use libcorenn::vec::VecData;
use half::f16;

// Assume db is an initialized CoreNN instance
// let db = CoreNN::create(...);

// Insert with automatic type conversion (f32)
let key1 = "document_001".to_string();
let vec1 = vec![0.3, 0.6, 0.9];
db.insert(&key1, &vec1);

// Insert multiple vectors concurrently (thread-safe)
use std::thread;
let db_clone = db.clone();

let handles: Vec<_> = (0..100).map(|i| {
    let db = db_clone.clone();
    thread::spawn(move || {
        let key = format!("doc_{:04}", i);
        let vec = vec![i as f32 * 0.1, i as f32 * 0.2, i as f32 * 0.3];
        db.insert(&key, &vec);
    })
}).collect();

for handle in handles {
    handle.join().unwrap();
}

// Insert with explicit VecData type for precision control (f16)
let vec_f16 = VecData::F16(vec![
    f16::from_f32(0.1),
    f16::from_f32(0.2),
    f16::from_f32(0.3),
]);
db.insert_vec(&"doc_f16".to_string(), vec_f16);
```

### Response
#### Success Response (200)
- **None** - The operation is usually fire-and-forget upon successful insertion.

#### Response Example
```rust
// No explicit response body for insert operations.
// Errors would be handled via Rust's Result type or panics.
println!("Vector inserted successfully.");
```
```

--------------------------------

### Load Exported Vectors from CoreNN Database (Python)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Loads vectors from a specified file path into a Python dictionary. Each line in the file is expected to contain a key followed by floating-point vector components. Dependencies include the numpy library.

```python
import numpy as np

def load_exported_vectors(filepath):
    """Load vectors exported from CoreNN database."""
    vectors = {}
    with open(filepath) as f:
        for line in f:
            parts = line.strip().split()
            key = parts[0]
            vector = np.array([float(x) for x in parts[1:]], dtype=np.float32)
            vectors[key] = vector
    return vectors

vectors = load_exported_vectors("vectors.txt")
print(f"Loaded {len(vectors)} vectors")
print(f"Vector dimension: {len(next(iter(vectors.values())))}")
```

--------------------------------

### HTTP REST API Server

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Run CoreNN as a standalone HTTP service, exposing vector operations through JSON endpoints for remote access.

```APIDOC
## HTTP REST API Server

### Description
Run CoreNN as a standalone HTTP service with JSON endpoints for remote vector operations.

### Start Server
```bash
# Start server with specified database path, port, address, and data type
corenn serve /path/to/db --port 4224 --addr 127.0.0.1 --dtype F32

# Available dtypes: BF16, F16, F32, F64
```

### Endpoints
- **GET /healthz**: Health check endpoint. Returns "OK" on success.
- **POST /insert**: Inserts vectors into the database. Accepts JSON payload with a list of vectors.
- **POST /query**: Queries for nearest neighbors. Accepts JSON payload with a query vector and number of neighbors (k).

### Request Examples

#### Health Check
```bash
curl http://localhost:4224/healthz
# Response: OK
```

#### Insert Vectors
```bash
curl -X POST http://localhost:4224/insert \
  -H "Content-Type: application/json" \
  -d '{ 
    "vectors": [
      {
        "key": "doc_001",
        "vector": [0.1, 0.2, 0.3, 0.4, 0.5]
      },
      {
        "key": "doc_002",
        "vector": [0.6, 0.7, 0.8, 0.9, 1.0]
      }
    ]
  }'
```

#### Query Nearest Neighbors
```bash
curl -X POST http://localhost:4224/query \
  -H "Content-Type: application/json" \
  -d '{
    "vector": [0.5, 0.5, 0.5, 0.5, 0.5],
    "k": 10
  }'

# Response Example:
# {
#   "results": [
#     ["doc_002", 0.31622776],
#     ["doc_001", 0.83666003]
#   ]
# }
```

### Python Client Example
```python
import requests
import numpy as np

BASE_URL = "http://localhost:4224"

def insert_vectors(vectors_dict):
    response = requests.post(
        f"{BASE_URL}/insert",
        json={"vectors": [
            {"key": k, "vector": v.tolist()} 
            for k, v in vectors_dict.items()
        ]}
    )
    response.raise_for_status()

def query_vectors(query_vec, k=10):
    response = requests.post(
        f"{BASE_URL}/query",
        json={"vector": query_vec.tolist(), "k": k}
    )
    response.raise_for_status()
    return response.json()["results"]

# Usage Example
vectors = {
    f"doc_{i}": np.random.randn(768).astype(np.float32)
    for i in range(100)
}
insert_vectors(vectors)

query = np.random.randn(768).astype(np.float32)
results = query_vectors(query, k=20)
print(f"Top result: {results[0][0]} at distance {results[0][1]}")
```
```

--------------------------------

### Vector Querying in Rust

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Queries k-nearest neighbors from a CoreNN database using a specified metric for distance calculation. Returns a sorted list of (key, distance) pairs. Supports single and batch queries with parallelization.

```rust
use libcorenn::CoreNN;

let db = CoreNN::open("/path/to/db");

// Query top-100 nearest neighbors
let query = vec![1.0, 1.3, 1.7];
let results = db.query(&query, 100);

// Results are sorted by distance (ascending)
for (i, (key, distance)) in results.iter().enumerate() {
    println!("Rank {}: {} (distance: {:.6})", i + 1, key, distance);
}

// Verify top result
assert_eq!(results[0].0.as_str(), "document_001");
assert!(results[0].1 < 1.0); // Distance less than 1.0

// Batch queries using multiple threads
use rayon::prelude::*;

let queries = vec![
    vec![1.0, 2.0, 3.0],
    vec![4.0, 5.0, 6.0],
    vec![7.0, 8.0, 9.0],
];

let all_results: Vec<Vec<(String, f64)>> = queries
    .par_iter()
    .map(|q| db.query(q, 10))
    .collect();

for (query_idx, results) in all_results.iter().enumerate() {
    println!("Query {}: {} results", query_idx, results.len());
}
```

--------------------------------

### Export Vectors from CoreNN Database (Bash)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

This bash command exports all vectors from a CoreNN database to a text file. The output format is a list of key-value pairs, where each line contains a vector key followed by its dimensions. This is useful for backing up data or performing external analysis.

```bash
# Export all vectors from database
corenn export-vectors /path/to/db --output vectors.txt

# Output format (text file):
# key1 vector_dim1 vector_dim2 vector_dim3 ...
# key2 vector_dim1 vector_dim2 vector_dim3 ...
```

--------------------------------

### Node.js Bindings - TypeScript Vector Operations

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Integrates type-safe vector operations into Node.js and TypeScript applications using Neon FFI bindings for native performance. Supports creating, inserting, and querying vector databases with various TypedArray types and includes error handling and an async wrapper.

```typescript
import { CoreNN, Cfg } from "@corenn/node";

// Create database with full type safety
const cfg: Cfg = {
    dim: 384,
    metric: "l2",  // Type: "l2" | "cosine"
    maxEdges: 64,
    compressionThreshold: 5_000_000,
    compressionMode: "trunc",  // Type: "pq" | "trunc"
    truncDims: 128,  // For Matryoshka embeddings
};

const db = CoreNN.create("/path/to/db", cfg);

// Batch insert with TypedArray for performance
const entries = [];
for (let i = 0; i < 1000; i++) {
    const vector = new Float32Array(384);
    for (let j = 0; j < 384; j++) {
        vector[j] = Math.random();
    }
    entries.push({
        key: `document_${i.toString().padStart(6, '0')}`,
        vector: vector,
    });
}

db.insert(entries);

// Float64Array also supported
const highPrecisionEntry = {
    key: "high_precision_doc",
    vector: new Float64Array([0.1, 0.2, 0.3, /* ... */]),
};
db.insert([highPrecisionEntry]);

// Later: open and query
const db = CoreNN.open("/path/to/db");

// Query with TypedArray
const queryVector = new Float32Array(384);
for (let i = 0; i < 384; i++) {
    queryVector[i] = Math.random();
}

const k = 100;
const results = db.query(queryVector, k);

// Results type: Array<{ key: string; distance: number }>
results.forEach((result, rank) => {
    console.log(`${rank + 1}. ${result.key} (distance: ${result.distance.toFixed(6)})`);
});

// Error handling with try-catch
try {
    const wrongDimVector = new Float32Array(128);  // Wrong dimension
    db.query(wrongDimVector, 10);
} catch (error) {
    console.error("Query failed:", error);
}

// Async wrapper for non-blocking operations
async function performQuery(queryVec: Float32Array) {
    return new Promise((resolve, reject) => {
        try {
            const results = db.query(queryVec, 50);
            resolve(results);
        } catch (error) {
            reject(error);
        }
    });
}

```

--------------------------------

### Python Client for CoreNN HTTP API

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Provides Python functions to interact with the CoreNN HTTP REST API server for inserting and querying vectors. Uses the `requests` library and `numpy` for vector manipulation, converting numpy arrays to lists for JSON serialization.

```python
# Python client example
import requests
import numpy as np

BASE_URL = "http://localhost:4224"

# Insert vectors
def insert_vectors(vectors_dict):
    response = requests.post(
        f"{BASE_URL}/insert",
        json={"vectors": [
            {"key": k, "vector": v.tolist()}
            for k, v in vectors_dict.items()
        ]}
    )
    response.raise_for_status()

# Query
def query_vectors(query_vec, k=10):
    response = requests.post(
        f"{BASE_URL}/query",
        json={"vector": query_vec.tolist(), "k": k}
    )
    response.raise_for_status()
    return response.json()["results"]

# Usage
vectors = {
    f"doc_{i}": np.random.randn(768).astype(np.float32)
    for i in range(100)
}
insert_vectors(vectors)

query = np.random.randn(768).astype(np.float32)
results = query_vectors(query, k=20)
print(f"Top result: {results[0][0]} at distance {results[0][1]}")

```

--------------------------------

### Vector Deletion and Compaction in Rust

Source: https://context7.com/wilsonzlin/corenn/llms.txt

Deletes vectors from a CoreNN database by their keys and triggers automatic compaction when a deletion threshold is met. Compaction reclaims space and maintains graph quality. Manual access to internal storage is also shown.

```rust
use libcorenn::CoreNN;

let db = CoreNN::open("/path/to/db");

// Delete single vector
db.delete(&"document_001".to_string());

// Bulk deletion
let keys_to_delete = vec!["doc_0010", "doc_0020", "doc_0030"];
for key in keys_to_delete {
    db.delete(&key.to_string());
}

// Compaction is triggered automatically when deletions exceed threshold
// (default: 1,000,000 deletions)
// During compaction:
// - Edges to deleted nodes are removed from graph
// - Deleted nodes are permanently removed from storage
// - Graph quality is maintained by re-pruning edges
// - Caches are cleared for freed nodes

// Manual access to internal database (advanced usage)
use libcorenn::store::Store;

let internal_db = db.internal_db();
// Access low-level storage operations
```

--------------------------------

### Calculate Recall from Evaluation Results (Python)

Source: https://context7.com/wilsonzlin/corenn/llms.txt

This Python function calculates recall@k for given predicted and ground truth files. It reads data from two files, computes the intersection of predicted items with the top-k ground truth items, and outputs the mean recall for specified k values. It depends on the numpy library.

```python
import numpy as np

def calculate_recall(predicted_file, ground_truth_file, k_values=[1, 10, 100]):
    """Calculate recall@k for evaluation results."""
    predicted = []
    with open(predicted_file) as f:
        for line in f:
            parts = line.strip().split(': ')
            ids = [int(x) for x in parts[1].split()]
            predicted.append(ids)

    ground_truth = []
    with open(ground_truth_file) as f:
        for line in f:
            parts = line.strip().split(': ')
            ids = [int(x) for x in parts[1].split()]
            ground_truth.append(ids)

    recalls = {k: [] for k in k_values}

    for pred, truth in zip(predicted, ground_truth):
        pred_set = set(pred)
        for k in k_values:
            truth_k = set(truth[:k])
            recall = len(pred_set.intersection(truth_k)) / k
            recalls[k].append(recall)

    for k, values in recalls.items():
        print(f"Recall@{k}: {np.mean(values):.4f}")

calculate_recall("output_results.txt", "ground_truth.txt")
```
Tokens:
Raw