NarrowDB

https://docs.rs/narrowdb/0.3.0
Admin
NarrowDB is a Rust database engine that provides SQL query support with columnar storage and...
Tokens:42,130
Snippets:238
Trust Score:9.1
Update:6 months ago
Show doc for...
Tokens:
Raw
### Open NarrowDB Database

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Opens an existing NarrowDB database or creates a new one at the specified path. Handles initialization of storage and tables, and optionally starts a background flush loop.

```rust
pub fn open(path: impl AsRef<Path>, options: DbOptions) -> Result<Self> {
        let auto_flush_interval = options.auto_flush_interval;
        let (storage, tables) = Storage::open(path, options)?;
        let inner = Arc::new(RwLock::new(DbInner { storage, tables }));

        let flush_handle = if let Some(interval) = auto_flush_interval {
            let weak = Arc::downgrade(&inner);
            let stop = Arc::new(AtomicBool::new(false));
            let stop_clone = stop.clone();
            let thread = std::thread::spawn(move || {
                background_flush_loop(weak, interval, stop_clone);
            });
            Some(FlushHandle {
                stop,
                thread: Some(thread),
            })
        } else {
            None
        };

        Ok(Self {
            inner,
            flush_handle: Mutex::new(flush_handle),
        })
    }
```

--------------------------------

### Get Database Path

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Retrieves the file system path of the NarrowDB database. Requires a read lock on the database inner state.

```rust
pub fn path(&self) -> Result<std::path::PathBuf> {
        let inner = self.inner.read().map_err(|_| anyhow!("lock poisoned"))?;
        Ok(inner.storage.path().to_path_buf())
    }
```

--------------------------------

### Execute SQL and Query Data in NarrowDB

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Demonstrates opening a database, executing SQL statements to create a table and insert data, and then querying the data using SQL. Includes assertions to verify results.

```rust
let path = std::env::temp_dir().join(format!("narrowdb-test-scalar-{}", SystemTime::now().duration_since(UNIX_EPOCH)?.as_nanos()));

        let db = NarrowDb::open(&path, DbOptions::default())?;
        db.execute_sql(
            "\n            CREATE TABLE logs (service TEXT, host TEXT, duration REAL);\n            INSERT INTO logs VALUES\n                ('api', 'a', 10.0),\n                ('api', 'b', 20.0),\n                ('worker', 'a', 30.0);\n            ",
        )?;

        let results = db.execute_sql(
            "SELECT host, service, COUNT(*) AS total FROM logs GROUP BY service, host ORDER BY host DESC LIMIT 10;",
        )?;
        let result = results.last().unwrap();
        assert_eq!(result.columns, vec!["host", "service", "total"]);
        assert_eq!(result.rows[0][0].to_string(), "b");
        assert_eq!(result.rows[0][1].to_string(), "api");

        std::fs::remove_file(path)?;
        Ok(())
```

--------------------------------

### Get Column Statistics for StoredRowGroup

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Returns a slice of `ColumnStats` for the `StoredRowGroup`.

```rust
pub fn stats(&self) -> &[ColumnStats] {
        &self.stats
    }
```

--------------------------------

### Get Column Index

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/compile.rs.html

Finds the index of a column in the schema by its name. Returns an error if the column is not found.

```rust
pub(super) fn column_index(schema: &Schema, column_name: &str) -> Result<usize> {
    schema
        .columns
        .iter()
        .position(|column| column.name == column_name)
        .with_context(|| format!("unknown column {column_name}"))
}
```

--------------------------------

### Database Initialization and Path

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Provides methods for opening a new NarrowDb instance and retrieving its storage path.

```APIDOC
## POST /websites/rs_narrowdb_0_3_0/open

### Description
Opens a new NarrowDb instance at the specified path with given options.

### Method
POST

### Endpoint
/websites/rs_narrowdb_0_3_0/open

### Parameters
#### Query Parameters
- **path** (string) - Required - The file path to open or create the database.
- **options** (DbOptions) - Required - Database configuration options.

### Request Body
```json
{
  "auto_flush_interval": "optional<Duration>",
  "max_open_files": "usize"
}
```

### Response
#### Success Response (200)
- **NarrowDb** (object) - The opened NarrowDb instance.

#### Response Example
```json
{
  "instance": "NarrowDb"
}
```

## GET /websites/rs_narrowdb_0_3_0/path

### Description
Retrieves the storage path of the current NarrowDb instance.

### Method
GET

### Endpoint
/websites/rs_narrowdb_0_3_0/path

### Response
#### Success Response (200)
- **path** (string) - The file path of the database.

#### Response Example
```json
{
  "path": "/path/to/database"
}
```
```

--------------------------------

### Run NarrowDB Benchmark

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/bench.rs.html

Opens a NarrowDB database, creates a table, and ingests a specified number of rows in batches. It measures the time taken for ingestion and flushing.

```rust
use std::path::Path;
use std::time::{Duration, Instant};

use anyhow::Result;

use crate::engine::{NarrowDb, QueryResult};
use crate::storage::DbOptions;
use crate::types::{BatchColumn, ColumnarBatch};

const SERVICES: [&str; 8] = [
    "api", "worker", "auth", "billing", "search", "ingest", "cdn", "edge",
];
const HOSTS: [&str; 16] = [
    "host-a", "host-b", "host-c", "host-d", "host-e", "host-f", "host-g", "host-h", "host-i",
    "host-j", "host-k", "host-l", "host-m", "host-n", "host-o", "host-p",
];
const LEVELS: [&str; 4] = ["info", "warn", "error", "debug"];
const MESSAGE_TEMPLATES: [&str; 16] = [
    "request completed",
    "request retried",
    "cache miss",
    "cache warm",
    "upstream timeout",
    "upstream reset",
    "auth failed",
    "auth refreshed",
    "job queued",
    "job started",
    "job finished",
    "rate limited",
    "disk spill",
    "checkpoint written",
    "connection reused",
    "connection opened",
];
const BATCH_SIZE: usize = 65_536;

pub fn run_benchmark(path: impl AsRef<Path>, rows: usize) -> Result<()> {
    let path = path.as_ref();
    if path.exists() {
        std::fs::remove_file(path)?;
    }

    let db = NarrowDb::open(
        path,
        DbOptions {
            row_group_size: 32_768,
            sync_on_flush: false,
            ..DbOptions::default()
        },
    )?;

    db.execute_sql(
        "CREATE TABLE logs (
            ts TIMESTAMP,
            level TEXT,
            service TEXT,
            host TEXT,
            request_id INT,
            status INT,
            duration REAL,
            bytes INT,
            message TEXT
        );",
```

```rust
    );

    let end_to_end_started = Instant::now();
    let mut ingest_elapsed = Duration::ZERO;
    for batch_start in (0..rows).step_by(BATCH_SIZE) {
        let batch_end = (batch_start + BATCH_SIZE).min(rows);
        let batch_rows = batch_end - batch_start;
        let mut ts = Vec::with_capacity(batch_rows);
        let mut level = Vec::with_capacity(batch_rows);
        let mut service = Vec::with_capacity(batch_rows);
        let mut host = Vec::with_capacity(batch_rows);
        let mut request_id = Vec::with_capacity(batch_rows);
        let mut status = Vec::with_capacity(batch_rows);
        let mut duration = Vec::with_capacity(batch_rows);
        let mut bytes = Vec::with_capacity(batch_rows);
        let mut message = Vec::with_capacity(batch_rows);

        for row in batch_start..batch_end {
            let ts_value = 1_700_000_000_i64 + row as i64;
            let level_value = if row % 17 == 0 {
                LEVELS[2]
            } else if row % 7 == 0 {
                LEVELS[1]
            } else {
                LEVELS[0]
            };
            let service_value = SERVICES[row % SERVICES.len()];
            let host_value = HOSTS[(row / 11) % HOSTS.len()];
            let request_id_value = row as i64;
            let status_value = if level_value == "error" {
                500 + (row % 8) as i64
            } else {
                200 + (row % 5) as i64
            };
            let duration_value = 4.0 + ((row % 1000) as f64 * 1.37);
            let bytes_value = 256 + ((row * 31) % 65_536) as i64;
            let message_value = MESSAGE_TEMPLATES[(row / 97) % MESSAGE_TEMPLATES.len()];

            ts.push(ts_value);
            level.push(level_value.to_string());
            service.push(service_value.to_string());
            host.push(host_value.to_string());
            request_id.push(request_id_value);
            status.push(status_value);
            duration.push(duration_value);
            bytes.push(bytes_value);
            message.push(message_value.to_string());
        }

        let batch = ColumnarBatch::new(vec![
            BatchColumn::Timestamp(ts),
            BatchColumn::String(level),
            BatchColumn::String(service),
            BatchColumn::String(host),
            BatchColumn::Int64(request_id),
            BatchColumn::Int64(status),
            BatchColumn::Float64(duration),
            BatchColumn::Int64(bytes),
            BatchColumn::String(message),
        ])?;

        let batch_started = Instant::now();
        db.insert_columnar_batch("logs", batch)?;
        ingest_elapsed += batch_started.elapsed();
    }
    let flush_started = Instant::now();
    db.flush_all()?;
    ingest_elapsed += flush_started.elapsed();
    let end_to_end_elapsed = end_to_end_started.elapsed();

    let file_size = std::fs::metadata(db.path()?)?.len();
    println!(
        "Ingested {rows} rows in {:?} end-to-end",
        end_to_end_elapsed
    );

```

--------------------------------

### Get DataType Tag

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.DataType.html

Returns a unique u8 tag for each DataType variant. This is useful for serialization and deserialization.

```rust
pub fn tag(self) -> u8
```

--------------------------------

### Get TypeId

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.DataType.html

Retrieves the unique TypeId for a given type. This is part of the Any trait implementation and is useful for runtime type introspection.

```rust
fn type_id(&self) -> TypeId
```

--------------------------------

### Create New Table

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Initializes a new Table with a given schema and database options. It pre-allocates space for pending rows based on the row group size.

```rust
pub(crate) fn new(schema: Schema, options: &DbOptions) -> Self {
        let pending = PendingBatch::new(&schema, options.row_group_size);
        Self {
            schema,
            row_groups: Vec::new(),
            pending,
        }
    }
```

--------------------------------

### Initialize StoredColumnSource

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Creates a new `StoredColumnSource` with optional null bitmap and data blobs.

```rust
fn new(nulls: Option<StoredBlob>, data: StoredBlob) -> Self {
        Self {
            nulls,
            data,
            decoded_column: OnceCell::new(),
            decoded_nulls: OnceCell::new(),
        }
    }
```

--------------------------------

### Create All and None SelectionBitmaps

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/bitmap.rs.html

Tests the creation of SelectionBitmaps representing all elements and no elements within a given size. Asserts counts, emptiness, and iteration behavior.

```rust
let all = SelectionBitmap::all(100);
assert_eq!(all.count(), 100);
assert_eq!(all.len(), 100);
assert!(!all.is_empty());
assert_eq!(all.iter_set().count(), 100);

let none = SelectionBitmap::none(100);
assert_eq!(none.count(), 0);
assert!(none.is_empty());
assert_eq!(none.iter_set().count(), 0);
```

--------------------------------

### Evaluate i64 vs. f64 Comparisons

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/scan.rs.html

Builds a selection bitmap for i64 columns compared against f64 target values, converting i64 to f64 for comparison.

```rust
fn eval_int64_vs_float(values: &[i64], target: f64, op: CompareOp, rows: usize) -> SelectionBitmap {
    let pred: fn(f64, f64) -> bool = match op {
        CompareOp::Eq => |a, b| a == b,
        CompareOp::NotEq => |a, b| a != b,
        CompareOp::Lt => |a, b| a < b,
        CompareOp::Lte => |a, b| a <= b,
        CompareOp::Gt => |a, b| a > b,
        CompareOp::Gte => |a, b| a >= b,
        _ => unreachable!(),
    };
    build_bitmap_from_predicate(rows, |i| pred(values[i] as f64, target))
}
```

--------------------------------

### Database Configuration Options

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Defines the configuration parameters for the database, including row group size and flush behavior.

```rust
#[derive(Debug, Clone)]
pub struct DbOptions {
    pub row_group_size: usize,
    pub sync_on_flush: bool,
    pub auto_flush_interval: Option<Duration>,
}

impl Default for DbOptions {
    fn default() -> Self {
        Self {
            row_group_size: 16_384,
            sync_on_flush: true,
            auto_flush_interval: None,
        }
    }
}
```

--------------------------------

### Create Table If Not Exists in NarrowDB

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Illustrates the use of `CREATE TABLE IF NOT EXISTS` in NarrowDB to prevent errors when attempting to create a table that already exists. Shows both successful and erroneous attempts.

```rust
let path = std::env::temp_dir().join(format!("narrowdb-test-ifne-{}", SystemTime::now().duration_since(UNIX_EPOCH)?.as_nanos()));

        let db = NarrowDb::open(&path, DbOptions::default())?;
        db.execute_sql("CREATE TABLE logs (ts TIMESTAMP, level TEXT);")?;

        // Should not error
        db.execute_sql("CREATE TABLE IF NOT EXISTS logs (ts TIMESTAMP, level TEXT);")?;

        // Should still error without IF NOT EXISTS
        let err = db.execute_sql("CREATE TABLE logs (ts TIMESTAMP, level TEXT);");
```

--------------------------------

### Print Query Result Preview

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/bench.rs.html

Prints the column names and the first five rows of a QueryResult.

```rust
fn print_result_preview(result: &QueryResult) {
    if result.columns.is_empty() {
        return;
    }
    println!("Columns: {}", result.columns.join(", "));
    for row in result.rows.iter().take(5) {
        let line = row
            .iter()
            .map(ToString::to_string)
            .collect::<Vec<_>>()
            .join(" | ");
        println!("  {line}");
    }
}
```

--------------------------------

### Core Re-exports and Modules

Source: https://docs.rs/narrowdb/0.3.0/index.html

Overview of the primary components exposed by the narrowdb crate.

```APIDOC
## Core Components

### Re-exports
- **NarrowDb** (engine::NarrowDb) - Main database engine interface
- **QueryResult** (engine::QueryResult) - Result structure for database queries
- **BatchColumn** (types::BatchColumn) - Column representation for batches
- **ColumnDef** (types::ColumnDef) - Definition of a database column
- **ColumnarBatch** (types::ColumnarBatch) - Batch of columnar data
- **DataType** (types::DataType) - Supported data types
- **Schema** (types::Schema) - Database schema definition
- **Value** (types::Value) - Generic value type

### Modules
- **engine** - Core database engine logic
- **types** - Data type and schema definitions

### Structs
- **DbOptions** - Configuration options for the database
```

--------------------------------

### Define DbOptions struct

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/struct.DbOptions.html

The primary configuration structure for database options.

```rust
pub struct DbOptions {
    pub row_group_size: usize,
    pub sync_on_flush: bool,
    pub auto_flush_interval: Option<Duration>,
}
```

--------------------------------

### Execute Query Benchmark

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/bench.rs.html

Executes a SQL query and prints the elapsed time and throughput.

```rust
fn run_query_benchmark(db: &NarrowDb, rows: usize, sql: &str, label: &str) -> Result<()> {
    let started = Instant::now();
    let results = db.execute_sql(sql)?;
    let elapsed = started.elapsed();
    let result = results.last().cloned().unwrap_or_else(QueryResult::empty);

    println!(
        "{label}: {:?} ({:.2} rows/sec scanned)",
        elapsed,
        rows as f64 / elapsed.as_secs_f64()
    );
    print_result_preview(&result);
    Ok(())
```

--------------------------------

### Ingest Columnar Batches into NarrowDB

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Demonstrates inserting data into a NarrowDB table using columnar batches, which is an efficient way to load structured data. Includes creating a table, inserting a batch, and querying the ingested data.

```rust
let path = std::env::temp_dir().join(format!("narrowdb-test-columnar-{}", SystemTime::now().duration_since(UNIX_EPOCH)?.as_nanos()));

        let db = NarrowDb::open(&path, DbOptions::default())?;
        db.execute_sql(
            "CREATE TABLE logs (ts TIMESTAMP, service TEXT, status INT, duration REAL);",
```

```rust
db.insert_columnar_batch(
            "logs",
            ColumnarBatch::new(vec![
                BatchColumn::Timestamp(vec![1, 2, 3]),
                BatchColumn::String(vec![
                    "api".to_string(),
                    "api".to_string(),
                    "worker".to_string(),
                ]),
                BatchColumn::Int64(vec![200, 500, 503]),
                BatchColumn::Float64(vec![10.0, 100.0, 250.0]),
            ])?,
        )?;
        db.flush_all()?;

        let results = db.execute_sql(
            "SELECT service, COUNT(*) AS total FROM logs WHERE status >= 500 GROUP BY service ORDER BY total DESC LIMIT 10;",
        )?;
        let result = results.last().unwrap();
        assert_eq!(result.rows.len(), 2);

        std::fs::remove_file(path)?;
        Ok(())
```

--------------------------------

### Clone Implementation for Schema

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/struct.Schema.html

Provides methods to duplicate Schema instances. `clone` creates a new Schema, while `clone_from` performs copy-assignment.

```rust
fn clone(&self) -> Schema
```

```rust
fn clone_from(&mut self, source: &Self)
```

--------------------------------

### Manage ColumnBuilder Lifecycle

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Provides methods to initialize, append values, check length, and finalize column data construction.

```rust
impl ColumnBuilder {
    pub fn with_capacity(data_type: DataType, capacity: usize) -> Self {
        match data_type {
            DataType::Int64 | DataType::Timestamp => {
                Self::Int64(Vec::with_capacity(capacity), Vec::with_capacity(capacity))
            }
            DataType::Float64 => {
                Self::Float64(Vec::with_capacity(capacity), Vec::with_capacity(capacity))
            }
            DataType::Bool => {
                Self::Bool(Vec::with_capacity(capacity), Vec::with_capacity(capacity))
            }
            DataType::String => {
                Self::String(Vec::with_capacity(capacity), Vec::with_capacity(capacity))
            }
        }
    }

    pub fn append(&mut self, value: Value) -> Result<()> {
        match (self, value) {
            (Self::Int64(values, nulls), Value::Int64(value)) => {
                values.push(value);
                nulls.push(true);
            }
            (Self::Int64(values, nulls), Value::Null) => {
                values.push(0);
                nulls.push(false);
            }
            (Self::Float64(values, nulls), Value::Float64(value)) => {
                values.push(value);
                nulls.push(true);
            }
            (Self::Float64(values, nulls), Value::Null) => {
                values.push(OrderedFloat(0.0));
                nulls.push(false);
            }
            (Self::Bool(values, nulls), Value::Bool(value)) => {
                values.push(value);
                nulls.push(true);
            }
            (Self::Bool(values, nulls), Value::Null) => {
                values.push(false);
                nulls.push(false);
            }
            (Self::String(values, nulls), Value::String(value)) => {
                values.push(value);
                nulls.push(true);
            }
            (Self::String(values, nulls), Value::Null) => {
                values.push(String::new());
                nulls.push(false);
            }
            (builder, value) => bail!("cannot append {value:?} into {builder:?}"),
        }
        Ok(())
    }

    pub fn len(&self) -> usize {
        match self {
            Self::Int64(values, _) => values.len(),
            Self::Float64(values, _) => values.len(),
            Self::Bool(values, _) => values.len(),
            Self::String(values, _) => values.len(),
        }
    }

    pub fn finish(&mut self) -> (ColumnData, Option<NullBitmap>) {
        match self {
            Self::Int64(values, nulls) => {
                let data = ColumnData::Int64(std::mem::take(values));
                let bitmap = if nulls.iter().all(|&p| p) {
                    None
                } else {
                    Some(NullBitmap::from_bools(nulls))
                };
                (data, bitmap)
            }
            Self::Float64(values, nulls) => {
                let data = ColumnData::Float64(std::mem::take(values));
                let bitmap = if nulls.iter().all(|&p| p) {
                    None
                } else {
                    Some(NullBitmap::from_bools(nulls))
                };
                (data, bitmap)
            }
            Self::Bool(values, nulls) => {
                let data = ColumnData::Bool(std::mem::take(values));
                let bitmap = if nulls.iter().all(|&p| p) {
                    None
                } else {
                    Some(NullBitmap::from_bools(nulls))
                };
                (data, bitmap)
            }
            Self::String(values, nulls) => {
```

--------------------------------

### String Conversion

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.Value.html

Documentation for the to_string method implementation.

```APIDOC
## fn to_string(&self) -> String

### Description
Converts the given value to a String.

### Method
N/A (Trait Method)

### Parameters
- **&self** - Required - The instance to convert.
```

--------------------------------

### PartialEq Implementation for Schema

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/struct.Schema.html

Allows comparison of two Schema instances for equality. The `eq` method tests for equality, while `ne` tests for inequality.

```rust
fn eq(&self, other: &Schema) -> bool
```

```rust
fn ne(&self, other: &Rhs) -> bool
```

--------------------------------

### Evaluate f64 Comparisons

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/scan.rs.html

Builds a selection bitmap for f64 columns using ordered floats for precise comparison based on the specified operation.

```rust
fn eval_float64(
    values: &[ordered_float::OrderedFloat<f64>],
    target: ordered_float::OrderedFloat<f64>,
    op: CompareOp,
    rows: usize,
) -> SelectionBitmap {
    let pred: fn(ordered_float::OrderedFloat<f64>, ordered_float::OrderedFloat<f64>) -> bool = match op {
        CompareOp::Eq => |a, b| a == b,
        CompareOp::NotEq => |a, b| a != b,
        CompareOp::Lt => |a, b| a < b,
        CompareOp::Lte => |a, b| a <= b,
        CompareOp::Gt => |a, b| a > b,
        CompareOp::Gte => |a, b| a >= b,
        _ => unreachable!(),
    };
    build_bitmap_from_predicate(rows, |i| pred(values[i], target))
}
```

--------------------------------

### Create New PendingBatch

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/storage.rs.html

Initializes a new PendingBatch with a specified capacity for each column, based on the provided schema.

```rust
pub(crate) fn new(schema: &Schema, capacity: usize) -> Self {
        Self {
            columns: schema
                .columns
                .iter()
                .map(|column| ColumnBuilder::with_capacity(column.data_type, capacity))
                .collect(),
        }
    }
```

--------------------------------

### TryFrom and TryInto Traits

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.Value.html

Documentation for fallible conversion traits used in the project.

```APIDOC
## impl<T, U> TryFrom<U> for T

### Description
Defines a fallible conversion from type U to type T.

### Parameters
- **value** (U) - Required - The value to convert.

### Response
- **Result<T, Error>** - The result of the conversion.

---

## impl<T, U> TryInto<U> for T

### Description
Defines a fallible conversion into type U from type T.

### Parameters
- **self** (T) - Required - The instance to convert.

### Response
- **Result<U, Error>** - The result of the conversion.
```

--------------------------------

### Define SelectionBitmap and core methods

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/bitmap.rs.html

Defines the SelectionBitmap structure and methods for initialization, intersection, and null-based masking.

```rust
1use crate::storage::NullBitmap;

/// Packed bit-vector: bit set = row selected.
/// Uses u64 words for wide SIMD-friendly operations.
pub(super) struct SelectionBitmap {
    words: Vec<u64>,
    len: usize,
    count: usize,
}

impl SelectionBitmap {
    /// All rows selected.
    pub fn all(len: usize) -> Self {
        let word_count = (len + 63) / 64;
        let mut words = vec![u64::MAX; word_count];
        // Clear unused trailing bits in the last word.
        let remainder = len % 64;
        if remainder > 0 && !words.is_empty() {
            *words.last_mut().unwrap() = (1u64 << remainder) - 1;
        }
        Self {
            words,
            len,
            count: len,
        }
    }

    /// No rows selected.
    pub fn none(len: usize) -> Self {
        let word_count = (len + 63) / 64;
        Self {
            words: vec![0u64; word_count],
            len,
            count: 0,
        }
    }

    pub fn is_empty(&self) -> bool {
        self.count == 0
    }

    pub fn count(&self) -> usize {
        self.count
    }

    #[allow(dead_code)]
    pub fn len(&self) -> usize {
        self.len
    }

    /// Intersect in-place: self &= other.
    pub fn intersect(&mut self, other: &SelectionBitmap) {
        debug_assert_eq!(self.len, other.len);
        for (a, b) in self.words.iter_mut().zip(other.words.iter()) {
            *a &= *b;
        }
        self.recount();
    }

    /// Clear bits where the null bitmap indicates null (bit clear in NullBitmap = null).
    /// After this, only non-null selected rows remain.
    pub fn mask_non_null(&mut self, nulls: &NullBitmap) {
        let null_bytes = nulls.data();
        // NullBitmap uses u8 words (set = present, clear = null).
        // We need to AND our u64 words with the corresponding 8 bytes from NullBitmap.
        for (word_idx, word) in self.words.iter_mut().enumerate() {
            let byte_offset = word_idx * 8;
            let mut mask: u64 = 0;
            for b in 0..8 {
                let idx = byte_offset + b;
                if idx < null_bytes.len() {
                    mask |= (null_bytes[idx] as u64) << (b * 8);
                }
            }
            *word &= mask;
        }
        self.recount();
    }

    /// Keep only bits where the null bitmap indicates null (bit clear in NullBitmap = null).
    pub fn mask_null(&mut self, nulls: &NullBitmap) {
        let null_bytes = nulls.data();
        for (word_idx, word) in self.words.iter_mut().enumerate() {
            let byte_offset = word_idx * 8;
            let mut mask: u64 = 0;
            for b in 0..8 {
                let idx = byte_offset + b;
                if idx < null_bytes.len() {
                    mask |= (null_bytes[idx] as u64) << (b * 8);
                }
            }
            // Invert: keep positions that are null (clear in NullBitmap).
            *word &= !mask;
        }
        // Clear any trailing bits beyond len.
        let remainder = self.len % 64;
        if remainder > 0 && !self.words.is_empty() {
            *self.words.last_mut().unwrap() &= (1u64 << remainder) - 1;
        }
        self.recount();
    }

    /// Mutable access to raw u64 words for vectorized filter producers.
    pub fn words_mut(&mut self) -> &mut [u64] {
        &mut self.words
    }

    /// Recompute cached popcount from words.
    pub fn recount(&mut self) {
        self.count = self.words.iter().map(|w| w.count_ones() as usize).sum();
    }

    /// Iterate over indices of set bits.
    pub fn iter_set(&self) -> BitmapIter<'_> {
        BitmapIter {
            words: &self.words,
            word_idx: 0,
            current: if self.words.is_empty() {
                0
            } else {
                self.words[0]
            },
            base: 0,
            len: self.len,
        }
    }
}
```

--------------------------------

### Select All Columns from Table

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Executes a SELECT * query. Ensures no GROUP BY clause is present. Scans the table and returns all columns for each row.

```rust
fn select_all(&self, table: &Table, plan: &SelectPlan) -> Result<QueryResult> {
        ensure!(
            plan.group_by.is_empty(),
            "SELECT * does not support GROUP BY"
        );
        let columns = table
            .schema
            .columns
            .iter()
            .map(|column| column.name.clone())
            .collect::<Vec<_>>();
        let filters = compile_filters(&table.schema, &plan.filters)?;
        let required_columns = (0..table.schema.columns.len()).collect::<Vec<_>>();
        let mapped = self.storage.mapped_bytes();
        let col_count = table.schema.columns.len();

        let mut rows = parallel_scan_table(
            table,
            mapped,
            &filters,
            &required_columns,
            |row_group, row| {
                (0..col_count)
                    .map(|idx| row_group.column(idx).value_at(row))
                    .collect()
            },
        )?;

        apply_order_by_and_limit(&mut rows, &columns, plan.order_by.as_ref(), plan.limit)?;
        Ok(QueryResult { columns, rows })
    }
```

--------------------------------

### Initialize Memory

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.DataType.html

Initializes memory with a given value. This is an unsafe operation and requires careful handling of memory.

```rust
unsafe fn init(init: <T as Pointable>::Init) -> usize
```

--------------------------------

### Create Table

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Creates a new table within the database with the specified schema. Requires a write lock on the database inner state.

```rust
pub fn create_table(&self, schema: Schema) -> Result<()> {
        let mut inner = self.inner.write().map_err(|_| anyhow!("lock poisoned"))?;
        inner.create_table_inner(schema)
    }
```

--------------------------------

### narrowdb Core Components

Source: https://docs.rs/narrowdb/0.3.0/narrowdb

Overview of the primary re-exported components available in the narrowdb crate.

```APIDOC
## Core Components

### Description
The narrowdb crate exposes several core components for database interaction and schema definition.

### Re-exports
- **NarrowDb** (engine::NarrowDb) - The primary database engine interface.
- **QueryResult** (engine::QueryResult) - Represents the result of a database query.
- **BatchColumn** (types::BatchColumn) - Represents a column within a batch.
- **ColumnDef** (types::ColumnDef) - Definition of a database column.
- **ColumnarBatch** (types::ColumnarBatch) - A batch of columnar data.
- **DataType** (types::DataType) - Supported data types for columns.
- **Schema** (types::Schema) - Database schema definition.
- **Value** (types::Value) - Represents a single data value.

### Structs
- **DbOptions** - Configuration options for the database engine.
```

--------------------------------

### Define SelectPlan Struct

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/sql.rs.html

Represents a SELECT SQL query. It includes the table name, filters, projections, grouping, ordering, and limit.

```rust
pub struct SelectPlan {
    pub table_name: String,
    pub filters: Vec<Filter>,
    pub projections: Vec<Projection>,
    pub group_by: Vec<String>,
    pub order_by: Option<OrderByPlan>,
    pub limit: Option<usize>,
}
```

--------------------------------

### Evaluate i64 Comparisons

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/scan.rs.html

Builds a selection bitmap for i64 columns based on a predicate function derived from the comparison operation and target value.

```rust
fn eval_int64(values: &[i64], target: i64, op: CompareOp, rows: usize) -> SelectionBitmap {
    let pred: fn(i64, i64) -> bool = match op {
        CompareOp::Eq => |a, b| a == b,
        CompareOp::NotEq => |a, b| a != b,
        CompareOp::Lt => |a, b| a < b,
        CompareOp::Lte => |a, b| a <= b,
        CompareOp::Gt => |a, b| a > b,
        CompareOp::Gte => |a, b| a >= b,
        _ => unreachable!(),
    };
    build_bitmap_from_predicate(rows, |i| pred(values[i], target))
}
```

--------------------------------

### Database Persistence and Query Test

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Integration test verifying that data can be persisted to a temporary file and queried correctly.

```rust
#[test]
    fn persists_and_queries() -> Result<()> {
        let path = std::env::temp_dir().join(format!(
            "narrowdb-test-{}.db",
            SystemTime::now().duration_since(UNIX_EPOCH)?.as_nanos()
        ));

        {
            let db = NarrowDb::open(
                &path,
                DbOptions {
                    row_group_size: 2,
                    sync_on_flush: true,
                    ..DbOptions::default()
                },
            )?;
            db.execute_sql(
                "
                CREATE TABLE logs (ts TIMESTAMP, level TEXT, service TEXT, status INT, duration REAL);
                INSERT INTO logs VALUES
                    (1, 'info', 'api', 200, 12.0),
                    (2, 'error', 'api', 500, 120.0),
                    (3, 'error', 'worker', 503, 250.0);
                ",
            )?;
            db.flush_all()?;
        }

        let db = NarrowDb::open(&path, DbOptions::default())?;
        let results = db.execute_sql(
            "SELECT service, COUNT(*) AS errors FROM logs WHERE level = 'error' GROUP BY service ORDER BY errors DESC LIMIT 2;",
        )?;
        assert_eq!(results.last().unwrap().rows.len(), 2);
        std::fs::remove_file(path)?;
        Ok(())
    }
```

--------------------------------

### Table Management

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

APIs for creating and managing tables within the NarrowDb instance.

```APIDOC
## POST /websites/rs_narrowdb_0_3_0/tables

### Description
Creates a new table with the specified schema.

### Method
POST

### Endpoint
/websites/rs_narrowdb_0_3_0/tables

### Parameters
#### Request Body
- **schema** (Schema) - Required - The schema definition for the new table.

### Request Example
```json
{
  "schema": {
    "columns": [
      {"name": "id", "type": "INT"},
      {"name": "name", "type": "STRING"}
    ]
  }
}
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message.

#### Response Example
```json
{
  "message": "Table created successfully."
}
```
```

--------------------------------

### Clone to Uninitialized Memory (Nightly)

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/enum.DataType.html

An experimental, nightly-only API that performs copy-assignment from self to uninitialized memory. Use with caution as it is unstable.

```rust
unsafe fn clone_to_uninit(&self, dest: *mut u8)
```

--------------------------------

### Build Bitmap from Predicate

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/scan.rs.html

Helper function to construct a SelectionBitmap by applying a given predicate to each row. It optimizes performance by processing data in 64-element chunks, leveraging auto-vectorization.

```rust
#[inline]
fn build_bitmap_from_predicate<F>(rows: usize, pred: F) -> SelectionBitmap
where
    F: Fn(usize) -> bool,
{
    let mut bitmap = SelectionBitmap::none(rows);
    let words = bitmap.words_mut();
    let full_words = rows / 64;
    for word_idx in 0..full_words {
        let base = word_idx * 64;
        let mut word: u64 = 0;
        for bit in 0..64 {
            word |= (pred(base + bit) as u64) << bit;
        }
        words[word_idx] = word;
    }
    // Handle the remainder.
    let remainder = rows % 64;
    if remainder > 0 {
        let base = full_words * 64;
        let mut word: u64 = 0;
        for bit in 0..remainder {
            word |= (pred(base + bit) as u64) << bit;
        }
        words[full_words] = word;
    }
    bitmap.recount();
    bitmap
}
```

--------------------------------

### NarrowDb Core Components

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/all.html

Overview of the primary structs and enums available in the NarrowDb 0.3.0 library.

```APIDOC
## Core Structs

- **DbOptions**: Configuration options for the database engine.
- **engine::NarrowDb**: The primary database engine interface.
- **engine::QueryResult**: Represents the result set returned from database queries.
- **types::ColumnDef**: Definition of a database column.
- **types::ColumnarBatch**: A batch of data organized in a columnar format.
- **types::Schema**: The schema definition for the database.

## Core Enums

- **types::BatchColumn**: Represents a column within a batch.
- **types::DataType**: Supported data types within the database.
- **types::Value**: Represents a single data value.
```

--------------------------------

### Define InsertPlan Struct

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/sql.rs.html

Represents the structure for an INSERT SQL command. It includes the table name and the rows of data to be inserted. Explicit column lists are not supported.

```rust
pub struct InsertPlan {
    pub table_name: String,
    pub rows: Vec<Vec<Value>>,
}
```

--------------------------------

### Schema Trait Implementations

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/types/struct.Schema.html

Details the various trait implementations for the Schema struct, including Clone, Debug, PartialEq, Eq, and several auto traits.

```APIDOC
## Trait Implementations for Schema

### impl Clone for Schema
- `fn clone(&self) -> Schema`: Returns a duplicate of the value.
- `fn clone_from(&mut self, source: &Self)`: Performs copy-assignment from `source`.

### impl Debug for Schema
- `fn fmt(&self, f: &mut Formatter<'_>) -> Result`: Formats the value using the given formatter.

### impl PartialEq for Schema
- `fn eq(&self, other: &Schema) -> bool`: Tests for `self` and `other` values to be equal.
- `fn ne(&self, other: &Rhs) -> bool`: Tests for `!=`.

### impl Eq for Schema

### Auto Trait Implementations
- `Freeze`
- `RefUnwindSafe`
- `Send`
- `Sync`
- `Unpin`
- `UnsafeUnpin`
- `UnwindSafe`

### Blanket Implementations
- `impl<T> Any for T`
- `impl<T> Borrow<T> for T`
- `impl<T> BorrowMut<T> for T`
- `impl<T> CloneToUninit for T`
- `impl<T> From<T> for T`
- `impl<T, U> Into<U> for T`
- `impl<T> IntoEither for T`
- `impl<T> Pointable for T`
- `impl<T> ToOwned for T`
- `impl<T, U> TryFrom<U> for T`
- `impl<T, U> TryInto<U> for T`
```

--------------------------------

### NarrowDb Core Operations

Source: https://docs.rs/narrowdb/0.3.0/narrowdb/engine/struct.NarrowDb.html

Provides methods for opening a database connection, managing table schemas, and inserting data into tables.

```APIDOC
## NarrowDb

### Struct NarrowDb

Represents the main database engine.

### Methods

#### `open(path: impl AsRef<Path>, options: DbOptions) -> Result<Self>`

Opens a connection to the NarrowDb database at the specified path with given options.

- **Method**: `pub fn open`
- **Endpoint**: N/A (Function call)

#### `path(&self) -> Result<PathBuf>`

Returns the path to the database file.

- **Method**: `pub fn path`
- **Endpoint**: N/A (Method call)

#### `create_table(&self, schema: Schema) -> Result<()>`

Creates a new table in the database based on the provided schema.

- **Method**: `pub fn create_table`
- **Endpoint**: N/A (Method call)

#### `insert_row(&self, table_name: &str, row: Vec<Value>) -> Result<()>`

Inserts a single row into the specified table.

- **Method**: `pub fn insert_row`
- **Endpoint**: N/A (Method call)

#### `insert_rows(&self, table_name: &str, rows: Vec<Vec<Value>>) -> Result<()>`

Inserts multiple rows into the specified table.

- **Method**: `pub fn insert_rows`
- **Endpoint**: N/A (Method call)

#### `insert_columnar_batch(&self, table_name: &str, batch: ColumnarBatch) -> Result<()>`

Inserts a batch of data in columnar format into the specified table.

- **Method**: `pub fn insert_columnar_batch`
- **Endpoint**: N/A (Method call)

#### `flush_table(&self, table_name: &str) -> Result<()>`

Flushes all pending writes for a specific table to disk.

- **Method**: `pub fn flush_table`
- **Endpoint**: N/A (Method call)

#### `flush_all(&self) -> Result<()>`

Flushes all pending writes for all tables to disk.

- **Method**: `pub fn flush_all`
- **Endpoint**: N/A (Method call)
```

--------------------------------

### Validate and Split Data Batches

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/types.rs.html

Methods for checking column type consistency and splitting a batch into a prefix.

```rust
            if column.data_type() != schema_column.data_type {
                bail!(
                    "column type mismatch for {}: expected {:?}, got {:?}",
                    schema_column.name,
                    schema_column.data_type,
                    column.data_type()
                )
            }
        }
        Ok(())
    }

    pub(crate) fn take_prefix(&mut self, len: usize) -> Result<Self> {
        if len > self.rows {
            bail!("requested prefix {len} exceeds batch size {}", self.rows)
        }
        let columns = self
            .columns
            .iter_mut()
            .map(|column| column.take_prefix(len))
            .collect();
        self.rows -= len;
        Ok(Self { columns, rows: len })
    }

    pub(crate) fn into_columns(self) -> Vec<BatchColumn> {
        self.columns
    }
}
```

--------------------------------

### Implement Display for Value

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/types.rs.html

Provides string representation for Value variants.

```rust
impl fmt::Display for Value {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            Self::Int64(value) => write!(f, "{value}"),
            Self::Float64(value) => write!(f, "{}", value.into_inner()),
            Self::Bool(value) => write!(f, "{value}"),
            Self::String(value) => write!(f, "{value}"),
            Self::Null => write!(f, "NULL"),
        }
    }
}
```

--------------------------------

### Execute Commands

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Processes a list of commands, ensuring writes are flushed before select operations.

```rust
fn execute_commands(&mut self, commands: Vec<Command>) -> Result<Vec<QueryResult>> {
        let mut results = Vec::new();

        for command in commands {
            match command {
                Command::CreateTable(schema) => {
                    self.create_table_inner(schema)?;
                    results.push(QueryResult::empty());
                }
                Command::CreateTableIfNotExists(schema) => {
                    if !self.tables.contains_key(&schema.table_name) {
                        self.create_table_inner(schema)?;
                    }
                    results.push(QueryResult::empty());
                }
                Command::Insert(plan) => {
                    self.insert_rows_inner(&plan.table_name, plan.rows)?;
                    results.push(QueryResult::empty());
                }
                Command::Select(plan) => {
                    self.flush_table_inner(&plan.table_name)?;
                    results.push(self.execute_select(&plan)?);
                }
                Command::Eval(plan) => {
                    results.push(execute_eval(plan)?);
                }
            }
        }

        Ok(results)
    }
```

--------------------------------

### Execute Single SQL Command

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/engine/mod.rs.html

Executes a single SQL statement and returns the first result set. If multiple result sets are produced, only the first is returned. Errors if no result set is found.

```rust
pub fn execute_one(&self, sql: &str) -> Result<QueryResult> {
        let mut results = self.execute_sql(sql)?;
        results.pop().context("no result from SQL statement")
    }
```

--------------------------------

### Parse SQL String to Commands

Source: https://docs.rs/narrowdb/0.3.0/src/narrowdb/sql.rs.html

Parses a given SQL string into a vector of `Command` enums. Uses SQLite dialect and the `sqlparser` crate. Requires `anyhow::Result` for error handling.

```rust
pub fn parse_sql(sql: &str) -> Result<Vec<Command>> {
    let dialect = SQLiteDialect {};
    let statements = Parser::parse_sql(&dialect, sql)?;
    statements.into_iter().map(parse_statement).collect()
}
```
Tokens:
Raw