### CompactionIterator usage example Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/iter.rs.html Demonstrates how to iterate over the CompactionIterator using a for loop. ```text let compaction_iter = CompactionIterator::new(...); for result in compaction_iter { let (key, value) = result?; // Write to output SSTable } ``` -------------------------------- ### Test Case Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/task.rs.html Example of initializing a test case for the task manager. ```rust #[test(tokio::test(flavor = "multi_thread"))] async fn test_wake_up_memtable() { let opts = Arc::new(Options::default()); ``` -------------------------------- ### Tree Creation and Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lsm.rs.html Details on how to create and initialize a new LSM Tree, including directory structure setup. ```APIDOC ## POST /api/tree/new ### Description Creates a new LSM tree with the specified options and initializes its directory structure. ### Method POST ### Endpoint /api/tree/new ### Parameters #### Request Body - **opts** (object) - Required - Options for configuring the LSM tree. - **path** (string) - Required - The base path for the LSM tree. - **enable_vlog** (boolean) - Optional - Enables the vlog feature. - **enable_versioning** (boolean) - Optional - Enables versioning. ### Request Example ```json { "opts": { "path": "/data/lsm_tree", "enable_vlog": true, "enable_versioning": false } } ``` ### Response #### Success Response (200) - **tree_id** (string) - Identifier for the newly created LSM tree. #### Response Example ```json { "tree_id": "tree_abc123" } ``` ``` -------------------------------- ### Prefix Compression Example Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/sstable/block.rs.html Demonstrates how prefix compression works with a sequence of keys and a specified restart interval, showing the resulting entries. ```text Key sequence: "apple", "apricot", "banana" Restart interval: 2 Entry 0 (restart point): shared=0, unshared=5, key_suffix="apple" Full key: "" + "apple" = "apple" Entry 1: shared=2, unshared=5, key_suffix="ricot" Full key: "ap" + "ricot" = "apricot" Entry 2 (new restart point because interval=2): shared=0, unshared=6, key_suffix="banana" Full key: "" + "banana" = "banana" ``` -------------------------------- ### Iterate Keys in a Range Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/transaction.rs.html Use this method to get keys and values within a specified range at the current timestamp. The iterator includes the start key but excludes the end key. Ensure you call `seek_first()` before iterating with `next()`. ```rust let mut iter = tx.range(b"a", b"z")?; iter.seek_first()?; while iter.valid() { let key = iter.key().user_key(); let ts = iter.key().timestamp(); let value = iter.value()?; iter.next()?; } // Or seek to a specific key: iter.seek(b"foo")? // Position at first version of "foo" ``` -------------------------------- ### Excluded Bound Seek Example Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/sstable/table.rs.html Example of seeking with an Excluded bound. ```text Example: Excluded("apple") with data [("apple",5), ("apple",3), ("banana",7)] seek(("apple", 0)): ("apple", 5) < ("apple", 0) → skip (5 > 0 descending) ("apple", 3) < ("apple", 0) → skip ("banana", 7) >= ("apple", 0) → land here ✓ Check: user_key == "apple"? No, it's "banana" → done Result: ("banana", 7) - first entry with user_key > "apple" ``` -------------------------------- ### Initialize BPlusTree Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/bplustree/tree.rs.html Initializes a new BPlusTree, setting up the file, header, and cache. It also handles the creation of the initial root node and writes the header to disk if the storage size is zero. ```rust let mut tree = BPlusTree { file, header, cache, compare, durability: Durability::Manual, }; // Initialize storage if it's a new tree if storage_size == 0 { let header_bytes = tree.header.serialize(); let mut buffer = vec![0u8; PAGE_SIZE]; buffer[..header_bytes.len()].copy_from_slice(&header_bytes); tree.file.write_at(0, &buffer)?; // Create initial root node let root = LeafNode::new(tree.header.root_offset); tree.write_node(&NodeType::Leaf(root))?; tree.file.sync_data()?; } else { // Read root node into cache tree.read_node(tree.header.root_offset)?; } Ok(tree) ``` -------------------------------- ### Included Bound Seek Example Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/sstable/table.rs.html Example of seeking with an Included bound. ```text Example: Included("banana") with data [("apple",5), ("banana",7), ("banana",2)] seek(("banana", MAX_SEQ)): ("apple", 5) < ("banana", MAX_SEQ) → skip ("banana", 7) >= ("banana", MAX_SEQ) → land here ✓ Result: ("banana", 7) - first entry with user_key >= "banana" ``` -------------------------------- ### Initialize VLogFile Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Creates a new VLogFile instance with the specified ID and path. ```rust fn new(_id: u32, path: PathBuf, _size: u64) -> Self { Self { path, } } ``` -------------------------------- ### Get Type ID Source: https://docs.rs/surrealkv/0.21.0/surrealkv/struct.InternalKeyComparator.html Gets the `TypeId` of the implementing type. This is part of the `Any` trait implementation. ```rust fn type_id(&self) -> TypeId> ``` -------------------------------- ### Set the starting sequence number for the batch Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/batch.rs.html Allows modification of the batch's starting sequence number. This is useful for re-assigning sequence numbers if a batch is being reused or re-processed. ```rust pub(crate) fn set_starting_seq_num(&mut self, seq_num: u64) { self.starting_seq_num = seq_num; } ``` -------------------------------- ### Implement Strategy Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/compaction/leveled.rs.html Provides methods to initialize the compaction strategy from existing configuration options. ```rust 36impl Default for Strategy { 37 fn default() -> Self { 38 let opts = Options::default(); 39 Self { 40 level0_file_num_trigger: opts.level0_max_files, 41 max_bytes_for_level: opts.max_bytes_for_level, 42 level_multiplier: opts.level_multiplier, 43 compaction_priority: CompactionPriority::default(), 44 } 45 } 46} 47 48impl Strategy { 49 /// Create a Strategy from Options 50 pub(crate) fn from_options(opts: Arc) -> Self { 51 Self { 52 level0_file_num_trigger: opts.level0_max_files, 53 max_bytes_for_level: opts.max_bytes_for_level, 54 level_multiplier: opts.level_multiplier, 55 compaction_priority: CompactionPriority::default(), 56 } 57 } 58 59 /// Create a Strategy from Options with a specific compaction priority 60 #[cfg(test)] 61 pub(crate) fn from_options_with_priority( 62 opts: Arc, 63 priority: CompactionPriority, 64 ) -> Self { 65 Self { 66 level0_file_num_trigger: opts.level0_max_files, 67 max_bytes_for_level: opts.max_bytes_for_level, 68 level_multiplier: opts.level_multiplier, 69 compaction_priority: priority, 70 } 71 } ``` -------------------------------- ### Initialize New BPlusTree and Verify Header Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/bplustree/tree.rs.html Creates a new BPlusTree on disk and verifies that the header information is correctly initialized by reading it directly from the file. ```rust let temp_file = NamedTempFile::new().unwrap(); let path = temp_file.path(); // Create new BPlusTree let _tree = BPlusTree::disk(path, Arc::new(TestComparator)).unwrap(); // Read header directly from file let mut file = File::open(path).unwrap(); let mut buffer = [0u8; 48]; file.read_exact(&mut buffer).unwrap(); let header = Header::deserialize(&buffer).unwrap(); assert_eq!(header.magic, MAGIC); assert_eq!(header.version, 1); assert_eq!(header.root_offset, PAGE_SIZE as u64); assert_eq!(header.free_page_count, 0); assert_eq!(header.trunk_page_head, 0); assert_eq!(header.total_pages, 2); ``` -------------------------------- ### Get Immutable Raw Pointer to Vec Buffer Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Value.html Use `as_ptr` to get a raw pointer to the vector's buffer. Ensure the vector outlives the pointer and do not mutate the memory through this pointer. ```rust let x = vec![1, 2, 4]; let x_ptr = x.as_ptr(); unsafe { for i in 0..x.len() { assert_eq!(*x_ptr.add(i), 1 << i); } } ``` ```rust unsafe { let mut v = vec![0, 1, 2]; let ptr1 = v.as_ptr(); let _ = ptr1.read(); let ptr2 = v.as_mut_ptr().offset(2); ptr2.write(2); // Notably, the write to `ptr2` did *not* invalidate `ptr1` // because it mutated a different element: let _ = ptr1.read(); } ``` -------------------------------- ### Get Mutable Raw Pointer to Vec Buffer Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Value.html Use `as_mut_ptr` to get a mutable raw pointer to the vector's buffer. The caller must ensure the vector outlives the pointer and handle potential reallocations. ```rust // Allocate vector big enough for 4 elements. let size = 4; let mut x: Vec = Vec::with_capacity(size); let x_ptr = x.as_mut_ptr(); // Initialize elements via raw pointer writes, then set length. unsafe { for i in 0..size { *x_ptr.add(i) = i as i32; } x.set_len(size); } assert_eq!(&*x, &[0, 1, 2, 3]); ``` ```rust unsafe { let mut v = vec![0]; let ptr1 = v.as_mut_ptr(); ptr1.write(1); let ptr2 = v.as_mut_ptr(); ptr2.write(2); // Notably, the write to `ptr2` did *not* invalidate `ptr1`: ptr1.write(3); } ``` ```rust use std::mem::{ManuallyDrop, MaybeUninit}; let mut v = ManuallyDrop::new(vec![0, 1, 2]); let ptr = v.as_mut_ptr(); let capacity = v.capacity(); let slice_ptr: *mut [MaybeUninit] = std::ptr::slice_from_raw_parts_mut(ptr.cast(), capacity); drop(unsafe { Box::from_raw(slice_ptr) }); ``` -------------------------------- ### Test Internal Key Separator with Misordered User Keys Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/comparator.rs.html Tests the `separator` method when the user keys are misordered (start key is lexicographically greater than the limit key). In this scenario, the method should return the start key unchanged. ```rust #[test] fn test_internal_key_short_separator_misordered() { let cmp = InternalKeyComparator::new(Arc::new(BytewiseComparator::default())); // When user keys are misordered (start > limit), return unchanged assert_eq!( ikey(b"foo", 100, InternalKeyKind::Set), cmp.separator( &ikey(b"foo", 100, InternalKeyKind::Set), &ikey(b"bar", 99, InternalKeyKind::Set) ) ); } ``` -------------------------------- ### VLog Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Creates a new VLog instance and triggers the pre-filling of existing file handles. ```rust pub(crate) fn new(opts: Arc) -> Result { let vlog = Self { path: opts.vlog_dir(), max_file_size: opts.vlog_max_file_size, checksum_level: opts.vlog_checksum_verification, next_file_id: AtomicU32::new(1), active_writer_id: AtomicU32::new(0), writer: RwLock::new(None), files_map: RwLock::new(HashMap::new()), file_handles: RwLock::new(HashMap::new()), opts, }; // PRE-FILL ALL EXISTING FILE HANDLES ON STARTUP vlog.prefill_file_handles()?; Ok(vlog) } ``` -------------------------------- ### Read Overflow Chain Data Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/bplustree/tree.rs.html Reads data sequentially from a chain of overflow pages starting from a given page offset. Returns an empty Bytes if the starting page is 0. Returns an error if an invalid node type is encountered in the chain. ```rust let mut result = BytesMut::new(); let mut current_offset = first_page; while current_offset != 0 { let node = self.read_node(current_offset)?; match node.as_ref() { NodeType::Overflow(overflow) => { result.extend_from_slice(&overflow.data); current_offset = overflow.next_overflow; } _ => { return Err(BPlusTreeError::InvalidOverflowChain(current_offset)); } } } Ok(result.freeze()) ``` -------------------------------- ### Core Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lsm.rs.html Initializes a new LSM tree with background task management. Sets up the database path and creates the inner core structure. ```APIDOC ## new ### Description Creates a new LSM tree with background task management. ### Method `pub(crate) fn new(opts: Arc) -> Result` ### Parameters #### Path Parameters None #### Query Parameters None #### Request Body None ### Request Example None ### Response #### Success Response (200) - `Self`: An instance of the Core struct representing the initialized LSM tree. #### Response Example ```json { "example": "Core { ... }" } ``` ### Error Handling - `Result`: Returns Ok(Self) on success or an Err on failure during initialization. ``` -------------------------------- ### get_at Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/transaction.rs.html Gets a value for a key at a specific timestamp. ```APIDOC ## get_at ### Description Gets a value for a key at a specific timestamp. Requires versioned queries to be enabled. ### Parameters #### Request Body - **key** (IntoBytes) - Required - The key to retrieve. - **timestamp** (u64) - Required - The timestamp for the versioned read. ### Response #### Success Response (200) - **value** (Option) - The value associated with the key if it exists. ``` -------------------------------- ### Create New Options Instance Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lib.rs.html Initializes a new `Options` instance using the default configuration. ```rust pub fn new() -> Self { Self::default() } ``` -------------------------------- ### Initialize VLog Writer Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Sets up the next file ID and initializes the active VLog writer based on existing files. ```rust if let Some(highest_file_id) = max_file_id { // Found existing files, set next_file_id to one past the highest existing file // ID self.next_file_id.store(highest_file_id + 1, Ordering::SeqCst); // Set the last file up as the active writer let writer = VLogWriter::new( &max_file_path.unwrap(), highest_file_id, self.opts.vlog_max_file_size, CompressionType::None as u8, )?; self.active_writer_id.store(highest_file_id, Ordering::SeqCst); *self.writer.write() = Some(writer); } ``` -------------------------------- ### Get Length of Vec Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Value.html Returns the number of elements currently in the vector. ```rust let a = vec![1, 2, 3]; assert_eq!(a.len(), 3); ``` -------------------------------- ### DiskBPlusTree Initialization and Configuration Source: https://docs.rs/surrealkv/0.21.0/surrealkv/bplustree/tree/type.DiskBPlusTree.html Methods for initializing and configuring a DiskBPlusTree. ```APIDOC ## DiskBPlusTree Initialization and Configuration ### `disk` Method #### Description Creates a new `DiskBPlusTree` instance associated with a specific file path. #### Method `pub fn disk>(path: P, compare: Arc) -> Result` #### Parameters - **path** (P: AsRef) - Required - The file path for the BPlusTree. - **compare** (Arc) - Required - The comparator function for ordering keys. #### Response - **Success Response (200)**: `Result` - A `DiskBPlusTree` instance on success, or a `BPlusTreeError` on failure. ### `with_file` Method #### Description Initializes a `BPlusTree` with a given file and comparator. This is a generic method that `DiskBPlusTree` can utilize. #### Method `pub fn with_file(file: F, compare: Arc) -> Result` #### Parameters - **file** (F: VfsFile) - Required - The file object to use for storage. - **compare** (Arc) - Required - The comparator function for ordering keys. #### Response - **Success Response (200)**: `Result` - A `BPlusTree` instance on success, or a `BPlusTreeError` on failure. ### `set_durability` Method #### Description Sets the durability level for the `BPlusTree`. #### Method `pub fn set_durability(&mut self, durability: Durability)` #### Parameters - **durability** (Durability) - Required - The durability setting to apply. ``` -------------------------------- ### Initialize VLogWriter Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Creates a new VLogWriter, handling file creation and optional header writing. ```rust pub(crate) fn new( path: &Path, file_id: u32, max_file_size: u64, compression: u8, ) -> Result { let file_exists = path.exists(); let file = OpenOptions::new().create(true).append(true).open(path)?; // Clone the fd before BufWriter consumes ownership. This cloned fd // is used to perform fsync outside the write lock. let sync_fd = Arc::new(file.try_clone()?); let current_offset = file.metadata()?.len(); let mut writer = BufWriter::new(file); // If this is a new file, write the header if !file_exists || current_offset == 0 { let header = VLogFileHeader::new(file_id, max_file_size, compression); let header_bytes = header.encode(); writer.write_all(&header_bytes)?; writer.flush()?; } // Update offset to account for header let current_offset = if !file_exists || current_offset == 0 { VLogFileHeader::SIZE as u64 } else { current_offset }; Ok(Self { writer, sync_fd, current_offset, file_id, bytes_written: current_offset, }) } ``` -------------------------------- ### Get Entry Count Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/sstable/block.rs.html Returns the number of entries stored in the block. ```rust 478 pub(crate) fn entries(&self) -> usize { 479 self.num_entries 480 } ``` -------------------------------- ### Initialize and Write Trunk Pages Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/bplustree/tree.rs.html Initializes trunk pages and writes them to the BPlusTree. This setup is used before performing page allocation tests. ```rust let mut trunk1 = TrunkPage::new(trunk1_offset); trunk1.add_free_page(4 * PAGE_SIZE as u64); // Free page 4 trunk1.next_trunk = trunk2_offset; bloop.write_trunk_page(&trunk1).unwrap(); // Create trunk2 with one free page, pointing to trunk3 let mut trunk2 = TrunkPage::new(trunk2_offset); trunk2.add_free_page(5 * PAGE_SIZE as u64); // Free page 5 trunk2.next_trunk = trunk3_offset; bloop.write_trunk_page(&trunk2).unwrap(); // Create trunk3 with one free page let mut trunk3 = TrunkPage::new(trunk3_offset); trunk3.add_free_page(6 * PAGE_SIZE as u64); // Free page 6 bloop.write_trunk_page(&trunk3).unwrap(); ``` -------------------------------- ### Initialize or Load LevelManifest Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/levels/mod.rs.html Handles manifest creation if the file is missing or loads an existing one from disk. ```rust if manifest_file_path.exists() { // Load existing manifest from file return Self::load_from_file(&manifest_file_path, opts); } // If no manifest exists, create a new one // Initialize levels with default values let levels = Self::initialize_levels(opts.level_count); // Start with next_table_id = 1 (0 is reserved) let next_table_id = Arc::new(AtomicU64::new(1)); let manifest = Self { path: manifest_file_path, levels, hidden_set: HashSet::with_capacity(10), next_table_id, manifest_format_version: MANIFEST_FORMAT_VERSION_V1, snapshots: Vec::new(), log_number: 0, last_sequence: 0, }; // Write levels to disk with the counter write_manifest_to_disk(&manifest)?; Ok(manifest) ``` -------------------------------- ### Slice Extraction Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Value.html Methods to get immutable and mutable slices of the entire vector. ```APIDOC ## GET /vector/as_slice ### Description Extracts an immutable slice containing the entire vector. Equivalent to `&s[..]`. ### Method GET ### Endpoint /vector/as_slice ### Parameters #### Query Parameters - **vector_ref** (Vec) - Required - A reference to the vector. ### Response #### Success Response (200) - **&[T]** - An immutable slice of the vector. ### Request Example ```json { "vector_ref": [1, 2, 3, 5, 8] } ``` ### Response Example ```json { "slice": [1, 2, 3, 5, 8] } ``` ## GET /vector/as_mut_slice ### Description Extracts a mutable slice of the entire vector. Equivalent to `&mut s[..]`. ### Method GET ### Endpoint /vector/as_mut_slice ### Parameters #### Query Parameters - **vector_mut_ref** (Vec) - Required - A mutable reference to the vector. ### Response #### Success Response (200) - **&mut [T]** - A mutable slice of the vector. ### Request Example ```json { "vector_mut_ref": [0, 0, 0] } ``` ### Response Example ```json { "mutable_slice": [0, 0, 0] } ``` ``` -------------------------------- ### Get Restart Point Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/sstable/block.rs.html Retrieves the offset of a specific restart point by its index. ```APIDOC ## BlockIterator::get_restart_point ### Description Gets a restart point offset by index. ### Method `get_restart_point` ### Endpoint N/A (Method on `BlockIterator` struct) ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Path Parameters - **index** (usize) - Required - The index of the restart point to retrieve. ### Request Example ```rust let offset = block_iterator.get_restart_point(5); ``` ### Response #### Success Response (usize) - **usize** - The offset of the specified restart point. ``` -------------------------------- ### SurrealKV Configuration Methods Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lib.rs.html Methods for configuring the SurrealKV database instance via the Options builder. ```APIDOC ## Configuration Methods ### Description These methods are used to configure the database instance before building. They follow a builder pattern, returning the modified instance. ### Parameters - **with_l0_no_compression** () - Configures L0 to use no compression. - **with_path** (PathBuf) - Sets the database storage path. - **with_level_count** (u8) - Sets the number of LSM levels. - **with_max_memtable_size** (usize) - Sets the maximum size of a memtable. - **with_block_cache_capacity** (u64) - Sets the unified block cache capacity in bytes. - **with_index_partition_size** (usize) - Configures the partitioned index size. - **with_vlog_max_file_size** (u64) - Sets the maximum size for VLog files. - **with_vlog_checksum_verification** (VLogChecksumLevel) - Sets the VLog checksum verification level. - **with_enable_vlog** (bool) - Enables or disables the Value Log (VLog). - **with_vlog_value_threshold** (usize) - Sets the threshold in bytes for storing values in VLog vs SSTables. - **with_versioning** (bool, u64) - Enables versioned queries and sets retention time in nanoseconds. - **with_versioned_index** (bool) - Enables the B+tree versioned index. - **with_flush_on_close** (bool) - Determines if memtables are flushed during shutdown. - **with_wal_recovery_mode** (WalRecoveryMode) - Sets the WAL recovery behavior. - **with_memtable_stall_threshold** (usize) - Sets the threshold for write stalls based on immutable memtables. - **with_l0_stall_threshold** (usize) - Sets the threshold for write stalls based on L0 file count. ``` -------------------------------- ### Get Memtable Stall Limit Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/stall.rs.html Retrieves the configured memtable stall limit. ```rust self.thresholds.memtable_limit ``` -------------------------------- ### Manage Directory Structure Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lsm.rs.html Creates the necessary subdirectories for SSTables, WAL, manifests, and optional VLog or versioning components. ```rust fn create_directory_structure(opts: &Options) -> Result<()> { // Create base directory create_dir_all(&opts.path)?; // Create all subdirectories create_dir_all(opts.sstable_dir())?; create_dir_all(opts.wal_dir())?; create_dir_all(opts.manifest_dir())?; // Create VLog directories if opts.enable_vlog { create_dir_all(opts.vlog_dir())?; } if opts.enable_versioning { create_dir_all(opts.versioned_index_dir())?; } Ok(()) } ``` -------------------------------- ### Disable compression in TreeBuilder Source: https://docs.rs/surrealkv/0.21.0/surrealkv/struct.TreeBuilder.html Example of configuring a TreeBuilder to disable data block compression. ```rust use surrealkv::TreeBuilder; let tree = TreeBuilder::new() .with_path("./data".into()) .without_compression() .build() .unwrap(); ``` -------------------------------- ### Initialize and Open WAL Instance Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/manager.rs.html Opens or creates a new WAL instance, ensuring directory preparation and stale file cleanup. ```rust pub(crate) fn open(dir: &Path, opts: Options) -> Result { // Ensure the options are valid opts.validate()?; // Ensure the directory exists with proper permissions Self::prepare_directory(dir, &opts)?; // Clean up any stale .wal.repair files from previous crashed repair attempts Self::cleanup_stale_repair_files(dir)?; // Determine the active log number let active_log_number = Self::calculate_active_log_number(dir)?; // Create the active Writer and sync fd let (active_writer, sync_fd) = Self::create_writer(dir, active_log_number, &opts)?; Ok(Self { active_writer, sync_fd, active_log_number, dir: dir.to_path_buf(), opts, closed: false, }) } ``` ```rust pub(crate) fn open_with_min_log_number( dir: &Path, min_log_number: u64, opts: Options, ) -> Result { // Ensure the options are valid opts.validate()?; // Ensure the directory exists with proper permissions Self::prepare_directory(dir, &opts)?; // Clean up any stale .wal.repair files from previous crashed repair attempts Self::cleanup_stale_repair_files(dir)?; // Use max of min_log_number and highest existing segment on disk. // This prevents writing to an old segment that could be skipped on recovery. // // Example scenario this fixes: // 1. Crash with segments #5, #6, #7 on disk // 2. Recovery replays all three into memtable // 3. Without this fix, new writes would go to segment #5 // 4. If memtable flushes (log_number = 6), then crash again // 5. Segment #5 would be skipped, losing the new writes let highest_on_disk = Self::calculate_active_log_number(dir).unwrap_or(0); let active_log_number = std::cmp::max(min_log_number, highest_on_disk); if active_log_number > min_log_number { log::info!( "WAL: advancing from min_log_number {} to {} (highest existing segment)", min_log_number, active_log_number ); } // Create the active Writer and sync fd at the calculated log number let (active_writer, sync_fd) = Self::create_writer(dir, active_log_number, &opts)?; Ok(Self { active_writer, sync_fd, active_log_number, dir: dir.to_path_buf(), opts, closed: false, }) } ``` -------------------------------- ### GET /capacity Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Value.html Returns the total number of elements the vector can hold without reallocating. ```APIDOC ## GET /capacity ### Description Returns the total number of elements the vector can hold without reallocating. ### Method GET ### Response #### Success Response (200) - **capacity** (usize) - The number of elements the vector can hold. ``` -------------------------------- ### Type Identification Source: https://docs.rs/surrealkv/0.21.0/surrealkv/enum.Error.html Provides a method to get the unique TypeId of a static, sized type. ```APIDOC ## GET /api/type_id ### Description Gets the `TypeId` of `self`. ### Method GET ### Endpoint /api/type_id ### Parameters #### Query Parameters - **self** (T) - Required - The object to get the TypeId from. ### Response #### Success Response (200) - **type_id** (TypeId) - The unique identifier for the type. ``` -------------------------------- ### Initialize SurrealKV Core Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lsm.rs.html Initializes the core SurrealKV structure, including setting up stall controllers, task managers, commit environments, and pipelines. It replays the Write-Ahead Log (WAL) and recovers the memtable if necessary. This process ensures data consistency and prepares the database for operation. ```rust let wal_seq_num_opt = Self::replay_wal_with_repair( &wal_path, min_wal_number, "Database startup", opts.wal_recovery_mode, opts.max_memtable_size, |memtable, wal_number| { // Flush intermediate memtable to SST during recovery let table_id = inner.level_manifest.read()?.next_table_id(); inner.flush_immutable_to_sst(Arc::clone(&memtable), table_id, wal_number)?; log::info!( "Recovery: flushed memtable to SST table_id={}, wal_number={}", table_id, wal_number ); Ok(()) }, )?; ``` ```rust if let Some(memtable) = recovered_memtable { let mut active_memtable = inner.active_memtable.write()?; *active_memtable = memtable; } ``` ```rust { let active_memtable = inner.active_memtable.read()?; let current_wal_number = inner.wal.read().get_active_log_number(); active_memtable.set_wal_number(current_wal_number); } ``` ```rust let max_seq_num = match wal_seq_num_opt { Some(wal_seq) => { let effective = std::cmp::max(manifest_last_seq, wal_seq); log::debug!( "WAL replayed: manifest_last_seq={}, wal_seq={}, using max={}", manifest_last_seq, wal_seq, effective ); effective } None => { log::debug!("WAL skipped or empty, using manifest_last_seq={}", manifest_last_seq); manifest_last_seq } }; ``` ```rust commit_pipeline.set_seq_num(max_seq_num); ``` ```rust inner.cleanup_orphaned_sst_files()?; ``` ```rust inner.cleanup_orphaned_vlog_files()?; ``` ```rust task_manager.wake_up_level(); ``` ```rust let core = Self { inner: Arc::clone(&inner), commit_pipeline: Arc::clone(&commit_pipeline), task_manager: Mutex::new(Some(task_manager)), write_stall, }; log::info!("=== LSM tree initialization complete ==="); Ok(core) ``` -------------------------------- ### Options Configuration Source: https://docs.rs/surrealkv/0.21.0/surrealkv/struct.Options.html The `Options` struct allows for detailed configuration of SurrealKV's behavior. You can customize aspects like block size, compression, versioning, and WAL recovery. ```APIDOC ## Struct Options ### Fields - `block_size` (usize) - `block_restart_interval` (usize) - `filter_policy` (Option>) - `comparator` (Arc) - `compression_per_level` (Vec) - `path` (PathBuf) - `level_count` (u8) - `max_memtable_size` (usize) - `index_partition_size` (usize) - `vlog_max_file_size` (u64) - `vlog_checksum_verification` (VLogChecksumLevel) - `enable_vlog` (bool) - If true, disables `VLog` creation entirely - `vlog_value_threshold` (usize) - If value size is less than this, it will be stored inline in `SSTable` - `enable_versioning` (bool) - If true, enables versioned queries with timestamp tracking - `enable_versioned_index` (bool) - If true, creates a B+tree index for timestamp-based queries. Requires `enable_versioning` to be true. When false, versioned queries will scan the LSM tree directly. - `versioned_history_retention_ns` (u64) - History retention period in nanoseconds (0 means no retention limit). Default: 0 (no retention limit) - `flush_on_close` (bool) - If true, flush active memtable to SSTable during shutdown. If false, skip flush for faster shutdown. DEFAULT: false - `wal_recovery_mode` (WalRecoveryMode) - Controls behavior when WAL corruption is detected during recovery. Default: TolerateCorruptedWithRepair (attempt repair and continue) - `level0_max_files` (usize) - Number of L0 files that trigger compaction. Default: 4 - `max_bytes_for_level` (u64) - Base size for level 1 in bytes. Default: 256MB - `level_multiplier` (f64) - Multiplier for calculating max bytes for each level. Level N max bytes = base * multiplier^(N-1). Default: 10.0 - `memtable_stall_threshold` (usize) - Number of immutable memtables that triggers write stall. When immutable memtable count >= this threshold, writes block until flushes complete. Default: 2 - `l0_stall_threshold` (usize) - Number of L0 files that triggers write stall. When L0 file count >= this threshold, writes block until compactions complete. Should be >= `level0_max_files` (compaction trigger). Default: 12 (3x `level0_max_files`) ### impl Options #### pub fn new() -> Self #### pub const fn with_block_size(self, value: usize) -> Self #### pub const fn with_block_restart_interval(self, value: usize) -> Self #### pub fn with_filter_policy(self, value: Option>) -> Self #### pub fn with_comparator(self, value: Arc) -> Self #### pub fn without_compression(self) -> Self Disables compression for data blocks in SSTables. This clears the `compression_per_level` vector, causing all levels to default to `CompressionType::None`. ##### Example ```rust use surrealkv::Options; let opts = Options::new().without_compression(); ``` #### pub fn with_compression_per_level(self, levels: Vec) -> Self Sets compression per level. Vector index corresponds to level number. If vector is shorter than level count, last compression type is used for higher levels. If vector is empty, global compression setting is used. ##### Example ```rust use surrealkv::{Options, CompressionType}; let opts = Options::new() .with_compression_per_level(vec![ CompressionType::None, // L0: no compression CompressionType::SnappyCompression, // L1+: Snappy compression ]); ``` #### pub fn with_l0_no_compression(self) -> Self Convenience method: no compression on L0, Snappy compression on other levels. Equivalent to `with_compression_per_level(vec![CompressionType::None, CompressionType::SnappyCompression])`. ##### Example ```rust use surrealkv::Options; let opts = Options::new().with_l0_no_compression(); ``` #### pub fn with_path(self, value: PathBuf) -> Self #### pub const fn with_level_count(self, value: u8) -> Self #### pub const fn with_max_memtable_size(self, value: usize) -> Self ``` -------------------------------- ### Get Active Log Number Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/manager.rs.html Retrieves the current active log number for the WAL. ```APIDOC ## GET /wal/log_number ### Description Returns the current log number of the active Write-Ahead Log file. ### Method GET ### Endpoint /wal/log_number ### Response #### Success Response (200) - **log_number** (u64) - The current active log number. #### Response Example ```json { "log_number": 12345 } ``` ``` -------------------------------- ### Get WAL Directory Path Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/manager.rs.html Retrieves the directory path where WAL files are stored. ```APIDOC ## GET /wal/dir ### Description Returns the absolute path to the directory where the Write-Ahead Log files are stored. ### Method GET ### Endpoint /wal/dir ### Response #### Success Response (200) - **path** (string) - The directory path for WAL files. #### Response Example ```json { "path": "/var/lib/surrealkv/wal" } ``` ``` -------------------------------- ### VLog File Initialization Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Initializes VLog files by checking for existing files, setting the next file ID, and preparing the active writer. ```APIDOC ## Initialize VLog Files ### Description This function initializes the VLog files. It checks for existing files, determines the next available file ID, and sets up the active writer for new entries. If existing files are found, it uses the highest existing file ID to set the `next_file_id` and prepares the last file as the active writer. ### Method Internal function (not directly exposed as an API endpoint) ### Endpoint N/A ### Parameters None ### Request Body None ### Request Example None ### Response #### Success Response (Ok(())) - Indicates successful initialization of VLog files. #### Response Example None ### Error Handling - Returns `Err(Error::Io)` if there's an issue opening VLog files. - Returns `Err(Error::Other)` for other I/O related errors during initialization. ``` -------------------------------- ### Prefill File Handles Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/vlog.rs.html Scans the VLog directory to populate the file handle cache and register existing files. ```rust fn prefill_file_handles(&self) -> Result<()> { let entries = match std::fs::read_dir(&self.path) { Ok(entries) => entries, Err(_) => return Ok(()), // Directory doesn't exist yet }; let mut max_file_id: Option = None; let mut max_file_path: Option = None; let mut file_handles = self.file_handles.write(); let mut files_map = self.files_map.write(); for entry in entries { let entry = entry?; let file_name = entry.file_name(); let file_name_str = file_name.to_string_lossy(); // Look for VLog files if let Some(file_id) = self.opts.extract_vlog_file_id(&file_name_str) { let file_path = entry.path(); let file_size = entry.metadata()?.len(); // Track the file with maximum ID for active writer setup if max_file_id.is_none_or(|current_max| file_id >= current_max) { max_file_id = Some(file_id); max_file_path = Some(file_path.clone()); } // Pre-open the file handle and validate header match File::open(&file_path) { Ok(file) => { // Validate file header for existing files if file_size > 0 { let mut header_data = vec![0u8; VLogFileHeader::SIZE]; vfs::File::read_at(&file, 0, &mut header_data)?; let header = VLogFileHeader::decode(&header_data)?; header.validate(file_id)?; } let handle = Arc::new(file); file_handles.insert(file_id, handle); // Also register in files_map files_map.insert( file_id, Arc::new(VLogFile::new(file_id, file_path, file_size)), ); } ``` -------------------------------- ### Get IOError Kind Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/mod.rs.html Accessor method to retrieve the io::ErrorKind from an IOError instance. ```rust pub(crate) fn kind(&self) -> io::ErrorKind { self.kind } ``` -------------------------------- ### Example: Versioning with Retention Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/iter.rs.html Illustrates data retention based on versioning and a time-based retention policy. Older versions are kept if they fall within the retention period, otherwise they are dropped. ```text Input: ("key1", seq=100, PUT, "v3", timestamp=now) ("key1", seq=50, PUT, "v2", timestamp=now-30min) ("key1", seq=20, PUT, "v1", timestamp=now-2hours) With versioning=true, retention=1hour, no snapshots: - seq=100: age=0, KEEP (latest) - seq=50: age=30min < 1hour, KEEP (within retention) - seq=20: age=2hours > 1hour, DROP (expired) Output: [seq=100, seq=50] delete_list: [seq=20] ``` -------------------------------- ### Get sequence number from InternalKeyRef Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lib.rs.html Retrieves the sequence number from the InternalKeyRef by decoding its trailer. ```rust pub fn seq_num(&self) -> u64 { InternalKey::seq_num_from_encoded(self.encoded) } ``` -------------------------------- ### Open WAL File with Options Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/manager.rs.html Opens a Write-Ahead Log (WAL) file with specified read, write, create, and append options. Includes platform-specific file mode settings for Unix. ```rust fn open_wal_file(file_path: &Path, opts: &Options) -> Result { let mut open_options = OpenOptions::new(); open_options.read(true).write(true).create(true).append(true); #[cfg(unix)] { use std::os::unix::fs::OpenOptionsExt; if let Some(file_mode) = opts.file_mode { open_options.mode(file_mode); } } Ok(open_options.open(file_path)?) } ``` -------------------------------- ### Get sequence number from InternalKey Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/lib.rs.html Extracts the sequence number from an InternalKey by converting its trailer. ```rust pub(crate) fn seq_num(&self) -> u64 { trailer_to_seq_num(self.trailer) } ``` -------------------------------- ### Skiplist::new Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/memtable/skiplist.rs.html Initializes a new Skiplist instance with a provided arena and comparison function, setting up head and tail sentinels. ```APIDOC ## Skiplist::new ### Description Constructs a new Skiplist, allocating head and tail sentinel nodes and linking them across all levels. ### Parameters - **arena** (Arc) - Required - The memory arena for node allocation. - **cmp** (Compare) - Required - The comparison logic for keys. ``` -------------------------------- ### Create Memtable Iterator Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/memtable/mod.rs.html Creates a new iterator for the memtable, starting from the first element. ```rust pub(crate) fn iter(&self) -> MemTableIterator<'_> { self.range(None, None) } ``` -------------------------------- ### Prepare Directory Permissions Source: https://docs.rs/surrealkv/0.21.0/src/surrealkv/wal/manager.rs.html Ensures a directory exists and sets its permissions. This function is primarily for Unix-based systems, as Windows handles directory permissions differently. ```rust fn prepare_directory(dir: &Path, opts: &Options) -> Result<()> { // Directory should already be created by Tree::new() // Set permissions on Unix only; Windows NTFS uses ACLs and // set_permissions on directories can fail with ERROR_ACCESS_DENIED. #[cfg(unix)] if let Ok(metadata) = fs::metadata(dir) { let mut permissions = metadata.permissions(); use std::os::unix::fs::PermissionsExt; permissions.set_mode(opts.dir_mode.unwrap_or(0o750)); fs::set_permissions(dir, permissions)?; } Ok(()) } ``` -------------------------------- ### unwrap_err() - Get Err value Source: https://docs.rs/surrealkv/0.21.0/surrealkv/type.Result.html Returns the contained `Err` value. Panics if the `Result` is an `Ok`. ```APIDOC ## GET /api/result/unwrap_err ### Description Returns the contained `Err` value. Panics if the `Result` is an `Ok`. ### Method GET ### Endpoint /api/result/unwrap_err ### Parameters #### Query Parameters - **value** (Result) - Required - The Result to unwrap. ### Request Example ```json { "value": "Err(\"emergency failure\")" } ``` ### Response #### Success Response (200) - **error** (E) - The error value contained within the `Err` variant. #### Response Example ```json { "error": "emergency failure" } ``` #### Error Response (Panic) Panics if the input is an `Ok` variant. ```