### Basic Usage and Database Creation in .NET Source: https://github.com/koculu/zonetree/blob/main/README.md Demonstrates the fundamental steps to initialize and use ZoneTree, including creating a new database instance. This snippet covers the initial setup required to start interacting with the database. ```.NET using ZoneTree; // Create a new database instance var database = new ZoneTree(); // You can now perform operations like Put, Get, Delete, etc. // Example: Putting a key-value pair database.Put("myKey", "myValue"); // Example: Getting a value by key var value = database.Get("myKey"); // Example: Deleting a key database.Delete("myKey"); ``` -------------------------------- ### Basic ZoneTree Setup Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/quick-start.md Demonstrates the most basic setup of a ZoneTree database, opening or creating it with default configurations for key and value types. ```C# using var zoneTree = new ZoneTreeFactory() .OpenOrCreate(); zoneTree.Upsert(39, "Hello Zone Tree"); ``` -------------------------------- ### Basic Usage and Database Creation in ZoneTree Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Demonstrates the fundamental steps to get started with ZoneTree, including creating a new database instance and performing basic operations. This snippet highlights the ease of integration into .NET applications. ```C# using ZoneTree; // Create a new database instance var database = new ZoneTree("myDatabase"); // Insert a key-value pair database.Insert("key1", 100); // Retrieve a value var value = database.Find("key1"); // Close the database when done database.Dispose(); ``` -------------------------------- ### Install ZoneTree NuGet Package Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/quick-start.md Installs the ZoneTree NuGet package into your .NET project using the dotnet CLI. ```shell dotnet add package ZoneTree ``` -------------------------------- ### Basic ZoneTree Usage Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md A simple C# example demonstrating how to create and use a ZoneTree instance for basic upsert operations. ```csharp using Tenray.ZoneTree; using var zoneTree = new ZoneTreeFactory() .OpenOrCreate(); zoneTree.Upsert(39, "Hello ZoneTree"); ``` -------------------------------- ### Basic ZoneTree Usage Source: https://github.com/koculu/zonetree/blob/main/README.md A simple C# example demonstrating how to create and use a ZoneTree instance for basic upsert operations. ```csharp using Tenray.ZoneTree; using var zoneTree = new ZoneTreeFactory() .OpenOrCreate(); zoneTree.Upsert(39, "Hello ZoneTree"); ``` -------------------------------- ### Multi-Part Disk Segment File Distribution Example Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/tuning-disk-segment.md Illustrates the file distribution and merge process in ZoneTree's multi-part disk segment mode, showing how data is distributed across files and how merges only require rewriting a subset of files. ```text - We have 3 files, f1, f2, f3 with random lengths between 3-5 | 1 2 5 | 7 8 9 11 | 15 16 | - now let's merge 13 to the disk segment | 1 2 5 | 7 8 9 11 13 | 15 16 | (only f2 is rewritten) - now let's merge 14 to the disk segment | 1 2 5 | 7 8 9 11 13 | 14 15 16 | (only f3 is rewritten) ``` -------------------------------- ### Classical Transactions Example Source: https://github.com/koculu/zonetree/blob/main/README.md Illustrates the classical approach to ZoneTree transactions, which involves manually managing transaction IDs, preparing, and committing. This method provides explicit control over the transaction lifecycle and includes basic error handling for aborted transactions. ```csharp using System; using System.Threading; using Tenray.ZoneTree; using Tenray.ZoneTree.Transaction; using var zoneTree = new ZoneTreeFactory() .OpenOrCreateTransactional(); try { var txId = zoneTree.BeginTransaction(); zoneTree.TryGet(txId, 3, out var value); zoneTree.Upsert(txId, 3, 9); var result = zoneTree.Prepare(txId); while (result.IsPendingTransactions) { Thread.Sleep(100); result = zoneTree.Prepare(txId); } zoneTree.Commit(txId); } catch (TransactionAbortedException) { // Handle aborted transaction (retry or cancel) } ``` -------------------------------- ### Exceptionless Transactions with Retry Logic Source: https://github.com/koculu/zonetree/blob/main/README.md Shows how to implement exceptionless transactions in ZoneTree, focusing on handling potential failures during upsert and prepare operations. This example includes a retry mechanism for the prepare step to manage pending transactions gracefully. ```csharp using System; using System.Threading.Tasks; using Tenray.ZoneTree.Transactional; public async Task ExecuteTransactionWithRetryAsync(ZoneTreeFactory zoneTreeFactory, int maxRetries = 3) { using var zoneTree = zoneTreeFactory.OpenOrCreateTransactional(); var transactionId = zoneTree.BeginTransaction(); // Execute the operations within the transaction var result = zoneTree.UpsertNoThrow(transactionId, 1, 100); if (result.IsAborted) return false; // Abort the transaction and return false on failure. result = zoneTree.UpsertNoThrow(transactionId, 2, 200); if (result.IsAborted) return false; // Abort the transaction and return false on failure. // Retry only the prepare step int retryCount = 0; while (retryCount <= maxRetries) { var prepareResult = zoneTree.PrepareNoThrow(transactionId); if (prepareResult.IsAborted) return false; // Abort the transaction and return false on failure. if (prepareResult.IsPendingTransactions) { await Task.Delay(100); // Simple delay before retrying retryCount++; continue; // Retry the prepare step } if (prepareResult.IsReadyToCommit) break; // Exit loop if ready to commit } // After successfully preparing, commit the transaction var commitResult = zoneTree.CommitNoThrow(transactionId); if (commitResult.IsAborted) return false; // Abort the transaction and return false on failure. // Transaction committed successfully return true; } ``` -------------------------------- ### Exceptionless Transactions with Retry Logic Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Shows how to implement exceptionless transactions in ZoneTree, focusing on handling potential failures during upsert and prepare operations. This example includes a retry mechanism for the prepare step to manage pending transactions gracefully. ```csharp using System; using System.Threading.Tasks; using Tenray.ZoneTree.Transactional; public async Task ExecuteTransactionWithRetryAsync(ZoneTreeFactory zoneTreeFactory, int maxRetries = 3) { using var zoneTree = zoneTreeFactory.OpenOrCreateTransactional(); var transactionId = zoneTree.BeginTransaction(); // Execute the operations within the transaction var result = zoneTree.UpsertNoThrow(transactionId, 1, 100); if (result.IsAborted) return false; // Abort the transaction and return false on failure. result = zoneTree.UpsertNoThrow(transactionId, 2, 200); if (result.IsAborted) return false; // Abort the transaction and return false on failure. // Retry only the prepare step int retryCount = 0; while (retryCount <= maxRetries) { var prepareResult = zoneTree.PrepareNoThrow(transactionId); if (prepareResult.IsAborted) return false; // Abort the transaction and return false on failure. if (prepareResult.IsPendingTransactions) { await Task.Delay(100); // Simple delay before retrying retryCount++; continue; // Retry the prepare step } if (prepareResult.IsReadyToCommit) break; // Exit loop if ready to commit } // After successfully preparing, commit the transaction var commitResult = zoneTree.CommitNoThrow(transactionId); if (commitResult.IsAborted) return false; // Abort the transaction and return false on failure. // Transaction committed successfully return true; } ``` -------------------------------- ### Classical Transactions Example Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Illustrates the classical approach to ZoneTree transactions, which involves manually managing transaction IDs, preparing, and committing. This method provides explicit control over the transaction lifecycle and includes basic error handling for aborted transactions. ```csharp using System; using System.Threading; using Tenray.ZoneTree; using Tenray.ZoneTree.Transaction; using var zoneTree = new ZoneTreeFactory() .OpenOrCreateTransactional(); try { var txId = zoneTree.BeginTransaction(); zoneTree.TryGet(txId, 3, out var value); zoneTree.Upsert(txId, 3, 9); var result = zoneTree.Prepare(txId); while (result.IsPendingTransactions) { Thread.Sleep(100); result = zoneTree.Prepare(txId); } zoneTree.Commit(txId); } catch (TransactionAbortedException) { // Handle aborted transaction (retry or cancel) } ``` -------------------------------- ### ZoneTree TTL with Sliding Expiration Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/TTL-support.md Illustrates the setup of a ZoneTree database with sliding expiration. This example shows how to configure the factory and then use `TryGetAndUpdate` to extend the expiration time of a key when it's accessed. ```C# using var zoneTree = new ZoneTreeFactory>() .SetValueSerializer(new StructSerializer>()) .SetIsDeletedDelegate((in int key, in TTLValue value) => value.IsExpired) .SetMarkValueDeletedDelegate(void (ref TTLValue value) => value.Expire()) .OpenOrCreate(); var expiration = DateTime.UtcNow.AddSeconds(30); zoneTree.Upsert(5, new TTLValue(99, expiration)); var found = zoneTree.TryGetAndUpdate( 5, out value, bool (ref TTLValue v) => v.SlideExpiration(TimeSpan.FromSeconds(15))); ``` -------------------------------- ### Zonetree Configuration Parameters Source: https://github.com/koculu/zonetree/blob/main/src/Playground/BenchmarkForAllModes-DSB-1MB.txt Configuration parameters for the Zonetree project, including thresholds for merge operations, segment sizes, and backup/compression settings. ```configuration ThresholdForMergeOperationStart: 2000000 MutableSegmentMaxItemCount: 1000000 EnableIncrementalBackup: False EnableDiskSegmentCompression: True WALCompressionBlockSize: 262144 DiskCompressionBlockSize: 1048576 DiskSegmentMaximumCachedBlockCount: 1 MinimumSparseArrayLength: 0 EnableParalelInserts: False ``` -------------------------------- ### Benchmark Environment Details Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/performance.md Details the hardware and software environment used for running the ZoneTree benchmarks, including BenchmarkDotNet version, operating system, CPU, memory, and storage. ```text BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000 Intel Core i7-6850K CPU 3.60GHz (Skylake), 1 CPU, 12 logical and 6 physical cores 64 GB DDR4 Memory SSD: Samsung SSD 850 EVO 1TB Config: 1M mutable segment size, 2M readonly segments merge-threshold ``` -------------------------------- ### ZoneTree Bottom Segment Merge Operation Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/terminology.md Explains how to initiate a merge operation for bottom disk segments. This is a background operation that can be started by calling a specific maintenance method with desired 'from-to' values. ```en To start a bottom segments merge operation: Call IZoneTree.Maintenance.StartBottomSegmentsMergeOperation with desired from-to values. ``` -------------------------------- ### ZoneTree Configuration Parameters Source: https://github.com/koculu/zonetree/blob/main/src/Playground/Compression-100M.txt This snippet details the configuration parameters for the ZoneTree project, covering aspects like merge operation thresholds, segment management, backup and compression settings, and WAL (Write-Ahead Logging) configurations. ```plaintext ThresholdForMergeOperationStart: 2000000 MutableSegmentMaxItemCount: 1000000 EnableIncrementalBackup: False EnableDiskSegmentCompression: True WALCompressionBlockSize: 262144 DiskCompressionBlockSize: 10485760 DiskSegmentMaximumCachedBlockCount: 1 MinimumSparseArrayLength: 0 EnableParalelInserts: True DiskSegmentMode: SingleDiskSegment CompressionMethod: LZ4 CompressionLevel: 0 ``` -------------------------------- ### Manual Merge of Disk Segments in ZoneTree Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/quick-start.md Demonstrates how to manually trigger merge operations for disk segments in ZoneTree after inserting a large volume of data. This includes moving the mutable segment forward and starting a merge operation. ```C# var dataPath = "data/mydatabase"; using var zoneTree = new ZoneTreeFactory() .SetDataDirectory(dataPath) .OpenOrCreate(); for (var i = 0; i < 10_000_000; ++i){ zoneTree.Insert(i, i+i); } zoneTree.Maintenance.MoveMutableSegmentForward(); zoneTree.Maintenance.StartMergeOperation()?.Join(); ``` -------------------------------- ### ZoneTree Benchmark Configuration Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/performance.md Defines the configuration parameters used for benchmarking ZoneTree, including disk compression block sizes, segment modes, and mutable segment item counts. ```csharp DiskCompressionBlockSize = 1024 * 1024 * 10; WALCompressionBlockSize = 1024 * 32 * 8; DiskSegmentMode = DiskSegmentMode.SingleDiskSegment; MutableSegmentMaxItemCount = 1_000_000; ThresholdForMergeOperationStart = 2_000_000; ``` -------------------------------- ### int-int Insert Performance Benchmarks Source: https://github.com/koculu/zonetree/blob/main/src/Playground/Compression-100M.txt Performance benchmarks for inserting integer key-value pairs. It compares different compression methods (LZ4, Brotli, Zstd, Gzip) and reports metrics such as memory usage, load time, insert time, merge time, and disk usage. ```json [ { "Section": "int-int insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "LZ4", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "759.26 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 77, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 39435, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 4115, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 43627 }, "MemoryUsageAtBegin": 92168, "MemoryUsageAtEnd": 2782339360 }, { "Section": "int-int insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Brotli", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "562.29 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 6, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 41915, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 3044, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 44965 }, "MemoryUsageAtBegin": 371592, "MemoryUsageAtEnd": 2551393384 }, { "Section": "int-int insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Zstd", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "561.89 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 5, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 45994, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 4441, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 50440 }, "MemoryUsageAtBegin": 415640, "MemoryUsageAtEnd": 2812703864 }, { "Section": "int-int insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Gzip", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "300.5 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 5, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 54500, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 38658, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 93163 }, "MemoryUsageAtBegin": 465504 } ] ``` -------------------------------- ### ZoneTree Benchmark Configuration Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Configuration settings used for ZoneTree benchmarks, including disk and WAL compression block sizes, disk segment mode, and mutable segment parameters. ```csharp DiskCompressionBlockSize = 10 * 1024 * 1024; // 10 MB WALCompressionBlockSize = 32 * 1024 * 8; // 256 KB DiskSegmentMode = DiskSegmentMode.SingleDiskSegment; MutableSegmentMaxItemCount = 1_000_000; ThresholdForMergeOperationStart = 2_000_000; ``` -------------------------------- ### Fluent Transactions Example Source: https://github.com/koculu/zonetree/blob/main/README.md Demonstrates how to use ZoneTree's Fluent API for transactional operations. This approach allows chaining operations and configuring retry logic in a readable manner. It handles upserts and checks within a single transaction block. ```csharp using System.Threading.Tasks; using Tenray.ZoneTree; using Tenray.ZoneTree.Transaction; using var zoneTree = new ZoneTreeFactory() .OpenOrCreateTransactional(); using var transaction = zoneTree .BeginFluentTransaction() .Do(tx => zoneTree.UpsertNoThrow(tx, 3, 9)) .Do(tx => { if (zoneTree.TryGetNoThrow(tx, 3, out var value).IsAborted) return TransactionResult.Aborted(); if (zoneTree.UpsertNoThrow(tx, 3, 21).IsAborted) return TransactionResult.Aborted(); return TransactionResult.Success(); }) .SetRetryCountForPendingTransactions(100) .SetRetryCountForAbortedTransactions(10); await transaction.CommitAsync(); ``` -------------------------------- ### ZoneTree Features Overview Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/features.md This section outlines the key features of the ZoneTree library, highlighting its capabilities in performance, data management, transaction support, and flexibility for .NET applications. ```.NET Works with .NET primitives, structs and classes. High Speed and Low Memory consumption. Crash Resilience Optimum disk space utilization. WAL and DiskSegment data compression. Very fast load/unload. Standard read/upsert/delete functions. Optimistic Transaction Support Atomic Read Modify Update Can work in memory. Can work with any disk device including cloud devices. Supports optimistic transactions. Supports Atomicity, Consistency, Isolation, Durability. Supports Read Committed Isolation. 4 different modes for write ahead log. Audit support with incremental transaction log backup. Live backup. Configurable amount of data that can stay in memory. Partially (with sparse arrays) or completely load/unload data on disk to/from memory. Forward/Backward iteration. Allow optional dirty reads. Embeddable. Optimized for SSDs. Exceptionless Transaction API. Fluent Transaction API with ready to use retry capabilities. Easy Maintenance. Configurable LSM merger. Transparent and simple implementation that reveals your database's internals. Fully open-source with unrestrictive MIT license. Transaction Log compaction. analyze / control transactions. Concurrency Control with minimum overhead by novel separation of Concurrency Stamps and Data. TTL support. Use your custom serializer for keys and values. Use your custom comparer. MultipleDiskSegments Mode to enable dividing data files into configurable sized chunks. Snapshot iterators. ``` -------------------------------- ### ZoneTree Write-Ahead Log (WAL) Modes Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Explanation of the four Write-Ahead Log (WAL) modes available in ZoneTree: Sync, Sync-Compressed, Async-Compressed, and None. Each mode details its durability, performance characteristics, and typical use cases. ```text ZoneTree offers four WAL modes to provide flexibility between performance and durability: 1. Sync Mode: - Durability: Maximum. - Performance: Slower write speed. - Use Case: Ensures data is not lost in case of crashes or power cuts. 2. Sync-Compressed Mode: - Durability: High, but slightly less than sync mode. - Performance: Faster write speed due to compression. - Use Case: Balances durability and performance; periodic jobs can persist decompressed tail records for added safety. 3. Async-Compressed Mode: - Durability: Moderate. - Performance: Very fast write speed; logs are written in a separate thread. - Use Case: Suitable where immediate durability is less critical but performance is paramount. 4. None Mode: - Durability: No immediate durability; relies on manual or automatic disk saves. - Performance: Maximum possible. - Use Case: Ideal for scenarios where performance is critical and data can be reconstructed or is not mission-critical. ``` -------------------------------- ### Fluent Transactions Example Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Demonstrates how to use ZoneTree's Fluent API for transactional operations. This approach allows chaining operations and configuring retry logic in a readable manner. It handles upserts and checks within a single transaction block. ```csharp using System.Threading.Tasks; using Tenray.ZoneTree; using Tenray.ZoneTree.Transaction; using var zoneTree = new ZoneTreeFactory() .OpenOrCreateTransactional(); using var transaction = zoneTree .BeginFluentTransaction() .Do(tx => zoneTree.UpsertNoThrow(tx, 3, 9)) .Do(tx => { if (zoneTree.TryGetNoThrow(tx, 3, out var value).IsAborted) return TransactionResult.Aborted(); if (zoneTree.UpsertNoThrow(tx, 3, 21).IsAborted) return TransactionResult.Aborted(); return TransactionResult.Success(); }) .SetRetryCountForPendingTransactions(100) .SetRetryCountForAbortedTransactions(10); await transaction.CommitAsync(); ``` -------------------------------- ### ZoneTree API Documentation Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/README-NUGET.md Provides comprehensive API references for ZoneTree, detailing its classes, methods, and functionalities for .NET developers. ```APIDOC ZoneTree API Documentation: This documentation covers the Tenray.ZoneTree namespace, providing details on classes, methods, and their usage within the ZoneTree library. Key Classes and Functionalities: - **ZoneTree**: The core data structure for efficient data management. - Methods: - `Add(TKey key, TValue value)`: Adds a key-value pair to the ZoneTree. - `Remove(TKey key)`: Removes a key and its associated value from the ZoneTree. - `Find(TKey key)`: Retrieves the value associated with a given key. - `Scan(TKey startKey, TKey endKey)`: Returns an enumerable collection of key-value pairs within a specified range. - **DiskSegment**: Manages data storage on disk for large datasets. - Properties: - `SegmentSize`: Configures the size of disk segments. - `CompressionType`: Specifies the compression algorithm to use. - Methods: - `Load()`: Loads the segment data from disk. - `Save()`: Saves the current segment data to disk. - **ZoneTreeBuilder**: Facilitates the creation and configuration of ZoneTree instances. - Methods: - `WithComparer(IComparer comparer)`: Sets a custom comparer for keys. - `WithDiskSegment(DiskSegment diskSegment)`: Integrates a pre-configured disk segment. - `Build()`: Constructs and returns a new ZoneTree instance. Refer to the official API documentation for detailed parameter descriptions, return types, and usage examples for each method and class. ``` -------------------------------- ### Forward and Backward Iteration Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/quick-start.md Shows how to iterate over data in ZoneTree in both forward and backward directions. The performance for both iteration directions is consistent. ```csharp using var zoneTree = new ZoneTreeFactory() // Additional stuff goes here .OpenOrCreate(); using var iterator = zoneTree.CreateIterator(); while(iterator.Next()) { var key = iterator.CurrentKey; var value = iterator.CurrentValue; } using var reverseIterator = zoneTree.CreateReverseIterator(); while(reverseIterator.Next()) { var key = reverseIterator.CurrentKey; var value = reverseIterator.CurrentValue; } ``` -------------------------------- ### ZoneTree Performance Benchmarks Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/performance.md A table summarizing insertion benchmark results for ZoneTree and RocksDb with integer and string data types, across different WAL and compression settings. Shows performance in milliseconds for 1M, 2M, 3M, and 10M records. ```text | Insert Benchmarks | 1M | 2M | 3M | 10M | | ------------------------------------- | ------- | -------- | -------- | -------- | | int-int ZoneTree async-compressed WAL | 267 ms | 464 ms | 716 ms | 2693 ms | | int-int ZoneTree sync-compressed WAL | 834 ms | 1617 ms | 2546 ms | 8642 ms | | int-int ZoneTree sync WAL | 2742 ms | 5533 ms | 8242 ms | 27497 ms | | | | str-str ZoneTree async-compressed WAL | 892 ms | 1833 ms | 2711 ms | 9443 ms | | str-str ZoneTree sync-compressed WAL | 1752 ms | 3397 ms | 5070 ms | 19153 ms | | str-str ZoneTree sync WAL | 3488 ms | 7002 ms | 10483 ms | 38727 ms | | | | int-int RocksDb sync-compressed WAL | 8059 ms | 16188 ms | 23599 ms | 61947 ms | | str-str RocksDb sync-compressed WAL | 8215 ms | 16146 ms | 23760 ms | 72491 ms | ``` -------------------------------- ### Zonetree Configuration Parameters Source: https://github.com/koculu/zonetree/blob/main/src/Playground/BenchmarkForAllModes.txt Configuration parameters for Zonetree, including thresholds for merge operations, segment item counts, and backup/compression settings. ```config ThresholdForMergeOperationStart: 2000000 MutableSegmentMaxItemCount: 1000000 EnableIncrementalBackup: False EnableDiskSegmentCompression: True WALCompressionBlockSize: 262144 DiskCompressionBlockSize: 10485760 DiskSegmentMaximumCachedBlockCount: 1 MinimumSparseArrayLength: 0 EnableParalelInserts: False ``` -------------------------------- ### Using InMemoryFileStreamProvider Source: https://github.com/koculu/zonetree/blob/main/src/ZoneTree/docs/ZoneTree/guide/quick-start.md Demonstrates the usage of InMemoryFileStreamProvider for storing ZoneTree data entirely in memory, which is beneficial for unit testing and temporary storage. ```csharp var provider = new InMemoryFileStreamProvider(); using var zoneTree = new ZoneTreeFactory(provider) .OpenOrCreate(); zoneTree.Upsert(1, "value"); ``` -------------------------------- ### ZoneTree Benchmark Configuration Source: https://github.com/koculu/zonetree/blob/main/README.md Configuration settings used for ZoneTree benchmarks, including disk and WAL compression block sizes, disk segment mode, and mutable segment parameters. ```csharp DiskCompressionBlockSize = 10 * 1024 * 1024; // 10 MB WALCompressionBlockSize = 32 * 1024 * 8; // 256 KB DiskSegmentMode = DiskSegmentMode.SingleDiskSegment; MutableSegmentMaxItemCount = 1_000_000; ThresholdForMergeOperationStart = 2_000_000; ``` -------------------------------- ### str-str Insert Performance Benchmarks Source: https://github.com/koculu/zonetree/blob/main/src/Playground/Compression-100M.txt Performance benchmarks for inserting string key-value pairs. It compares different compression methods (LZ4, Brotli, Zstd, Gzip) and reports metrics such as memory usage, load time, insert time, merge time, and disk usage. ```json [ { "Section": "str-str insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "LZ4", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "1.56 GB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 16, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 136992, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 41595, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 178593 }, "MemoryUsageAtBegin": 483512, "MemoryUsageAtEnd": 16463360000 }, { "Section": "str-str insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Brotli", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "508.73 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 6, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 119369, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 42646, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 162021 }, "MemoryUsageAtBegin": 500000, "MemoryUsageAtEnd": 11744053248 }, { "Section": "str-str insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Zstd", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "301.77 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 7, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 125968, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 59974, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 185949 }, "MemoryUsageAtBegin": 510000, "MemoryUsageAtEnd": 12750000000 }, { "Section": "str-str insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Gzip", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "622.38 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 5, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 141428, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 119064, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 260497 }, "MemoryUsageAtBegin": 517000, "MemoryUsageAtEnd": 12800000000 } ] ``` -------------------------------- ### Zonetree Insert and Iterate Performance (string, string) Source: https://github.com/koculu/zonetree/blob/main/src/Playground/BenchmarkForAllModes-DSB-1MB.txt Performance benchmarks for inserting and iterating string key-value pairs. Tests include different data sizes (1M, 2M, 3M, 10M) with no compression. ```benchmark None Insert 1M Loaded in: 14 Completed in: 794 Merged in: 606 None Iterate 1M Loaded in: 15 Completed in: 451 ``` ```benchmark None Insert 2M Loaded in: 4 Completed in: 1390 Merged in: 1120 None Iterate 2M Loaded in: 7 Completed in: 844 ``` ```benchmark None Insert 3M Loaded in: 4 Completed in: 2098 Merged in: 1725 None Iterate 3M Loaded in: 7 Completed in: 1249 ``` ```benchmark None Insert 10M Loaded in: 5 Completed in: 7460 Merged in: 11634 None Iterate 10M Loaded in: 6 ``` -------------------------------- ### ZoneTree API Documentation Source: https://github.com/koculu/zonetree/blob/main/README.md Provides comprehensive API references for ZoneTree, detailing its classes, methods, and functionalities for .NET developers. ```APIDOC ZoneTree API Documentation: This documentation covers the Tenray.ZoneTree namespace, providing details on classes, methods, and their usage within the ZoneTree library. Key Classes and Functionalities: - **ZoneTree**: The core data structure for efficient data management. - Methods: - `Add(TKey key, TValue value)`: Adds a key-value pair to the ZoneTree. - `Remove(TKey key)`: Removes a key and its associated value from the ZoneTree. - `Find(TKey key)`: Retrieves the value associated with a given key. - `Scan(TKey startKey, TKey endKey)`: Returns an enumerable collection of key-value pairs within a specified range. - **DiskSegment**: Manages data storage on disk for large datasets. - Properties: - `SegmentSize`: Configures the size of disk segments. - `CompressionType`: Specifies the compression algorithm to use. - Methods: - `Load()`: Loads the segment data from disk. - `Save()`: Saves the current segment data to disk. - **ZoneTreeBuilder**: Facilitates the creation and configuration of ZoneTree instances. - Methods: - `WithComparer(IComparer comparer)`: Sets a custom comparer for keys. - `WithDiskSegment(DiskSegment diskSegment)`: Integrates a pre-configured disk segment. - `Build()`: Constructs and returns a new ZoneTree instance. Refer to the official API documentation for detailed parameter descriptions, return types, and usage examples for each method and class. ``` -------------------------------- ### SyncCompressed Insert Performance Source: https://github.com/koculu/zonetree/blob/main/src/Playground/BenchmarkForAllModes-DSB-1MB.txt Benchmarks for synchronous insertion operations with compression. Measures time taken to load, complete, and merge data for various sizes. ```benchmark SyncCompressed Insert 1M Loaded in: 6 Completed in: 1752 Merged in: 0 SyncCompressed Insert 2M Loaded in: 6 Completed in: 3454 Merged in: 0 SyncCompressed Insert 3M Loaded in: 6 Completed in: 5211 Merged in: 0 SyncCompressed Insert 10M Loaded in: 7 Completed in: 19104 Merged in: 7161 ``` -------------------------------- ### String-to-String Insertion with Brotli Compression Source: https://github.com/koculu/zonetree/blob/main/src/Playground/Compression-100M.txt Details the performance metrics for string-to-string insertions using Brotli compression. It includes disk usage, memory consumption before and after the operation, and the time taken for different stages like loading, insertion, and merging. ```json { "Section": "str-str insert", "Name": "Insert", "Options": { "WAL": "None", "Compression": "Brotli", "Count": 100000000 }, "AdditionalStats": { "Disk Usage": "508.73 MB" }, "Stages": { "Stages": { "Loaded In": { "ElapsedMilliseconds": 6, "Name": "Loaded In" }, "Inserted In": { "ElapsedMilliseconds": 119369, "Name": "Inserted In" }, "Merged In": { "ElapsedMilliseconds": 42646, "Name": "Merged In" } }, "TotalElapsedMilliseconds": 162021 }, "MemoryUsageAtBegin": 498600, "MemoryUsageAtEnd": 12020563792 } ``` -------------------------------- ### Sync Insert Performance Source: https://github.com/koculu/zonetree/blob/main/src/Playground/BenchmarkForAllModes-DSB-1MB.txt Benchmarks for synchronous insertion operations without compression. Measures time taken to load, complete, and merge data for various sizes. ```benchmark Sync Insert 1M Loaded in: 4 Completed in: 3514 Merged in: 0 Sync Insert 2M Loaded in: 4 Completed in: 7188 Merged in: 0 Sync Insert 3M Loaded in: 4 Completed in: 10747 Merged in: 0 Sync Insert 10M Loaded in: 4 Completed in: 38499 Merged in: 5497 ```