SlateDB

https://slatedb.io/docs/
Admin
SlateDB is an embedded storage engine built on object storage using a log-structured merge-tree...
Tokens:28,134
Snippets:323
Trust Score:7.3
Update:4 months ago
Show doc for...
Tokens:
Raw
### Go Usage Example

Source: https://slatedb.io/docs/get-started/quickstart.md

A minimal example demonstrating basic put and get operations in Go.

```go
package main
async function main() {
  const store = ObjectStore.resolve("memory:///");
  let db;

  try {
    const builder = new DbBuilder("demo-db", store);
    try {
      db = await builder.build();
    } finally {
      builder.dispose();
    }

    const key = Buffer.from("hello");
    const value = Buffer.from("world");

    await db.put(key, value);

    const read = await db.get(key);
    assert.deepEqual(read, value);

    console.log(Buffer.from(read).toString("utf8"));
  } finally {
    if (db != null) {
      await db.shutdown();
      db.dispose();
    }
    store.dispose();
  }
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});
```

--------------------------------

### Node.js Installation

Source: https://slatedb.io/docs/get-started/quickstart.md

Install the SlateDB Node.js binding.

```bash
npm install @slatedb/uniffi
```

--------------------------------

### Go Installation

Source: https://slatedb.io/docs/get-started/quickstart.md

Install the SlateDB Go binding.

```bash
go get slatedb.io/slatedb-go
```

--------------------------------

### Python Usage Example

Source: https://slatedb.io/docs/get-started/quickstart.md

An example demonstrating basic put, get, write batch, scan, and transaction operations in Python.

```python
import asyncio

from slatedb.uniffi import (
    DbBuilder,
    IsolationLevel,
    ObjectStore,
    WriteBatch,
)

async def main() -> None:
    store = ObjectStore.resolve("memory:///")
    builder = DbBuilder("demo-db", store)
    db = await builder.build()

    try:
        await db.put(b"user:1", b"Alice")
        value = await db.get(b"user:1")
        assert value == b"Alice"

        batch = WriteBatch()
        batch.put(b"user:2", b"Bob")
        batch.put(b"user:3", b"Carol")
        await db.write(batch)

        scan = await db.scan_prefix(b"user:")
        while (row := await scan.next()) is not None:
            print(row.key, row.value)

        tx = await db.begin(IsolationLevel.SERIALIZABLE_SNAPSHOT)
        await tx.put(b"user:4", b"Dora")
        await tx.commit()
    finally:
        await db.shutdown()

asyncio.run(main())
```

--------------------------------

### Python Installation

Source: https://slatedb.io/docs/get-started/quickstart.md

Install the SlateDB Python package.

```bash
pip install slatedb
```

--------------------------------

### Java Installation (Maven)

Source: https://slatedb.io/docs/get-started/quickstart.md

Add the SlateDB Java dependency using Maven.

```xml
<dependency>
  <groupId>io.slatedb</groupId>
  <artifactId>slatedb-uniffi</artifactId>
  <version>&lt;version&gt;</version>
</dependency>
```

--------------------------------

### Java Installation (Gradle)

Source: https://slatedb.io/docs/get-started/quickstart.md

Add the SlateDB Java dependency using Gradle.

```gradle
dependencies {
    implementation("io.slatedb:slatedb-uniffi:<version>")
}
```

--------------------------------

### Rust Installation

Source: https://slatedb.io/docs/get-started/quickstart.md

Install the SlateDB Rust library and Tokio.

```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
```

--------------------------------

### Install LocalStack CLI

Source: https://slatedb.io/docs/tutorials/s3.md

Installs the LocalStack CLI using Homebrew.

```bash
brew install localstack/tap/localstack-cli
```

--------------------------------

### Configuration Example

Source: https://slatedb.io/docs/operations/configuration.md

This example shows all three configuration layers in one place: Settings, Builder configuration, and Per-call options.

```rust
use slatedb::
config::{DbReaderOptions, ReadOptions, Settings};
use slatedb::db_cache::foyer::FoyerCache;
use slatedb::object_store::{memory::InMemory, ObjectStore};
use slatedb::Db;
use slatedb::DbReader;
use slatedb::Error;
use std::sync::Arc;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let object_store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());

    // 1. Serializable runtime settings.
    let mut settings = Settings::default();
    settings.flush_interval = Some(Duration::from_millis(250));

    // 2. Builder configuration for process-local components and wiring.
    let db = Db::builder("example", Arc::clone(&object_store))
        .with_settings(settings)
        .with_db_cache(Arc::new(FoyerCache::new()))
        .build()
        .await?;

    // 3. Per-call options for one operation.
    let _value = db
        .get_with_options(
            b"key",
            &ReadOptions::default().with_cache_blocks(false),
        )
        .await?;

    // Reader-specific options configure the DbReader instance itself.
    let mut reader_options = DbReaderOptions::default();
    reader_options.skip_wal_replay = true;

    let _reader = DbReader::builder("example", object_store)
        .with_options(reader_options)
        .build()
        .await?;

    Ok(())
}
```

--------------------------------

### Install AWS CLI

Source: https://slatedb.io/docs/tutorials/s3.md

Installs the AWS CLI using Homebrew.

```bash
brew install awscli
```

--------------------------------

### Start LocalStack

Source: https://slatedb.io/docs/tutorials/s3.md

Starts LocalStack in detached mode.

```bash
localstack start -d
```

--------------------------------

### Bencher Benchmark Script Example

Source: https://slatedb.io/docs/operations/benchmarks.md

The benchmark-db.sh script serves as an example for configuring and running the bencher tool.

```bash
#!/bin/bash

# Example script for running SlateDB bencher

# Configuration variables (example)
OBJECT_STORE_URL="http://localhost:9000"
ACCESS_KEY="minioadmin"
SECRET_KEY="minioadmin"
BUCKET_NAME="slate-benchmarks"

# Ensure bucket exists (or create it)
# aws s3api create-bucket --endpoint-url $OBJECT_STORE_URL --bucket $BUCKET_NAME --region us-east-1

# Run bencher
# slatedb-bencher \
#   --endpoint $OBJECT_STORE_URL \
#   --access-key $ACCESS_KEY \
#   --secret-key $SECRET_KEY \
#   --bucket $BUCKET_NAME \
#   --num-puts 10000 \
#   --num-gets 10000 \
#   --put-size 1024 \
#   --get-size 1024

echo "Benchmark script executed (example)."

```

--------------------------------

### Partitioned SSTables After Compaction

Source: https://slatedb.io/docs/design/overview.md

Example content of SSTables after compaction, demonstrating range partitioning and removal of older versions.

```text
SSTable 1:
  key1:value1b
  key43:value43
  key44:value44
  key56:value56b
  key77:value77

SSTable 2:
  key89:value89
  key99:value99
```

--------------------------------

### Get Operation with L0 and Sorted Runs

Source: https://slatedb.io/docs/design/overview.md

A sequence diagram illustrating the get() operation's traversal through MemTables, L0 SSTables, and Sorted Runs.

```mermaid
sequenceDiagram
  box Memory
    participant C as Client
    participant MT as MemTable (mutable)
    participant IMT as MemTable (immutable)
  end
  box Disk
    participant L0 as L0 SSTable
    participant SR as SR (sorted run)
  end

  C->>SR: get()
  MT-->>C: key-value pair (if found)
  IMT-->>C: key-value pair (if found)
  L0-->>C: key-value pair (if found)
  SR-->>C: key-value pair (if found)
```

--------------------------------

### Install SlateDB CLI using Cargo

Source: https://slatedb.io/docs/operations/cli.md

Installs the SlateDB CLI using Cargo, ensuring a locked version.

```bash
cargo install --locked slatedb-cli
```

--------------------------------

### Get Operation Across Data Structures

Source: https://slatedb.io/docs/design/overview.md

A sequence diagram illustrating how a get() operation searches across MemTables and SSTables.

```mermaid
sequenceDiagram
  box Memory
    participant C as Client
    participant MT as MemTable (mutable)
    participant IMT as MemTable (immutable)
  end
  box Disk
    participant SST as SSTable
  end

  C->>SST: get()
  MT-->>C: key-value pair (if found)
  IMT-->>C: key-value pair (if found)
  SST-->>C: key-value pair (if found)
```

--------------------------------

### Compacted SST (key order)

Source: https://slatedb.io/docs/design/sorted-string-table.md

Example of Compacted SST content showing key order.

```text
key=apple   seq=42  value=green
key=banana  seq=41  value=yellow
key=carrot  seq=43  value=orange
```

--------------------------------

### Unpartitioned SSTables Before Compaction

Source: https://slatedb.io/docs/design/overview.md

Example content of multiple unpartitioned SSTables, showing overlapping key ranges and different versions of values.

```text
SSTable 1:
  key1:value1a
  key56:value56a
  key99:value99

SSTable 2:
  key43:value43
  key56:value56b
  key89:value89

SSTable 3:
  key1:value1b
  key44:value44
  key77:value77
```

--------------------------------

### Opening a DbReader and reading a key

Source: https://slatedb.io/docs/design/readers.md

This example opens a database, flushes one key to object storage, then opens a DbReader and reads that key back.

```rust
use slatedb::{
    Db,
    DbReader,
    Error,
};
use slatedb::object_store::{memory::InMemory, ObjectStore};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Error> {
    let object_store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());

    let db = Db::open("example", Arc::clone(&object_store)).await?;
    db.put(b"key", b"value").await?;
    db.flush().await?;

    let reader = DbReader::builder("example", object_store).build().await?;
    assert_eq!(reader.get(b"key").await?, Some("value".into()));
    Ok(())
}
```

--------------------------------

### WAL SST (sequence order)

Source: https://slatedb.io/docs/design/sorted-string-table.md

Example of WAL SST content showing sequence order.

```text
seq=41  key=banana  value=yellow
seq=42  key=apple   value=green
seq=43  key=carrot  value=orange
```

--------------------------------

### Setting WriteOnInsertion Policy

Source: https://slatedb.io/docs/operations/foyer-cache.md

Example of setting the write policy to WriteOnInsertion using HybridCacheBuilder.

```rust
use foyer::{
    BlockEngineConfig,
    DeviceBuilder,
    FsDeviceBuilder,
    HybridCacheBuilder,
    HybridCachePolicy,
    PsyncIoEngineConfig,
};

let cache = HybridCacheBuilder::new()
    // ...
    .with_policy(HybridCachePolicy::WriteOnInsertion)
    .storage()
    // ...
    .build()
    .await
    .unwrap();
```

--------------------------------

### Using DefaultMetricsRecorder

Source: https://slatedb.io/docs/operations/metrics.md

Example of how to use DefaultMetricsRecorder for metrics collection, especially useful in tests or scenarios without a dedicated metrics backend. It demonstrates initializing the recorder, building a Db with it, performing operations, and then snapshotting and printing the metrics.

```rust
use slatedb_common::metrics::DefaultMetricsRecorder;
use std::sync::Arc;

let recorder = Arc::new(DefaultMetricsRecorder::new());
let db = Db::builder("test_db", object_store)
    .with_metrics_recorder(recorder.clone())
    .build()
    .await
    .unwrap();

// ... perform operations ...

let snapshot = recorder.snapshot();
for metric in snapshot.all() {
    println!("{}: {:?} {{:?}}", metric.name, metric.labels, metric.value);
}
```

--------------------------------

### Create a project

Source: https://slatedb.io/docs/tutorials/gcs.md

Creates a new Rust project.

```bash
cargo init slatedb-gcs
cd slatedb-gcs
```

--------------------------------

### Create a new Rust project

Source: https://slatedb.io/docs/tutorials/s3.md

Creates a new Rust project and navigates into the project directory.

```bash
cargo init slatedb-playground
cd slatedb-playground
```

--------------------------------

### Run the code

Source: https://slatedb.io/docs/tutorials/s3.md

Compiles and runs the Rust project.

```bash
cargo run
```

--------------------------------

### Create a new Rust project

Source: https://slatedb.io/docs/tutorials/abs.md

Commands to initialize a new Rust project and navigate into its directory.

```bash
cargo init slatedb-abs
cd slatedb-abs
```

--------------------------------

### String Concatenation Merge Operator Example

Source: https://slatedb.io/docs/design/merge-operators.md

An example demonstrating a merge operator that concatenates string fragments. It shows how to implement the MergeOperator trait and use it with Db::merge.

```rust
use bytes::Bytes;
use slatedb::{Db, Error, MergeOperator, MergeOperatorError};
use slatedb::object_store::{memory::InMemory, ObjectStore};
use std::sync::Arc;

struct StringConcatMergeOperator;

impl MergeOperator for StringConcatMergeOperator {
    fn merge(
        &self,
        _key: &Bytes,
        existing_value: Option,
        operand: Bytes,
    ) -> Result {
        let mut result = existing_value.unwrap_or_default().as_ref().to_vec();
        result.extend_from_slice(&operand);
        Ok(Bytes::from(result))
    }
}

#[tokio::main]
async fn main() -> Result<(), Error> {
    let object_store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
    let db = Db::builder("example", object_store)
        .with_merge_operator(Arc::new(StringConcatMergeOperator))
        .build()
        .await?;

    db.merge(b"greeting", b"hello, ").await?;
    db.merge(b"greeting", b"world").await?;

    let value = db.get(b"greeting").await?.unwrap();
    assert_eq!(value.as_ref(), b"hello, world");
    Ok(())
}
```

--------------------------------

### Create Storage bucket and Service Account

Source: https://slatedb.io/docs/tutorials/gcs.md

These steps create a storage bucket and generate service account credentials.

```bash
# Set bucket name and project ID
BucketName=
ProjectID=

# Login to Google Cloud
gcloud auth login

# Set the default project
gcloud config set project $ProjectID

# Create a storage bucket (if needed)
gcloud storage buckets create gs://$BucketName --location=us-central1

# Create a service account for authentication
ServiceAccountName=slatedb-gcs-test
gcloud iam service-accounts create $ServiceAccountName \
    --display-name="SlateDB GCS Test Service Account"

# Grant the service account access to the project (for storage operations)
gcloud projects add-iam-policy-binding $ProjectID \
    --member="serviceAccount:$ServiceAccountName@$ProjectID.iam.gserviceaccount.com" \
    --role="roles/storage.objectAdmin"

# Create and download a key for the service account
gcloud iam service-accounts keys create ~/slatedb-gcs-key.json \
    --iam-account=$ServiceAccountName@$ProjectID.iam.gserviceaccount.com
```

--------------------------------

### Setting SST Block Size

Source: https://slatedb.io/docs/operations/tuning.md

Example of setting the SST block size for optimizing read performance.

```Rust
DbBuilder::with_sst_block_size(4096)
```

--------------------------------

### Initial MemTable Write to Disk

Source: https://slatedb.io/docs/design/overview.md

A sequence diagram illustrating the initial process of writing a MemTable to disk, showing a potential blocking issue.

```mermaid
sequenceDiagram
  box Memory
    participant C as Client
    participant MT as MemTable (mutable)
  end
  box Disk
    participant SST as SSTable
  end

  C->>MT: put()
  MT-->>C: update OK
  MT->>SST: write
  SST-->>C: write OK
```

--------------------------------

### Create Storage Account and List Keys

Source: https://slatedb.io/docs/tutorials/abs.md

Commands to set storage account names, log in to Azure, create a resource group, create an Azure Storage account, create a storage container, and list the account keys.

```bash
# Set storage account names
StorageAccountName=
ContainerName=
ResourceGroupName=

# Login
az login

# Create Resource Group in the default subscription.
az group create --name $ResourceGroupName --location westus

# Create Azure Storage account.
az storage account create --name $StorageAccountName --resource-group $ResourceGroupName --location westus --sku Standard_LRS

# Create a storage container
az storage container create --name $ContainerName --account-name $StorageAccountName

# Get the keys.
az storage account keys list --resource-group $ResourceGroupName --account-name $StorageAccountName
```

--------------------------------

### Add dependencies

Source: https://slatedb.io/docs/tutorials/gcs.md

Adds SlateDB and required dependencies to Cargo.toml.

```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/gcp
cargo add anyhow
```

--------------------------------

### Fixed-Length Components Example

Source: https://slatedb.io/docs/operations/data-modeling.md

Concatenating fixed-length components directly for a 12-byte key composed of a 4-byte topic ID and an 8-byte offset.

```rust
let mut key = Vec::with_capacity(12);
key.extend_from_slice(&topic_id.to_be_bytes());
key.extend_from_slice(&offset.to_be_bytes());
```

--------------------------------

### Create a Db with no compactor

Source: https://slatedb.io/docs/tutorials/standalone-compactor.md

To prevent `Db` from starting the embedded compactor, set `Settings::compactor_options` to `None` and use `Db::builder`.

```rust
use slate_db::Db;
use slate_db::settings::Settings;

let db = Db::builder()
    .settings(Settings { compactor_options: None, ..Default::default() })
    .build()
    .await;

// ...

db.close().await;
```

--------------------------------

### Add Dependencies

Source: https://slatedb.io/docs/tutorials/abs.md

Command to add SlateDB, tokio, object-store with Azure feature, and anyhow to the Cargo.toml file.

```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/azure
cargo add anyhow
```

--------------------------------

### Configuring Filter Policies

Source: https://slatedb.io/docs/operations/tuning.md

Example of configuring filter policies for SST construction and evaluation using Bloom filters or custom policies.

```Rust
DbBuilder::with_filter_policies(vec![BloomFilterPolicy::new(10)])
```

--------------------------------

### Build SlateDB CLI from source

Source: https://slatedb.io/docs/operations/cli.md

Clones the SlateDB repository and builds the CLI from source using Cargo.

```bash
git clone https://github.com/slatedb/slatedb.git
cd slatedb
cargo build --release -p slatedb-cli
```

--------------------------------

### FoyerHybridCache Configuration with Large Buffer Pool and Queue

Source: https://slatedb.io/docs/operations/foyer-cache.md

Example of configuring FoyerHybridCache with a large memory tier, a persistent disk tier, and tuned flusher and buffer pool settings to avoid backpressure.

```rust
use foyer::{
    BlockEngineConfig,
    DeviceBuilder,
    FsDeviceBuilder,
    HybridCacheBuilder,
    PsyncIoEngineConfig,
};
use slatedb::db_cache::CachedEntry;
use slatedb::db_cache::foyer_hybrid::FoyerHybridCache;

let cache = HybridCacheBuilder::new()
    .with_name("slatedb_block_cache")
    .memory(8 * 1024 * 1024 * 1024)  // 8 GB memory tier
    .with_weighter(|_, v: &CachedEntry| v.size())
    .storage()
    .with_io_engine_config(PsyncIoEngineConfig::new())
    .with_engine_config(
        BlockEngineConfig::new(
            FsDeviceBuilder::new("/data/slatedb-cache")
                .with_capacity(100 * 1024 * 1024 * 1024)  // 100 GB disk tier
                .build()
                .unwrap(),
        )
        .with_block_size(64 * 1024)
        .with_flushers(4)
        .with_buffer_pool_size(256 * 1024 * 1024)         // large buffer pool
        .with_submit_queue_size_threshold(1024 * 1024 * 1024), // large queue size
    )
    .build()
    .await
    .unwrap();

let cache = FoyerHybridCache::new_with_cache(cache);
```

--------------------------------

### Run the standalone compactor

Source: https://slatedb.io/docs/operations/cli.md

Executes the SlateDB compactor as a standalone process, specifying environment file and database path.

```bash
slatedb --env-file .env --path <db-path> run-compactor
```

--------------------------------

### Rust example of stamping a write with a specific sequence number

Source: https://slatedb.io/docs/design/writes.md

This Rust code snippet demonstrates how to use `WriteOptions` to assign a custom sequence number (42 in this case) to a `put` operation in SlateDB. It highlights the use of `seqnum` and `Default::default()` for other options.

```rust
use slatedb::config::{PutOptions, WriteOptions};

// Stamp this write with seqnum = 42 (e.g., the offset returned by your
// external WAL after appending the record).
let opts = WriteOptions { seqnum: 42, ..Default::default() };
db.put_with_options(b"key", b"value", &PutOptions::default(), &opts).await?;
```

--------------------------------

### Add dependencies

Source: https://slatedb.io/docs/tutorials/s3.md

Adds SlateDB and other necessary dependencies to the Cargo.toml file.

```bash
cargo add slatedb tokio --features tokio/macros,tokio/rt-multi-thread
cargo add object-store --features object-store/aws
cargo add anyhow
```

--------------------------------

### MemTable Write with Immutable MemTables

Source: https://slatedb.io/docs/design/overview.md

A sequence diagram showing the improved process with immutable MemTables, allowing writes to continue in the background.

```mermaid
sequenceDiagram
  box Memory
    participant C as Client
    participant MT as MemTable (mutable)
    participant IMT as MemTable (immutable)
  end
  box Disk
    participant SST as SSTable
  end

  C->>MT: put()
  MT-->>C: update OK
  MT->>IMT: freeze
  IMT->>SST: write
```
Tokens:
Raw