remote-store

https://docs.remotestore.dev
Admin
A unified Python library providing a single interface for file storage across local filesystems, S3,...
Tokens:1,070,348
Snippets:7,532
Trust Score:7.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### Quick start with S3Backend

Source: https://docs.remotestore.dev/stable

Quick start example using S3Backend to write and read a file.

```python
from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # 'hello'
```

--------------------------------

### Initial Navigation Structure for Getting Started

Source: https://docs.remotestore.dev/stable/explanation/design/audits/audit-012-docs-structure

The original navigation configuration where 'Tutorial' was incorrectly nested under 'Getting Started'.

```yaml
- Getting Started:
    - Tutorial: getting-started.md
    - Examples: examples/
```

--------------------------------

### AsyncStore quick start with MemoryBackend

Source: https://docs.remotestore.dev/stable/guides/async

Shows the minimal async setup: create an AsyncStore with a MemoryBackend, write bytes, read them back, and print the result. The backend is auto-wrapped via SyncBackendAdapter, delegating calls to the default executor with asyncio.to_thread().

```python
import asyncio

from remote_store.aio import AsyncStore
from remote_store.backends import MemoryBackend

async def main() -> None:
    async with AsyncStore(MemoryBackend(), root_path="reports") as store:
        result = await store.write("summary.txt", b"Q1 results", overwrite=True)
        print(f"wrote {result.size} bytes")
        data = await store.read_bytes("summary.txt")
        print(data.decode())

asyncio.run(main())
```

--------------------------------

### Quick Start: upload, download, and transfer

Source: https://docs.remotestore.dev/stable/guides/transfer-operations

Shows the basic usage of the three transfer functions with an in-memory backend. Use this to get started with uploading, downloading, and transferring files between stores.

```python
from remote_store import Store, upload, download, transfer
from remote_store.backends import MemoryBackend

store = Store(backend=MemoryBackend())

# Upload a local file to the store
upload(store, "local/report.csv", "reports/report.csv")

# Download a remote file to a local path
download(store, "reports/report.csv", "local/copy.csv")

# Transfer between two stores
other = Store(backend=MemoryBackend())
transfer(store, "reports/report.csv", other, "archive/report.csv")

```

--------------------------------

### Full Quickstart Python script demonstrating direct Store construction and Registry configuration

Source: https://docs.remotestore.dev/stable/tutorial/examples/quickstart

The complete quickstart script imports tempfile, Registry, RegistryConfig, Store, and LocalBackend. It defines demo_direct (three-line Store construction plus write/read) and demo_registry (Registry.from_dict declarative config, with a backend named 'main' and store named 'data'). The main block uses tempfile.TemporaryDirectory and prints labels for each demonstration. Run with Python; output lines are 'Hello, world!' from the direct path and then 'Hello, world!' from the registry path.

```python
"""Quickstart — Minimal config, write, and read.

Demonstrates two ways to get started:
1. Direct construction — three lines, no config
2. Registry with declarative config — for multi-backend applications

---
see_also:
  - label: Getting Started
    url: ../../tutorial/getting-started.md
    note: step-by-step guide
"""

from __future__ import annotations

import tempfile

from remote_store import Registry, RegistryConfig, Store
from remote_store.backends import LocalBackend


def demo_direct(root: str) -> None:
    """Simplest usage: construct a Store directly."""
    store = Store(LocalBackend(root=root))
    store.write_text("hello.txt", "Hello, world!")
    print(store.read_text("hello.txt"))  # 'Hello, world!'


def demo_registry(root: str) -> None:
    """Registry usage: declarative config, multiple stores."""
    config = RegistryConfig.from_dict(
        {
            "backends": {"main": {"type": "local", "options": {"root": root}}},
            "stores": {"data": {"backend": "main", "root_path": ""}},
        }
    )

    with Registry(config) as registry:
        store = registry.get_store("data")
        store.write_text("hello.txt", "Hello, world!")
        print(store.read_text("hello.txt"))  # 'Hello, world!'


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        print("-- Direct construction --")
        demo_direct(f"{tmp}/direct")

        print("-- Registry config --")
        demo_registry(f"{tmp}/registry")

```

--------------------------------

### Setup: Install dependencies for medallion_dagster example

Source: https://docs.remotestore.dev/stable/tutorial/examples/medallion-dagster

Navigate to the example directory and install remote-store with the dagster, arrow, otel, and requests extras, plus polars, dagster-webserver, and opentelemetry-sdk. Requires Python 3.10+ and network access to data.geo.admin.ch.

```bash
cd examples/medallion_dagster

# Install remote-store with required extras + showcase dependencies
pip install -e "../../[dagster,arrow,otel,requests]" polars dagster-webserver opentelemetry-sdk

```

--------------------------------

### SFTPBackend.to_key example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/010-native-path-resolution

Example usage of SFTPBackend.to_key. Strips the configured base_path. If the input does not start with the base_path, it is returned unchanged.

```python
backend = SFTPBackend(host="srv", base_path="/srv/sftp")
backend.to_key("/srv/sftp/data/file.txt")   # → "data/file.txt"
backend.to_key("data/file.txt")             # → "data/file.txt" (no prefix, unchanged)
```

--------------------------------

### Quick start: ping() with LocalBackend

Source: https://docs.remotestore.dev/stable/guides/health-check

Creates a Store with a LocalBackend and calls ping() to verify connectivity. ping() raises on failure and is silent on success.

```python
from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="/data/inbox"))
store.ping()  # raises on failure, silent on success

```

--------------------------------

### Install ParquetDatasetStore with PyArrow

Source: https://docs.remotestore.dev/stable/guides/parquet-datasets

Install the remote-store package with the arrow extra to get PyArrow support required by ParquetDatasetStore.

```bash
pip install "remote-store[arrow]"
```

--------------------------------

### LocalBackend.to_key example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/010-native-path-resolution

Example usage of LocalBackend.to_key. Strips the backend's filesystem root directory. If the input does not start with the root, it is returned unchanged.

```python
backend = LocalBackend("/tmp/store")
backend.to_key("/tmp/store/data/file.txt")  # → "data/file.txt"
backend.to_key("data/file.txt")             # → "data/file.txt" (no prefix, unchanged)
```

--------------------------------

### Initialize Store with LocalBackend

Source: https://docs.remotestore.dev/stable/explanation/design/research/research-docs-landing-page

Demonstrates initializing a Store with a LocalBackend and performing basic read/write operations relative to a root path.

```python
from remote_store import Store
from remote_store.backends.local import LocalBackend

store = Store(LocalBackend(root="/tmp/data"))
store.write_text("hello.txt", "Hello, world!")
print(store.read_text("hello.txt"))  # 'Hello, world!'
```

--------------------------------

### Go Example test function with // Output:

Source: https://docs.remotestore.dev/stable/explanation/design/research/research-example-testing

Place in a _test.go file. The function name must start with Example. The trailing // Output: comment is compared against stdout by go test. Without it, the example is compiled but not executed.

```go
func ExampleReverse() {
    fmt.Println(stringutil.Reverse("hello"))
    // Output: olleh
}

```

--------------------------------

### Main execution block

Source: https://docs.remotestore.dev/stable/tutorial/examples/s3-listing-strategies

The entry point for the demonstration, setting up the registry and executing the various listing strategies.

```python
if __name__ == "__main__":
    registry = setup_registry()

    with registry:
        store = registry.get_store("demo")

        try:
            setup_sample_data(store)
            demo_shallow_listing(store)
            demo_recursive_listing(store)
            demo_filtering_on_stream(store)
            demo_performance_comparison(store)
            demo_antipattern(store)

        finally:
            cleanup(store)

    print("Done! See docs-src/guides/backends/s3.md for detailed performance analysis.")
```

--------------------------------

### ParquetDatasetStore read/write example

Source: https://docs.remotestore.dev/stable/reference/api/extensions/parquet

Install with `pip install "remote-store[arrow]"`. Creates a `ParquetDatasetStore` and writes a dataset to "silver/orders", then reads it back.

```python
from remote_store.ext.parquet import ParquetDatasetStore

pds = ParquetDatasetStore(store)
manifest = pds.write_dataset(table, "silver/orders")
table = pds.read_dataset("silver/orders")

```

--------------------------------

### Documentation file examples

Source: https://docs.remotestore.dev/stable/explanation/design/authoring

Examples showing how to apply markers or rely on directory defaults for different types of documentation files.

```markdown
<!-- A new guide. docs-src/ is docs-only by default; no marker needed. -->
# Streaming reads
...

<!-- A repo-root contributor doc. Explicit dual marker; bridge emits docs/contributing/. -->
<!-- doc: dual dest=contributing.md -->
# Contributing
...

<!-- A Claude-only operational file. Explicit repo-only marker; never reaches the docs site. -->
<!-- doc: repo-only -->
# Claude Code Reference
...

<!-- A new spec. sdd/specs/ has a directory default; no marker needed. -->
# Spec 048 -- Some Topic
...
```

--------------------------------

### Example: sync_safe_unwrap implementation and usage

Source: https://docs.remotestore.dev/stable/reference/api/aio/adapters

Example showing a backend implementing _SyncSafeHandleProvider and using adapter.unwrap to retrieve a sync-safe client.

```python
class _SafeBackend(AsyncBackend, _SyncSafeHandleProvider):
    def sync_safe_unwrap(self, type_hint):
        return self._sync_client

adapter = AsyncBackendSyncAdapter(_SafeBackend())
client = adapter.unwrap(SyncClient)

```

--------------------------------

### Streaming read in remote_store: Backend.read() example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/006-streaming-io

Example of streaming a read operation from a backend: open a stream and read a 4096-byte chunk. The stream is positioned at the start, is not guaranteed to be seekable, and the caller must close it. For seekability use Store.read_seekable().

```python
stream = backend.read("data.bin")
chunk = stream.read(4096)

```

--------------------------------

### Demonstrate Store.child() functionality

Source: https://docs.remotestore.dev/stable/tutorial/examples/store-child

This script demonstrates creating child stores, writing to them, verifying path isolation, chaining child stores, and testing close semantics and context manager behavior.

```python
"""Store.child() — Runtime sub-scoping: create child stores that share a backend but isolate paths.

Demonstrates:
- Creating child stores with child(subpath)
- Children share the parent's backend (no new connections)
- Path isolation: children only see their own namespace
- Chaining: parent.child("a").child("b") == parent.child("a/b")
- Close semantics: closing a child does NOT close the backend
- Context manager usage with children
"""

from __future__ import annotations

import tempfile

from remote_store import BackendConfig, Registry, RegistryConfig, Store, StoreProfile


def demo(store: Store) -> None:
    """Child store creation, isolation, chaining, close semantics."""
    # --- Create child stores ---
    reports = store.child("reports")
    archive = store.child("archive")
    print(f"Parent:  {store!r}")
    print(f"Child 1: {reports!r}")
    print(f"Child 2: {archive!r}")

    # --- Write via children ---
    reports.write("q1.csv", b"revenue,100\n")
    reports.write("q2.csv", b"revenue,200\n")
    archive.write("2024/summary.txt", b"Year-end summary")
    print("\nWrote 3 files across 2 child stores.")

    # --- Path isolation ---
    print("\nreports/ files:")
    for f in reports.list_files("", recursive=True):
        print(f"  {f.path}")

    print("archive/ files:")
    for f in archive.list_files("", recursive=True):
        print(f"  {f.path}")

    print("Parent (all files):")
    for f in store.list_files("", recursive=True):
        print(f"  {f.path}")

    # --- Read from child ---
    content = reports.read_bytes("q1.csv")
    print(f"\nreports/q1.csv: {content.decode().strip()}")

    # --- Chaining ---
    deep = store.child("archive").child("2024")
    direct = store.child("archive/2024")
    print(f"\nChained:  {deep!r}")
    print(f"Direct:   {direct!r}")
    print(f"Equal:    {deep == direct}")

    content = deep.read_bytes("summary.txt")
    print(f"Read via chained child: {content.decode()}")

    # --- Close semantics ---
    child = store.child("reports")
    child.close()
    print(f"\nParent works after child.close(): {store.read_bytes('reports/q1.csv').decode().strip()}")

    # --- Context manager ---
    with store.child("reports") as scoped:
        info = scoped.get_file_info("q1.csv")
        print(f"\nContext manager child: {info.size} bytes")
    print(f"Parent still works: {store.exists('reports/q1.csv')}")


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        config = RegistryConfig(
            backends={"local": BackendConfig(type="local", options={"root": tmp})},
            stores={"data": StoreProfile(backend="local", root_path="data")},
        )

        with Registry(config) as registry:
            store = registry.get_store("data")
            demo(store)

    print("\nDone!")
```

--------------------------------

### Python S3 Listing Strategies Example

Source: https://docs.remotestore.dev/stable/tutorial/examples/s3-listing-strategies

Complete example demonstrating shallow and recursive listing with remote-store S3 backend, including setup, sample data, and iteration patterns. Requires RS_S3_BUCKET environment variable set; optional RS_S3_KEY, RS_S3_SECRET, and RS_S3_ENDPOINT for MinIO.

```python
"""S3 Listing Strategies -- Shallow vs. recursive listing: cost tradeoffs, iterator patterns, and MinIO endpoint usage.

Demonstrates:
- Shallow listing (direct children only) -- O(n_children) cost
- Recursive listing (flat stream) -- O(n_total_files) cost
- When to use each approach
- Why parallelization is wrong for large buckets
- Impact of bucket structure on listing performance

Prerequisites:
- pip install "remote-store[s3]"
- An S3-compatible service with sample data

Environment variables:
    RS_S3_BUCKET    S3 bucket name (required)
    RS_S3_KEY       AWS access key ID
    RS_S3_SECRET    AWS secret access key
    RS_S3_ENDPOINT  Custom endpoint URL (e.g. http://localhost:9000 for MinIO)

---
see_also:
  - label: S3 Backend
    url: ../../guides/backends/s3.md
    note: listing strategies and performance
"""

from __future__ import annotations

import os
import sys
import time
from typing import TYPE_CHECKING

from remote_store import (
    BackendConfig,
    FileInfo,
    FolderEntry,
    Registry,
    RegistryConfig,
    StoreProfile,
)

if TYPE_CHECKING:
    from remote_store import Store

BUCKET = os.environ.get("RS_S3_BUCKET", "")

if not BUCKET:
    print("Set RS_S3_BUCKET to run this example.\nOptional: RS_S3_KEY, RS_S3_SECRET, RS_S3_ENDPOINT\n")
    sys.exit(1)


def setup_registry() -> Registry:
    """Create a registry connected to S3."""
    options: dict[str, object] = {"bucket": BUCKET}
    if val := os.environ.get("RS_S3_KEY"):
        options["key"] = val
    if val := os.environ.get("RS_S3_SECRET"):
        options["secret"] = val
    if val := os.environ.get("RS_S3_ENDPOINT"):
        options["endpoint_url"] = val

    config = RegistryConfig(
        backends={"s3": BackendConfig(type="s3", options=options)},
        stores={"demo": StoreProfile(backend="s3", root_path="listing-demo")},
    )
    return Registry(config)


def setup_sample_data(store: Store) -> None:
    """Create sample directory structure for demonstration."""
    print("Setting up sample data...")

    # Create a nested structure:
    # listing-demo/
    #   file_at_root.txt
    #   level1/
    #     file1.txt
    #     file2.txt
    #     level2/
    #       file3.txt
    #       level3/
    #         file4.txt
    #   another/
    #     orphan.txt

    files = [
        "file_at_root.txt",
        "level1/file1.txt",
        "level1/file2.txt",
        "level1/level2/file3.txt",
        "level1/level2/level3/file4.txt",
        "another/orphan.txt",
    ]

    for path in files:
        store.write(path, f"Content of {path}".encode())

    print(f"  Created {len(files)} files\n")


def demo_shallow_listing(store: Store) -> None:
    """Shallow listing: list only direct children."""
    print("[1] SHALLOW LISTING (direct children only)")
    print("    Code: store.iter_children(path)")
    print("    Cost: 1-N API calls (1 per 1000 items, with pagination)\n")

    print("    Direct children of 'listing-demo/':")
    for entry in store.iter_children(""):
        if isinstance(entry, FileInfo):
            print(f"      FILE  {entry.name} ({entry.size} bytes)")
        elif isinstance(entry, FolderEntry):
            print(f"      DIR   {entry.name}/")

    print("\n    Direct children of 'listing-demo/level1/':")
    for entry in store.iter_children("level1"):
        if isinstance(entry, FileInfo):
            print(f"      FILE  {entry.name} ({entry.size} bytes)")
        elif isinstance(entry, FolderEntry):
            print(f"      DIR   {entry.name}/")

    print()


def demo_recursive_listing(store: Store) -> None:
    """Recursive listing: flat stream of all files."""
    print("[2] RECURSIVE LISTING (flat stream, all files)")
    print("    Code: store.list_files(path, recursive=True)")
    print("    Cost: 1-N API calls (1 per 1000 items, S3 handles pagination)\n")

    print("    All files under 'listing-demo/' (recursive):")
    count = 0
    total_size = 0
    for file_info in store.list_files("", recursive=True):
        count += 1
        total_size += file_info.size
        print(f"      FILE  {file_info.path} ({file_info.size} bytes)")

    print(f"\n    Total: {count} files, {total_size} bytes\n")


def demo_filtering_on_stream(store: Store) -> None:
    """Filter files while streaming (memory-efficient)."""
    print("[3] FILTERING ON STREAM (memory-efficient)")
    print("    Process files without loading entire listing into memory\n")

    print("    Files matching pattern (*.txt):")
    for file_info in store.list_files("", recursive=True):
        if str(file_info.path).endswith(".txt"):
            print(f"      FILE  {file_info.path}")

    print()


```

--------------------------------

### Initialize PostgreSQL Backend

Source: https://docs.remotestore.dev/stable/guides/backends/sql-blob

Connect to a PostgreSQL database by providing the connection URL to the SQLBlobBackend.

```python
backend = SQLBlobBackend(url="postgresql://user:pass@localhost/mydb")
store = Store(backend=backend)
```

--------------------------------

### Store.to_key Composition Example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/010-native-path-resolution

Demonstrates how the backend conversion and root path stripping are composed for an SFTP backend.

```python
# SFTP backend with base_path="/srv/sftp", store with root_path="data"
store = Store(backend=sftp, root_path="data")

# Full chain: "/srv/sftp/data/reports/q1.csv"
#   → backend.to_key → "data/reports/q1.csv"
#   → strip root_path → "reports/q1.csv"
store.to_key("/srv/sftp/data/reports/q1.csv")  # → "reports/q1.csv"
```
Tokens:
Raw