remote-store

https://docs.remotestore.dev/stable/
Admin
remote-store provides a unified, portable API for file storage across local filesystems, S3, SFTP,...
Tokens:1,188,323
Snippets:8,161
Trust Score:7.5
Update:1 month ago
Show doc for...
Tokens:
Raw
### Quick start with S3 backend

Source: https://docs.remotestore.dev/stable

Quick start example: creates a Store with an S3Backend and writes/reads a file. This is the same API as the local backend example.

```python
from remote_store import Store
from remote_store.backends import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # 'hello'
```

--------------------------------

### Quick start with S3 backend

Source: https://docs.remotestore.dev/stable/explanation/design/research/research-docs-landing-page

Minimal example of using the S3 backend to write and read a text file. Requires the S3 backend package.

```python
from remote_store import Store
from remote_store.backends.s3 import S3Backend

store = Store(S3Backend(bucket="my-bucket"))
store.write_text("file.txt", "hello")
print(store.read_text("file.txt"))  # 'hello'
```

--------------------------------

### AsyncStore quick start with MemoryBackend

Source: https://docs.remotestore.dev/stable/guides/async

Shows the minimal async setup: create an AsyncStore with a MemoryBackend, write bytes, read them back, and print the result. The backend is auto-wrapped via SyncBackendAdapter, delegating calls to the default executor with asyncio.to_thread().

```python
import asyncio

from remote_store.aio import AsyncStore
from remote_store.backends import MemoryBackend

async def main() -> None:
    async with AsyncStore(MemoryBackend(), root_path="reports") as store:
        result = await store.write("summary.txt", b"Q1 results", overwrite=True)
        print(f"wrote {result.size} bytes")
        data = await store.read_bytes("summary.txt")
        print(data.decode())

asyncio.run(main())
```

--------------------------------

### delete_folder: delete a folder with recursive and missing_usage

Source: https://docs.remotestore.dev/stable/reference/api/store

usage notes and caveats: requires Capability.DELETE and can throw various exceptions depending on conditions. 

```python
delete_folder(path, recursive, missing_背景;)
```

--------------------------------

### open_atopen_ttic: context manager yields writable binary stream with atomic commit (with example)ea

Source: https://docs.remotestore.dev/stable/reference/api/store

Use this context manager when you need to write a file atomically. If your code raises an interrupted operation, the partial write is discarded.blew (overwrite=True) overwrites the existing file; if `overfallse

```python: 
open_atomic(path, path: str, *, overwrite: 
```

```python
with store.ope_atomic(
```

--------------------------------

### Full Quickstart Python script demonstrating direct Store construction and Registry configuration

Source: https://docs.remotestore.dev/stable/tutorial/examples/quickstart

The complete quickstart script imports tempfile, Registry, RegistryConfig, Store, and LocalBackend. It defines demo_direct (three-line Store construction plus write/read) and demo_registry (Registry.from_dict declarative config, with a backend named 'main' and store named 'data'). The main block uses tempfile.TemporaryDirectory and prints labels for each demonstration. Run with Python; output lines are 'Hello, world!' from the direct path and then 'Hello, world!' from the registry path.

```python
"""Quickstart — Minimal config, write, and read.

Demonstrates two ways to get started:
1. Direct construction — three lines, no config
2. Registry with declarative config — for multi-backend applications

---
see_also:
  - label: Getting Started
    url: ../../tutorial/getting-started.md
    note: step-by-step guide
"""

from __future__ import annotations

import tempfile

from remote_store import Registry, RegistryConfig, Store
from remote_store.backends import LocalBackend


def demo_direct(root: str) -> None:
    """Simplest usage: construct a Store directly."""
    store = Store(LocalBackend(root=root))
    store.write_text("hello.txt", "Hello, world!")
    print(store.read_text("hello.txt"))  # 'Hello, world!'


def demo_registry(root: str) -> None:
    """Registry usage: declarative config, multiple stores."""
    config = RegistryConfig.from_dict(
        {
            "backends": {"main": {"type": "local", "options": {"root": root}}},
            "stores": {"data": {"backend": "main", "root_path": ""}},
        }
    )

    with Registry(config) as registry:
        store = registry.get_store("data")
        store.write_text("hello.txt", "Hello, world!")
        print(store.read_text("hello.txt"))  # 'Hello, world!'


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        print("-- Direct construction --")
        demo_direct(f"{tmp}/direct")

        print("-- Registry config --")
        demo_registry(f"{tmp}/registry")

```

--------------------------------

### Setup: Install dependencies for medallion_dagster example

Source: https://docs.remotestore.dev/stable/tutorial/examples/medallion-dagster

Navigate to the example directory and install remote-store with the dagster, arrow, otel, and requests extras, plus polars, dagster-webserver, and opentelemetry-sdk. Requires Python 3.10+ and network access to data.geo.admin.ch.

```bash
cd examples/medallion_dagster

# Install remote-store with required extras + showcase dependencies
pip install -e "../../[dagster,arrow,otel,requests]" polars dagster-webserver opentelemetry-sdk

```

--------------------------------

### Run commands for medallion_dagster

Source: https://docs.remotestore.dev/stable/explanation/design/research/research-medallion-dagster-showcase

Commands to run the medallion_dagster project: change to the example directory, install dependencies with pip, and start Dagster dev server with definitions.py.

```bash
cd examples/medallion_dagster
pip install -e "../../[dagster,arrow,otel]" polars duckdb
dagster dev -f definitions.py
```

--------------------------------

### Install ParquetDatasetStore with PyArrow

Source: https://docs.remotestore.dev/stable/guides/parquet-datasets

Install the remote-store package with the arrow extra to get PyArrow support required by ParquetDatasetStore.

```bash
pip install "remote-store[arrow]"
```

--------------------------------

### Quick start: write and read Parquet datasets

Source: https://docs.remotestore.dev/stable/guides/parquet-datasets

Shows how to create a ParquetDatasetStore, write a table, read it back, and read with column projection.

```python
import pyarrow as pa
from remote_store import Store
from remote_store.backends import LocalBackend
from remote_store.ext.parquet import ParquetDatasetStore

store = Store(LocalBackend("/data/warehouse"))
pds = ParquetDatasetStore(store)

# Write
table = pa.table({"id": [1, 2, 3], "name": ["Alice", "Bob", "Charlie"]})
manifest = pds.write_dataset(table, "silver/customers")

# Read
customers = pds.read_dataset("silver/customers")

# Read with column projection
ids_only = pds.read_dataset("silver/customers", columns=["id"])
```

--------------------------------

### SFTPBackend.to_key example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/010-native-path-resolution

Example usage of SFTPBackend.to_key. Strips the configured base_path. If the input does not start with the base_path, it is returned unchanged.

```python
backend = SFTPBackend(host="srv", base_path="/srv/sftp")
backend.to_key("/srv/sftp/data/file.txt")   # → "data/file.txt"
backend.to_key("data/file.txt")             # → "data/file.txt" (no prefix, unchanged)
```

--------------------------------

### Initialize SFTPBackend with Private Key

Source: https://docs.remotestore.dev/stable/llms-api.txt

Example of loading a private key from a file and initializing an SFTPBackend instance with an automatic host key policy.

```python
key = SFTPUtils.load_private_key("~/.ssh/id_rsa", from_file=True)
backend = SFTPBackend(
    host="sftp.example.com",
    pkey=key,
    host_key_policy=SFTPUtils.HostKeyPolicy.AUTO_ADD,
)
```

--------------------------------

### LocalBackend.to_key example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/010-native-path-resolution

Example usage of LocalBackend.to_key. Strips the backend's filesystem root directory. If the input does not start with the root, it is returned unchanged.

```python
backend = LocalBackend("/tmp/store")
backend.to_key("/tmp/store/data/file.txt")  # → "data/file.txt"
backend.to_key("data/file.txt")             # → "data/file.txt" (no prefix, unchanged)
```

--------------------------------

### Streaming read example

Source: https://docs.remotestore.dev/stable/explanation/design/specs/006-streaming-io

Example of a streaming read. The returned stream is positioned at the start and the caller is responsible for consuming and closing it. The stream is not guaranteed to be seekable.

```python
stream = backend.read("data.bin")
chunk = stream.read(4096)

```

--------------------------------

### Quick start: ping() with LocalBackend

Source: https://docs.remotestore.dev/stable/guides/health-check

Creates a Store with a LocalBackend and calls ping() to verify connectivity. ping() raises on failure and is silent on success.

```python
from remote_store import Store
from remote_store.backends import LocalBackend

store = Store(LocalBackend(root="/data/inbox"))
store.ping()  # raises on failure, silent on success

```

--------------------------------

### ParquetDatasetStore read/write example

Source: https://docs.remotestore.dev/stable/reference/api/extensions/parquet

Install with `pip install "remote-store[arrow]"`. Creates a `ParquetDatasetStore` and writes a dataset to "silver/orders", then reads it back.

```python
from remote_store.ext.parquet import ParquetDatasetStore

pds = ParquetDatasetStore(store)
manifest = pds.write_dataset(table, "silver/orders")
table = pds.read_dataset("silver/orders")

```

--------------------------------

### Configure and use the local backend with Registry

Source: https://docs.remotestore.dev/stable/guides/backends/local

Shows how to configure the local backend with a root directory and a store profile, then write a file using the registry. The root option is required. The local backend is built-in and requires no extra dependencies.

```python
from remote_store import BackendConfig, RegistryConfig, Registry, StoreProfile

config = RegistryConfig(
    backends={"local": BackendConfig(type="local", options={"root": "/data"})},
    stores={"files": StoreProfile(backend="local", root_path="files")},
)

with Registry(config) as registry:
    store = registry.get_store("files")
    store.write_text("readme.txt", "Hello!")

```

--------------------------------

### Module docstring example

Source: https://docs.remotestore.dev/stable/explanation/design/design

Every module starts with a 1-2 sentence docstring explaining why it exists.

```python
"""Normalized error hierarchy for remote_store."""
```

--------------------------------

### Store.__init__(backend, root_path="")

Source: https://docs.remotestore.dev/stable/llms-api.txt

Initializes a Store instance with a backend and optional root path.

```APIDOC
## Store.__init__(backend, root_path="")

### Description
Creates a logical remote folder scoped to a root path. All path arguments are validated and prefixed with `root_path` before being delegated to the backend. Supports the context-manager protocol.

### Method
Constructor

### Signature
```python
def __init__(self, backend: Backend, root_path: str = "") -> None
```

### Parameters
- **backend** (Backend) - Required - Backend instance (Local, S3, SFTP, Azure, Memory).
- **root_path** (str) - Optional - Prefix prepended to every path. `""` means the backend root.
```

--------------------------------

### Demonstrate Store.child() functionality

Source: https://docs.remotestore.dev/stable/tutorial/examples/store-child

This script demonstrates creating child stores, writing to them, verifying path isolation, chaining child stores, and testing close semantics and context manager behavior.

```python
"""Store.child() — Runtime sub-scoping: create child stores that share a backend but isolate paths.

Demonstrates:
- Creating child stores with child(subpath)
- Children share the parent's backend (no new connections)
- Path isolation: children only see their own namespace
- Chaining: parent.child("a").child("b") == parent.child("a/b")
- Close semantics: closing a child does NOT close the backend
- Context manager usage with children
"""

from __future__ import annotations

import tempfile

from remote_store import BackendConfig, Registry, RegistryConfig, Store, StoreProfile


def demo(store: Store) -> None:
    """Child store creation, isolation, chaining, close semantics."""
    # --- Create child stores ---
    reports = store.child("reports")
    archive = store.child("archive")
    print(f"Parent:  {store!r}")
    print(f"Child 1: {reports!r}")
    print(f"Child 2: {archive!r}")

    # --- Write via children ---
    reports.write("q1.csv", b"revenue,100\n")
    reports.write("q2.csv", b"revenue,200\n")
    archive.write("2024/summary.txt", b"Year-end summary")
    print("\nWrote 3 files across 2 child stores.")

    # --- Path isolation ---
    print("\nreports/ files:")
    for f in reports.list_files("", recursive=True):
        print(f"  {f.path}")

    print("archive/ files:")
    for f in archive.list_files("", recursive=True):
        print(f"  {f.path}")

    print("Parent (all files):")
    for f in store.list_files("", recursive=True):
        print(f"  {f.path}")

    # --- Read from child ---
    content = reports.read_bytes("q1.csv")
    print(f"\nreports/q1.csv: {content.decode().strip()}")

    # --- Chaining ---
    deep = store.child("archive").child("2024")
    direct = store.child("archive/2024")
    print(f"\nChained:  {deep!r}")
    print(f"Direct:   {direct!r}")
    print(f"Equal:    {deep == direct}")

    content = deep.read_bytes("summary.txt")
    print(f"Read via chained child: {content.decode()}")

    # --- Close semantics ---
    child = store.child("reports")
    child.close()
    print(f"\nParent works after child.close(): {store.read_bytes('reports/q1.csv').decode().strip()}")

    # --- Context manager ---
    with store.child("reports") as scoped:
        info = scoped.get_file_info("q1.csv")
        print(f"\nContext manager child: {info.size} bytes")
    print(f"Parent still works: {store.exists('reports/q1.csv')}")


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmp:
        config = RegistryConfig(
            backends={"local": BackendConfig(type="local", options={"root": tmp})},
            stores={"data": StoreProfile(backend="local", root_path="data")},
        )

        with Registry(config) as registry:
            store = registry.get_store("data")
            demo(store)

    print("\nDone!")
```

--------------------------------

### See also block

Source: https://docs.remotestore.dev/stable/explanation/design/documentation

Footer section required on every API page, linking to the primary guide or a matching example.

```markdown
## See also

- [Guide name](https://github.com/haalfi/remote-store/blob/master/guide-path.md) — one-line description
- [Example name](https://github.com/haalfi/remote-store/blob/master/examples/example-path.md) — one-line description

```
Tokens:
Raw