### Install and run the development server

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-js/examples/react-wasm/README.md

Install dependencies and start the Vite development server for the React WASM example.

```bash
npm install
npm run dev
```

--------------------------------

### Development setup with uv

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/CONTRIBUTING.md

Commands to set up the development environment using uv, including starting local dev object stores, installing dependencies, configuring maturin-import-hook, building the Rust extension, and running tests, type checking, and linting.

```bash
# Bring up local dev object stores used by certain tests, otherwise you will get some failures
docker compose up -d

# Install all development dependencies (includes test dependencies, mypy, ruff, maturin)
uv sync

# Configure maturin-import-hook for fast incremental Rust compilation
uv run -m maturin_import_hook site install

# Build the Rust extension
uv run maturin develop --uv

# Activate the virtual environment
source .venv/bin/activate

# Run tests
uv run pytest

# Run type checking
uv run mypy python tests

# Run linting
uv run ruff check python
```

--------------------------------

### Manual setup with pip

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/CONTRIBUTING.md

Alternative setup steps using pip and a virtual environment, including installing maturin, installing dev dependencies, and building the Rust extension. Note that modern pip (>=21.3) supports dependency groups.

```bash
python3 -m venv .venv
source .venv/bin/activate

# Install maturin
pip install maturin

# Install dev dependencies (includes test, mypy, ruff)
pip install --group dev

# Build the Rust extension
maturin develop
```

--------------------------------

### Venv setup for development

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Creates a virtual environment, installs maturin and development dependencies, and builds the Rust extension with 'maturin develop'.

```bash
python3 -m venv .venv
source .venv/bin/activate

# Install maturin and dependencies
pip install maturin
pip install --group dev

# Build the Rust extension
maturin develop
```

--------------------------------

### uv setup for development

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Installs all development dependencies with uv, configures the maturin-import-hook for fast incremental Rust compilation, and builds the Rust extension with 'maturin develop --uv'.

```bash
# Install all development dependencies (includes test dependencies, mypy, ruff, maturin)
uv sync

# Configure maturin-import-hook for fast incremental Rust compilation
uv run -m maturin_import_hook site install

# Build the Rust extension
uv run maturin develop --uv
```

--------------------------------

### Start MkDocs development server

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Starts the MkDocs development server with live reload at http://127.0.0.1:8000. Run from the pixi shell.

```bash
just docs-serve
```

--------------------------------

### Full chunk-manifests configuration example

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/005-manifest-split.md

Full configuration example for chunk-manifests, showing manifest sets, rules, and preload settings. Demonstrates mutually exclusive max-manifest-size and arrays-per-manifest, overflow-to, cardinality, and ordered rules with regex path matching.

```yaml
chunk-manifests:
  sets:                           # the lists of manifest sets and their properties
                                  # it's a list because order is important

    - coord1:                     # a name for the set, used later with rules

        max-manifest-size: 10000  # this set will contain manifests with <= 10k references

        arrays-per-manifest: null # pack this many arrays in each manifest
                                  # max-manifest-size and arrays-per-manifest are
                                  # mutually exclusive

        overflow-to: coord2       # if an array cannot fit in the set, send it to set coord2
                                  # if no overflow-to is indicated it goes to
                                  # the default set (see below)

        cardinality: 2            # this set cannot have more than two manifest files
                                  # cardinality = null means unlimited

    - coord2:
        max-manifest-size: 10000
                                  # overflow-to: default       # by default
                                  # arrays-per-manifest: null  # by default
                                  # cardinality: 1             # by default

    - big-array:
        max-manifest-size: null
        arrays-per-manifest: 1
        cardinality: null

    - default:                  # default set must always exist, from defaults if needed
                                # max-size: 1_000_000 by default (we'll have to tune)
                                # default set is the only one that can have members that
                                # go beyond the max-size
                                # (in case of large arrays that don't fit)
                                # default always has cardinality = null

  rules:                        # what arrays go to each manifest set, an ordered list

    # each rule has a target (the manifest set) and a set of conditions that are and-ed
    # rules are applied in the order declared, and they break on the first match

    - path: .*/(latitude|longitude|time)  # an optional regex matching on path

      metadata-chunks: [0, 500]           # arrays having number of chunks in this range
                                          # both extremes can be null
                                          # in the future we'll implement initialized-chunks
                                          # that will be useful for sparse arrays

      target: coord1                      # arrays that match will go to this manifest set


    - metadata-chunks: [0, 200]           # arrays < 200 chunks will go to coord2
                                          # but only if they didn't get assigned by the rule above
      target: coord2


    - metadata-chunks: [2000000, null]    # huge arrays have their special manifest set
      target: big-array


  preload:                      # what manifests to asynchronously preload on session start
    max-manifest-size: 50000    # don't preload manifests larger than this
    max-manifests: 10           # don't preload more than this many manifests
    arrays:                     # currently we support selecting by path only
                                # we will refine with more power if needed

      - path: .*/time           # will preload manifests for all arrays matching this regex
      - path: .*/latitude
      - path: .*/longitude
```

--------------------------------

### Install icechunk with WASM binary

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-js/README.md

Install the package with the `--cpu=wasm32` flag to get the WASM binary.

```bash
npm install @earthmover/icechunk --cpu=wasm32
```

--------------------------------

### Install dependencies and run Icechunk CLI commands

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-js/examples/node-cli/README.md

Install the necessary dependencies and execute CLI commands to create a repository, list branches, and list tags.

```bash
npm install

# Create a new repo with sample groups and arrays
node --experimental-strip-types main.ts create ./my-repo

# List branches
node --experimental-strip-types main.ts list-branches ./my-repo

# List tags
node --experimental-strip-types main.ts list-tags ./my-repo
```

--------------------------------

### Install just, cargo-nextest, and cargo-deny

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Install the just command runner, cargo-nextest, and cargo-deny from source. These tools are used for build tasks, running tests, and code quality checks respectively. If using pixi with an activated shell, these installations can be skipped.

```bash
cargo install just
```

```bash
cargo install cargo-nextest
```

```bash
cargo install cargo-deny
```

--------------------------------

### Install pre-commit hooks

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Installs pre-commit and sets up the git hooks. The configuration runs fast Python and Rust checks on every commit, medium Rust checks before push, and manual full CI checks.

```bash
pip install pre-commit
pre-commit install
```

--------------------------------

### Install pooch and netCDF4 for tutorial data

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/xarray.md

Required to download Xarray tutorial data.

```shell
pip install pooch netCDF4
```

--------------------------------

### Install icechunk_v1 with uv

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Install the renamed icechunk_v1 package using uv. First enter the icechunk-python directory, then run third-wheel sync. This downloads icechunk v1 from PyPI, renames it to icechunk_v1, and installs it into your environment.

```bash
cd icechunk-python
uv run third-wheel sync -v
```

--------------------------------

### Install and test icechunk-js

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-js/README.md

Standard development workflow for the icechunk-js package: install dependencies, build, and run tests.

```bash
cd icechunk-js
yarn install
yarn build
yarn test
```

--------------------------------

### Conda/Mamba setup for development

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Creates a conda environment with Python 3.12, Rust, and Zarr, activates it, installs maturin and development dependencies, and builds the Rust extension with 'maturin develop'.

```bash
mamba create -n icechunk python=3.12 rust zarr
mamba activate icechunk

# Install maturin and dependencies
pip install maturin
pip install --group dev

# Build the Rust extension
maturin develop
```

--------------------------------

### Create Icechunk repository and session

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/understanding/parallel.md

Sets up an Icechunk repository and a writable session, then loads example data. This is the first step in a parallel write workflow.

```python
import xarray as xr
import tempfile
import icechunk as ic

ds = xr.tutorial.open_dataset("rasm").isel(time=slice(24))
repo = ic.Repository.create(ic.local_filesystem_storage(tempfile.TemporaryDirectory().name))
session = repo.writable_session("main")
```

--------------------------------

### Install virtualizarr and dependencies

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/virtual.md

Install required packages: virtualizarr (>=2.4.0), icechunk, fsspec, s3fs, and h5py.

```shell
pip install "virtualizarr>=2.4.0" icechunk fsspec s3fs h5py
```

--------------------------------

### Install icechunk_v1 with pixi

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Install the renamed icechunk_v1 package using pixi. This downloads icechunk v1 from PyPI, renames it to icechunk_v1, and installs it into your environment.

```bash
pixi run third-wheel sync -v
```

--------------------------------

### Install virtualizarr and icechunk

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/zarr.md

Install the required packages. VirtualiZarr must be at least version 2.5.0 (2.5.1 for sharded v3 arrays).

```shell
pip install "virtualizarr>=2.5.1" icechunk
```

--------------------------------

### Build and run gc_bench list on EC2

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk/examples/gc_bench/README.md

Commands to build the gc_bench example with the bench profile and run the `list` subcommand against an S3 bucket. Requires rustup (stable), cloning the repo, and temporary AWS credentials. The bench profile writes to target/release, not target/bench.

```sh
# on the instance: install rustup (stable), clone the repo at this commit, then
cargo build --profile bench --features logs --example gc_bench

# temporary credentials, e.g. from `aws sts assume-role` or the console
export AWS_ACCESS_KEY_ID=... AWS_SECRET_ACCESS_KEY=... AWS_SESSION_TOKEN=...

# the bench profile writes to target/release, not target/bench
B=./target/release/examples/gc_bench
$B list --s3-bucket <bucket> --s3-prefix <prefix> --s3-region <region> --mode single --duration-secs 30
$B list --s3-bucket <bucket> --s3-prefix <prefix> --s3-region <region> --mode asset-manager --streams 64 --duration-secs 30
$B list --s3-bucket <bucket> --s3-prefix <prefix> --s3-region <region> --mode asset-manager --streams 128 --duration-secs 30
```

--------------------------------

### Initialize Icechunk store and session

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/dask.md

Initialize an Icechunk store with a local filesystem, create a repository, and open a writable session. This setup is required before using Icechunk's distributed write functions with Dask.

```python
import icechunk as ic
import tempfile

# initialize the icechunk store
storage = ic.local_filesystem_storage(tempfile.TemporaryDirectory().name)
repo = ic.Repository.create(storage)
session = repo.writable_session("main")
```

--------------------------------

### Benchmark output for getsize comparison

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/benchmarks/README.md

Example output from the `getsize` benchmark comparison. Shows median times for `test_time_getsize_key` and `test_time_getsize_prefix` across datasets, with the main branch (0034_main_3a) and current (NOW) results. The prefix benchmark shows a significant speedup (31.10x).

```text
-------- benchmark 'test_time_getsize_key era5-single': 2 tests --------
Name (time in us)                                       Median
------------------------------------------------------------------------
test_time_getsize_key[era5-single] (0034_main_3a)     110.2501 (1.05)
test_time_getsize_key[era5-single] (NOW)              104.7080 (1.0)
------------------------------------------------------------------------

-------- benchmark 'test_time_getsize_key gb-128mb': 2 tests --------
Name (time in us)                                    Median
---------------------------------------------------------------------
test_time_getsize_key[gb-128mb] (0034_main_3a)     106.5000 (1.05)
test_time_getsize_key[gb-128mb] (NOW)              101.2499 (1.0)
---------------------------------------------------------------------

-------- benchmark 'test_time_getsize_key gb-8mb': 2 tests --------
Name (time in us)                                  Median
-------------------------------------------------------------------
test_time_getsize_key[gb-8mb] (0034_main_3a)     105.6660 (1.05)
test_time_getsize_key[gb-8mb] (NOW)              100.8749 (1.0)
-------------------------------------------------------------------

------- benchmark 'test_time_getsize_prefix era5-single': 2 tests --------
Name (time in ms)                                         Median
--------------------------------------------------------------------------
test_time_getsize_prefix[era5-single] (0034_main_3a)     68.8355 (31.10)
test_time_getsize_prefix[era5-single] (NOW)               2.2133 (1.0)
--------------------------------------------------------------------------
```

--------------------------------

### Set up WASM compiler for wasm32-wasip1-threads

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Set up a WASM-capable C toolchain for compiling icechunk to wasm32-wasip1-threads on macOS. Install the Rust target, install LLVM via Homebrew, configure environment variables for the target, and verify the build with cargo check. The repository sets the required tokio unstable cfg automatically.

```bash
rustup target add wasm32-wasip1-threads
```

```bash
brew install llvm
```

```bash
export PATH="/opt/homebrew/opt/llvm/bin:$PATH"
export CC_wasm32_wasip1_threads=/opt/homebrew/opt/llvm/bin/clang
export CXX_wasm32_wasip1_threads=/opt/homebrew/opt/llvm/bin/clang++
export AR_wasm32_wasip1_threads=/opt/homebrew/opt/llvm/bin/llvm-ar
```

```bash
cargo check -p icechunk --no-default-features --target wasm32-wasip1-threads
```

--------------------------------

### Import Xarray and Icechunk

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/xarray.md

Imports needed for the examples.

```python
import xarray as xr
import icechunk as ic
```

--------------------------------

### Install @earthmover/icechunk for default platform

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/icechunk-js.md

Installs the Icechunk JavaScript package and the zarrita library for the default platform (native Rust bindings via NAPI-RS).

```bash
npm install @earthmover/icechunk zarrita
```

--------------------------------

### VCC configuration example

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/014-virtual-chunk-ref-efficiency.md

Example of a VCC configuration in the repository config. The VCC has a name 'my-virtual-icechunk' and a URL prefix 's3://testbucket/my-repo/chunks'. The store is an S3-compatible store with region 'us-east-1' and anonymous access disabled.

```yaml
  s3:
    name: my-virtual-icechunk
    url_prefix: s3://testbucket/my-repo/chunks
    store: !s3_compatible
      region: us-east-1
      anonymous: false
```

--------------------------------

### Example redirect server using http.server

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/storage.md

Minimal HTTP server that returns 302 redirects with a Location header for known repository paths. Serves as the server side for redirect storage; any HTTP server returning a 3xx redirect works.

```python
from http.server import BaseHTTPRequestHandler, HTTPServer

REPOS = {
    "/my-repo": "s3://my-bucket/icechunk/my-repo?region=us-east-1",
    "/public-era5": "gs://era5-zarr/icechunk/era5",
}


class RedirectHandler(BaseHTTPRequestHandler):
    def do_GET(self) -> None:
        location = REPOS.get(self.path)
        if location is None:
            self.send_response(404)
            self.end_headers()
            return

        self.send_response(302)
        self.send_header("Location", location)
        self.end_headers()


HTTPServer(("127.0.0.1", 8989), RedirectHandler).serve_forever()
```

--------------------------------

### Zarr No Version Tracking Example

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/overview.md

Shows how Zarr lacks version tracking, making it impossible to reproduce past analyses after data updates.

```python
# January: Run analysis for paper
forecast = zarr.open_group("s3://public-weather/forecast")
results_jan = analyze(forecast["temperature"][:])
# Published results based on this data

# March: Try to reproduce results
forecast = zarr.open_group("s3://public-weather/forecast")
results_mar = analyze(forecast["temperature"][:])
# Data was updated! Results don't match!
# No way to know what changed or get back to January's data
```

--------------------------------

### Example workflow for experimenting with manifest splitting

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/performance.md

Workflow for experimenting with manifest splitting: define a new config (split size 5), open the repo, create a branch, rewrite manifests on that branch, benchmark reads, persist the config with save_config, reset main to the experiment branch, and verify the persisted config is restored on open.

```python
# first define a new config
split_config = ic.config.ManifestSplittingConfig.from_dict(
    {ic.config.ManifestSplitCondition.AnyArray(): {ic.config.ManifestSplitDimCondition.Any(): 5}}
)
repo_config = ic.config.RepositoryConfig(
    manifest=ic.config.ManifestConfig(splitting=split_config),
)
# open the repo with the new config.
repo = ic.Repository.open(storage, config=repo_config)
```

```python
repo.create_branch("split-experiment-1", repo.lookup_branch("main"))
snap = repo.rewrite_manifests(
    f"rewrite_manifests with new config", branch="split-experiment-1"
)
print(repo.list_manifest_files(snap))
```

```python
store = repo.readonly_session("main").store
store_split = repo.readonly_session("split-experiment-1").store
# ...
```

```python
repo.save_config()
```

```python
repo.reset_branch("main", repo.lookup_branch("split-experiment-1"))
```

```python
print(ic.Repository.open(storage).config.manifest)
```

--------------------------------

### Icechunk Atomic Consistency Example

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/overview.md

Demonstrates using an Icechunk transaction to atomically update multiple variables, ensuring readers always see a consistent snapshot.

```python
# Writer process
with repo.transaction("main", message="Add forecast") as store:
    forecast = zarr.open_group(store)
    forecast["temperature"][:, new_time] = temp_data
    forecast["pressure"][:, new_time] = press_data
    forecast["humidity"][:, new_time] = humid_data
# All variables updated atomically

# Reader process (concurrent with writer)
session = repo.readonly_session("main")
forecast = zarr.open_group(session.store)
# Always sees a complete, consistent snapshot
# Either all old values OR all new values, never mixed
```

--------------------------------

### Run read performance benchmark with hyperfine

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/008-no-copy-serialization-formats.md

Command to run the read benchmark for the example program `multithreaded_get_chunk_refs.rs`, using one warmup run.

```sh
nix run nixpkgs#hyperfine -- \
  --warmup 1 \
  'cargo run --release --example multithreaded_get_chunk_refs -- --read /tmp/test-perf'
```

--------------------------------

### Run write performance benchmark with hyperfine

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/008-no-copy-serialization-formats.md

Command to run the write benchmark for the example program `multithreaded_get_chunk_refs.rs`, preparing a clean directory and using one warmup run.

```sh
nix run nixpkgs#hyperfine -- \
  --prepare 'rm -rf /tmp/test-perf' \
  --warmup 1 \
  'cargo run --release --example multithreaded_get_chunk_refs -- --write /tmp/test-perf'
```

--------------------------------

### List manifest files in Icechunk 2.0

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/breaking-changes-2.md

In Icechunk 2.0, use Repository.list_manifest_files() to get manifest files for a snapshot. The old SnapshotInfo.manifests field and Repository.manifest_files() method were removed or renamed.

```python
manifests = repo.list_manifest_files(snapshot_id)
```

--------------------------------

### Open Repo with Custom Config

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/getting-started/howto.md

Opens a repository with a custom configuration, here setting the chunk cache size to 100 MB. Many other configuration options are available; see the Configuration guide for details.

```python
config = icechunk.RepositoryConfig.default()
config.caching = icechunk.CachingConfig(num_bytes_chunks=100_000_000)
repo = icechunk.Repository.open(storage, config=config)
```

--------------------------------

### Read the operations log

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/understanding/repository-features.md

Create an in-memory Icechunk repository, make commits, create a branch, and iterate over the ops log to print each update's timestamp and kind. The example uses icechunk, zarr, and numpy to demonstrate the ops log API.

```python
import icechunk as ic
import zarr
import numpy as np

# Create a repository and make some changes
repo = ic.Repository.create(ic.in_memory_storage())
session = repo.writable_session("main")
root = zarr.group(session.store)
root.create_array("temperature", shape=(100,), dtype="f4")
session.commit("Add temperature array")

session = repo.writable_session("main")
arr = zarr.open_array(session.store, path="temperature")
arr[:] = np.random.randn(100).astype("f4")
session.commit("Write temperature data")

repo.create_branch("develop", repo.lookup_branch("main"))

for update in repo.ops_log():
    print(f"{update.updated_at}  {update.kind}")
```

--------------------------------

### Configure virtual chunk credentials with containers_credentials

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/configuration.md

Configure virtual chunk container credentials using the containers_credentials helper and pass them to Repository.open via the authorize_virtual_chunk_access parameter. The example uses explicit S3 credentials for two buckets; passing None as a credential is deprecated and may expose private credentials.

```python
credentials = ic.credentials.containers_credentials(
    { "s3://my_s3_bucket": ic.credentials.s3_credentials(bucket="my-s3-bucket", region="us-east-1"),
      "s3://my_other_s3_bucket": ic.credentials.s3_credentials(bucket="my-other-s3-bucket", region="us-west-2"),
    }
)

repo = ic.Repository.open(
    storage=storage,
    config=config,
    authorize_virtual_chunk_access=credentials,
)
```

--------------------------------

### Configure Vite for WASM build

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/icechunk-js.md

Required setup for using the WASM build with Vite. Install the two plugins and add them to the Vite configuration file.

```bash
npm install vite-plugin-wasm vite-plugin-top-level-await
```

```typescript
// vite.config.ts
import wasm from 'vite-plugin-wasm'
import topLevelAwait from 'vite-plugin-top-level-await'

export default {
  plugins: [wasm(), topLevelAwait()],
}
```

--------------------------------

### Run benchmarks with S3 latency via ICECHUNK_BENCH_LATENCY_MS

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk/benches/README.md

Run benchmarks against S3 (MinIO) behind toxiproxy with a specified downstream latency in milliseconds. Requires `docker compose up -d` to start MinIO and toxiproxy. The latency toxic is applied only during timed `get_chunk` iterations, not during setup.

```sh
ICECHUNK_BENCH_LATENCY_MS=100 cargo bench -- get_chunks
```

--------------------------------

### Uncooperative distributed writes with multiprocessing and icechunk

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/understanding/parallel.md

Example of uncooperative distributed writes using multiprocessing. The script creates a repository, spawns two worker processes that repeatedly attempt to write to the same branch, handling ConflictError by retrying. The output shows the order of writes is not guaranteed. Note: using the 'fork' start method will deadlock; set 'forkserver' as shown.

```python
import multiprocessing as mp
import icechunk as ic
import zarr


def get_storage():
    return ic.local_filesystem_storage(tempfile.TemporaryDirectory().name)


def worker(i):
    print(f"Stated worker {i}")
    storage = get_storage()
    repo = ic.Repository.open(storage)
    # keep trying until it succeeds
    while True:
        try:
            session = repo.writable_session("main")
            z = zarr.open(session.store, mode="r+")
            print(f"Opened store for {i} | {dict(z.attrs)}")
            a = z.attrs.get("done", [])
            a.append(i)
            z.attrs["done"] = a
            session.commit(f"wrote from worker {i}")
            break
        except ic.ConflictError:
            print(f"Conflict for {i}, retrying")
            pass


def main():
    # This is necessary on linux systems
    mp.set_start_method('forkserver')
    storage = get_storage()
    repo = ic.Repository.create(storage)
    session = repo.writable_session("main")

    zarr.create(
        shape=(10, 10),
        chunks=(5, 5),
        store=session.store,
        overwrite=True,
    )
    session.commit("initialized dataset")

    p1 = mp.Process(target=worker, args=(1,))
    p2 = mp.Process(target=worker, args=(2,))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

    session = repo.readonly_session(branch="main")
    z = zarr.open(session.store, mode="r")
    print(z.attrs["done"])
    print(list(repo.ancestry(branch="main")))


if __name__ == "__main__":
    main()
```

```sh
Stated worker 1
Stated worker 2
Opened store for 1 | {}
Opened store for 2 | {}
Conflict for 1, retrying
Opened store for 1 | {'done': [2]}
[2, 1]
[SnapshotInfo(id="MGPV1YE1SY0799AZFFB0", parent_id=YAN3D2N7ANCNKCFN3JSG, written_at=datetime.datetime(2025,3,4,21,40,57,19985, tzinfo=datetime.timezone.utc), message="wrote from..."), SnapshotInfo(id="YAN3D2N7ANCNKCFN3JSG", parent_id=0M5H3J6SC8MYBQYWACC0, written_at=datetime.datetime(2025,3,4,21,40,56,734126, tzinfo=datetime.timezone.utc), message="wrote from..."), SnapshotInfo(id="0M5H3J6SC8MYBQYWACC0", parent_id=WKKQ9K7ZFXZER26SES5G, written_at=datetime.datetime(2025,3,4,21,40,56,47192, tzinfo=datetime.timezone.utc), message="initialize..."), SnapshotInfo(id="WKKQ9K7ZFXZER26SES5G", parent_id=None, written_at=datetime.datetime(2025,3,4,21,40,55,868277, tzinfo=datetime.timezone.utc), message="Repository...")]
```

--------------------------------

### Create benchmark datasets with pytest -m setup_benchmarks

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/benchmarks/README.md

Creates the benchmark datasets required for running read benchmarks. Must be re-run whenever the Dataset or format changes; as of Jan 20, 2025 it takes about 3 minutes.

```sh
pytest -nauto -m setup_benchmarks benchmarks/
```

--------------------------------

### Build static documentation site

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Builds the static documentation site to the docs/.site directory.

```bash
just docs-build
```

--------------------------------

### Install Icechunk JS with npm

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-js/README.md

Install the Icechunk JS library from npm. For browser usage, you must install the WASM package with `--cpu=wasm32`.

```bash
npm install @earthmover/icechunk@alpha
```

--------------------------------

### Install Xarray >= 2025.1.1

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/xarray.md

Required before using Xarray with Icechunk. Install with pip.

```shell
pip install "xarray>=2025.1.1"
```

--------------------------------

### Install zarr-python for rectilinear chunks

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/zarr.md

Installs an unreleased development version of zarr-python required for rectilinear chunk grids.

```bash
pip install zarr @ git+https://github.com/maxrjones/zarr-python.git@main
```

--------------------------------

### Import icechunk and create a Repository with S3 storage

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/index.md

Shows the top-level import and usage of icechunk, including creating a Repository with S3 storage using credentials from the environment. Also demonstrates accessing submodules for configuration and conflict resolution.

```python
import icechunk as ic

# Top-level: Repository, storage factories, exceptions, utilities
repo = ic.Repository.create(ic.s3_storage(bucket="my-bucket", prefix="my-prefix", from_env=True))

# Submodules for everything else
config = ic.config.RepositoryConfig(...)
solver = ic.conflicts.BasicConflictSolver(...)
```

--------------------------------

### Install Icechunk with conda

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/getting-started/quickstart.md

Install Icechunk using conda from conda-forge. Requires Zarr Python 3 for Zarr V3 support.

```bash
conda install -c conda-forge icechunk
```

--------------------------------

### Create cloud storage backends with Storage

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/icechunk-js.md

Shows how to create storage backends for all major cloud providers in Node.js (or NAPI-compatible runtimes). Each factory method takes provider-specific arguments such as bucket, prefix, credentials, and options. The HTTP backend is read-only.

```typescript
import { Storage } from '@earthmover/icechunk'

// Local filesystem
const local = await Storage.newLocalFilesystem('/path/to/repo')

// Amazon S3
const s3 = Storage.newS3('my-bucket', 'my-prefix', credentials, options)

// Google Cloud Storage
const gcs = Storage.newGcs('my-bucket', 'my-prefix', credentials, config)

// Azure Blob Storage
const azure = await Storage.newAzureBlob('account', 'container', 'prefix', credentials, config)

// Cloudflare R2
const r2 = Storage.newR2('my-bucket', 'my-prefix', accountId, credentials, options)

// Tigris
const tigris = Storage.newTigris('my-bucket', 'my-prefix', credentials, options)

// HTTP (read-only)
const http = Storage.newHttp('https://example.com/repo')
```

--------------------------------

### Install Icechunk with pip

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/getting-started/quickstart.md

Install Icechunk using pip. Requires Python and Zarr Python 3 for Zarr V3 support.

```bash
python -m pip install icechunk
```

--------------------------------

### Open repository with split size 5

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/performance.md

Opens the same storage with a different splitting config where 5 chunk references are in a single manifest, and prints the new repository's manifest config.

```python
split_config = ic.config.ManifestSplittingConfig.from_dict(
    {ic.config.ManifestSplitCondition.AnyArray(): {ic.config.ManifestSplitDimCondition.Any(): 5}}
)
repo_config = ic.config.RepositoryConfig(manifest=ic.config.ManifestConfig(splitting=split_config))
new_repo = ic.Repository.open(storage, config=repo_config)
print(new_repo.config.manifest)
```

--------------------------------

### Create Icechunk repo with local filesystem

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/configuration.md

Creates a new Icechunk repository on the local filesystem using the provided configuration. Requires an existing `config` object.

```python
repo = ic.Repository.create(
    storage=ic.local_filesystem_storage("/path/to/my/dataset"),
    config=config
)
```

--------------------------------

### Skip dataset re-creation with --force-setup=False

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/benchmarks/README.md

Avoids re-creating datasets when possible by setting --force-setup=False. Use this flag to skip dataset creation if the datasets already exist.

```sh
pytest -nauto -m setup_benchmarks --force-setup=False benchmarks/
```

--------------------------------

### Open Icechunk repo from local filesystem

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/configuration.md

Opens an existing Icechunk repository from the local filesystem using `IcechunkStore.open`. If a config is provided, it is merged with any persisted config, with the provided config taking precedence.

```python
storage = ic.local_filesystem_storage("/path/to/my/dataset")
store = ic.zarr.IcechunkStore.open(
    storage=storage,
    config=config,
)
```

--------------------------------

### Check out a branch in Icechunk

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/icechunk-for-git-users.md

Shows how to create read-only and read-write sessions based on a branch tip.

```python
# check out a branch for read-only access
session = repo.readonly_session(branch="my-new-branch")
# readonly_session accepts a branch name by default
session = repo.readonly_session("my-new-branch")
# check out a branch for read-write access
session = repo.writable_session("my-new-branch")
```

--------------------------------

### List manifest files for snapshot

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/performance.md

Lists the manifest files associated with the snapshot using repo.list_manifest_files.

```python
print(repo.list_manifest_files(snap))
```

--------------------------------

### Look up a branch snapshot with repo.lookup_branch

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/icechunk-for-git-users.md

Shows how to get the snapshot id that a branch points to.

```python
repo.lookup_branch("my-new-branch")

# '198273178639187'
```

--------------------------------

### Append data with append_dim='time'

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/xarray.md

Gets a new session and appends ds2 to the store along the time dimension.

```python
# we have to get a new session after committing
session = repo.writable_session("main")
to_icechunk(ds2, session, append_dim='time')
```

--------------------------------

### Compare benchmarks across multiple stores

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/benchmarks/README.md

Runs the `test_write_simple` benchmark across multiple stores (`s3`, `s3_ob`, `gcs`) on the `main` ref. The output shows median times for each store, with the fastest normalized to 1.0.

```sh
python benchmarks/runner.py --skip-setup --pytest '-k test_write_simple' --where 's3|s3_ob|gcs' main
```

```text
-------- benchmark 'test_write_simple_1d simple-1d': 3 tests --------
Name (time in s)                                     Median
---------------------------------------------------------------------
test_write_simple_1d[simple-1d] (g/gcs_main_95e)     5.2314 (3.15)
test_write_simple_1d[simple-1d] (g/s3_main_95ef)     1.6622 (1.0)
test_write_simple_1d[simple-1d] (g/s3_ob_main_9)     1.6909 (1.02)
---------------------------------------------------------------------
```

--------------------------------

### Custom exception for rebase failure

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/002-rebase-commit-improvements.md

Example of accessing snapshot_id and conflicts from a RebaseFailedError to list conflict paths and reasons.

```python
try:
    session.rebase(solver=BasicConflictSolver())
    commit_id = session.commit("merged")
except RebaseFailedError as e:
    # just an example of what we could pull for data, the message would be more informative
    print(f"Failed to rebase at snapshot {e.snapshot_id} because of conflicts:")
    for conflict in e.conflicts:
        print(f"{c.path}: {c.reason}")
```

--------------------------------

### Custom exception for commit conflict

Source: https://github.com/earth-mover/icechunk/blob/main/design-docs/002-rebase-commit-improvements.md

Example of accessing expected_parent and actual_parent attributes from a ConflictError to provide detailed information.

```python
try:
    commit_id = session.commit("yay")
except ConflictError as e:
    # just an example of what we could pull for data, the message would be more informative
    print(f"Commit failed: expected parent: {e.expected_parent}, actual parent: {e.actual_parent}")
```

--------------------------------

### Create repository with data and show tree

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/moving-nodes.md

Creates an in-memory Icechunk repository, adds a group and array, commits, and prints the tree before a move. Uses a writable session.

```python
import icechunk as ic
import zarr
import numpy as np

# Create a repository with some data
repo = ic.Repository.create(ic.in_memory_storage())
session = repo.writable_session("main")
root = zarr.group(session.store)
root.create_group("data/raw")
arr = root.create_array("data/raw/temperature", shape=(100,), dtype="f4")
arr[:] = np.random.randn(100)
session.commit("Initial structure")

print("Before:")
print(root.tree())
```

--------------------------------

### Create repository and commit initial data

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/understanding/version-control.md

Creates an in-memory Icechunk repository, opens a writable session on the 'main' branch, adds a root group attribute, creates a 10x10 int32 array named 'data', and commits the changes.

```python
import icechunk as ic
import numpy as np
import zarr

repo = ic.Repository.create(ic.in_memory_storage())
session = repo.writable_session("main")
root = zarr.create_group(session.store)
root.attrs["foo"] = "bar"
root.create_array("data", shape=(10, 10), chunks=(1, 1), dtype=np.int32)
print(session.commit(message="Add foo attribute and data array"))
```

--------------------------------

### Icechunk Permanent References Example

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/overview.md

Demonstrates using Icechunk tags to reference immutable data versions, guaranteeing reproducible analyses.

```python
# January: Run analysis using specific tagged version
import icechunk as ic
repo = ic.Repository.open(
    ic.s3_storage(bucket="public-weather", prefix="forecast")
)
session = repo.readonly_session(tag="2025-01-15")
forecast = zarr.open_group(session.store)
results_jan = analyze(forecast["temperature"][:])
# Record in paper: "Analysis used tag 2025-01-15"

# March: Reproduce results using same tagged version
session = repo.readonly_session(tag="2025-01-15")
forecast = zarr.open_group(session.store)
results_mar = analyze(forecast["temperature"][:])
# Exact same data, guaranteed reproducible results! ✓
```

--------------------------------

### Create a branch in Icechunk

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/icechunk-for-git-users.md

Shows how to create a branch in Icechunk, create a writable session, and commit changes.

```python
# We create the branch
repo.create_branch("my-new-branch", repo.lookup_branch("main"))
# create a writable session
session = repo.writable_session("my-new-branch")
...  # make some changes
# commit the changes
session.commit("My new branch")
```

--------------------------------

### Import libraries and set xarray display options

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/ingestion/glad-ingest.ipynb

Sets up the Python environment for the GLAD ingestion workflow, importing dask, icechunk, numpy, pandas, and xarray, and configuring xarray to display arrays as text.

```python
import itertools

import dask
import dask.array
import icechunk as ic
import numpy as np
import pandas as pd
import xarray as xr

xr.set_options(display_style="text")
```

--------------------------------

### Get commit history with repo.ancestry

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/icechunk-js.md

Retrieves the commit history for a branch using `repo.ancestry` and iterates over the returned snapshots, logging each snapshot's ID and message.

```typescript
const history = await repo.ancestry({ branch: 'main' })
for (const snapshot of history) {
  console.log(`${snapshot.id} ${snapshot.message}`)
}
```

--------------------------------

### Run tiered pre-commit checks

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/reference/contributing.md

Tiered pre-commit commands for code quality checks: fast (format/lint), medium (compilation and deps), and full CI (all tests and examples).

```bash
# Fast checks (~3 seconds) - format and lint only
just pre-commit-fast

# Medium checks (~2-3 minutes) - includes compilation and deps
just pre-commit

# Full CI checks (~5+ minutes) - includes all tests and examples
just pre-commit-ci
```

--------------------------------

### Configure StorageRetriesSettings

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/flaky-networks.md

Configures retry behavior for all storage operations by setting the `retries` attribute on the default repository config. The example overrides the defaults with `max_tries=20`, `initial_backoff_ms=200`, and `max_backoff_ms=60_000`.

```python
import icechunk

config = icechunk.RepositoryConfig.default()
config.storage.retries = icechunk.StorageRetriesSettings(
    max_tries=20,
    initial_backoff_ms=200,
    max_backoff_ms=60_000,
)

```

--------------------------------

### Create Dask array for distributed write

Source: https://github.com/earth-mover/icechunk/blob/main/icechunk-python/docs/docs/guides/dask.md

Create a Dask array of shape (100, 100) with chunks of (20, 20). This array is the source for the distributed write example.

```python
import dask.array as da
shape = (100, 100)
dask_chunks = (20, 20)
dask_array = da.random.random(shape, chunks=dask_chunks)
```