VirtualiZarr

https://github.com/zarr-developers/virtualizarr
Admin
VirtualiZarr creates virtual Zarr stores from archival scientific data formats using Xarray syntax,...
Tokens:64,978
Snippets:546
Trust Score:8.6
License:Apache-2.0
Update:1 month ago
Show doc for...
Tokens:
Raw
### Install documentation dependencies

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Install requirements for building the project documentation locally.

```shell
pip install '-e .[docs]'
```

--------------------------------

### Installing VirtualiZarr

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Install the development version of VirtualiZarr from GitHub.

```bash
# we want some small unreleased features, should arrive in v2.6.3
!pip install git+https://github.com/zarr-developers/VirtualiZarr.git
```

--------------------------------

### Serve documentation locally

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Command to start a live-reloading documentation server.

```bash
pixi run serve-docs
```

--------------------------------

### Install all file parser dependencies

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Convenience command to install all available file format parsers.

```shell
pip install "virtualizarr[all_parsers]"
```

--------------------------------

### Install VirtualiZarr via pip

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Standard installation command for the base package.

```shell
pip install virtualizarr
```

--------------------------------

### Build documentation locally with Pixi

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Commands to install the documentation environment and build the site.

```bash
pixi install --environment docs
pixi run build-docs
```

--------------------------------

### Install VirtualiZarr with GRIB support

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/api/parsers/grib.md

Install the necessary dependencies to enable GRIB parsing capabilities.

```shell
pip install "virtualizarr[grib]"
```

--------------------------------

### Install optional dependency groups

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Install specific sets of dependencies for reading or writing formats using bracket notation.

```shell
pip install "virtualizarr[hdf, icechunk]"
```

--------------------------------

### Install pre-commit hooks

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Commands to install the pre-commit tool and its associated hooks for code formatting and linting.

```bash
python -m pip install pre-commit
```

```bash
pre-commit install
```

--------------------------------

### Install test dependencies

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Install requirements for running the local test suite in editable mode.

```shell
pip install '-e .[test]'
```

--------------------------------

### Install VirtualiZarr via conda

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/getting_started/installation.md

Installation command for users of the conda-forge channel.

```shell
conda install -c conda-forge virtualizarr
```

--------------------------------

### Open virtual dataset with explicit configuration

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/migration_guide.md

V2 requires explicit Parser and ObjectStoreRegistry configuration. Use these examples to configure S3 or local storage access.

```python
from obstore.store import S3Store
from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset
from virtualizarr.parsers import HDFParser

bucket = "nex-gddp-cmip6"
store = S3Store(
    bucket=bucket,
    region="us-west-2",
    skip_signature=True # required for this specific example data because the data is in a public bucket, so the S3Store shouldn't fetch and use credentials.
)
registry = ObjectStoreRegistry({f"s3://{bucket}": store})
parser = HDFParser()
vds = open_virtual_dataset(
    url=f"s3://{bucket}/NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc",
    registry=registry,
    parser=parser
)
print(vds)
```

```python


from obstore.store import LocalStore
from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset
from virtualizarr.parsers import HDFParser

from pathlib import Path

store_path = Path.cwd()
file_path = str(store_path / "tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc")
file_url = f"file://{file_path}"

store = LocalStore(prefix=store_path)
registry = ObjectStoreRegistry({file_url: store})
parser = HDFParser()

vds = open_virtual_dataset(
    url=file_url,
    registry=registry,
    parser=parser
)
    print(vds)
```

--------------------------------

### Run tests in alternative environments

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Execute tests using specific dependency configurations or list installed versions.

```bash
pixi run --environment min-deps run-tests # Test with the minimal set of dependencies installed
# List which versions are installed in the `min-deps` environment
pixi list --environment min-deps
```

--------------------------------

### Configure ObjectStore for Cloud and Local Data

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/how_to/usage.md

Examples for initializing ObjectStore instances across different storage providers including S3, GCS, Azure, R2, HTTP, and CEPH/OSN.

```python
import xarray as xr
from obstore.store import from_url

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser
from obspec_utils.registry import ObjectStoreRegistry

bucket = "s3://nex-gddp-cmip6"
path = "NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc"
url = f"{bucket}/{path}"
store = from_url(bucket, region="us-west-2", skip_signature=True)
registry = ObjectStoreRegistry({bucket: store})
```

```python
import xarray as xr
from obstore.store import from_url

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser
from obspec_utils.registry import ObjectStoreRegistry

bucket = "gs://data-bucket"
path = "file-path/data.nc"
url = f"{bucket}/{path}"
store = from_url(bucket)
registry = ObjectStoreRegistry({bucket: store})
```

```python
import xarray as xr
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import from_url

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

bucket = "abfs://data-container"
path = "file-path/data.nc"
url = f"{bucket}/{path}"
store = from_url(bucket)
registry = ObjectStoreRegistry({bucket: store})
```

```python

import xarray as xr
from obstore.store import S3Store
from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

endpoint = "https://f0b62eebfbdde1133378bfe3958325f6.r2.cloudflarestorage.com"
access_key_id = "<access_key_id>"
secret_access_key = "<secret_access_key>"
path = "<path_to_files>"
scheme = "s3://"
bucket_name = "<bucket_name>"
bucket = f"{scheme}{bucket_name}"

# create anon s3 store
store = S3Store.from_url(f"{bucket}", endpoint=endpoint, skip_signature=True)

# create s3 store with aws-style credentials
store = S3Store.from_url(f"{bucket}", endpoint=endpoint, access_key_id=access_key_id, secret_access_key=secret_access_key)

registry = ObjectStoreRegistry({f"{bucket}": store})
```

```python

import xarray as xr
from obstore.store import from_url
from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

# This examples uses a NetCDF file of CMIP6 from ESGF.
bucket  = 'https://esgf-data.ucar.edu'
path = 'thredds/fileServer/esg_dataroot/CMIP6/CMIP/NCAR/CESM2/historical/r3i1p1f1/day/tas/gn/v20190308/tas_day_CESM2_historical_r3i1p1f1_gn_19200101-19291231.nc'
store = from_url(bucket)
registry = ObjectStoreRegistry({bucket: store})
```

```python

import xarray as xr
from obstore.store import S3Store
from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser

endpoint = "https://nyu1.osn.mghpcc.org"
access_key_id = "<access_key_id>"
secret_access_key = "<secret_access_key>"
path = "<path_to_files>"
scheme = "s3://"
bucket_name = "<bucket_name>"
bucket = f"{scheme}{bucket_name}"

# create anon s3 store
store = S3Store.from_url(f"{bucket}", endpoint=endpoint, skip_signature=True)

# create s3 store with aws-style credentials
store = S3Store.from_url(f"{bucket}", endpoint=endpoint, access_key_id = access_key_id, secret_access_key=secret_access_key)

registry = ObjectStoreRegistry({f"{bucket}": store})
```

--------------------------------

### Verifying VirtualiZarr version

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Check the installed version of the VirtualiZarr library.

```python
import virtualizarr as vz
vz.__version__
```

--------------------------------

### Example Chunk Manifest Structure

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/data_structures.md

A dictionary representation of a chunk manifest mapping chunk keys to file paths, offsets, and lengths.

```python
{
    "0.0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
    "0.0.1": {"path": "s3://bucket/foo.nc", "offset": 200, "length": 100},
    "0.1.0": {"path": "s3://bucket/foo.nc", "offset": 300, "length": 100},
    "0.1.1": {"path": "s3://bucket/foo.nc", "offset": 400, "length": 100},
}
```

--------------------------------

### Open Virtual Multi-File Dataset

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Example usage of open_virtual_mfdataset with the preprocessing pipeline.

```python
vmds = vz.open_virtual_mfdataset(
    urls,
    registry=registry,
    parser=vz.parsers.HDFParser(),
    loadable_variables=list(DEFAULT_LOADABLE_VARIABLES),
    preprocess=preprocess,
    concat_dim="t", combine="nested",
)
```

--------------------------------

### Initialize ObjectStore and Registry

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/index.md

Create an object store from a URL and register it for use with VirtualiZarr.

```python
bucket = "s3://nex-gddp-cmip6"
path = "NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc"
store = obstore.store.from_url(bucket, region="us-west-2", skip_signature=True)
registry = ObjectStoreRegistry({bucket: store})
```

--------------------------------

### Initialize Object Store Registry

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Sets up the object store registry if one is not provided.

```python
if registry is None:
    store = obs.store.from_url(BUCKET, region="us-east-1", skip_signature=True)
    registry = ObjectStoreRegistry({BUCKET: store})
```

--------------------------------

### Run basic GOES-16 virtualization

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/README.md

Executes the baseline virtualization script for GOES-16 satellite data without optimizations.

```bash
uv run --script examples/V2/goes_basic.py
```

--------------------------------

### Initialize Arraylake Client

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Sets up the Arraylake client and authenticates the session.

```python
import arraylake as al
import xarray as xr
import matplotlib.pyplot as plt


client = al.Client()
client.login()
```

--------------------------------

### Initializing Repository with Caching

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Retrieves a repository instance with specific manifest and caching configurations.

```python
ic_repo = al_client.get_repo(
    name="earthmover-public/goes-16",
    config=ic.RepositoryConfig(
        manifest=ic.ManifestConfig(splitting=split_config),
        caching=caching_config,
    ),
)
```

--------------------------------

### Checking Icechunk version

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Verify the installed version of Icechunk.

```python
# Icechunk v2.0.5 has some important optimizations for this workload
ic.__version__
```

--------------------------------

### Setting minimum year

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Define the starting year for operational GOES-16 data.

```python
# GOES-16 went operational in 2017; anything earlier is a sentinel/test directory.
MIN_YEAR = 2017
```

--------------------------------

### Manual Era Ingestion

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Example of manually triggering ingestion for a specific era.

```python
# ingest the first of 4 eras
ingest_all_days(
    ic_repo,
    branch="v2-ingest",
    group="ABI-L2-MCMIPF/pre-2017-07-24",
    start=None,
    end=LAST_EARLY_PREOP_FILE,
    #loop_end_day=5,                # just the first 5 days
    verify_every=1000,
    zarr_async_concurrency=64,
)
```

--------------------------------

### Run optimized GOES-16 virtualization

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/README.md

Executes the virtualization script using CachingReadableStore and SplittingReadableStore for improved metadata access.

```bash
uv run --script examples/V2/goes_with_caching_stores.py
```

--------------------------------

### Import required libraries

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/its_live.ipynb

Initializes the environment with necessary dependencies for virtual dataset handling and object storage access.

```python
import os

import numpy as np
import obstore
import xarray as xr
from obspec_utils.registry import ObjectStoreRegistry

import virtualizarr as vz
from virtualizarr.parsers import HDFParser
```

--------------------------------

### Resetting Repository Branch

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Example of resetting a repository branch to a specific snapshot ID.

```python
# nuke the repo back to it's starting point
# ic_repo.reset_branch(
#     branch="v2-ingest",
#     snapshot_id="1CECHNKREP0F1RSTCMT0",   # universal icechunk initial-empty snapshot
#     #from_snapshot_id=ic_repo.lookup_branch("v2-ingest"),  # safety 
# )
```

--------------------------------

### Configure object store and parser

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/its_live.ipynb

Sets up the object store registry and HDF parser, excluding specific variables that are not needed.

```python
store = obstore.store.from_url(bucket, region="us-west-2", skip_signature=True)
registry = ObjectStoreRegistry({bucket: store})
parser = HDFParser(drop_variables=["mapping", "img_pair_info"])
```

--------------------------------

### Test against upstream development versions

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Install and test using unreleased development versions of upstream libraries.

```bash
pixi install --environment upstream
pixi run -e upstream install-upstream-overrides
pixi run --environment upstream run-tests
```

--------------------------------

### Compare virtualization performance

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/README.md

Runs both basic and optimized scripts sequentially to compare virtualization execution times.

```bash
uv run --script examples/V2/goes_basic.py
uv run --script examples/V2/goes_with_caching_stores.py
```

--------------------------------

### Iterate Archive Days

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Yields chronological (year, doy) tuples for the archive, supporting optional start and end clipping.

```python
def iter_days(
    *,
    start: str | None = None,
    end: str | None = None,
) -> Iterator[tuple[int, int]]:
    """Yield ``(year, doy)`` for every day in the archive, in chronological order.

    Days are inclusively clipped to those at or after ``start`` and at or before
    ``end``. ``start``/``end`` are full file URLs (or anything compatible with
    :func:`_parse_url_to_day`).
    """
    start_day = _parse_url_to_day(start) if start else None
    end_day = _parse_url_to_day(end) if end else None

    for year in list_years():
        if start_day and year < start_day[0]:
            continue
        if end_day and year > end_day[0]:
            return
        for doy in list_days_in_year(year):
            current = (year, doy)
            if start_day and current < start_day:
                continue
            if end_day and current > end_day:
                return
            yield current
```

--------------------------------

### Configure cast_value Codec with Scalar Mapping

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/custom_parsers.md

Example configuration for handling fill values during type casting using scalar_map.

```json
{
    "name": "cast_value",
    "configuration": {
        "data_type": "uint8",
        "rounding": "nearest-even",
        "scalar_map": {
            "encode": [["NaN", 0]],
            "decode": [[0, "NaN"]]
        }
    }
}
```

--------------------------------

### Inspecting Repository Configuration

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Displays the current repository configuration settings.

```python
ic_repo.config
```

--------------------------------

### Configure and create an Icechunk repository

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/its_live.ipynb

Sets up a local Icechunk repository configured to access public S3 data as virtual chunks.

```python
import shutil

import icechunk as ic

# the referenced chunk data lives on this public, anonymous S3 bucket
url_prefix = "s3://its-live-data/"
store_path = "its_live_cube.icechunk"

# start fresh so this cell is re-runnable
shutil.rmtree(store_path, ignore_errors=True)

# register a virtual chunk container so icechunk knows how to read the s3 refs
config = ic.RepositoryConfig.default()
config.set_virtual_chunk_container(
    ic.VirtualChunkContainer(
        url_prefix, ic.s3_store(region="us-west-2", anonymous=True)
    )
)
repo = ic.Repository.create(
    storage=ic.local_filesystem_storage(store_path),
    config=config,
    authorize_virtual_chunk_access=ic.containers_credentials(
        {url_prefix: ic.s3_credentials(anonymous=True)}
    ),
)
```

--------------------------------

### Open a virtual dataset

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/how_to/usage.md

Initialize a parser and open a virtual dataset using the specified registry and URL.

```python
parser = HDFParser()
vds = open_virtual_dataset(
  url=f"{bucket}/{path}",
  parser=parser,
  registry=registry,
)
```

--------------------------------

### List and Filter Archive Files

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Retrieves and deduplicates file URLs from the archive for a specific day, applying optional start and end constraints.

```python
    store = _store()
    day_prefix = f"{PRODUCT}{year}/{doy:03d}/"
    urls: list[str] = []
    for page in obs.list(store, prefix=day_prefix):
        for obj in page:
            path = obj["path"]
            if path.endswith(".nc") and not _is_aborted_scan(path):
                urls.append(f"{BUCKET}/{path}")
    urls.sort(key=_scan_start_token)

    # Deduplicate by scan-start token: some NOAA hour prefixes get listed
    # under both `.../H/` and `.../HH/` (single-digit and zero-padded), so
    # obstore returns the same underlying file twice. Keep the first occurrence.
    seen: set[str] = set()
    deduped: list[str] = []
    for u in urls:
        tok = _scan_start_token(u)
        if tok not in seen:
            seen.add(tok)
            deduped.append(u)
    urls = deduped

    if start is not None:
        urls = [u for u in urls if _scan_start_token(u) >= _scan_start_token(start)]
    if end is not None:
        urls = [u for u in urls if _scan_start_token(u) <= _scan_start_token(end)]
    return urls
```

--------------------------------

### Define Zarr V3 Codec Chain for Packing

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/custom_parsers.md

Example configuration for packing a float32 array into uint8 using scale_offset and cast_value codecs.

```json
{
    "data_type": "float32",
    "codecs": [
        {
            "name": "scale_offset",
            "configuration": {
                "offset": 1000,
                "scale": 0.1
            }
        },
        {
            "name": "cast_value",
            "configuration": {
                "data_type": "uint8",
                "out_of_range": "wrap"
            }
        },
        "bytes"
    ]
}
```

--------------------------------

### Open documentation on MacOS

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Terminal command to open the built documentation in a web browser.

```bash
open docs/_build/html/index.html
```

--------------------------------

### Determine Codec Era

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Determines whether to use 'pre' or 'post' codec based on the scan start time derived from dataset encoding or attributes.

```python
    for candidate in (
        (ds.encoding.get("source") or "").rsplit("/", 1)[-1],
        ds.attrs.get("dataset_name") or "",
    ):
        if not candidate:
            continue
        try:
            scan_start = _parse_scan_start_to_datetime(candidate)
        except (ValueError, IndexError):
            continue
        return "post" if scan_start >= CODEC_CHANGE_DATETIME else "pre"
    raise ValueError(
        "Cannot determine codec era: neither ds.encoding['source'] nor "
        "ds.attrs['dataset_name'] yields a parseable scan-start token."
    )
```

--------------------------------

### Configure Zarr Concurrency and Ingestion Range

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Sets global Zarr concurrency and calculates the subset of days to ingest based on provided start and end indices.

```python
if zarr_async_concurrency is not None:
    import zarr
    zarr.config.set({"async.concurrency": zarr_async_concurrency})
    print(
        f"zarr async concurrency set to {zarr_async_concurrency} "
        f"(process-global; affects write path on the driver and verify "
        f"reads on workers if forwarded via dask.delayed).",
        flush=True,
    )

parser = vz.parsers.HDFParser()
loadable_variables = list(loadable_variables)

all_days_to_ingest = days_to_ingest(start=start, end=end)
end_idx = loop_end_day if loop_end_day is not None else len(all_days_to_ingest)
selected = all_days_to_ingest[loop_start_day:end_idx]
print(
    f"Ingesting {len(selected)} day(s) "
    f"[{loop_start_day}:{end_idx}] of {len(all_days_to_ingest)} total available.",
    flush=True,
)
```

--------------------------------

### Initialize Schema and Auto-Resume Logic

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Checks for existing schema to determine if this is the first write and queries the last committed day to support idempotent re-runs.

```python
is_first_write = not _schema_exists(repo, branch, group)
if is_first_write:
    where = f"group {group!r}" if group else "root group"
    print(
        f"No existing schema on '{branch}' at {where} — the first iteration "
        "will create it (no append_dim).",
        flush=True,
    )

# Auto-resume: query the existing dataset's last t value and skip every
# already-committed day. Makes re-runs idempotent — rerunning with an
# overlapping `loop_start_day` no longer double-ingests.
skip_through: tuple[int, int] | None = None
last_committed_t: np.datetime64 | None = None
if resume and not is_first_write:
    info = _last_committed_day(repo, branch, group)
    if info is not None:
        skip_through, last_committed_t = info
        print(
            f"Auto-resuming: last committed day is "
            f"{skip_through[0]}-{skip_through[1]:03d} "
            f"(last t = {last_committed_t}). Skipping any selected days "
            f"at or before that.",
            flush=True,
        )
```

--------------------------------

### Setting Up Coiled Cluster

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Configures and initializes a Coiled cluster for distributed processing with specific worker types and spot policies.

```python
cluster = coiled.Cluster(
    name="GOES",
    n_workers=[1000, 1500],  # roughly highest I can do without an AWS account quota increase - sad
    region="us-east-1",
    #worker_vm_types="m8g.medium",
    worker_vm_types=[
        # spread across c6/c7/c8 + m6/m7/m8, two sizes each = ~18 spot pools
        "t4g.small", "t4g.medium",# "t4g.large",
        "c6g.medium",
        "m6g.medium",
        "c7g.medium",
        "m7g.medium",
        "c8g.medium",
        "m8g.medium",
        #"c6g.large", "c7g.large", "c8g.large",
        #"m6g.large", "m7g.large", "m8g.large",
        #"c6g.xlarge", "c7g.xlarge", "c8g.xlarge",
    ],
    scheduler_vm_types=["c8g.2xlarge", "m7g.2xlarge", "m8g.2xlarge"],  # (one time I hit AWS capacity constraints so let Coiled choose)
    spot_policy="spot",  # important to avoid 10x cost spikes!
    #use_best_zone=True,
    allow_cross_zone=True,
    wait_for_workers=False,
)
client = cluster.get_client()
client
```

--------------------------------

### Create a ChunkManifest with inlined data

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/data_structures.md

Initialize a manifest by providing a dictionary of entries where inlined chunks include a 'data' key containing raw bytes.

```python
from virtualizarr.manifests import ChunkManifest

manifest = ChunkManifest(
    entries={
        "0.0": {"path": "s3://bucket/foo.nc", "offset": 100, "length": 100},
        "0.1": {"path": "", "offset": 0, "length": 4, "data": b"\x00\x01\x02\x03"},
    }
)
```

--------------------------------

### Import VirtualiZarr and dependencies

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/index.md

Import the necessary classes and functions for VirtualiZarr and object store management.

```python
import icechunk
import obstore

from obspec_utils.registry import ObjectStoreRegistry

from virtualizarr import open_virtual_dataset, open_virtual_mfdataset
from virtualizarr.parsers import HDFParser
```

--------------------------------

### Locate and fetch GOES-16 files

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Logic to identify candidate files within a time window and download them to a temporary file for processing.

```python
# Generate candidate URLs from this day and the immediately preceding
    # day (in case ts is just past midnight).
    candidates: list[str] = []
    for offset_days in (-1, 0):
        date = (py_dt + datetime.timedelta(days=offset_days)).date()
        doy = date.timetuple().tm_yday
        try:
            candidates.extend(list_day_files(date.year, doy))
        except Exception:
            pass

    window_start = py_dt - _FILENAME_TOKEN_TO_T_OFFSET_WINDOW
    window_end = py_dt
    filtered: list[str] = []
    for url in candidates:
        token_t = parse_scan_start_to_datetime(url).astype("datetime64[us]").astype(datetime.datetime)
        if window_start <= token_t <= window_end:
            filtered.append(url)

    if not filtered:
        raise FileNotFoundError(
            f"No candidate NetCDF in {py_dt.date()} whose filename-token-time "
            f"falls within {_FILENAME_TOKEN_TO_T_OFFSET_WINDOW} before t={t}"
        )

    for url in filtered:
        tmp = _fetch_source_to_tempfile(url, registry=registry)
        try:
            file_t = _read_t_from_netcdf(tmp)
            if file_t == ts:
                return url
        finally:
            try:
                os.unlink(tmp)
            except OSError:
                pass

    raise FileNotFoundError(
        f"None of {len(filtered)} candidate file(s) near t={t} had a matching "
        f"internal t value. Candidates tried: {[u.rsplit('/', 1)[-1] for u in filtered]}"
    )
```

```python
def _fetch_source_to_tempfile(url: str, *, registry: ObjectStoreRegistry | None = None) -> str:
    """Download a single S3 NetCDF to a temp path; return the path."""
    if not url.startswith(BUCKET + "/"):
        raise ValueError(f"URL must live under {BUCKET}; got {url!r}")
    key = url[len(BUCKET) + 1:]

    if registry is not None:
        store, _ = registry.resolve(url)
    else:
        store = obs.store.from_url(BUCKET, region="us-east-1", skip_signature=True)

    blob = obs.get(store, key).bytes().to_bytes()
    fd, path = tempfile.mkstemp(suffix=".nc", prefix="mcmipf_verify_")
    with os.fdopen(fd, "wb") as f:
        f.write(blob)
    return path
```

--------------------------------

### Open a GRIB file as a virtual dataset

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/api/parsers/grib.md

Initialize the GribberishParser and use it with open_virtual_dataset to access GRIB data.

```python
from obspec_utils.registry import ObjectStoreRegistry
from obstore.store import LocalStore

from gribberish.virtualizarr import GribberishParser
from virtualizarr import open_virtual_dataset

registry = ObjectStoreRegistry({"file://": LocalStore()})
parser = GribberishParser()
vds = open_virtual_dataset(
    url="file:///path/to/file.grib2", parser=parser, registry=registry
)
```

--------------------------------

### Repository Configuration Output

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

The resulting configuration object showing caching, storage, and manifest settings.

```text
Result:
icechunk.config.RepositoryConfig(
    inline_chunk_threshold_bytes=None,
    get_partial_values_concurrency=None,
    max_concurrent_requests=None,
    num_updates_per_repo_info_file=None,
    compression=None,
    caching=icechunk.config.CachingConfig(
        num_snapshot_nodes=2,
        num_chunk_refs=50000000,
        num_transaction_changes=None,
        num_bytes_attributes=None,
        num_bytes_chunks=None,
    ),
    storage=icechunk.storage.StorageSettings(
        unsafe_use_conditional_create=None,
        unsafe_use_conditional_update=None,
        unsafe_use_metadata=None,
        storage_class=None,
        metadata_storage_class=None,
        chunks_storage_class=None,
        minimum_size_for_multipart_upload=None,
        concurrency=None,
        retries=None,
        timeouts=None,
    ),
    manifest=icechunk.config.ManifestConfig(
        preload=None,
        splitting=icechunk.config.ManifestSplittingConfig(
            split_sizes=[(icechunk.config.ManifestSplitCondition.name_matches("CMI"), [(icechunk.config.ManifestSplitDimCondition.dimension_name("y"), 12), (icechunk.config.ManifestSplitDimCondition.dimension_name("x"), 12), (icechunk.config.ManifestSplitDimCondition.dimension_name("t"), 2160)]), (icechunk.config.ManifestSplitCondition.name_matches("DQF"), [(icechunk.config.ManifestSplitDimCondition.dimension_name("y"), 12), (icechunk.config.ManifestSplitDimCondition.dimension_name("x"), 12), (icechunk.config.ManifestSplitDimCondition.dimension_name("t"), 2160)])],
        ),
        virtual_chunk_location_compression=None,
    ),
    repo_update_retries=None,
    virtual_chunk_containers={
        "s3://noaa-goes16/": icechunk.virtual.VirtualChunkContainer(
            name=None,
            url_prefix="s3://noaa-goes16/",
            store=icechunk.config.ObjectStoreConfig.S3(icechunk.storage.S3Options(
                region="us-east-1",
                endpoint_url=None,
                allow_http=False,
                anonymous=False,
                force_path_style=False,
                network_stream_timeout_seconds=None,
                requester_pays=False,
                checksum_algorithm=None,
            )),
        ),
    },
)
```

--------------------------------

### Create Arraylake Repository with Manifest Config

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Initializes an Arraylake repository with the specified manifest splitting configuration and virtual chunk access.

```python
# we want the manifest splitting to be permanently attached to this repo, so it definitely gets used by any writes
ic_repo = al_client.create_repo(
    name="earthmover-public/goes-16",
    bucket_config_nickname="earthmover",  # bucket in us-east-1
    authorize_virtual_chunk_access={"s3://noaa-goes16/": "noaa-goes16"},
    config=ic.RepositoryConfig(
        manifest=ic.ManifestConfig(splitting=split_config),
    ),
)
```

--------------------------------

### Configure Icechunk repository

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/index.md

Set up an Icechunk repository configuration with a virtual chunk container.

```python
config = icechunk.RepositoryConfig.default()
container = icechunk.VirtualChunkContainer(
    url_prefix="s3://nex-gddp-cmip6/",
    store=icechunk.s3_store(region="us-west-2", anonymous=True),
)
config.set_virtual_chunk_container(container)
```

--------------------------------

### Open virtual datasets for manual concatenation

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/how_to/usage.md

Initialize virtual datasets from individual file URLs before combining them.

```python
url_1 = "s3://nex-gddp-cmip6/NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2015_v2.0.nc"
url_2 = "s3://nex-gddp-cmip6/NEX-GDDP-CMIP6/ACCESS-CM2/ssp126/r1i1p1f1/tasmax/tasmax_day_ACCESS-CM2_ssp126_r1i1p1f1_gn_2016_v2.0.nc"

vds1 = open_virtual_dataset(url=url_1, registry=registry, parser=parser)
vds2 = open_virtual_dataset(url=url_2, registry=registry, parser=parser)
```

--------------------------------

### Constructing an Empty ChunkManifest

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/data_structures.md

Initialize an empty manifest with a specified chunk grid shape.

```python
from virtualizarr.manifests import ChunkManifest

manifest = ChunkManifest(entries={}, shape=(4, 8))
```

--------------------------------

### Implementing a Custom Parser with Configuration

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/explanation/custom_parsers.md

Use a class-based approach to store configuration options as attributes, which are then accessed during the __call__ execution.

```python
class CustomParser:
    def __init__(self, option_1: bool = False,  option_2: int | None = None) -> None:
        self.option_1 = option_1
        self.option_2 = option_2

    def __call__(self, url: str, registry: ObjectStoreRegistry) -> ManifestStore:
        # access the file's contents, e.g. using the ObjectStore instance in the registry
        store, path_in_store = registry.resolve(url)
        readable_file = obstore.open_reader(store, path_in_store)

        # parse the file contents to extract its metadata
        # this is generally where the format-specific logic lives
        manifestgroup: ManifestGroup = extract_metadata(readable_file, option_1=self.option_1, option_2=self.option_2)

        # construct the Manifeststore from the parsed metadata and the object store registry
        return ManifestStore(group=manifestgroup, registry=registry)

parser = CustomParser(option_1=True, option_2=6)
vds = vz.open_virtual_dataset(
    url,
    registry=registry,
    parser=parser,
)
```

--------------------------------

### Define Preprocessing Constants and Loadable Variables

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

Sets up constants for GOES-16 grid dimensions and defines the default list of variables to be loaded during processing.

```python
X_SIZE = 5424
Y_SIZE = 5424
N_BANDS = 16
ALL_CHANNELS = range(1, 17)
REFLECTIVE_CHANNELS = range(1, 7)   # C01-C06 — reflectance_factor stats
EMISSIVE_CHANNELS = range(7, 17)    # C07-C16 — brightness_temperature stats

DATETIME_NS = np.dtype("datetime64[ns]")


def _ch(i: int) -> str:
    return f"C{i:02d}"


# Recommended loadable_variables — every low-dimensional variable.
DEFAULT_LOADABLE_VARIABLES: tuple[str, ...] = (
    "x", "y", "t", "band",
    "x_image", "y_image",
    "x_image_bounds", "y_image_bounds", "time_bounds",
    "goes_imager_projection",
    "nominal_satellite_height",
    "nominal_satellite_subpoint_lat",
    "nominal_satellite_subpoint_lon",
    "geospatial_lat_lon_extent",
    "percent_uncorrectable_GRB_errors",
    "percent_uncorrectable_L0_errors",
    "algorithm_product_version_container",
    "dynamic_algorithm_input_data_container",
    *(f"band_id_{_ch(i)}"             for i in ALL_CHANNELS),
    *(f"band_wavelength_{_ch(i)}"     for i in ALL_CHANNELS),
    *(f"outlier_pixel_count_{_ch(i)}" for i in ALL_CHANNELS),
    *(f"{s}_reflectance_factor_{_ch(i)}"
      for s in ("min", "max", "mean", "std_dev")
      for i in REFLECTIVE_CHANNELS),
    *(f"{s}_brightness_temperature_{_ch(i)}"
      for s in ("min", "max", "mean", "std_dev")
      for i in EMISSIVE_CHANNELS),
)
```

--------------------------------

### Run project tests with pixi

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/about/contributing.md

Execute the test suite using the pixi environment manager, with options for network-dependent or offline tests.

```bash
# Run all tests
pixi run --environment test run-tests
# Skip tests that require a network connection
pixi run --environment test run-tests-no-network
```

--------------------------------

### Ingest and Verify GOES-16 Data

Source: https://github.com/zarr-developers/virtualizarr/blob/main/examples/V2/goes-16-ingest.ipynb

This snippet fetches a source file, opens it as a lazy Xarray dataset, and compares it against an Icechunk repository. It uses a custom timing context manager to track performance of the fetch, load, and comparison steps.

```python
timings: dict[str, float] = {}

def _time(name: str):
    @contextlib.contextmanager
    def _cm():
        t0 = time.perf_counter()
        yield
        timings[name] = time.perf_counter() - t0
    return _cm()

with _time("fetch source"):
    tmp = _fetch_source_to_tempfile(source_url, registry=registry)
try:
    with xr.open_dataset(tmp, engine="h5netcdf", chunks=None) as src_ds_lazy:
        file_t = _ensure_datetime64_ns(src_ds_lazy["t"].values)

        if t is not None and _ensure_datetime64_ns(t) != file_t:
            raise ValueError(
                f"t={t!r} disagrees with the internal t of {source_url!r} (={file_t})."
                f" (Hint: the filename `_s_` token is scan-command time, ~minutes "
                f"earlier than the actual t.)"
            )
        t_ns = file_t

        chunks_arg: dict | None = None if force_synchronous else {"y": -1, "x": -1}
        ro = repo.readonly_session(branch=branch)
        with xr.open_zarr(
            ro.store, group=group,
            chunks=chunks_arg, zarr_format=3,
        ) as ic_ds_lazy:
            if "t" not in ic_ds_lazy.dims:
                raise ValueError(f"Group {group!r} has no 't' dim; cannot index by timestamp")
            if t_ns not in ic_ds_lazy["t"].values:
                raise KeyError(
                    f"t={t_ns} (from source {source_url.rsplit('/', 1)[-1]}) not present "
                    f"in icechunk group {group!r} on branch {branch!r}. "
                    f"Available range: {ic_ds_lazy['t'].values[0]} → {ic_ds_lazy['t'].values[-1]}, "
                    f"n_t={ic_ds_lazy.sizes['t']}"
                )
            ic_slice_lazy = ic_ds_lazy.sel(t=t_ns)

            with _time("load icechunk"):
                ic_slice = ic_slice_lazy.load()

            with _time("compare"):
                report = _run_comparison(
                    ic_slice=ic_slice, src_ds=src_ds_lazy, t=t_ns, source_url=source_url,
                    group=group, variables=variables, skip_variables=skip_variables,
                    verbose=False,  # caller handles the success print below
                )
```

--------------------------------

### Use the .vz Xarray Accessor

Source: https://github.com/zarr-developers/virtualizarr/blob/main/docs/migration_guide.md

Use the .vz shorthand for Xarray operations instead of the deprecated virtualize accessor.

```python
vds.vz.to_icechunk(icechunk_store)
```
Tokens:
Raw