### Include Example README in Docs Source: https://github.com/zarr-developers/zarr-python/blob/main/examples/README.md This example shows how to include an example's README.md file directly into a documentation page using a markdown include directive. ```markdown # docs/user-guide/examples/my_example.md --8<-- "examples/my_example/README.md" ## Source Code ```python --8<-- "examples/my_example/my_example.py" ``` ``` -------------------------------- ### Run Custom Data Type Example with UV Source: https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/README.md Execute the custom data type example script using the UV ASGI server. ```bash uv run examples/custom_dtype/custom_dtype.py ``` -------------------------------- ### Run Custom Data Type Example Source: https://github.com/zarr-developers/zarr-python/blob/main/examples/custom_dtype/README.md Execute the custom data type example script using Python. ```bash python examples/custom_dtype/custom_dtype.py ``` -------------------------------- ### Install Hatch and Show Environments Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Install the hatch build tool and list available development environments. Hatch is used to manage Zarr's development environments and dependencies. ```bash pip install hatch hatch env show # list all available environments ``` -------------------------------- ### Install Prek Hooks Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Install the configured prek hooks into your local Git repository. These hooks will run automatically on commit. ```bash prek install ``` -------------------------------- ### Install Zarr using pip Source: https://github.com/zarr-developers/zarr-python/blob/main/README.md Install the Zarr package from PyPI using pip. This is the standard method for installing Python packages. ```bash pip install zarr ``` -------------------------------- ### Get Help for Individual Zarr Commands Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/cli.md Use this to get detailed help for specific commands like 'migrate' or 'remove-metadata'. ```bash zarr migrate --help ``` ```bash zarr remove-metadata --help ``` -------------------------------- ### Create and Use CacheStore Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/experimental.md Demonstrates how to create a CacheStore wrapping a source store with an in-memory cache. This setup improves performance for repeated data access, especially with remote stores. ```python import zarr from zarr.storage import LocalStore import numpy as np from tempfile import mkdtemp from zarr.experimental.cache_store import CacheStore # Create a local store and a separate cache store local_store_path = mkdtemp(suffix='.zarr') source_store = LocalStore(local_store_path) cache_store = zarr.storage.MemoryStore() # In-memory cache cached_store = CacheStore( store=source_store, cache_store=cache_store, max_size=256*1024*1024 # 256MB cache ) # Create an array using the cached store zarr_array = zarr.zeros((100, 100), chunks=(10, 10), dtype='f8', store=cached_store, mode='w') # Write some data to force chunk creation zarr_array[:] = np.random.random((100, 100)) ``` -------------------------------- ### Start Python Interpreter Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Command to start an interactive Python session. Used for checking Python interpreter versions and dependencies. ```console python ``` -------------------------------- ### Create Zarr Array on S3 Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/quick-start.md Example of creating a Zarr array directly to an Amazon S3 bucket. This requires the `s3fs` library to be installed and configured. ```python import zarr import numpy as np z = zarr.create_array( "s3://example-bucket/foo", shape=(100, 100), chunks=(10, 10), dtype="f4" ) z[:, :] = np.random.random((100, 100)) ``` -------------------------------- ### Install Zarr-Python with Conda Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/index.md Install the Zarr-Python library using Conda. This is an alternative installation method. ```bash conda install --channel conda-forge zarr ``` -------------------------------- ### Create FsspecStore with Explicit Filesystem Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Explicitly create an fsspec filesystem (e.g., S3) and then wrap it with FsspecStore. This allows for more control over filesystem configuration. Requires 's3fs' to be installed. ```python # Note: requires s3fs to be installed import fsspec fs = fsspec.filesystem( 's3', anon=True, asynchronous=True, client_kwargs={'endpoint_url': "https://noaa-nwm-retro-v2-zarr-pds.s3.amazonaws.com"} ) store = zarr.storage.FsspecStore(fs) print(store) ``` -------------------------------- ### Open Zarr Array with V3 Metadata (Python) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/cli.md Demonstrates how to open a Zarr array using v3 metadata in Python after a migration. This requires the `zarr` library to be installed. ```python import zarr # create a small array to open (stands in for the migrated store) zarr.create_array("data/cli-demo.zarr", shape=(4, 4), chunks=(2, 2), dtype="i4", overwrite=True) zarr_with_v3_metadata = zarr.open("data/cli-demo.zarr", zarr_format=3) ``` -------------------------------- ### Install Zarr with optional GPU support via pip Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/installation.md Install Zarr with the 'gpu' extra dependency group for GPU acceleration. Use this when GPU support is needed. ```console pip install "zarr[gpu]" ``` -------------------------------- ### Create and Open LocalStore Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Use LocalStore to store Zarr data in a nested directory structure on the local filesystem. Data is read-only in this example. ```python store = zarr.storage.LocalStore('data/foo/bar', read_only=True) group = zarr.open_group(store=store, mode='r') print(group) ``` -------------------------------- ### Create Git Tag for zarr-metadata Release Source: https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-metadata/README.md Example of creating and pushing a git tag for a zarr-metadata release. The tag format must be zarr_metadata-v. ```bash git tag zarr_metadata-v0.2.0 git push origin zarr_metadata-v0.2.0 ``` -------------------------------- ### Create and populate a Zarr array Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Initializes a Zarr array and populates it with data. Used as a prerequisite for subsequent indexing examples. ```python import zarr import numpy as np data = np.arange(10) ** 2 z = zarr.create_array(store='data/example-12.zarr', shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` -------------------------------- ### Install Zarr with optional remote support via pip Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/installation.md Install Zarr with the 'remote' extra dependency group for remote data store support. Use this when interacting with remote storage. ```console pip install "zarr[remote]" ``` -------------------------------- ### Install Prek Git Hooks Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Install the prek pre-commit hook manager locally. This tool enforces code standards and PEP8 compliance before commits. ```bash uv tool install prek ``` ```bash pip install prek ``` -------------------------------- ### Illustrative Zarr Group Creation Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md This is an example of how to create a Zarr group. It is illustrative and includes a placeholder for further operations. ```python import zarr g = zarr.group() # etc. ``` -------------------------------- ### Example Python Interpreter Output Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md An example of the output from a Python interpreter session, showing version and build information. Useful for debugging and reporting. ```ansi Python 3.12.7 | packaged by conda-forge | (main, Oct 4 2024, 15:57:01) [Clang 17.0.6 ] on darwin ``` -------------------------------- ### Install Latest Zarr-Python 2.x Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/v3_migration.md Use this command to install the latest available release of Zarr-Python 2.x. Security and bug fixes will be provided for this version for at least six months after the Zarr-Python 3 release. ```console $ pip install "zarr==2.*" ``` -------------------------------- ### Open Group with Implicit FsspecStore (S3) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Implicitly creates a read-only FsspecStore for an S3 location. Requires `s3fs` to be installed. Pass `storage_options` for authentication or configuration. ```python # Implicitly create a read-only FsspecStore # Note: requires s3fs to be installed group = zarr.open_group( store='s3://noaa-nwm-retro-v2-zarr-pds', mode='r', storage_options={'anon': True} ) print(group) ``` -------------------------------- ### Create and populate a multidimensional Zarr array Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Initializes a 2D Zarr array and populates it with data. This serves as a base for multidimensional indexing examples. ```python data = np.arange(15).reshape(3, 5) z = zarr.create_array(store='data/example-13.zarr', shape=data.shape, dtype=data.dtype) z[:] = data print(z[:]) ``` -------------------------------- ### Install latest nightly Zarr build using pip Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/installation.md Install the latest development build of Zarr from the scientific-python nightly wheels index. Use this for testing the newest features, but be aware that nightly builds may be unstable. ```console pip install --pre --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple zarr ``` -------------------------------- ### Open Remote FsspecStore from URL Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Utilize FsspecStore to access Zarr data on remote storage systems like S3. The filesystem type is inferred from the URL scheme. Requires 's3fs' to be installed. ```python # Note: requires s3fs to be installed store = zarr.storage.FsspecStore.from_url( 's3://noaa-nwm-retro-v2-zarr-pds', read_only=True, storage_options={'anon': True} ) group = zarr.open_group(store=store, mode='r') print(group) ``` -------------------------------- ### Execute Code Block in Documentation Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Example of an executable code block in Markdown documentation using 'exec="true"'. This code will be run when the documentation is built. ```markdown ```python exec="true" print("Hello world") ``` ``` -------------------------------- ### Get Array View with Different Memory Order Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Demonstrates using the `with_config` method to obtain a view of an existing Zarr array with a different memory layout ('F' for Fortran order). ```python import zarr arr = zarr.create_array({}, shape=(10,), dtype='int8', config={"write_empty_chunks": True}) arr_f = arr.with_config({"order": "F"}) print(arr_f.config) ``` -------------------------------- ### Get Complete Array Compression Info Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Inspect the underlying store for detailed compression diagnostics using the .info_complete() method. This can be slow for large arrays. ```python print(z.info_complete()) ``` -------------------------------- ### Access data using oindex (rows and columns) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Demonstrates accessing specific elements by row and column indices using the `oindex` property. This example selects rows 0 and 2, and columns 1 and 3. ```python print(z.oindex[[0, 2], [1, 3]]) # select rows [0, 2] and columns [1, 4] ``` -------------------------------- ### Create Group and Arrays for Zarr Tree Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Sets up a Zarr group containing multiple arrays with different shapes, chunking, and data types. This example demonstrates the hierarchical structure of Zarr stores. ```python root = zarr.create_group('data/example-19.zarr') foo = root.create_array(name='foo', shape=(1000, 100), chunks=(10, 10), dtype='float32') bar = root.create_array(name='bar', shape=(100,), dtype='int32') foo[:, :] = np.random.random((1000, 100)) bar[:] = np.arange(100) print(root.tree()) ``` -------------------------------- ### Create ZipStore Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Use ZipStore to store Zarr hierarchy contents within a single Zip file. This example creates a new Zip file in write mode. ```python store = zarr.storage.ZipStore('data.zip', mode='w') array = zarr.create_array(store=store, shape=(2,), dtype='float64') print(array) ``` -------------------------------- ### Access Remote ObjectStore (S3) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Access remote Zarr data using ObjectStore backed by an S3Store from 'obstore'. This example opens a group in read-only mode. The 'obstore' library and 's3fs' are required. ```python from zarr.storage import ObjectStore from obstore.store import S3Store s3_store = S3Store('noaa-nwm-retro-v2-zarr-pds', skip_signature=True, region="us-west-2") store = zarr.storage.ObjectStore(store=s3_store, read_only=True) group = zarr.open_group(store=store, mode='r') print(group.info) ``` -------------------------------- ### Get NumPy dtype string Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/data_types.md Demonstrates how to get the string representation of a NumPy dtype, which is used as the Zarr V2 identifier for many data types. This example shows a 64-bit integer. ```python import zarr import numpy as np import json store = {} np_dtype = np.dtype('int64') print(np_dtype.str) ``` -------------------------------- ### Create and Populate Array for Block Indexing Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Initializes a Zarr array with specified shape, dtype, and chunk size, then populates it with data. This setup is necessary before performing block indexing operations. ```python import numpy as np import zarr data = np.arange(100).reshape(10, 10) z = zarr.create_array(store='data/example-17.zarr', shape=data.shape, dtype=data.dtype, chunks=(3, 3)) z[:] = data ``` -------------------------------- ### Get data using orthogonal selection (columns) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Selects specific columns from a 2D Zarr array using `get_orthogonal_selection`. This example selects the second and fourth columns. ```python print(z.get_orthogonal_selection((slice(None), [1, 3]))) # select second and fourth columns) ``` -------------------------------- ### Serve Documentation Locally Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Serve a continuously updating version of the documentation locally for development. Access it at http://0.0.0.0:8000/. ```bash hatch --env docs run serve ``` -------------------------------- ### Get data using orthogonal selection (rows) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Selects specific rows from a 2D Zarr array using `get_orthogonal_selection`. This example selects the first and third rows. ```python print(z.get_orthogonal_selection(([0, 2], slice(None)))) # select first and third rows ``` -------------------------------- ### Get data using orthogonal selection (rows and columns) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Selects specific rows and columns from a 2D Zarr array using `get_orthogonal_selection`. This example selects rows 0 and 2, and columns 1 and 3. ```python print(z.get_orthogonal_selection(([0, 2], [1, 3]))) # select rows [0, 2] and columns [1, 4] ``` -------------------------------- ### Build Documentation Locally Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Build the project's documentation locally using Hatch. The output will be available in the 'docs/_build/html' directory. ```bash hatch --env docs run build ``` -------------------------------- ### Install Zarr using Conda Source: https://github.com/zarr-developers/zarr-python/blob/main/README.md Install the Zarr package from the conda-forge channel using Conda. This is an alternative installation method for users of the Anaconda distribution. ```bash conda install -c conda-forge zarr ``` -------------------------------- ### Initialize Zarr Environment Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/quick-start.md Sets up the environment by removing old data, importing necessary libraries, and suppressing specific warnings for Zarr v3 compatibility. ```python import shutil shutil.rmtree('data', ignore_errors=True) import numpy as np from pprint import pprint import io import warnings warnings.filterwarnings( "ignore", message="Numcodecs codecs are not in the Zarr version 3 specification*", category=UserWarning ) np.random.seed(0) ``` -------------------------------- ### Display Zarr CLI Help Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/cli.md Run this command to see all available commands in the Zarr CLI. ```bash zarr --help ``` -------------------------------- ### Demonstrate Cache Effectiveness Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/experimental.md Illustrates the performance benefits of CacheStore by comparing the access times for the first (cache miss) and subsequent (cache hit) reads of array data. Also verifies cache statistics. ```python import numpy as np import time from tempfile import mkdtemp import zarr import zarr.storage from zarr.experimental.cache_store import CacheStore # Create test data with dual-store cache local_store_path = mkdtemp(suffix='.zarr') source_store = zarr.storage.LocalStore(local_store_path) cache_store = zarr.storage.MemoryStore() cached_store = CacheStore( store=source_store, cache_store=cache_store, max_size=256*1024*1024 ) zarr_array = zarr.zeros((100, 100), chunks=(10, 10), dtype='f8', store=cached_store, mode='w') zarr_array[:] = np.random.random((100, 100)) # Demonstrate cache effectiveness with repeated access start = time.time() data = zarr_array[20:30, 20:30] # First access (cache miss) first_access = time.time() - start print(f"First access took {first_access}") start = time.time() data = zarr_array[20:30, 20:30] # Second access (cache hit) second_access = time.time() - start print(f"Second access took {second_access}") # Check cache statistics info = cached_store.cache_info() assert info['cached_keys'] > 0 # Should have cached keys assert info['current_size'] > 0 # Should have cached data print(f"Cache contains {info['cached_keys']} keys with {info['current_size']} bytes") ``` -------------------------------- ### Replace init_group with zarr.open_group Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/v3_migration.md Use zarr.open_group for initializing groups instead of the removed zarr.storage.init_group helper function. ```diff ```diff - from zarr.storage import init_group - init_group(store, overwrite=True, path="my/path") + import zarr + zarr.open_group(store, mode="w", path="my/path") ``` ``` -------------------------------- ### Create ObjectStore with MemoryStore Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Utilize ObjectStore with an in-memory backend from 'obstore' for Zarr data. This is an experimental feature. ```python from zarr.storage import ObjectStore from obstore.store import MemoryStore store = ObjectStore(MemoryStore()) array = zarr.create_array(store=store, shape=(2,), dtype='float64') print(array) ``` -------------------------------- ### Configure CacheStore with Max Size Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/experimental.md Demonstrates setting the maximum size of the cache in bytes. Use `None` for an unlimited cache size, but be cautious about memory usage. ```python # 256MB cache with size limit cache = CacheStore( store=source_store, cache_store=cache_store, max_size=256*1024*1024 ) # Unlimited cache size (use with caution) cache = CacheStore( store=source_store, cache_store=cache_store, max_size=None ) ``` -------------------------------- ### Array Creation and Access Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/api/zarr/api/synchronous.md Demonstrates how to create and access Zarr arrays synchronously. ```APIDOC ## Array Creation and Access ### Description This section covers the synchronous creation of Zarr arrays and how to access their data and metadata. ### Methods - `zarr.open(store, mode='r', **kwargs)`: Open an existing Zarr array or group. - `zarr.create(shape, chunks, dtype, store, overwrite=False, **kwargs)`: Create a new Zarr array. ### Parameters #### `zarr.open` Parameters - **store** (MutableMapping): The storage backend (e.g., a dictionary, a directory path). - **mode** (str): The mode to open the array in ('r', 'r+', 'w', 'a'). Defaults to 'r'. - ****kwargs**: Additional keyword arguments. #### `zarr.create` Parameters - **shape** (tuple): The shape of the array. - **chunks** (tuple): The chunk shape of the array. - **dtype**: The data type of the array elements. - **store** (MutableMapping): The storage backend. - **overwrite** (bool): If True, overwrite existing array. Defaults to False. - ****kwargs**: Additional keyword arguments. ### Examples #### Opening an existing array: ```python import zarr # Assuming 'my_array.zarr' is an existing Zarr array array = zarr.open('my_array.zarr', mode='r') print(array.shape) ``` #### Creating a new array: ```python import zarr import numpy as np shape = (100, 100) chunks = (50, 50) dtype = np.float64 # Create array in memory store = {{}} array = zarr.create(shape, chunks=chunks, dtype=dtype, store=store) # Create array on disk # array = zarr.create(shape, chunks=chunks, dtype=dtype, store='my_new_array.zarr') ``` ``` -------------------------------- ### Create Group and Arrays Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/consolidated_metadata.md Demonstrates creating a Zarr group and several arrays within it using an in-memory store. This sets up the initial structure before consolidating metadata. ```python import zarr import warnings warnings.filterwarnings("ignore", category=UserWarning) group = zarr.create_group(store="memory://consolidated-metadata-demo") print(group) array = group.create_array(shape=(1,), name='a', dtype='float64') print(array) ``` ```python array = group.create_array(shape=(2, 2), name='b', dtype='float64') print(array) ``` ```python array = group.create_array(shape=(3, 3, 3), name='c', dtype='float64') print(array) ``` -------------------------------- ### Get Info for a Specific Group Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/groups.md Retrieve diagnostic information for a specific subgroup by accessing its `info` attribute. ```python print(foo.info) ``` -------------------------------- ### Open Zarr Array from Zip File Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/quick-start.md Demonstrates how to open and read data from a Zarr array stored within a ZipStore in read-only mode. ```python # Open the ZipStore in read-only mode store = zarr.storage.ZipStore("data/example-5.zip", read_only=True) z = zarr.open_array(store, mode='r') # read the data as a NumPy Array print(z[:]) ``` -------------------------------- ### List Available Prek Hooks Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Display a list of all available pre-commit hooks managed by prek. ```bash prek list ``` -------------------------------- ### Create a Basic Zarr Array Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/quick-start.md Demonstrates creating a 2D Zarr array with specified shape, chunking, and data type, then populating it with random data and printing its information. ```python import zarr import numpy as np # Create a 2D Zarr array z = zarr.create_array( store="data/example-1.zarr", shape=(100, 100), chunks=(10, 10), dtype="f4" ) # Assign data to the array z[:, :] = np.random.random((100, 100)) print(z.info) ``` -------------------------------- ### Get Basic Array Info Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/groups.md Access basic diagnostic information for an array, such as its shape and data type, using the `info` attribute. ```python print(baz.info) ``` -------------------------------- ### Open or Create a Group on Disk Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/groups.md Use `zarr.open_group` to open an existing group from a directory or create a new one if it doesn't exist. Specify the path and mode ('w' for write/create). ```python root = zarr.open_group('data/group.zarr', mode='w') print(root) ``` -------------------------------- ### Get Basic Group Info Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/groups.md Access diagnostic information about a group, including its members and their basic properties, via the `info` attribute. ```python root = zarr.group(store="memory://diagnostics-demo") foo = root.create_group('foo') bar = foo.create_array(name='bar', shape=1000000, chunks=100000, dtype='int64') bar[:] = 42 baz = foo.create_array(name='baz', shape=(1000, 1000), chunks=(100, 100), dtype='float32') baz[:] = 4.2 print(root.info) ``` -------------------------------- ### Rendered Advanced Executable Code Block Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md The rendered output of the advanced executable code block example, showing the code and its execution result. ```python print("Hello world") ``` -------------------------------- ### Zarr Array Creation with Blosc Compression Source: https://github.com/zarr-developers/zarr-python/blob/main/bench/compress_normal.txt Demonstrates the creation of a large Zarr array with specific Blosc compression options. This is used to measure initialization and compression overhead. ```python zarr.core.Array((200000000,), uint16, chunks=(1000000,), order=C) compression: blosc; compression_opts: {'clevel': 5, 'cname': 'lz4', 'shuffle': 2} nbytes: 381.5M; nbytes_stored: 294; ratio: 1360544.2; initialized: 0/200 ``` -------------------------------- ### Pickle/Unpickle Zarr Array with Local Store Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/performance.md Demonstrates pickling and unpickling a Zarr array backed by a local store. Ensures the unpickled array is identical to the original. ```python import pickle data = np.arange(100000) z1 = zarr.create_array(store='data/perf-example-2.zarr', shape=data.shape, chunks=data.shape, dtype=data.dtype) z1[:] = data s = pickle.dumps(z1) z2 = pickle.loads(s) assert z1 == z2 print(np.all(z1[:] == z2[:])) ``` -------------------------------- ### Get Default Array Order Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/config.md Retrieve the current value of the 'array.order' configuration setting. This shows the default or currently set array ordering. ```python import zarr print(zarr.config.get('array.order')) ``` -------------------------------- ### Registering a Custom Codec via Entrypoints Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/extending.md Declare custom codecs in your package's pyproject.toml under the [project.entry-points."zarr.codecs"] section. This allows Zarr to automatically discover and load your custom codec. ```toml [project.entry-points."zarr.codecs"] "custompackage.fancy_codec" = "custompackage:FancyCodec" ``` -------------------------------- ### Importing Codecs Directly from numcodecs Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/v3_migration.md In Zarr-Python 3, codecs must be imported directly from the numcodecs library, not from zarr. This example shows the correct import. ```python from numcodecs import Blosc # instead of: # from zarr import Blosc ``` -------------------------------- ### Get Array Compression Info Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Print compression diagnostics for a Zarr array using the .info property. This provides a quick overview of the compressor used. ```python print(z.info) ``` -------------------------------- ### Run Tests in a Specific Hatch Environment Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Execute the unit tests within a specified hatch development environment. This verifies the integrity of the development setup. ```bash hatch env run --env test.py3.12-optional run ``` -------------------------------- ### Store Zarr Array in a Zip File Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/quick-start.md Shows how to create and write a Zarr array to a ZipStore, which packages the array into a single zip file. The store must be explicitly closed after use. ```python # Store the array in a ZIP file store = zarr.storage.ZipStore("data/example-5.zip", mode="w") z = zarr.create_array( store=store, shape=(100, 100), chunks=(10, 10), dtype="f4" ) # write to the array z[:, :] = np.random.random((100, 100)) # the ZipStore must be explicitly closed store.close() ``` -------------------------------- ### Shim 2: `__getattr__` for `zarr.core.metadata.v3` Source: https://github.com/zarr-developers/zarr-python/blob/main/design/chunk-grid.md This shim provides a similar mapping for imports originating from `zarr.core.metadata.v3`, ensuring compatibility for projects like VirtualiZarr that use this import path. ```python # Shim 2: __getattr__ in metadata/v3.py (~12 lines) # Same pattern as Shim 1, but for imports from zarr.core.metadata.v3. ``` -------------------------------- ### Get data using a boolean mask selection Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Retrieves specific elements from a Zarr array using a boolean mask. The mask must have the same shape as the array. ```python sel = np.zeros_like(z, dtype=bool) sel[2] = True sel[5] = True print(z.get_mask_selection(sel)) ``` -------------------------------- ### Get default scalar value for ZDType Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/data_types.md Retrieve the default scalar value for a given Zarr data type (ZDType). This is typically the zero-equivalent for numeric types. ```python default_value = int8.default_scalar() assert default_value == np.int8(0) ``` -------------------------------- ### Get Complete Array Info Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/groups.md Obtain detailed diagnostic information for an array, including its data type, shape, chunking, and storage details, using `info_complete()`. ```python print(bar.info_complete()) ``` -------------------------------- ### Memory Store with Persistent Cache Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/experimental.md Set up a CacheStore with an in-memory primary store and a local directory for persistent caching. This is beneficial for scenarios where data is generated in memory but needs durable caching. ```python from tempfile import mkdtemp from zarr.storage import MemoryStore, LocalStore from zarr.experimental.cache_store import CacheStore memory_store = MemoryStore() local_store_path = mkdtemp(suffix='.zarr') persistent_cache = LocalStore(local_store_path) cached_store = CacheStore( store=memory_store, cache_store=persistent_cache, max_size=256*1024*1024 ) ``` -------------------------------- ### Open Zarr Store and Print Chunk Sizes Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/examples/rectilinear_chunks.ipynb Opens a Zarr store from a given path and prints the chunk sizes. This is useful for verifying that the chunking strategy was maintained during a write operation. ```python roundtrip = xr.open_zarr(output_path, zarr_format=3, consolidated=False) print("Round-trip chunk sizes:", roundtrip.chunks) ``` -------------------------------- ### Write Zarr V3 with Rectilinear Chunks Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/examples/rectilinear_chunks.ipynb Writes a variable-chunked dataset to a local Zarr V3 store with rectilinear chunk grids enabled. Ensure the 'zarr' library is installed. ```python from pathlib import Path import tempfile import zarr import numcodecs import numpy as np # Assume chunk_sizes is defined elsewhere, e.g.: # chunk_sizes = np.array([25, 645, 1510, 2363, 3203, 74, 769, 3963, 4096, 233, 1603, 2450, 4096, 4096, 3327, 4047, 4096, 4096, 1278, 2113, 4096, 3879, 4096, 3842, 2173, 983, 4046, 2187, 4095, 1369, 4096, 4096, 4096, 3515, 1395, 4096, 3622, 4096, 4096, 3875, 4096, 4096, 4096, 2034, 4096, 358, 3991, 4096, 4096, 4096, 2714, 1210, 4096, 4096, 92, 3826, 4096, 2629, 4096, 1438, 4096, 353, 4078, 3410, 2407, 226, 132, 2738, 1223, 23]) # Placeholder for chunk_sizes if not provided if 'chunk_sizes' not in locals(): chunk_sizes = np.array([4096, 4096]) # Example default output_path = Path(tempfile.mkdtemp()) / "healpix_rectilinear.zarr" encoding = { "da": {"chunks": [chunk_sizes.tolist()]}, "cell_ids": {"chunks": [chunk_sizes.tolist()]}, } # Create a dummy array for demonstration if 'da' is not defined if 'da' not in locals(): # Create a dummy array with a shape compatible with chunk_sizes if possible # For this example, let's assume a 2D array where the first dim is covered by chunk_sizes # and the second dim is also covered by chunk_sizes for simplicity. # In a real scenario, 'da' would be your actual data array. array_shape = (len(chunk_sizes), len(chunk_sizes)) # Example shape da = np.random.rand(*array_shape) da.to_zarr(output_path, zarr_format=3, mode="w", encoding=encoding, consolidated=False) print(f"Written to: {output_path}") ``` -------------------------------- ### Access data using oindex (columns) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Demonstrates accessing specific columns of a 2D Zarr array using the `oindex` property. This example selects the second and fourth columns. ```python print(z.oindex[:, [1, 3]]) # select second and fourth columns ``` -------------------------------- ### Set data using orthogonal selection Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Modifies specific elements in a 2D Zarr array using `set_orthogonal_selection`. This example updates elements at specified row and column indices. ```python z.set_orthogonal_selection(([0, 2], [1, 3]), [[-1, -2], [-3, -4]]) ``` -------------------------------- ### Get multidimensional data using a boolean mask Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/arrays.md Retrieves elements from a multidimensional Zarr array using a boolean mask. The mask must match the array's shape. ```python sel = np.zeros_like(z, dtype=bool) sel[0, 1] = True sel[2, 3] = True print(z.get_mask_selection(sel)) ``` -------------------------------- ### Open Group with Implicit LocalStore (String Path) Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/user-guide/storage.md Opens a group using an implicit LocalStore created from a string path. This is a common way to access existing Zarr data on the local filesystem. ```python group = zarr.open_group(store='data/foo/bar') print(group) ``` -------------------------------- ### Validate Zarr Array Metadata with Pydantic Source: https://github.com/zarr-developers/zarr-python/blob/main/packages/zarr-metadata/README.md Demonstrates how to load and validate Zarr v3 array metadata from a JSON file using Pydantic's TypeAdapter. Ensure you have pydantic installed. ```python import json from pydantic import TypeAdapter from zarr_metadata.v3.array import ArrayMetadataV3 with open("zarr.json", "rb") as f: raw = json.load(f) metadata = TypeAdapter(ArrayMetadataV3).validate_python(raw) ``` -------------------------------- ### Run Test Suite with Coverage Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/contributing.md Execute the test suite with coverage reporting and generate an XML report. This is required to ensure 100% test coverage before code acceptance. ```bash hatch env run --env test.py3.12-optional run-coverage ``` -------------------------------- ### zarr.open_like Source: https://github.com/zarr-developers/zarr-python/blob/main/docs/api/zarr/open.md Opens a Zarr array or group, creating a new one if it doesn't exist, based on an existing array or group. ```APIDOC ## zarr.open_like ### Description Opens a Zarr array or group, creating a new one if it doesn't exist, with the same properties as a provided template array or group. ### Parameters #### Path Parameters - **template** (Array or Group) - An existing Zarr array or group to use as a template. - **store** (Store or string) - The store for the new array or group. - **mode** (string, optional) - The mode to open the store in. Defaults to 'w' (write). - **...kwargs** - Additional keyword arguments. ### Example ```python import zarr # Assume 'existing_array' is an opened Zarr array new_array = zarr.open_like(existing_array, 'new_array.zarr', mode='w') ``` ```