### Array Construction Examples Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Array.Array-element_pointer-shape-order.html Examples demonstrating array creation with C-style and Fortran-style layout orders. ```cpp int data[6] = {1, 2, 3, 4, 5, 6}; auto c_array = Array(&data[0], {2, 3}); EXPECT_EQ(MakeArray({{1, 2, 3}, {4, 5, 6}}), array); auto f_array = Array(&data[0], {3, 2}, fortran_order); EXPECT_EQ(MakeArray({{1, 4}, {2, 5}, {3, 6}}), array); ``` -------------------------------- ### Relative Path Example Source: https://google.github.io/tensorstore/driver/image/jpeg/index.html Example demonstrating the use of the 'path' member for specifying an additional path relative to the kvstore. ```string "path/to/data" ``` -------------------------------- ### Zip Driver Example JSON Source: https://google.github.io/tensorstore/kvstore/zip/index.html An example of how to specify the zip driver in a JSON configuration. ```APIDOC ## Example JSON specifications ```json { "driver": "zip", "kvstore": "gs://my-bucket/path/to/file.zip" } ``` ``` -------------------------------- ### Define base_url examples Source: https://google.github.io/tensorstore/kvstore/http/index.html Examples of valid base_url strings for the HTTP driver. ```json "https://example.com" ``` ```json "https://example.com:8000/path/to/data" ``` ```json "https://example.com:8000/path/to/data?query=value" ``` -------------------------------- ### Install TensorStore Python API from local source Source: https://google.github.io/tensorstore/installation.html Installs TensorStore from a local git repository checkout. This command invokes Bazel to build the C++ extension module. Ensure build prerequisites are installed. ```bash git clone https://github.com/google/tensorstore cd tensorstore python3 -m pip install . ``` -------------------------------- ### Install TensorStore Python API from PyPI on Windows Source: https://google.github.io/tensorstore/installation.html Alternative command for installing the TensorStore Python API from PyPI on Windows systems. ```bash py -3 -m pip install tensorstore -vv ``` -------------------------------- ### Example of HalfOpenInterval Usage in C++ Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.HalfOpenInterval.html Demonstrates the effect of HalfOpenInterval on dimension selection and input domain. The example shows how specifying start, stop, and strides transforms the domain and equivalent input indices. ```cpp Dims(0, 2).HalfOpenInterval({1, 8}, {4, 3}, {1, -2}) ``` -------------------------------- ### Constructing IndexDomains Source: https://google.github.io/tensorstore/python/api/tensorstore.IndexDomain.__init__-components.html Examples showing how to initialize an IndexDomain by specifying rank or shape. ```python >>> ts.IndexDomain(rank=5) { (-inf*, +inf*), (-inf*, +inf*), (-inf*, +inf*), (-inf*, +inf*), (-inf*, +inf*) } >>> ts.IndexDomain(shape=[2, 3]) { [0, 2), [0, 3) } ``` -------------------------------- ### N5 URL representation examples Source: https://google.github.io/tensorstore/driver/n5/index.html Examples showing the mapping between URL strings and JSON configuration objects for N5 datasets. ```json {"driver": "n5", "kvstore": {"driver": "file", "path": "/tmp/dataset.n5/"} } ``` ```json {"driver": "n5", "kvstore": {"driver": "file", "path": "/tmp/dataset.n5/path/within/hierarchy/"} } ``` -------------------------------- ### Install Build Dependencies Source: https://google.github.io/tensorstore/installation.html Install required system packages on Debian-based systems. ```bash sudo apt-get install build-essential git nasm python3 python3-dev python3-pip python3-venv ``` -------------------------------- ### Configure Zlib Compressor Source: https://google.github.io/tensorstore/driver/zarr/index.html Example configuration for Zlib or Gzip compression. ```json {"id": "gzip", "level": 9} ``` -------------------------------- ### S3 Bucket Endpoint Example Source: https://google.github.io/tensorstore/kvstore/s3/index.html Example of an S3 bucket URL, potentially including a virtual-hosted-style endpoint. ```json "https://mybucket.cwobject.com" ``` -------------------------------- ### Example: MurmurHash3 Configuration Source: https://google.github.io/tensorstore/kvstore/neuroglancer_uint64_sharded/index.html Example JSON specification for opening the neuroglancer_uint64_sharded driver with murmurhash3_x86_128 hash and a 1GB cache pool. ```APIDOC ### Example: Opening with murmurhash3_x86_128 hash and 1GB cache ```json { "driver": "neuroglancer_uint64_sharded", "kvstore": "gs://my-bucket/path/to/sharded/data/", "metadata": { "@type": "neuroglancer_uint64_sharded_v1", "hash": "murmurhash3_x86_128", "preshift_bits": 0, "minishard_bits": 3, "shard_bits": 3, "data_encoding": "raw", "minishard_index_encoding": "gzip" }, "context": { "cache_pool": {"total_bytes_limit": 1000000000} } } ``` ``` -------------------------------- ### Example: Identity Hash Configuration Source: https://google.github.io/tensorstore/kvstore/neuroglancer_uint64_sharded/index.html Example JSON specification for opening the neuroglancer_uint64_sharded driver with an identity hash and a 1GB cache pool. ```APIDOC ## Example JSON specifications ### Example: Opening with identity hash and 1GB cache ```json { "driver": "neuroglancer_uint64_sharded", "kvstore": "gs://my-bucket/path/to/sharded/data/", "metadata": { "@type": "neuroglancer_uint64_sharded_v1", "hash": "identity", "preshift_bits": 1, "minishard_bits": 3, "shard_bits": 3, "data_encoding": "raw", "minishard_index_encoding": "gzip" }, "context": { "cache_pool": {"total_bytes_limit": 1000000000} } } ``` ``` -------------------------------- ### Install TensorStore Python API from PyPI Source: https://google.github.io/tensorstore/installation.html Installs the latest published version of the TensorStore Python API using pip. Use the -vv option to show progress. This is the recommended method for most users. ```bash python3 -m pip install tensorstore -vv ``` -------------------------------- ### DimExpression::Stride Usage Examples Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.Stride.html Examples of how to invoke the Stride method using dimension selection and stride vectors. ```cpp Dims(0, 2).Stride({-2, 3}) ``` ```cpp Stride({1, 2, 3}) ``` ```cpp Stride(5) ``` -------------------------------- ### ChainResult Example with Result-wrapped and Plain Values Source: https://google.github.io/tensorstore/cpp/api/tensorstore.ChainResult.html Demonstrates using ChainResult with an initial value that is already a Result, and another example with a plain initial value. Assumes the existence of functions func1, func2, and func3 with appropriate signatures. ```cpp float func1(int x); Result func2(float x); bool func3(std::string_view x); Result y1 = ChainResult(Result(3), func1, func2, func3); Result y2 = ChainResult(3, func1, func2, func3); ``` -------------------------------- ### Downsample Driver Schema Example Source: https://google.github.io/tensorstore/driver/downsample/index.html An example of a TensorStore schema configuration using the `downsample` driver. This includes specifying downsample factors, the downsample method, and the base TensorStore configuration. ```json { "driver": "downsample", "downsample_factors": [1, 2, 2], "downsample_method": "mean", "base": {"driver": "zarr", "kvstore": {"driver": "gcs", "bucket": "my-bucket"}} } ``` -------------------------------- ### Set up Python virtual environment and dependencies Source: https://google.github.io/tensorstore/installation.html Creates a Python virtual environment and installs necessary dependencies like pip, setuptools, and numpy for building TensorStore from source. ```bash python3 -m venv ts-venv source ts-venv/bin/activate python3 -m pip install --upgrade pip setuptools numpy ``` -------------------------------- ### Get BoxView Origin in C++ Source: https://google.github.io/tensorstore/cpp/api/tensorstore.BoxView.origin.html Retrieves the origin array of length `rank()` from a `BoxView`. This is useful for understanding the starting coordinates of the view. ```cpp #include "tensorstore/box.h" span tensorstore::BoxView::origin() const; ``` -------------------------------- ### Build TensorStore Documentation Source: https://google.github.io/tensorstore/installation.html Run the documentation build tool using bazelisk. ```bash python3 bazelisk.py run //tools/docs:build_docs -- --output /tmp/tensorstore-docs ``` -------------------------------- ### Get TensorStore Spec Transform Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Spec.transform.html Retrieves the transform applied on top of the driver for a TensorStore specification. No setup or imports are required beyond including the necessary header. ```cpp #include "tensorstore/spec.h" const IndexTransform<>& tensorstore::Spec::transform() const; ``` -------------------------------- ### Get Index Expression with Inclusive and Exclusive Bounds Source: https://google.github.io/tensorstore/python/api/tensorstore.IndexDomain.index_exp.html Generates an index expression from inclusive minimum and exclusive maximum bounds, resulting in slices with specified start and end points. ```python >>> ts.IndexDomain(inclusive_min=[1, 2], exclusive_max=[5, 10]).index_exp (slice(1, 5, None), slice(2, 10, None)) ``` -------------------------------- ### Get Output Rank of Transform Source: https://google.github.io/tensorstore/cpp/api/tensorstore.OutputIndexMapRange.size.html Use this method to retrieve the output rank of the transform represented by `OutputIndexMapRange`. No specific setup or imports are required beyond including the necessary header. ```cpp #include "tensorstore/index_space/output_index_map.h" StaticOrDynamicRank tensorstore::OutputIndexMapRange::size() const; ``` -------------------------------- ### Open and Use Memory KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.html Demonstrates opening a memory-based KvStore, writing a key-value pair, reading it back, and listing available keys. Use this for in-memory storage or testing. ```python store = await ts.KvStore.open({'driver': 'memory'}) await store.write(b'a', b'value') KvStore.TimestampedStorageGeneration(...) await store.read(b'a') KvStore.ReadResult(state='value', value=b'value', stamp=KvStore.TimestampedStorageGeneration(...)) await store.read(b'b') KvStore.ReadResult(state='missing', value=b'', stamp=KvStore.TimestampedStorageGeneration(...)) await store.list() [b'a'] ``` -------------------------------- ### Open KvStore with Path Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.path.html Demonstrates opening a KvStore with a specified path prefix using the 'gcs' driver. The path is stored and can be retrieved later. ```python >>> store = await ts.KvStore.open({ ... 'driver': 'gcs', ... 'bucket': 'my-bucket', ... 'path': 'path/to/object' ... }) >>> store.spec() KvStore.Spec({'bucket': 'my-bucket', 'driver': 'gcs', 'path': 'path/to/object'}) >>> store.path 'path/to/object' ``` -------------------------------- ### TranslateBoxSlice Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.TranslateBoxSlice.html Example demonstrating the effect of TranslateBoxSlice on dimension selection. ```cpp Dims(0, 2).TranslateBoxSlice(BoxView({1, 4}, {3, 4})) ``` -------------------------------- ### Read and List Keys in KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.read.html Demonstrates reading values from a memory-backed store, handling missing keys, and listing keys. ```python >>> store = await ts.KvStore.open({'driver': 'memory'}) >>> await store.write(b'a', b'value') KvStore.TimestampedStorageGeneration(...) >>> await store.read(b'a') KvStore.ReadResult(state='value', value=b'value', stamp=KvStore.TimestampedStorageGeneration(...)) >>> store[b'a'] b'value' >>> await store.read(b'b') KvStore.ReadResult(state='missing', value=b'', stamp=KvStore.TimestampedStorageGeneration(...)) >>> store[b'b'] Traceback (most recent call last): ... KeyError... ``` ```python >>> store[b'a'] = b'value' >>> store[b'b'] = b'value' >>> store.list().result() ``` -------------------------------- ### SizedInterval Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.SizedInterval.html Example demonstrating the effect of SizedInterval on dimension selection. ```cpp Dims(0, 2).SizedInterval({1, 8}, {3, 2}, {1, -2}) ``` -------------------------------- ### KvStore Constructor with Driver Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.KvStore.KvStore-driver.html Constructs a KvStore using only a driver. This is a basic initialization. ```cpp tensorstore::kvstore::KvStore::KvStore(DriverPtr driver); ``` -------------------------------- ### Corresponding IndexDomain Example Source: https://google.github.io/tensorstore/driver/zarr3/index.html The IndexDomain representation corresponding to the example Zarr v3 metadata. ```json { "exclusive_max": [[1000], [2000], [3000]], "inclusive_min": [0, 0, 0], "labels": ["x", "y", "z"] } ``` -------------------------------- ### Configure and Access Context Resources in C++ Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Context.html Demonstrates creating a context from a JSON specification, retrieving a resource, and executing tasks using the resource's executor. ```cpp auto spec_result = Context::Spec::FromJson({ {"data_copy_concurrency", {{"limit", 4}}}, {"data_copy_concurrency#a", {{"limit", 8}}}, }); if (!spec_result) { // Handle error } auto resource_spec_result = Context::Resource ::FromJson("data_copy_concurrency"); if (!resource_spec_result) { // Handle error } auto context = Context(*spec_result); auto resource_result = context.GetResource( resource_spec_result->value()); if (!resource_result) { // Handle error } // Can access executor member of context resource. (*resource_result)->executor([] { // task body }); // Access other executor. auto resource2 = context.GetResource( Context::Resource ::FromJson("data_copy_concurrency#a").value()).value(); resource2->executor([] { /* task body */ }); ``` -------------------------------- ### StridedLayout Example Usage Source: https://google.github.io/tensorstore/cpp/api/tensorstore.StridedLayout.html Demonstrates how to create and use the StridedLayout class with different configurations. ```APIDOC ## Example Usage ```cpp tensorstore::StridedLayout<2> x(tensorstore::c_order, sizeof(int), {3, 4}); EXPECT_EQ(3, x.rank()); // Indexing with `operator()` requires all dimensions be specified. EXPECT_EQ((4 + 2) * sizeof(int), x(1, 2)); EXPECT_EQ((4 + 2) * sizeof(int), x({1, 2})); // Indexing with `operator[]` supports partial indexing. EXPECT_EQ(4 * sizeof(int), x[1]); EXPECT_EQ((4 + 2) * sizeof(int), x[{1, 2}]); // Create a StridedLayoutView that refers to x. tensorstore::StridedLayoutView<2> x_ref = x; ``` ``` -------------------------------- ### All Unlabeled Dimensions Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.IndexDomain.operator-call-transform.html Example demonstrating slicing when both transform and domain have only unlabeled dimensions. ```text * transform: `[0, 5), [1, 7)` * domain: `[2, 4), [3, 6)` * result: `[2, 4), [3, 6)` ``` -------------------------------- ### Define File KvStore via JSON Source: https://google.github.io/tensorstore/kvstore/file/index.html Examples of mapping file URLs to their corresponding JSON configuration objects. ```json {"driver": "file", "path": "/tmp/dataset/"} ``` ```json {"driver": "file", "path": "C:/Users/abc/dataset/"} ``` -------------------------------- ### Downsample Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Downsample-Spec.html Example of calling the Downsample function with a braced list for factors. ```cpp Downsample(base_spec, {2, 3}, DownsampleMethod::kMean) ``` -------------------------------- ### MarkBoundsExplicit Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.MarkBoundsExplicit.html An example demonstrating the effect of calling MarkBoundsExplicit on selected dimensions. ```cpp Dims(0, 2).MarkBoundsExplicit() ``` -------------------------------- ### Construct a Batch View Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Batch.View.View-no_batch.html Includes the necessary header and demonstrates the default and no-batch constructors for Batch::View. ```cpp #include "tensorstore/batch.h" ``` ```cpp constexpr tensorstore::Batch::View::View(); ``` ```cpp constexpr tensorstore::Batch::View::View(no_batch_t); ``` -------------------------------- ### Initialize and index a StridedLayout Source: https://google.github.io/tensorstore/cpp/api/tensorstore.StridedLayout.html Demonstrates creating a 2D StridedLayout with C-order, calculating byte offsets, and creating a view. ```cpp tensorstore::StridedLayout<2> x(tensorstore::c_order, sizeof(int), {3, 4}); EXPECT_EQ(3, x.rank()); // Indexing with `operator()` requires all dimensions be specified. EXPECT_EQ((4 + 2) * sizeof(int), x(1, 2)); EXPECT_EQ((4 + 2) * sizeof(int), x({1, 2})); // Indexing with `operator[]` supports partial indexing. EXPECT_EQ(4 * sizeof(int), x[1]); EXPECT_EQ((4 + 2) * sizeof(int), x[{1, 2}]); // Create a StridedLayoutView that refers to x. tensorstore::StridedLayoutView<2> x_ref = x; ``` -------------------------------- ### ClosedInterval Usage Examples Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.ClosedInterval.html Examples demonstrating the effect of ClosedInterval on dimension selection and domain transformation. ```cpp Dims(0, 2).ClosedInterval({1, 8}, {4, 3}, {1, -2}) ``` ```cpp Dims(0, 2).ClosedInterval({1, 9}, {4, 3}, {1, -2}) ``` -------------------------------- ### Open KvStore with Spec and Options Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.Open.html Opens a KvStore using a pre-parsed `kvstore::Spec` object and an `OpenOptions` object. This is useful when the specification is already available in a structured format. ```cpp #include "tensorstore/kvstore/kvstore.h" Future tensorstore::kvstore::Open(Spec spec, OpenOptions&& options); ``` -------------------------------- ### DimSelection Initialization and Examples Source: https://google.github.io/tensorstore/python/api/tensorstore.DimSelection.html Demonstrates how to construct DimSelection objects using various indexing methods with ts.d. ```APIDOC ## DimSelection Construction Examples ### Description Examples of creating `DimSelection` objects using `tensorstore.d` with different indexing specifications. ### Examples ```python >>> import tensorstore as ts # Select dimensions by index >>> ts.d[0, 1, 2] d[0,1,2] # Select dimensions using slices and labels >>> ts.d[0:1, 2, "x"] d[0:1,2,'x'] # Select dimensions using lists of indices >>> ts.d[[0, 1], [2]] d[0,1,2] # Combine list indexing with dimension selection >>> ts.d[[0, 1], ts.d[2, 3]] d[0,1,2,3] ``` ``` -------------------------------- ### Writing and Deleting Keys in KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.write.html Demonstrates opening a memory-backed store, writing a value, reading it back, and deleting the key by writing None. ```python >>> store = await ts.KvStore.open({'driver': 'memory'}) >>> await store.write(b'a', b'value') KvStore.TimestampedStorageGeneration(...) >>> await store.read(b'a') KvStore.ReadResult(state='value', value=b'value', stamp=KvStore.TimestampedStorageGeneration(...)) >>> await store.write(b'a', None) KvStore.TimestampedStorageGeneration(...) >>> await store.read(b'a') KvStore.ReadResult(state='missing', value=b'', stamp=KvStore.TimestampedStorageGeneration(...)) ``` -------------------------------- ### MapFutureValue Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.MapFutureValue.html Example demonstrating how to combine two integer futures using an inline executor. ```cpp Future future_a = ...; Future future_b = ...; Future mapped_future = MapFutureValue( InlineExecutor{}, [](int a, int b) { return a + b; }, future_a, future_b); ``` -------------------------------- ### Create and Index TensorStore Dataset with vindex Source: https://google.github.io/tensorstore/python/api/tensorstore.TensorStore.vindex.html Demonstrates creating a Zarr dataset in memory and using the vindex method with array indexing for a virtual view. Ensure the TensorStore library is imported as 'ts'. ```python >>> dataset = await ts.open( ... { ... 'driver': 'zarr', ... 'kvstore': { ... 'driver': 'memory' ... } ... }, ... dtype=ts.uint32, ... shape=[60, 70, 80], ... create=True) >>> view = dataset.vindex[:, [5, 10, 20], [7, 8, 10]] >>> view TensorStore({ 'context': { 'cache_pool': {}, 'data_copy_concurrency': {}, 'memory_key_value_store': {}, }, 'driver': 'zarr', 'dtype': 'uint32', 'kvstore': {'driver': 'memory'}, 'metadata': { 'chunks': [60, 70, 80], 'compressor': { 'blocksize': 0, 'clevel': 5, 'cname': 'lz4', 'id': 'blosc', 'shuffle': -1, }, 'dimension_separator': '.', 'dtype': '>> import tensorstore as ts >>> import numpy as np >>> dataset = ts.open({ ... 'driver': 'n5', ... 'kvstore': { ... 'driver': 'file', ... 'path': 'tmp/dataset/', ... }, ... 'metadata': { ... 'compression': { ... 'type': 'gzip' ... }, ... 'dataType': 'uint32', ... 'dimensions': [1000, 20000], ... 'blockSize': [100, 100], ... }, ... 'create': True, ... 'delete_existing': True, ... }).result() ``` -------------------------------- ### StaticRankCast Usage Examples Source: https://google.github.io/tensorstore/cpp/api/tensorstore.StaticRankCast.html Examples demonstrating checked and unchecked static rank casting for different TensorStore types. ```cpp StaticRank<3> r_static = StaticRankCast<3, unchecked>(DimensionIndex(3)); Box<> box = ...; Result> checked_result = StaticRankCast<3>(box); Box<3> unchecked_result = StaticRankCast<3, unchecked>(box); ``` -------------------------------- ### TranslateBackwardBy Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.TranslateBackwardBy.html Example demonstrating the effect of shifting dimensions 0 and 2 by offsets 10 and 20. ```cpp Dims(0, 2).TranslateBackwardBy({10, 20}) ``` -------------------------------- ### Construct KVStore Spec with Driver Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.Spec.Spec-driver.html Constructs a KVStore Spec from a driver with an empty path. Include the necessary header for Spec and DriverSpecPtr. ```cpp #include "tensorstore/kvstore/spec.h" tensorstore::kvstore::Spec::Spec(DriverSpecPtr driver); ``` -------------------------------- ### Transpose Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.Transpose-target_dimensions.html Example showing the effect of transposing dimensions 2 and 0 to indices 1 and 2. ```cpp Dims(2, 0).Transpose({1, 2}) ``` -------------------------------- ### OuterIndexArraySlice Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.OuterIndexArraySlice.html Example demonstrating the effect of slicing dimensions 2 and 0 with specific index arrays. ```cpp Dims(2, 0).OuterIndexArraySlice(MakeArray({{2, 3}, {4, 5}}), MakeArray({6, 7})) ``` -------------------------------- ### Open KvStore with Spec and Variadic Options Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.Open.html Opens a KvStore using a `kvstore::Spec` and a variadic sequence of options. This overload allows specifying options directly as arguments, offering flexibility in configuration. ```cpp template requires IsCompatibleOptionSequence static Future tensorstore::kvstore::Open(Spec spec, Option&&... option); ``` -------------------------------- ### List keys in a memory-backed KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.list.html Opens an in-memory key-value store, adds two key-value pairs, and then lists all keys. This demonstrates basic usage of KvStore.list(). ```python >>> store = ts.KvStore.open({'driver': 'memory'}).result() >>> store[b'a'] = b'value' >>> store[b'b'] = b'value' >>> store.list().result() [b'a', b'b'] ``` -------------------------------- ### Build pre-built extension module and create wheel Source: https://google.github.io/tensorstore/installation.html Skips building the extension module directly and instead uses a pre-built module. First, build the extension, then use pip to create a wheel package. ```bash python3 setup.py build_ext -b /tmp/prebuilt TENSORSTORE_PREBUILT_DIR=/tmp/prebuilt pip wheel . ``` -------------------------------- ### Zarr v3 Metadata Example Source: https://google.github.io/tensorstore/driver/zarr3/index.html Example Zarr v3 metadata structure used to define an array's properties. ```json { "zarr_format": 3, "node_type": "array", "shape": [1000, 2000, 3000], "dimension_names": ["x", "y", "z"], "chunk_grid": {"name": "regular", "configuration": {"chunk_shape": [100, 200, 300]}}, "chunk_key_encoding": {"name": "default"}, "data_type": "uint16", "codecs": [{"name": "bytes", "configuration": {"endian": "little"}}], "fill_value": 0 } ``` -------------------------------- ### KvStore Open Options Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.OpenOptions.html The `OpenOptions` struct contains driver-agnostic options that can be specified when opening a `KvStore`. Refer to the `kvstore::Open` documentation for detailed usage. ```APIDOC ## KvStore Open Options ### Description Driver-agnostic options that may be specified when opening a `KvStore`. ### Method N/A (Struct definition) ### Endpoint N/A (Struct definition) ### Parameters N/A (Struct definition) ### Request Example N/A (Struct definition) ### Response N/A (Struct definition) ``` -------------------------------- ### Open KvStore from URL Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.open.html Use this when opening a key-value store directly from a URL string. Requires asynchronous execution. ```python kvstore = await ts.KvStore.open('memory://abc/') kvstore.spec() ``` -------------------------------- ### Mixed Labeled and Unlabeled Dimensions Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.IndexDomain.operator-call-transform.html Example demonstrating slicing with a mix of labeled and unlabeled dimensions in both transform and domain. ```text * transform: `"x": [0, 10), "": [0, 10), "": [0, 10), "y": [0, 10)` * domain: `"y": [1, 6), "": [2, 7), "x": [3, 8), "": [4, 9)` * result: `"x": [3, 8), "": [2, 7), "": [4, 9), "y": [1, 6)` ``` -------------------------------- ### TranslateBy Usage Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.TranslateBy.html A practical example showing the effect of shifting dimensions 0 and 2 by offsets of 10 and 20. ```cpp Dims(0, 2).TranslateBy({10, 20}) ``` -------------------------------- ### Retrieve KvStore string representation Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.__repr__.html Demonstrates opening a file-based KvStore and viewing its string representation. ```python >>> kvstore = await ts.KvStore.open({ ... 'driver': 'file', ... 'path': 'tmp/data/' ... }) >>> kvstore KvStore({ 'context': { 'file_io_concurrency': {}, 'file_io_locking': {}, 'file_io_mode': {}, 'file_io_sync': True, }, 'driver': 'file', 'path': 'tmp/data/', }) ``` -------------------------------- ### Define Inclusive Start Index Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimRangeSpec.inclusive_start.html Specifies the inclusive start index for a dimension range, defaulting based on the step direction. ```cpp std::optional tensorstore::DimRangeSpec::inclusive_start; ``` -------------------------------- ### Verifying Unit parsing behavior Source: https://google.github.io/tensorstore/cpp/api/tensorstore.Unit.Unit-string.html Examples demonstrating how various string inputs are parsed into multiplier and base unit components. ```cpp EXPECT_EQ(Unit(4, "nm"), Unit("4nm")); EXPECT_EQ(Unit(4e-3, "nm"), Unit("4e-3nm")); EXPECT_EQ(Unit(4e-3, "nm"), Unit("+4e-3nm")); EXPECT_EQ(Unit(-4e-3, "nm"), Unit("-4e-3nm")); EXPECT_EQ(Unit(4.5, "nm"), Unit("4.5nm")); EXPECT_EQ(Unit(1, "nm"), Unit("nm")); EXPECT_EQ(Unit(4, ""), Unit("4")); EXPECT_EQ(Unit(1, ""), Unit("")); ``` -------------------------------- ### S3 Server Endpoint Example Source: https://google.github.io/tensorstore/kvstore/s3/index.html Example of an S3 server endpoint URL. This is used when not connecting to the public Amazon S3 endpoints. ```json "http://localhost:1234" ``` -------------------------------- ### Create and Index a TensorStore Dataset Source: https://google.github.io/tensorstore/python/api/tensorstore.TensorStore.__getitem__-indices.html Opens a TensorStore dataset using Zarr driver and memory kvstore, then creates a virtual view using NumPy-style indexing. Data is not read until `read` is called on the view. ```python >>> dataset = await ts.open( ... { ... 'driver': 'zarr', ... 'kvstore': { ... 'driver': 'memory' ... } ... }, ... dtype=ts.uint32, ... shape=[70, 80], ... create=True) >>> view = dataset[[5, 10, 20], 6:10] >>> view ``` -------------------------------- ### Construct KvStore.Spec from JSON or URL Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.Spec.__new__.html Create a specification object by providing either a dictionary representing the driver configuration or a URL string. ```python >>> spec = ts.KvStore.Spec({'driver': 'file', 'path': 'tmp/data/'}) >>> spec KvStore.Spec({'driver': 'file', 'path': 'tmp/data/'}) ``` ```python >>> spec = ts.KvStore.Spec('file:///path/to/data/') >>> spec KvStore.Spec({'driver': 'file', 'path': '/path/to/data/'}) ``` -------------------------------- ### Get IndexDomain Shape Source: https://google.github.io/tensorstore/python/api/tensorstore.IndexDomain.shape.html Access the shape attribute of an IndexDomain object to get its dimensions. This is useful for understanding the size and structure of the domain. ```python >>> domain = ts.IndexDomain(inclusive_min=[1, 2, 3], shape=[3, 4, 5]) >>> domain.shape (3, 4, 5) ``` -------------------------------- ### Accessing TensorStore KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.TensorStore.kvstore.html Demonstrates opening a TensorStore with a memory-backed kvstore and accessing the kvstore property. ```python >>> store = await ts.open( ... { ... 'driver': 'n5', ... 'kvstore': { ... 'driver': 'memory', ... 'path': 'abc/', ... } ... }, ... create=True, ... shape=[100, 200], ... dtype=ts.uint32, ... ) >>> store.kvstore KvStore({'context': {'memory_key_value_store': {}}, 'driver': 'memory', 'path': 'abc/'}) ``` -------------------------------- ### Select Interval with Slice Source: https://google.github.io/tensorstore/python/indexing.html Use a slice object `start:stop` to select an interval. The `start` value is inclusive, and the `stop` value is exclusive. ```python >>> a = ts.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9], dtype=ts.int32) >>> a[1:5] ``` -------------------------------- ### Start DimExpression with a dimension range Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.html Initiates a DimExpression with a specified range of dimensions. Supports inclusive start, optional exclusive stop, and a step value. ```cpp auto tensorstore::DimRange( DimensionIndex inclusive_start, std::optional exclusive_stop = std::nullopt, const DimensionIndex step = 1); ``` -------------------------------- ### Create and Use a Transaction Source: https://google.github.io/tensorstore/python/api/tensorstore.Transaction.html Demonstrates creating a transaction, opening a dataset, performing writes within the transaction, and observing changes with transactional and non-transactional reads. ```python >>> txn = ts.Transaction() >>> store = ts.open({ ... 'driver': 'n5', ... 'kvstore': { ... 'driver': 'file', ... 'path': 'tmp/dataset/' ... }, ... 'metadata': { ... 'dataType': 'uint16', ... 'blockSize': [2, 3], ... 'dimensions': [5, 6], ... 'compression': { ... 'type': 'raw' ... } ... }, ... 'create': True, ... 'delete_existing': True ... }).result() >>> store.with_transaction(txn)[1:4, 2:5] = 42 >>> store.with_transaction(txn)[0:2, 4] = 43 ``` ```python >>> store.with_transaction(txn).read().result() array([[ 0, 0, 0, 0, 43, 0], [ 0, 0, 42, 42, 43, 0], [ 0, 0, 42, 42, 42, 0], [ 0, 0, 42, 42, 42, 0], [ 0, 0, 0, 0, 0, 0]], dtype=uint16) ``` ```python >>> store.read().result() array([[0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0]], dtype=uint16) ``` -------------------------------- ### Get TensorStore IndexTransform Input Shape Source: https://google.github.io/tensorstore/python/api/tensorstore.IndexTransform.input_shape.html Use the `input_shape` property to get the shape of the input domain for an IndexTransform. This is an alias for the `shape` property of the `domain`. ```python transform = ts.IndexTransform(input_shape=[3, 4, 5]) transform.input_shape ``` -------------------------------- ### Open or Create a TensorStore Source: https://google.github.io/tensorstore/python/api/tensorstore.open.html Initializes a new TensorStore using the Zarr driver in memory with specified shape, dtype, and chunk layout. ```python >>> store = await ts.open( ... { ... 'driver': 'zarr', ... 'kvstore': { ... 'driver': 'memory' ... } ... }, ... create=True, ... dtype=ts.int32, ... shape=[1000, 2000, 3000], ... chunk_layout=ts.ChunkLayout(inner_order=[2, 1, 0]), ... ) >>> store TensorStore({ 'context': { 'cache_pool': {}, 'data_copy_concurrency': {}, 'memory_key_value_store': {}, }, 'driver': 'zarr', 'dtype': 'int32', 'kvstore': {'driver': 'memory'}, 'metadata': { 'chunks': [101, 101, 101], 'compressor': { 'blocksize': 0, 'clevel': 5, 'cname': 'lz4', 'id': 'blosc', 'shuffle': -1, }, 'dimension_separator': '.', 'dtype': '>> transform = ts.IndexTransform(input_shape=[3, 4, 5], ... input_labels=["x", "y", "z"]) >>> transform.input_rank 3 ``` -------------------------------- ### Accessing and updating Schema codec Source: https://google.github.io/tensorstore/python/api/tensorstore.Schema.codec.html Demonstrates initializing a schema, checking the default codec, and updating it with specific Zarr driver configurations. ```python >>> schema = ts.Schema() >>> print(schema.codec) None >>> schema.update(codec=ts.CodecSpec({ ... 'driver': 'zarr', ... 'compressor': None ... })) >>> schema.update(codec=ts.CodecSpec({'driver': 'zarr', 'filters': None})) >>> schema.codec CodecSpec({'compressor': None, 'driver': 'zarr', 'filters': None}) ``` -------------------------------- ### Initialize a tiff TensorStore via JSON Source: https://google.github.io/tensorstore/driver/image/tiff/index.html Example configuration for opening a TIFF file stored in a Google Cloud Storage bucket. ```json {"driver": "tiff", "kvstore": "gs://my-bucket/path-to-image.tiff"} ``` -------------------------------- ### S3 Host Header Override Example Source: https://google.github.io/tensorstore/kvstore/s3/index.html Example of overriding the HTTP host header for S3 requests. This is useful for testing with local services like LocalStack. ```json "mybucket.s3.af-south-1.localstack.localhost.com" ``` -------------------------------- ### Unsafe DimExpression Construction Example Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.html This example demonstrates an unsafe way to construct a DimExpression by storing it in a variable that outlives the temporary values it references. Avoid this pattern. ```c++ auto do_not_use = Dims(std::string("x"), std::string("y")).IndexSlice({1, 2}); ``` -------------------------------- ### Open and Access KvStore Base Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.base.html Demonstrates opening an 'ocdbt' driver key-value store with a 'memory://' base and accessing its base property. This is useful for understanding the structure of adapter key-value stores. ```python >>> store = await ts.KvStore.open({ ... 'driver': 'ocdbt', ... 'base': 'memory://' ... }) >>> store.base KvStore({'context': {'memory_key_value_store': {}}, 'driver': 'memory'}) ``` -------------------------------- ### Get KvStore URL Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.url.html Access the `url` property of an opened KvStore object to get its string representation. This is useful for identifying the store's location and configuration. ```python store = await ts.KvStore.open({ 'driver': 'gcs', 'bucket': 'my-bucket', 'path': 'path/to/object' }) store.url ``` -------------------------------- ### Synchronously read a key from KvStore Source: https://google.github.io/tensorstore/python/api/tensorstore.KvStore.__getitem__.html Demonstrates opening a memory-backed KvStore, setting a value, and reading it back. Note that accessing a non-existent key raises a KeyError. ```python >>> store = ts.KvStore.open({'driver': 'memory'}).result() >>> store[b'a'] = b'value' >>> store[b'a'] b'value' >>> store[b'b'] Traceback (most recent call last): ... KeyError... ``` -------------------------------- ### KvStore Constructor with Driver, Path, and Transaction Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.KvStore.KvStore-driver.html Constructs a KvStore with a driver, an optional path, and an optional transaction. The path specifies a location within the driver, and the transaction can be omitted. ```cpp explicit tensorstore::kvstore::KvStore::KvStore( DriverPtr driver, std::string path, Transaction transaction = no_transaction ); ``` -------------------------------- ### TranslateTo Example Usage Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.TranslateTo.html Demonstrates how to use the TranslateTo method with a specific origin vector. This example shifts dimensions 0 and 2 to origins 10 and 20 respectively. ```cpp Dims(0, 2).TranslateTo({10, 20}) ``` -------------------------------- ### Create a new N5 TensorStore with schema constraints Source: https://google.github.io/tensorstore/python/api/tensorstore.open.html Demonstrates applying schema constraints to an N5 driver, showing how storage characteristics are specified independently of the format. ```python >>> store = await ts.open( ... { ... 'driver': 'n5', ... 'kvstore': { ... 'driver': 'memory' ... }, ... }, ... create=True, ... dtype=ts.float32, ... shape=[1000, 2000, 3000], ... chunk_layout=ts.ChunkLayout( ... chunk_shape=[10, None, None], ... chunk_aspect_ratio=[None, 2, 1], ... chunk_elements=10000000, ... ), ... ) >>> store ``` -------------------------------- ### TensorStore IndexArraySlice Example Usage Source: https://google.github.io/tensorstore/cpp/api/tensorstore.DimExpression.IndexArraySlice.html Example demonstrating the use of IndexArraySlice with multiple index arrays for slicing dimensions. Ensure index arrays have compatible shapes and ranks. ```cpp Dims(0, 2).IndexArraySlice(MakeArray({{1, 2, 3}, {4, 5, 6}}), MakeArray({{7, 8, 9}, {0, 1, 2}})) ``` -------------------------------- ### Reorder all selected dimensions to start at a positive index Source: https://google.github.io/tensorstore/python/api/tensorstore.DimExpression.transpose.html A single non-negative target index reorders all selected dimensions to start at that index. For `k > 1` dimensions, `i` is equivalent to `i:i+k`. ```python >>> transform = ts.IndexTransform(input_shape=[2, 3, 4, 5], ... input_labels=["a", "b", "c", "d"]) >>> transform[ts.d['a', 'd'].transpose[1]] Rank 4 -> 4 index space transform: Input domain: 0: [0, 3) "b" 1: [0, 2) "a" 2: [0, 5) "d" 3: [0, 4) "c" Output index maps: out[0] = 0 + 1 * in[1] out[1] = 0 + 1 * in[0] out[2] = 0 + 1 * in[3] out[3] = 0 + 1 * in[2] ``` -------------------------------- ### Retrieve context resources using __getitem__ Source: https://google.github.io/tensorstore/python/api/tensorstore.Context.__getitem__.html Demonstrates accessing context resources by key, including specific resource IDs and general resource types. ```python >>> context = ts.Context( ... {'cache_pool#a': { ... 'total_bytes_limit': 10000000 ... }}) >>> context['cache_pool#a'] Context.Resource({'total_bytes_limit': 10000000}) >>> context['cache_pool'] Context.Resource({}) ``` -------------------------------- ### Get exclusive_max of a Dimension Source: https://google.github.io/tensorstore/python/api/tensorstore.Dim.exclusive_max.html Access the `exclusive_max` property of a `ts.Dim` object to get the exclusive upper bound of the interval. This is equal to `inclusive_max + 1` or a special value for unbounded intervals. ```python >>> ts.Dim(inclusive_min=5, inclusive_max=10).exclusive_max 11 ``` ```python >>> ts.Dim(exclusive_max=5).exclusive_max 5 ``` ```python >>> ts.Dim().exclusive_max 4611686018427387904 ``` -------------------------------- ### Run TensorStore Tests Source: https://google.github.io/tensorstore/installation.html Execute the test suite using bazelisk. ```bash python3 bazelisk.py test //... ``` -------------------------------- ### Get KvStore Spec with Options Source: https://google.github.io/tensorstore/cpp/api/tensorstore.kvstore.KvStore.spec.html Use this function to get a Spec for re-opening a KvStore. Options like ContextBindingMode can be specified to control how context resources are handled in the returned Spec. ```cpp #include "tensorstore/kvstore/kvstore.h" template requires IsCompatibleOptionSequence Result tensorstore::kvstore::KvStore::spec(Option&&... option) const; ``` ```cpp Result tensorstore::kvstore::KvStore::spec( SpecRequestOptions&& options) const; ``` -------------------------------- ### Create and Use a Promise/Future Pair Source: https://google.github.io/tensorstore/python/api/tensorstore.Promise.html Demonstrates the basic usage of creating a Promise and Future pair, setting a result, and retrieving it. Ensure the Promise is used to set the result or exception. ```python >>> promise, future = ts.Promise.new() >>> future.done() False >>> promise.set_result(5) >>> future.done() True >>> future.result() 5 ``` -------------------------------- ### Get Input Shape of IndexTransform Source: https://google.github.io/tensorstore/cpp/api/tensorstore.IndexTransform.input_shape.html Use the `input_shape()` method on a valid `IndexTransform` object to get a span of extents for its input dimensions. The transform must be valid before calling this method. ```cpp #include "tensorstore/index_space/index_transform.h" span tensorstore:: IndexTransform::input_shape() const; ```