hdf5plugin

https://hdf5plugin.readthedocs.io/en/latest/
Admin
hdf5plugin provides HDF5 compression filters (Blosc, Blosc2, BitShuffle, BZip2, FciDecomp, LZ4,...
Tokens:72,792
Snippets:1,010
Trust Score:9.6
Update:3 months ago
Show doc for...
Tokens:
Raw
### HDF5 Plugin Configuration Example

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/e06bcdf56854b1c0ccf86a948b1e7045/hdf5plugin_EuropeanHUG2022.ipynb

Demonstrates the basic setup and configuration required for HDF5 plugins.

```python
import hdf5plugin
import h5py

# Example usage of hdf5plugin
with h5py.File('example.h5', 'w') as f:
    f.create_dataset('data', data=[1, 2, 3], **hdf5plugin.Blosc(cname='lz4', clevel=5))
```

--------------------------------

### Install notebook requirements

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/e06bcdf56854b1c0ccf86a948b1e7045/hdf5plugin_EuropeanHUG2022.ipynb

Install the necessary dependencies for the demonstration notebook.

```bash
# Notebook requirements
# A recent version of Pillow is required!
# %pip install numpy h5py hdf5plugin h5glance rise jupyterlab matplotlib ipympl Pillow
```

--------------------------------

### HDF5 Plugin Usage Example

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2021.ipynb.txt

Example demonstrating the basic usage of the HDF5 plugin.

```python
import hdf5plugin
import h5py

# Example code snippet from the source
f = h5py.File('example.h5', 'r')
data = f['dataset']
print(data[:])
```

--------------------------------

### Install hdf5plugin from source directory

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/install.rst.txt

Install hdf5plugin from the source directory, allowing for custom build options.

```bash
pip install . --no-binary hdf5plugin
```

--------------------------------

### Initialize plotting environment

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2023/benchmark.html

Setup matplotlib and import necessary libraries for data visualization.

```python
# Initialization
# %matplotlib inline
%matplotlib widget

import numpy
from matplotlib import pyplot as plt

import hdf5plugin
```

--------------------------------

### Install hdf5plugin from source

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/install.rst.txt

Install hdf5plugin from source to enable compilation optimizations specific to your host machine. The [--user] flag is optional.

```bash
pip install hdf5plugin --no-binary hdf5plugin [--user]
```

--------------------------------

### HDF5 Plugin Data Access Example

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2022.ipynb.txt

Example demonstrating how to access and process data using the HDF5 plugin.

```python
import h5py
import hdf5plugin

# Open an HDF5 file with the plugin
with h5py.File('data.h5', 'r') as f:
    dataset = f['dataset_name']
    data = dataset[:]
```

--------------------------------

### Install hdf5plugin on Debian/Ubuntu

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2023/presentation.html

Installation command for Debian 12 and Ubuntu 23.04.

```bash
apt-get install python3-hdf5plugin
```

--------------------------------

### Benchmark HDF5 Compression

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2023/benchmark.ipynb.txt

Example of setting up and running a benchmark for HDF5 compression plugins.

```python
import h5py
import hdf5plugin
import numpy as np
import time

# Create a dummy dataset
data = np.random.rand(1000, 1000)

# Write with a plugin
with h5py.File('test.h5', 'w') as f:
    f.create_dataset('data', data=data, **hdf5plugin.Blosc(cname='lz4', clevel=5))
```

--------------------------------

### Example HDF5_PLUGIN_PATH output

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2022.html

Displays the expected output after setting the HDF5_PLUGIN_PATH and listing the plugin directory contents.

```text
HDF5_PLUGIN_PATH=/venv/ub20.04/lib/python3.8/site-packages/hdf5plugin/plugins
libh5blosc.so
libh5bshuf.so
libh5fcidecomp.so
libh5lz4.so
libh5zfp.so
libh5zstd.so
```

--------------------------------

### Install hdf5plugin using pip

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/install.rst.txt

Use this command to install the pre-built hdf5plugin package from PyPI. The [--user] flag is optional.

```bash
pip install hdf5plugin [--user]
```

--------------------------------

### Install hdf5plugin via bash

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2022.html

Use pip to install the hdf5plugin package.

```bash
pip3 install hdf5plugin
```

--------------------------------

### Setup conda environment for testing

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/contribute.rst.txt

Create and activate a dedicated conda environment for development and testing.

```bash
conda create -n hdf5plugins_env python=3.XX
conda activate hdf5plugins_env
```

--------------------------------

### Install hdf5plugin and dependencies

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/3d56356ea95a767c3e4274beb79341e4/hdf5plugin_EuropeanHUG2021.ipynb

Install the necessary packages including numpy, h5py, and hdf5plugin via pip.

```bash
%pip install numpy h5py hdf5plugin h5glance rise jupyterlab matplotlib ipympl Pillow
```

--------------------------------

### Configure compression filters

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2023/presentation.ipynb.txt

Example of applying specific compression filters provided by the plugin when creating a dataset.

```python
import h5py
import hdf5plugin

with h5py.File("output.h5", "w") as f:
    f.create_dataset("data", data=data, **hdf5plugin.Blosc(cname='zstd', clevel=5))
```

--------------------------------

### Benchmark HDF5 Plugin

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/594fcc4a4f6d40f9213ff2f85f815f91/benchmark.ipynb

Example code for running benchmarks on the HDF5 plugin.

```python
import h5py
import hdf5plugin
import time

# Example benchmark code
with h5py.File('data.h5', 'w') as f:
    f.create_dataset('data', data=range(1000000), **hdf5plugin.Blosc(cname='lz4', clevel=5))
```

--------------------------------

### Write Compressed Data

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2022.ipynb.txt

Example of creating a dataset with a specific compression filter provided by the plugin.

```python
import h5py
import hdf5plugin
import numpy as np

data = np.arange(1000)
with h5py.File('test.h5', 'w') as f:
    f.create_dataset('data', data=data, **hdf5plugin.Blosc(cname='zstd', clevel=5))
```

--------------------------------

### Get help for hdf5plugin filters

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2022.html

Access documentation for specific compression filters.

```python
help(hdf5plugin.Blosc)
```

--------------------------------

### Install test requirements

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/contribute.rst.txt

Install necessary build and test dependencies via conda and pip.

```bash
conda install compilers
pip install py-cpuinfo setuptools
```

--------------------------------

### Install hdf5plugin

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2023/benchmark.html

Commands to install the hdf5plugin package via pip or conda.

```bash
pip install hdf5plugin or conda install -c conda-forge hdf5plugin
```

```bash
pip install hdf5plugin --no-binary hdf5plugin
```

--------------------------------

### Compressing data with HDF5 plugins

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/e06bcdf56854b1c0ccf86a948b1e7045/hdf5plugin_EuropeanHUG2022.ipynb

Example demonstrating how to apply compression filters to HDF5 datasets using the hdf5plugin library.

```python
import h5py
import hdf5plugin

with h5py.File('example.h5', 'w') as f:
    f.create_dataset('data', data=data, **hdf5plugin.Blosc(cname='lz4', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE))
```

--------------------------------

### Reading compressed HDF5 data

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/e06bcdf56854b1c0ccf86a948b1e7045/hdf5plugin_EuropeanHUG2022.ipynb

Example showing how to read datasets that have been compressed with HDF5 plugins.

```python
import h5py
import hdf5plugin

with h5py.File('example.h5', 'r') as f:
    data = f['data'][:]
```

--------------------------------

### Register HDF5 Plugin

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/e06bcdf56854b1c0ccf86a948b1e7045/hdf5plugin_EuropeanHUG2022.ipynb

Example of how to register a custom HDF5 plugin for data compression.

```python
import hdf5plugin
import h5py

# Register the plugin
hdf5plugin.register_plugin()

# Use the plugin with h5py
with h5py.File('data.h5', 'w') as f:
    f.create_dataset('data', data=data, compression='gzip')
```

--------------------------------

### Get help for Blosc2

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2023/presentation.html

Display help information for the Blosc2 compression filter.

```python
help(hdf5plugin.Blosc2)
```

--------------------------------

### Write Compressed Data with HDF5

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2021.ipynb.txt

Example of creating an HDF5 file and writing a dataset using a specific compression filter provided by the plugin.

```python
import h5py
import hdf5plugin
import numpy as np

data = np.arange(100)
with h5py.File('example.h5', 'w') as f:
    f.create_dataset('data', data=data, **hdf5plugin.Blosc(cname='lz4', clevel=5))
```

--------------------------------

### Write compressed data with Blosc

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2021.ipynb.txt

Example of creating a dataset using the Blosc compression filter.

```python
import h5py
import hdf5plugin
import numpy as np

data = np.arange(100)
with h5py.File('test.h5', 'w') as f:
    f.create_dataset('data', data=data, **hdf5plugin.Blosc())
```

--------------------------------

### Use hdf5plugin with h5py

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2021.ipynb.txt

Example of using hdf5plugin with h5py to write compressed data.

```python
import h5py
import hdf5plugin
import numpy as np

f = h5py.File('test.h5', 'w')
f.create_dataset('data', data=np.arange(100), **hdf5plugin.Blosc(cname='lz4', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE))
f.close()
```

--------------------------------

### Use hdf5plugin with h5py

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2022.ipynb.txt

Example of reading an HDF5 file using h5py with hdf5plugin registered.

```python
import h5py
import hdf5plugin

f = h5py.File('data.h5', 'r')
data = f['dataset']
print(data[:])
```

--------------------------------

### Retrieve HDF5 Plugin Path via Command Line

Source: https://hdf5plugin.readthedocs.io/en/latest/usage.html

Use this command to print the directory path where the HDF5 plugins are installed.

```bash
python -c "import hdf5plugin; print(hdf5plugin.PLUGIN_PATH)"
```

--------------------------------

### Write compressed data with h5py

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/3d56356ea95a767c3e4274beb79341e4/hdf5plugin_EuropeanHUG2021.ipynb

Example of creating an HDF5 file and writing a dataset using the Blosc compression filter provided by hdf5plugin.

```python
import h5py
import hdf5plugin
import numpy as np

f = h5py.File('test.h5', 'w')
data = np.arange(100)
f.create_dataset('data', data=data, **hdf5plugin.Blosc(cname='lz4', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE))
f.close()
```

--------------------------------

### Write compressed HDF5 file

Source: https://hdf5plugin.readthedocs.io/en/latest/_downloads/3d56356ea95a767c3e4274beb79341e4/hdf5plugin_EuropeanHUG2021.ipynb

Example of creating an HDF5 file using the Blosc compression filter provided by hdf5plugin.

```python
import h5py
import hdf5plugin
import numpy

f = h5py.File('test.h5', 'w')
f.create_dataset('data', data=numpy.arange(100), **hdf5plugin.Blosc(cname='lz4', clevel=5, shuffle=hdf5plugin.Blosc.SHUFFLE))
f.close()
```

--------------------------------

### Install hdf5plugin with OpenMP disabled

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/install.rst.txt

Disable OpenMP compilation when installing hdf5plugin by setting the HDF5PLUGIN_OPENMP environment variable to False. This can be done either globally before the pip install command or when installing from the source directory.

```bash
HDF5PLUGIN_OPENMP=False pip install hdf5plugin --no-binary hdf5plugin
```

```bash
HDF5PLUGIN_OPENMP=False pip install .
```

--------------------------------

### Build the project

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/contribute.rst.txt

Compile the project from the source directory.

```bash
python setup.py build
```

--------------------------------

### Build documentation

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/contribute.rst.txt

Build the HTML documentation using Sphinx.

```bash
python setup.py build
PYTHONPATH=build/lib.<os>-<machine>-<pyver>/ sphinx-build -b html doc/ build/html
```

--------------------------------

### Install hdf5plugin using conda

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/install.rst.txt

Use this command to install the hdf5plugin package from the conda-forge channel.

```bash
conda install -c conda-forge hdf5plugin
```

--------------------------------

### Download Demo Dataset

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2023/hdf5_compressed_chunk_direct_read.ipynb.txt

Downloads the sample HDF5 file to shared memory.

```bash
# Download dataset
!wget -O /dev/shm/kevlar.h5 http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5
```

--------------------------------

### Import and List Available Plugins

Source: https://hdf5plugin.readthedocs.io/en/latest/_sources/hdf5plugin_EuropeanHUG2022.ipynb.txt

Import the hdf5plugin module to register filters and list the available compression plugins.

```python
import hdf5plugin
import h5py

print(hdf5plugin.filters.keys())
```

--------------------------------

### Download benchmark dataset

Source: https://hdf5plugin.readthedocs.io/en/latest/hdf5plugin_EuropeanHUG2023/benchmark.html

Use wget to download the required HDF5 dataset to the local environment.

```bash
!wget -O /dev/shm/kevlar.h5 http://www.silx.org/pub/pyFAI/pyFAI_UM_2020/data_ID13/kevlar.h5
```

--------------------------------

### Basic Compression and Decompression

Source: https://hdf5plugin.readthedocs.io/en/latest/usage.html

Demonstrates the standard workflow for creating a compressed dataset using LZ4 and subsequently reading it back.

```python
import numpy
import h5py
import hdf5plugin

# Compression
f = h5py.File('test.h5', 'w')
f.create_dataset('data', data=numpy.arange(100), compression=hdf5plugin.LZ4())
f.close()

# Decompression
f = h5py.File('test.h5', 'r')
data = f['data'][()]
f.close()
```
Tokens:
Raw