LiberTEM

https://libertem.github.io/LiberTEM/
Admin
LiberTEM is an open-source platform for high-throughput distributed processing of large-scale...
Tokens:814,699
Snippets:5,227
Trust Score:9.7
Update:1 month ago
Show doc for...
Tokens:
Raw
### Continuous live preview with setup and cancellation

Source: https://libertem.github.io/LiberTEM-live/examples.html

Full live preview example with setup, cancellation, and timeout handling. Connects to a DECTRIS DCU, creates a LiveContext with BQLive2DPlot, and runs SumUDF with plots. Handles UDFRunCancelled by continuing to the next acquisition.

```python
from libertem.viz.bqp import BQLive2DPlot
from libertem_live.api import LiveContext
from libertem.udf.sum import SumUDF
from libertem.exceptions import UDFRunCancelled

ctx = LiveContext(plot_class=BQLive2DPlot)

# connect to the DECTRIS DCU, and set up a shared memory area:
with ctx.make_connection('dectris').open(
    api_host="127.0.0.1",
    api_port=DCU_API_PORT,
    data_host="127.0.0.1",
    data_port=DCU_DATA_PORT,
    buffer_size=2048,
    bytes_per_frame=64*512,
) as conn:
    while True:
        pending_aq = conn.wait_for_acquisition(timeout=10.0)
        if pending_aq is None:
            continue
        aq = ctx.make_acquisition(
            conn=conn,
            nav_shape=(128, 128),
            frames_per_partition=512,
            pending_aq=pending_aq,
        )
        try:
            res = ctx.run_udf(dataset=aq, udf=SumUDF(), plots=True)
            # do something with the result here
        except UDFRunCancelled:
            # this acquisiiton was cancelled, wait for the next one:
            continue

```

--------------------------------

### Build documentation live with tox -e docs

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Starts the live documentation building process. Requires manual installation of pandoc on the build system. The documentation is served at http://localhost:8008 and rebuilt on changes to documentation source files; changes to Python source code do not trigger a rebuild.

```shell
$ tox -e docs

```

--------------------------------

### Start client development server with npm run start

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Starts a development server that watches source files and rebuilds automatically. Press the 'o' key to open the LiberTEM GUI in the default browser.

```shell
$ npm run start
```

--------------------------------

### Install LiberTEM-blobfinder from PyPI

Source: https://libertem.github.io/LiberTEM-blobfinder/index.html

Quick installation command from PyPI. Installs the package with the 'udf' extra.

```bash
$ pip install libertem-blobfinder[udf]

```

--------------------------------

### Start the GUI with libertem-server

Source: https://libertem.github.io/LiberTEM/usage.html

Start the LiberTEM web interface from the command line after activating the virtualenv or conda environment where LiberTEM is installed. In most situations this automatically launches a webpage in your browser; if not, access it by default at http://localhost:9000.

```bash
(libertem) $ libertem-server

```

--------------------------------

### Install LiberTEM-holo GUI components

Source: https://libertem.github.io/LiberTEM-holo/index.html

Installation command for the GUI components using the gui extra.

```bash
$ pip install libertem-holo[gui]
```

--------------------------------

### Install client dependencies with npm install

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Run from the client directory to install build tools and dependencies. Keeping dependencies current avoids build failures; removing node_modules may be needed in some cases.

```shell
$ cd client/
$ npm install
```

--------------------------------

### Install LiberTEM in a virtual environment

Source: https://libertem.github.io/LiberTEM/index.html

Commands to create a virtual environment and install LiberTEM with optional PyTorch and CuPy support.

```bash
$ virtualenv -p python3 ~/libertem-venv/
$ source ~/libertem-venv/bin/activate
(libertem-venv) $ python -m pip install "libertem[torch]"

# optional for GPU support
# See also https://docs.cupy.dev/en/stable/install.html
(libertem-venv) $ python -m pip install cupy
```

--------------------------------

### Install LiberTEM-holo with GPU support

Source: https://libertem.github.io/LiberTEM-holo/index.html

Installation steps for GPU-accelerated reconstruction, requiring the gpu extra and appropriate CuPy/CUDA toolkit versions.

```bash
$ conda create -n holo python=3.12
$ conda activate holo
$ pip install libertem-holo[gpu] cupy-cuda13x 'cuda-toolkit[all]<14'
```

--------------------------------

### Install and run Jupyter in LiberTEM environment

Source: https://libertem.github.io/LiberTEM/_sources/tips.rst.txt

Install Jupyter into the LiberTEM virtual environment and launch a local notebook that uses that environment.

```shell
(libertem) $ python -m pip install jupyter
```

```shell
(libertem) $ jupyter notebook
```

--------------------------------

### Instantiate AsiTpx3Acquisition

Source: https://libertem.github.io/LiberTEM-live/_modules/libertem_live/detectors/asi_tpx3/acquisition.html

Example showing how to instantiate an AsiTpx3Acquisition using the LiveContext.make_connection and make_acquisition methods.

```python
>>> with ctx.make_connection('asi_tpx3').open(
...     data_port=TPX3_PORT,
... ) as conn:
...     pending_aq = conn.wait_for_acquisition(1.0)
...     aq = ctx.make_acquisition(
...         conn=conn,
...         pending_aq=pending_aq,
...     )
```

--------------------------------

### Quality-guided phase unwrapping

Source: https://libertem.github.io/LiberTEM-holo/_modules/libertem_holo/base/unwrap/quality.html

Unwraps phase guided by a quality map. Includes a usage example.

```python
def quality_unwrap(
    phase: np.ndarray[tuple[int, int]],
    quality: np.ndarray[tuple[int, int]],
) -> np.ndarray[tuple[int, int]]:
    """Unwrap phase guided by a quality map.

    Examples:
    ---------
    >>> import numpy as np
    >>> from libertem_holo.base.unwrap import derivative_variance, quality_unwrap
    >>> rng = np.random.default_rng()
    >>> phase = rng.random(128*128).reshape(128, 128)
    >>> dv = derivative_variance(phase)
    >>> unwrapped = quality_unwrap(phase, dv)

    Parameters:
    -----------
    phase
        Input phase as a 2D ndarray
    quality
        Inverse score quality map for each pixel of the phase:
        lower values are higher quality
    """
    # quality is lowest => best
    assert -np.pi <= phase.min() <= np.pi
    assert -np.pi <= phase.max() <= np.pi
    assert phase.ndim == 2
    assert phase.shape == quality.shape
    img_shape = phase.shape

    # Flat views and results array
    flat_quality = quality.ravel()
    flat_phase = phase.ravel()
    flat_uw_phase = phase.copy().ravel()  # result array
    flat_to_q = np.ones(flat_phase.shape, dtype=np.int8)

    pos = np.argmin(flat_quality)
    heap = [(flat_quality[pos], pos, pos)]
    flat_to_q[pos] = 0  # first position already in q
```

--------------------------------

### Initialize connection and control

Source: https://libertem.github.io/LiberTEM-live/merlin-lowlevel-example.html

Set up the QdConnection and MerlinControl instances.

```python
conn = QdConnection(
    data_host=MERLIN_DATA_SOCKET[0],
    data_port=MERLIN_DATA_SOCKET[1],
    frame_stack_size=16,
    shm_handle_path=socket_path,
    drain=False,
    recovery_strategy="immediate_reconnect",
    huge=False,
)
control = MerlinControl(host=MERLIN_CONTROL_SOCKET[0], port=MERLIN_CONTROL_SOCKET[1])
```

--------------------------------

### Start client with custom API host and port

Source: https://libertem.github.io/LiberTEM/contributing.html

Use environment variables API_HOST and API_PORT to influence where web API requests end up, for example when running the API server on a non-standard port or another host. This command starts the development server with those settings.

```bash
$ API_HOST=127.0.0.1 API_PORT=9001 npm run start

```

--------------------------------

### Run a single test file with pytest

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Run a subset of tests with pytest, for example a single test file. Requires the test requirements to be installed.

```shell
(libertem) $ pytest tests/test_analysis_masks.py
```

--------------------------------

### Start the LiberTEM GUI server

Source: https://libertem.github.io/LiberTEM/_sources/usage.rst.txt

Launch the web-based interface from the command line after activating the appropriate environment.

```shell
(libertem) $ libertem-server
```

--------------------------------

### Run Refinement Example

Source: https://libertem.github.io/LiberTEM-blobfinder/_modules/libertem_blobfinder/udf/refinement.html

Demonstrates how to execute the refinement process using a dataset and the run_refine function.

```python
>>> dataset = ctx.load(
...     filetype="memory",
...     data=np.zeros(shape=(2, 2, 128, 128), dtype=np.float32)
... )
>>> (result, used_indices) = run_refine(
...     ctx, dataset,
...     zero=(64, 64), a=(1, 0), b=(0, 1),
...     match_pattern=libertem_blobfinder.common.patterns.RadialGradient(radius=4),
...     matcher=grm.Matcher()
... )
>>> result['centers'].data  #doctest: +ELLIPSIS
array(...)
```

--------------------------------

### SERDataSet Usage Example (ctx.load)

Source: https://libertem.github.io/LiberTEM/_modules/libertem/io/dataset/ser.html

Loads a SER dataset using `ctx.load('ser', ...)`. The `path` parameter points to the .ser file. In this context, the navigation shape and signal shape can be optionally specified, and `sync_offset` can skip frames from the start or insert blank frames.

```python
>>> ds = ctx.load("ser", path="/path/to/file.ser")  # doctest: +SKIP
```

--------------------------------

### Robust continuous live preview with connection handling

Source: https://libertem.github.io/LiberTEM-live/_sources/examples.rst.txt

A production-ready implementation that includes setup, connection management, timeouts, and UDF cancellation handling.

```python
from libertem.viz.bqp import BQLive2DPlot
from libertem_live.api import LiveContext
from libertem.udf.sum import SumUDF
from libertem.exceptions import UDFRunCancelled

ctx = LiveContext(plot_class=BQLive2DPlot)

# connect to the DECTRIS DCU, and set up a shared memory area:
with ctx.make_connection('dectris').open(
    api_host="127.0.0.1",
    api_port=DCU_API_PORT,
    data_host="127.0.0.1",
    data_port=DCU_DATA_PORT,
    buffer_size=2048,
    bytes_per_frame=64*512,
) as conn:
    while True:
        pending_aq = conn.wait_for_acquisition(timeout=10.0)
        if pending_aq is None:
            continue
        aq = ctx.make_acquisition(
            conn=conn,
            nav_shape=(128, 128),
            frames_per_partition=512,
            pending_aq=pending_aq,
        )
        try:
            res = ctx.run_udf(dataset=aq, udf=SumUDF(), plots=True)
            # do something with the result here
        except UDFRunCancelled:
            # this acquisiiton was cancelled, wait for the next one:
            continue
```

--------------------------------

### SSH port forwarding for LiberTEM with conda or virtualenv

Source: https://libertem.github.io/LiberTEM/_sources/tips.rst.txt

Use SSH port forwarding to access LiberTEM on a remote host via localhost:9000. The first command activates a conda environment before starting the server; the second uses a virtualenv path. Both assume LiberTEM is already installed on the remote host.

```shell
 $ ssh -L 9000:localhost:9000 <remote-hostname> "conda activate libertem; libertem-server"
```

```shell
 $ ssh -L 9000:localhost:9000 <remote-hostname> "/path/to/virtualenv/bin/libertem-server"
```

--------------------------------

### Open detector connection (generic example)

Source: https://libertem.github.io/LiberTEM-live/_sources/usage.rst.txt

Shows the general pattern for opening a connection to a detector system using make_connection().open() with key-value parameters for network hostnames, IP addresses, and ports.

```python
conn = ctx.make_connection('your_detector_type').open(
    key=value,
    ...
)
```

--------------------------------

### Enable ipywidgets notebook extension in Jupyter

Source: https://libertem.github.io/LiberTEM/_sources/tips.rst.txt

Manually activate the ipywidgets notebook extension in classic Jupyter Notebook. Required for examples using ipywidgets, notably BQLive2DPlot. For JupyterHub and possibly JupyterLab, bqplot and bqplot-image-gl must be installed in both the notebook and server environments, which may require admin privileges.

```shell
 $ jupyter nbextension enable --py widgetsnbextension
```

--------------------------------

### Connect to ASI TPX3 and run a SumUDF

Source: https://libertem.github.io/LiberTEM-live/_sources/detectors/asi_tpx3.rst.txt

This example demonstrates how to establish a connection to the ASI TPX3 detector, wait for an acquisition, and execute a SumUDF on the live data stream.

```python
from libertem.viz.bqp import BQLive2DPlot
from libertem_live.api import LiveContext
from libertem.udf.sum import SumUDF

ctx = LiveContext(plot_class=BQLive2DPlot)

with ctx.make_connection('asi_tpx3').open(
    data_host="127.0.0.1",
    data_port=TPX3_DATA_PORT,
    chunks_per_stack=16,
    bytes_per_chunk=1500000,
    buffer_size=2048,  # MiB
) as conn:

    # If the timeout is hit, pending_aq is None.
    # In a real situation, make sure to test for this,
    # for example by looping until a pending acquisition
    pending_aq = conn.wait_for_acquisition(timeout=10.0)

    # prepare for acquisition
    aq = ctx.make_acquisition(
        conn=conn,
        frames_per_partition=4096,
        pending_aq=pending_aq,
    )

    # run one or more UDFs on the live data stream:
    ctx.run_udf(dataset=aq, udf=SumUDF())
```

--------------------------------

### Setup LiveContext and connection

Source: https://libertem.github.io/LiberTEM-live/_sources/usage.rst.txt

Initializes the LiveContext and establishes a connection to the Dectris API.

```python
from libertem_live.api import LiveContext
from libertem.udf.sum import SumUDF

ctx = LiveContext()

conn = ctx.make_connection('dectris').open(
    api_host="127.0.0.1",
    api_port=DCU_API_PORT,
    data_host="127.0.0.1",
    data_port=DCU_DATA_PORT,
)
```

--------------------------------

### Install client dependencies with npm install

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Installs the JavaScript/TypeScript dependencies for the client. Run this before running the client tests.

```shell
$ cd client/
$ npm install

```

--------------------------------

### Initialize camera client

Source: https://libertem.github.io/LiberTEM-live/merlin-lowlevel-example.html

Create the camera client using the shared memory socket path.

```python
cam_client = CamClient(handle_path=socket_path)
```

--------------------------------

### Install LiberTEM-holo with Conda

Source: https://libertem.github.io/LiberTEM-holo/index.html

Standard installation steps for creating a new environment and installing the library via pip.

```bash
$ conda create -n holo python=3.12
$ conda activate holo
$ pip install libertem-holo
```

--------------------------------

### Start Dask scheduler on localhost

Source: https://libertem.github.io/LiberTEM/_sources/deployment/clustercontainer.rst.txt

Starts a Dask scheduler on localhost for a custom cluster. Run this before starting workers.

```shell
(libertem) $ dask-scheduler --host localhost
```

--------------------------------

### Create LiveContext

Source: https://libertem.github.io/LiberTEM-live/_sources/usage.rst.txt

Initializes the main entry point for LiberTEM-live, starting worker processes and preparing them for data reception. Keep the instance around as long as possible to avoid repeated startup overhead.

```python
from libertem_live.api import LiveContext

ctx = LiveContext()
```

--------------------------------

### Install LiberTEM with extra dependencies

Source: https://libertem.github.io/LiberTEM/_sources/dev/setup.rst.txt

Install LiberTEM in editable mode with extra dependencies such as torch and cupy. This works just like installing from PyPI.

```shell
(libertem) $ python -m pip install -e .[torch,cupy]
```

--------------------------------

### MemoryAcquisition usage example

Source: https://libertem.github.io/LiberTEM-live/reference/memory.html

Shows how to create a memory connection, define hooks, create an acquisition, run a SumUDF, and see the output. The example uses numpy random data with shape (23, 42, 51, 67) and prints 'Triggering! (23, 42)' when the acquisition is ready.

```python
>>> import numpy as np
>>> from libertem_live.api import Hooks
...
>>> data = np.random.random((23, 42, 51, 67))
...
>>> class MyHooks(Hooks):
...     def on_ready_for_data(self, env):
...         print(f"Triggering! {env.aq.shape.nav}")
...
>>> conn = ctx.make_connection('memory').open(
...    data=data
... )
...
>>> aq = ctx.make_acquisition(
...     conn=conn,
...     hooks=MyHooks(),
... )
...
>>> udf = SumUDF()
>>> ctx.run_udf(dataset=aq, udf=udf, plots=True)
Triggering! (23, 42)
{'intensity': <BufferWrapper kind=sig dtype=float64 extra_shape=()>}
```

--------------------------------

### Start passive connection

Source: https://libertem.github.io/LiberTEM-live/merlin-lowlevel-example.html

Start the background thread for the connection.

```python
# start the background thread:
conn.start_passive()
```

--------------------------------

### Example metadata output

Source: https://libertem.github.io/LiberTEM-holo/io.html

Example output from printing hologram metadata.

```text
shape: (20, 3838, 3710)
shape of the first image: (3838, 3710)
pixel size: 1.671157330274582e-10 m
total expsure time: 120.0 s

```

--------------------------------

### Merlin camera setup routine (merlin_setup)

Source: https://libertem.github.io/LiberTEM-live/merlin-example.html

Configures a Merlin detector for acquisition, setting dwell time, counter depth, trigger, and file saving options. The simulator accepts all commands but does not respond like a real detector. If save_path is provided, it enables file output with raw format and timestamping disabled.

```python
def merlin_setup(c: MerlinControl, dwell_time=1e-3, depth=6, save_path=None):
    print("Setting Merlin acquisition parameters")
    # Here go commands to control the camera and the rest of the setup
    # to perform an acquisition.

    # The Merlin simulator currently accepts all kinds of commands
    # and doesn't respond like a real Merlin detector.
    c.set('CONTINUOUSRW', 1)
    c.set('ACQUISITIONTIME' , dwell_time * 1e3)  # Time in miliseconds
    c.set('COUNTERDEPTH', depth)

    # Soft trigger for testing
    # For a real STEM acquisition the trigger setup has to be adapted for the given instrument.
    # See the MerlinEM User Manual for more details on trigger setup
    c.set('TRIGGERSTART', 5)

    c.set('RUNHEADLESS', 1)
    c.set('FILEFORMAT', 2)  # 0 binary, 2 raw binary

    if save_path is not None:
        c.set('IMAGESPERFILE', 256)
        c.set('FILEENABLE', 1)
        c.set('USETIMESTAMPING', 0)  # raw format with timestamping is buggy, we need to do it ourselves
        c.set('FILEFORMAT', 2)  # raw format, less overhead?
        c.set('FILEDIRECTORY', save_path)
    else:
        c.set('FILEENABLE', 0)

    print("Finished Merlin setup.")
```

--------------------------------

### Install LiberTEM with CuPy support

Source: https://libertem.github.io/LiberTEM/_sources/install.rst.txt

Install LiberTEM with CuPy support using pip. This requires a build chain and can be time-consuming; the recommended method is to install a precompiled CuPy binary package.

```shell
(libertem-venv) $ python -m pip install "libertem[cupy]"
```

--------------------------------

### Start LiberTEM worker with libertem-worker

Source: https://libertem.github.io/LiberTEM/_sources/deployment/clustercontainer.rst.txt

Starts a LiberTEM worker that connects to the scheduler at tcp://localhost:8786. This is the easiest way to start workers with the appropriate GPU resource tags and environment settings.

```shell
(libertem) $ libertem-worker tcp://localhost:8786
```

--------------------------------

### Run UDF with progress bar and sample output

Source: https://libertem.github.io/LiberTEM/_sources/udf/profiling.rst.txt

Run a UDF with progress=True to display a progress bar that shows per-partition progress. The example output indicates 18 partitions and 4.29 seconds per partition.

```text
res = ctx.run_udf(udf=fit_udf, dataset=ds, roi=roi, progress=True)
> 100%|██████████| 18/18 [01:17<00:00,  4.29s/it]
```

--------------------------------

### Install pre-commit hooks

Source: https://libertem.github.io/LiberTEM/_sources/contributing.rst.txt

Installs the pre-commit hooks so they run automatically before each commit.

```shell
$ pre-commit install --install-hooks

```
Tokens:
Raw