### Install EXtra-data using pip

Source: https://github.com/european-xfel/extra-data/blob/master/README.md

Install the extra_data package from PyPI for use in other environments with Python 3.

```bash
pip install extra_data
```

--------------------------------

### Get Run Overview with .info()

Source: https://github.com/european-xfel/extra-data/blob/master/docs/inspection.ipynb

Use the `.info()` method on a RunDirectory object to get a summary of the data, including the number of trains, duration, detector modules, instrument sources, and control sources.

```python
from extra_data import RunDirectory

run = RunDirectory("/gpfs/exfel/exp/XMPL/201750/p700000/raw/r0010")
run.info()
```

--------------------------------

### Install EXtra-data from PyPI

Source: https://github.com/european-xfel/extra-data/blob/master/docs/index.md

Install the EXtra-data package from PyPI for use in other Python 3 environments. Additional dependencies can be installed using extras like [bridge] or [complete].

```bash
pip install extra_data
```

```bash
pip install "extra_data[bridge]"
```

```bash
pip install "extra_data[complete]"
```

--------------------------------

### Import necessary libraries

Source: https://github.com/european-xfel/extra-data/blob/master/docs/parallel_example.ipynb

Imports required for the parallel processing example.

```python
from getpass import getuser
import h5py
import subprocess
```

--------------------------------

### Install extra_data with bridge dependencies

Source: https://github.com/european-xfel/extra-data/blob/master/docs/streaming.md

Install the extra_data package with optional dependencies required for streaming data via Karabo Bridge. This command ensures all necessary components are available.

```shell
pip install extra_data[bridge]
```

--------------------------------

### Getting Data by Source and Key

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Illustrates how to select a source and then retrieve specific keys from it. This method is useful for exploring available keys within a source.

```default
xgm = run['SPB_XTD9_XGM/DOOCS/MAIN']
xgm.keys()
beam_x = xgm['beamPosition.ixPos']
```

--------------------------------

### Get Run Information with H5tool

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Use the h5tool to get verbose information about a run directory, including details about trains, detectors, and scanning parameters.

```bash
h5tool --info -verbose $dir
```

--------------------------------

### Inspect Run Contents with lsxfel CLI

Source: https://context7.com/european-xfel/extra-data/llms.txt

Provides command-line examples for using `lsxfel` to inspect proposal directories, run directories, or HDF5 files. Use this for quick checks of data availability, source details, and data counts without writing Python code.

```shell
# Inspect a full proposal directory
lsxfel /gpfs/exfel/exp/XMPL/201750/p700000

# Inspect a run directory
lsxfel /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002

# Filter to sources matching a substring or glob
lsxfel /path/to/run -s BAM
lsxfel /path/to/run -s '*/BAM*:output'

# Show detailed keys and shapes for selected sources
lsxfel /path/to/run --detail '*XGM*'

# Show data counts per train
lsxfel /path/to/run --counts

# Show which data aggregator saves each source
lsxfel /path/to/run --aggregators
```

--------------------------------

### Open a Data Run and Show Info

Source: https://github.com/european-xfel/extra-data/blob/master/docs/iterate_trains.ipynb

Opens a data run for a specific proposal and run number. Use this to get an overview of the data before processing.

```python
from extra_data import open_run

run = open_run(proposal=700000, run=2)
run.info()  # Show overview info about this data
```

--------------------------------

### Install EXtra-data on Maxwell Cluster

Source: https://github.com/european-xfel/extra-data/blob/master/docs/index.md

Load the exfel and exfel-python modules to use EXtra-data in the pre-configured Python environment on the Maxwell cluster.

```bash
module load exfel exfel-python
```

--------------------------------

### Get HDF5 virtual dataset

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Create an HDF5 virtual dataset for a given source and key. Data is loaded on-demand. The dataset can be written to a specified file path or a temporary file.

```python
ds = run.get_virtual_dataset(source='source_name', key='key_name', filename='path/to/virtual_dataset.h5')
```

--------------------------------

### Get available data keys

Source: https://github.com/european-xfel/extra-data/blob/master/docs/xpd_examples2.ipynb

Retrieve the keys for available data from a specific instrument source. This helps identify the datasets you can access.

```python
sa1_data['SA1_XTD2_XGM/XGM/DOOCS:output'].keys()
```

--------------------------------

### Access AGIPD1M Detector Data

Source: https://context7.com/european-xfel/extra-data/llms.txt

Provides examples for interacting with the AGIPD1M detector, a 16-module detector. It shows how to load pixel data as labeled xarray, select pulse ranges, use Dask for lazy loading, and access corrected data with masks applied. Use this for detailed analysis of AGIPD1M data.

```python
from extra_data import open_run
from extra_data.components import AGIPD1M, LPD1M, DSSC1M, JUNGFRAU, identify_multimod_detectors

run = open_run(700000, 1, data='proc')

# AGIPD-1M: 16 modules, each 512×128 pixels
agipd = AGIPD1M(run)
# or: agipd = AGIPD1M(run, detector_name='SPB_DET_AGIPD1M-1', modules={0,1,2}, min_modules=8)

# Load pixel data as labelled xarray (trains × pulses × modules × slow-px × fast-px)
arr = agipd['image.data'].ndarray()     # shape (n_trains, n_pulses, 16, 512, 128)
xarr = agipd.get_array('image.data')   # same but with pulse-ID coordinates

# Select pulse range (first 10 pulses)
arr10 = agipd.get_array('image.data', pulses=np.s_[:10])

# Lazy Dask array for large data
da = agipd.get_dask_array('image.data')

# Corrected data with mask applied (bad pixels → NaN)
masked = agipd.masked_data()
arr_m = masked.ndarray()

# Train-by-train iteration
for tid, data in agipd.trains(pulses=np.s_[:32], require_all=True):
    images = data['image.data']   # shape (16, 32, 512, 128)

# Check which modules have data per train
avail = agipd.data_availability()   # bool array (16, n_trains)

# Write frames to a new HDF5 file
agipd.write_frames('selected.h5',
    trains=np.array([142844490, 142844491]),
    pulses=np.array([0, 0])
)

# Write virtual CXI file for downstream analysis tools (e.g. Cheetah, CrystFEL)
agipd.write_virtual_cxi('agipd_run1.cxi')

# Auto-detect which multi-module detector is in a run
det_name, det_class = identify_multimod_detectors(run, single=True)
det = det_class(run, detector_name=det_name)

# Stack modules from a train dict (lower-level helper)
from extra_data import stack_detector_data
_, train_data = run.select('*/DET/*', 'image.data').train_from_index(0)
stacked = stack_detector_data(train_data, 'image.data', modules=16)
# stacked.shape → (16, n_pulses, 512, 128)
```

--------------------------------

### Load and Select Data Using Aliases

Source: https://context7.com/european-xfel/extra-data/llms.txt

Demonstrates loading run data with aliases defined in a YAML file and selecting specific data streams based on these aliases. Use this when your data sources are organized with logical names.

```python
run_aliased2 = run.with_aliases('/path/to/aliases.yml')

# Select using alias
sel = run_aliased.alias.select('xgm', 'beamPosition.*')

# Only keep aliased sources
sel2 = run.only_aliases({'xgm': 'SA1_XTD2_XGM/XGM/DOOCS'}, require_all=True)

# Drop all aliases from the collection
clean = run_aliased.drop_aliases()
```

--------------------------------

### Open Run and Display Info

Source: https://github.com/european-xfel/extra-data/blob/master/docs/xpd_examples.ipynb

Opens a specified run for a given proposal and displays overview information about the data contained within the run. This is a prerequisite for further data analysis.

```python
run = open_run(proposal=900027, run=67)
run.info()  # Show overview info about the data
```

--------------------------------

### get_dask_array

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Gets a Dask array for a data field, optionally labelled.

```APIDOC
## get_dask_array(source, key, labelled=False)

### Description
Get a Dask array for a data field defined by source and key.
see [`KeyData.dask_array()`](#extra_data.KeyData.dask_array) for details.
```

--------------------------------

### Loading Data as NumPy Array

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Demonstrates how to load data into a NumPy array. The 'roi' parameter can be used to specify a region of interest for partial loading.

```default
ndarray(roi=(), out=None)
```

--------------------------------

### data_availability

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Get an array indicating what image data is available for each module and train.

```APIDOC
## data_availability(module_gaps=False)

### Description
Get an array indicating what image data is available.

Returns a boolean array (modules, entries), True where a module has data
for a given train, False for missing data.

### Parameters
#### Query Parameters
- **module_gaps** (bool) - If True, indicates gaps in module data.
```

--------------------------------

### Stream Data from Files with Karabo Bridge

Source: https://github.com/european-xfel/extra-data/blob/master/docs/cli.md

Use `karabo-bridge-serve-files` to stream data from specified files in the Karabo bridge format. This command is recommended for streaming from files, while `karabo-bridge-serve-run` is preferred for entire run directories. Filter sources and keys using `--source` and `--key` respectively.

```shell
karabo-bridge-serve-files /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0005 4321
```

--------------------------------

### Get Number of Pulses

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Returns the total number of pulses in the run data.

```python
size_pulses()
```

--------------------------------

### Open a run from a filesystem path

Source: https://context7.com/european-xfel/extra-data/llms.txt

Open a run directly from a directory path containing HDF5 files, useful when the standard proposal directory layout is not available. Provides access to run metadata and train timestamps.

```python
from extra_data import RunDirectory

run = RunDirectory("/gpfs/exfel/exp/XMPL/201750/p700000/raw/r0001")

# Inspect metadata
meta = run.run_metadata()
# {'proposalNumber': '700000', 'runNumber': '1', 'creationDate': '2017-...', ...}

# Get timestamps for each train
import numpy as np
timestamps = run.train_timestamps()         # numpy array of datetime64
ts_labelled = run.train_timestamps(labelled=True)  # pandas Series indexed by train ID
ts_local = run.train_timestamps(labelled=True, euxfel_local_time=True)  # Europe/Berlin TZ

print(f"Run has {len(run.train_ids)} trains")
print(f"Start: {timestamps[0]}, End: {timestamps[-1]}")
```

--------------------------------

### Get Number of Trains

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Returns the total number of trains in the run data.

```python
size_trains()
```

--------------------------------

### Get Number of Indices

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Returns the total number of indices available in the run data.

```python
size_index()
```

--------------------------------

### Get Size Information

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Retrieve size-related information for indices, train IDs, or pulse IDs.

```python
size
```

--------------------------------

### Open a Run and Load Data

Source: https://github.com/european-xfel/extra-data/blob/master/docs/index.md

Open a specific run using its proposal and run number, then load data for a given source and key as a NumPy array.

```python
from extra_data import open_run

run = open_run(proposal=700000, run=1)
```

```python
arr = run["SA3_XTD10_PES/ADC/1:network", "digitizers.channel_4_A.raw.samples"].ndarray()
```

--------------------------------

### Create virtual CXI detector files with extra-data-make-virtual-cxi

Source: https://context7.com/european-xfel/extra-data/llms.txt

Build virtual CXI files from multi-module detector data for compatibility with tools like CrystFEL. Allows specifying output file, minimum modules, fill values, and excluding suspect trains.

```shell
# Basic usage: write to proposal scratch directory (default)
extra-data-make-virtual-cxi /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0003
```

```shell
# Specify output file
extra-data-make-virtual-cxi /path/to/run -o my_run.cxi
```

```shell
# Require at least 12 of 16 modules
extra-data-make-virtual-cxi /path/to/run --min-modules 12
```

```shell
# Set fill value for missing data
extra-data-make-virtual-cxi /path/to/run --fill-value data 0 --fill-value mask 0
```

```shell
# Exclude suspect train IDs
extra-data-make-virtual-cxi /path/to/run --exc-suspect-trains
```

--------------------------------

### Get pandas DataFrame from DataCollection

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Retrieve a pandas DataFrame for specified data fields. Timestamps can be excluded.

```python
df = run.get_dataframe(fields=[
    ("*_XGM/*", "*.i[xy]Pos"),
    ("*_XGM/*", "*.photonFlux")
])
```

--------------------------------

### get_data_counts(source, key)

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Get the count of data points for a specific source and key across all trains.

```APIDOC
## get_data_counts(source, key)

Get a count of data points in each train for the given data field.

Returns a pandas series with an index of train IDs.

* **Parameters:**
  * **source** (*str*) – Source name, e.g. “SPB_DET_AGIPD1M-1/DET/7CH0:xtdf”
  * **key** (*str*) – Key of parameter within that device, e.g. “image.data”.
```

--------------------------------

### select_trains with by_id

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Example of selecting trains excluding a specific train ID that might cause issues.

```APIDOC
## select_trains(by_id[:2**64-1])

### Description
Selects all trains except for the one with the maximum possible ID (2^64 - 1), which can cause issues with methods like `info()`.

### Parameters
#### Path Parameters
- **by_id** (slice) - Represents a slice of train IDs. `[:2**64-1]` selects all IDs up to, but not including, the maximum possible value.

### Usage
```python
from extra_data import by_id
sel = run.select_trains(by_id[:2**64-1])
```
```

--------------------------------

### Stream Run Data with Karabo Bridge

Source: https://github.com/european-xfel/extra-data/blob/master/docs/cli.md

Utilize `karabo-bridge-serve-run` to stream data from a specific proposal and run number in the Karabo bridge format. Specify the port, include patterns for desired sources, and optionally allow partial trains or append detector modules. Use `--use-infiniband` for potential performance gains on compatible networks.

```shell
#                     Proposal run
karabo-bridge-serve-run 700000 40 --port 4321 \
     --include 'SPB_IRDA_JF4M/DET/JNGFR*:daqOutput' \
     --include '*/MOTOR/*[*Position]'
```

--------------------------------

### Serve a saved run over ZeroMQ

Source: https://github.com/european-xfel/extra-data/blob/master/docs/streaming.md

Use this command to stream data from a specific proposal run. Ensure the specified port is available and configure include patterns for desired data streams.

```shell
#                     Proposal run
karabo-bridge-serve-run 700000 40 --port 4545 \
     --include 'SPB_IRDA_JF4M/DET/JNGFR*:daqOutput' \
     --include '*/MOTOR/*[*Position]'
```

--------------------------------

### Get Detector Data Shape

Source: https://github.com/european-xfel/extra-data/blob/master/docs/lpd_data.ipynb

Prints the shape of the retrieved detector data module. This helps in understanding the dimensions of the data.

```python
data_module0.shape
```

--------------------------------

### Select Trains by Index

Source: https://github.com/european-xfel/extra-data/blob/master/docs/lpd_data.ipynb

Filters the run data to include only the first three trains. This is useful for keeping examples concise.

```python
run = run.select_trains(by_index[:3])
```

--------------------------------

### Accessing Data by Alias

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Demonstrates how to access data using aliases defined in an extra-data-aliases.yml file. Ensure the alias is present and valid.

```default
run.alias["xgm"]
run.alias["energy-kev"]
run.alias["agipd3-data"]
run.alias["agipd3-mask"]
```

--------------------------------

### RunDirectory()

Source: https://context7.com/european-xfel/extra-data/llms.txt

Opens a data run directly from a directory path, useful when the standard proposal directory layout is not used.

```APIDOC
## RunDirectory()

### Description
Open a run directly from a directory path containing EuXFEL HDF5 files, without requiring EuXFEL's standard proposal directory layout.

### Method
`extra_data.RunDirectory(path)`

### Parameters
#### Path Parameters
- **path** (str) - Required - The filesystem path to the run directory.

### Request Example
```python
from extra_data import RunDirectory

run = RunDirectory("/gpfs/exfel/exp/XMPL/201750/p700000/raw/r0001")
```

### Response
#### Success Response (RunDirectory)
- **run_metadata** (dict) - Metadata about the run.
- **train_ids** (list) - List of train IDs in the run.
- **train_timestamps** (numpy.ndarray or pandas.Series) - Timestamps for each train.

### Response Example
```python
# Inspect metadata
meta = run.run_metadata()
# {'proposalNumber': '700000', 'runNumber': '1', 'creationDate': '2017-...', ...}

# Get timestamps for each train
import numpy as np
timestamps = run.train_timestamps()         # numpy array of datetime64
ts_labelled = run.train_timestamps(labelled=True)  # pandas Series indexed by train ID
ts_local = run.train_timestamps(labelled=True, euxfel_local_time=True)  # Europe/Berlin TZ

print(f"Run has {len(run.train_ids)} trains")
print(f"Start: {timestamps[0]}, End: {timestamps[-1]}")
```
```

--------------------------------

### Get Number of Trains with Data

Source: https://github.com/european-xfel/extra-data/blob/master/docs/aligning_trains.ipynb

Retrieve the number of trains for which the detector recorded any data. This is determined by the `train_ids` attribute of the detector component.

```python
len(dssc.train_ids)
```

--------------------------------

### Write Virtual CXI File

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Create a virtual CXI file that provides access to detector data without copying it. Requires HDF5 1.10 or later. Fill values for missing data can be specified.

```python
det.write_virtual_cxi('output.cxi')
```

--------------------------------

### KeyData.train_from_index(i, keep_dims=False)

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Retrieves data for a train by its index (starting from 0) as a NumPy array. Returns a tuple of (train ID, array).

```APIDOC
## train_from_index(i, keep_dims=False)

### Description
Get data for a train by index (starting at 0) as a numpy array. Returns (train ID, array).

### Parameters
#### Path Parameters
* **i** (int) - Required - The index of the train to retrieve data for.
#### Query Parameters
* **keep_dims** (bool) - Optional - False (default) removes the train axis for single elements. True preserves this dimension.
```

--------------------------------

### Create Dask Client

Source: https://github.com/european-xfel/extra-data/blob/master/docs/dask_averaging.ipynb

Instantiate a Dask client connected to the configured cluster. This client is used to submit tasks and monitor computations.

```python
client = Client(cluster)
print("Created dask client:", client)
```

--------------------------------

### Initialize RunDirectory

Source: https://github.com/european-xfel/extra-data/blob/master/docs/parallel_example.ipynb

Initializes a RunDirectory object to access data from the specified run path.

```python
run = RunDirectory('/gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002/')
```

--------------------------------

### Get Train Data by Index

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Retrieve all data for the nth train in the file using its index. Supports filtering by devices and controlling output format.

```python
tid, data = run.train_from_index(5)
```

--------------------------------

### Get Train Data by ID

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Retrieve all data for a specific train using its unique ID. Supports filtering by devices and controlling output format.

```python
tid, data = run.train_from_id(12345)
```

--------------------------------

### Select Slurm partition

Source: https://github.com/european-xfel/extra-data/blob/master/docs/parallel_example.ipynb

Defines the Slurm partition to use for job submission. 'upex' is for external users, and 'exfel' is for staff. Comment out the 'Staff' line if you are an external user.

```python
partition = 'upex'   # External users
partition = 'exfel'  # Staff
```

--------------------------------

### Initialize Run Object

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Instantiate a Run object by providing the path to a run directory. This object is used to access and manage run data.

```python
r = Run(datapath)
```

--------------------------------

### Select data using a dictionary of source aliases and key sets

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Use this to select specific keys from one aliased source and all keys from another. An empty set for a source alias will retrieve all its keys.

```python
sel = run.select({'agipd': {'image.data'}, 'sa1-xgm': set()})
```

--------------------------------

### Stream run data over ZeroMQ with karabo-bridge-serve-run

Source: https://context7.com/european-xfel/extra-data/llms.txt

Re-stream run data using the Karabo bridge protocol for testing live analysis. Supports filtering sources, appending detector data, and different socket types.

```shell
# Stream JUNGFRAU detector + motor positions from run 40 on port 4321
karabo-bridge-serve-run 700000 40 --port 4321 \
    --include 'SPB_IRDA_JF4M/DET/JNGFR*:daqOutput' \
    --include '*/MOTOR/*[*Position]'
```

```shell
# Use PUB socket, allow partial trains, append multi-module detector data
karabo-bridge-serve-run 700000 40 --port 4545 \
    --include '*/DET/*' \
    --append-detector-modules \
    --allow-partial \
    --socket-type PUB
```

```python
# Receiving side (Python karabo-bridge client)
# from karabo_bridge import Client
# c = Client('tcp://127.0.0.1:4321')
# data, meta = c.next()
```

--------------------------------

### Define Data Chunks for Parallel Processing

Source: https://github.com/european-xfel/extra-data/blob/master/docs/parallel_example.ipynb

Divides the total number of trains into chunks, defining the start and end indices for each chunk to be processed by a worker.

```python
N_proc = 4
cuts = [int(xgm_vds.shape[0] * i / N_proc) for i in range(N_proc + 1)]
chunks = list(zip(cuts[:-1], cuts[1:]))
```

--------------------------------

### Get a Single Train by ID or Index

Source: https://github.com/european-xfel/extra-data/blob/master/docs/iterate_trains.ipynb

Retrieves a single train's data using either its unique train ID or its index within the run.

```python
tid, data = sel.train_from_id(79726787)
```

```python
tid, data = sel.train_from_index(36)
```

--------------------------------

### DataCollection.get_run_value() / get_run_values()

Source: https://context7.com/european-xfel/extra-data/llms.txt

Read the per-run snapshot of a control device property — the value recorded once at the start of the run, including properties not saved to CONTROL during the run.

```APIDOC
## DataCollection.get_run_value() / get_run_values()

### Description
Read the per-run snapshot of a control device property — the value recorded once at the start of the run, including properties not saved to CONTROL during the run.

### Method
`get_run_value(source, key)` or `get_run_values(source)`

### Parameters
- `source` (str) - The source name to query.
- `key` (str) - The specific property key to retrieve (for `get_run_value`).

### Request Example
```python
run = extra_data.open_run(700000, 1)

# Single value
trigger_mode = run.get_run_value(
    'HED_OPT_PAM/CAM/SAMPLE_CAM_4',
    'triggerMode'
)
print(trigger_mode)   # e.g. 1

# All run values for a source
all_vals = run.get_run_values('HED_OPT_PAM/CAM/SAMPLE_CAM_4')
print(all_vals.keys())   # dict_keys(['triggerMode', 'exposureTime', ...])

# Also available on SourceData
src = run['SA1_XTD2_XGM/XGM/DOOCS']
val = src.run_value('pulseEnergy.photonFlux')
vals = src.run_values()
```

### Response
- `value` (any) - The single run value or a dictionary of all run values for the source.
```

--------------------------------

### Select data using source alias and key glob pattern

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Use this to select all keys matching a pattern for a specific aliased source. Ensure the source alias is correctly defined.

```python
sel = run.alias.select('sa1-xgm', 'data.intensity*')
```

--------------------------------

### Import Libraries for Data Analysis

Source: https://github.com/european-xfel/extra-data/blob/master/docs/xpd_examples2.ipynb

Imports necessary libraries for plotting, numerical operations, and data handling with xarray. This is a common setup for data analysis tasks.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import xarray as xr

from extra_data import open_run
```

--------------------------------

### Access Data by Relative Train and Pulse ID

Source: https://github.com/european-xfel/extra-data/blob/master/docs/design.org

Retrieve data for a specific pulse within a relative train. Relative train IDs start from 0 for each run.

```python
r.data(t = 1, pulse = 3)
```

--------------------------------

### List Keys for a Specific Source

Source: https://github.com/european-xfel/extra-data/blob/master/docs/inspection.ipynb

View the available data keys within a specific instrument or control source. This helps in identifying the exact data streams you can access.

```python
run['SA1_XTD2_XGM/XGM/DOOCS:output'].keys()
```

--------------------------------

### get_dask_array

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Get a labelled Dask array of detector data. Dask allows for lazy, parallelized computing, suitable for large data volumes. Data is not loaded until a computation is triggered.

```APIDOC
## get_dask_array(key, fill_value=None, astype=None)

### Description
Get a labelled Dask array of detector data.

Dask does lazy, parallelised computing, and can work with large data
volumes. This method doesn’t immediately load the data: that only
happens once you trigger a computation.

### Parameters
#### Path Parameters
- **key** (str) - Required - The data to get, e.g. ‘data.adc’ for pixel values.
#### Query Parameters
- **fill_value** (int or float, optional) - Value to use for missing values. If None (default) the fill value is 0 for integers and np.nan for floats.
- **astype** (Type) - data type of the output array. If None (default) the dtype matches the input array dtype
```

--------------------------------

### Displaying Loaded Aliases

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Shows how to view all loaded aliases and their validity status. An '✗' indicates an invalid alias.

```default
Loaded aliases:
✗ xgm: SA42_XTD1_XGM/XGM/DOOCS

  MID_DET_AGIPD1M-1/DET/3CH0:xtdf:
    agipd3-data: image.data
    agipd3-mask: image.mask
```

--------------------------------

### List data files

Source: https://github.com/european-xfel/extra-data/blob/master/docs/parallel_example.ipynb

Lists the HDF5 sequence files that contain the data for the run.

```bash
!ls /gpfs/exfel/exp/XMPL/201750/p700000/raw/r0002/RAW-R0034-DA01-S*.h5
```

--------------------------------

### Check Data Availability

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Get a boolean array indicating the availability of image data for each module across trains. Useful for identifying trains or modules with missing data before attempting to read them.

```python
from extra_data import AGIPDDetector

det = AGIPDDetector("path/to/your/data.h5")

# Get data availability, including module gaps
availability = det.data_availability(module_gaps=True)

# Get data availability without module gaps information
availability_no_gaps = det.data_availability(module_gaps=False)
```

--------------------------------

### Verify alignment after selection

Source: https://github.com/european-xfel/extra-data/blob/master/docs/aligning_trains.ipynb

After using `.select(..., require_all=True)`, reload the data and verify that the train IDs now match perfectly between the selected sources.

```python
intensity_sase3 = sel['SA3_XTD10_XGM/XGM/DOOCS:output', 'data.intensityTD']
intensity_scs = sel['SCS_BLU_XGM/XGM/DOOCS:output', 'data.intensityTD']
print(f"# trains measured: {intensity_sase3.shape[0]}, {intensity_scs.shape[0]}")

train_ids_eq = intensity_sase3.train_id_coordinates() == intensity_scs.train_id_coordinates()
print("Train IDs all match:", train_ids_eq.all())
```

--------------------------------

### DataCollection.select

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Select a subset of sources and keys from this data.

```APIDOC
## DataCollection.select(seln_or_source_glob, key_glob='*')

### Description
Select a subset of sources and keys from this data.

### Parameters
#### Path Parameters
- **seln_or_source_glob** (str or list) - Description of the source to select.
- **key_glob** (str) - Description of the keys to select within the source. Defaults to '*'.

### Returns
A new [`DataCollection`](#extra_data.DataCollection) object for the selected data.
```

--------------------------------

### Get Dask Array for Detector Module

Source: https://github.com/european-xfel/extra-data/blob/master/docs/dask_averaging.ipynb

Retrieves a Dask array for a specific detector module's image data. This operation does not load the data into memory but provides its shape and structure.

```python
run.get_dask_array('SPB_DET_AGIPD1M-1/DET/0CH0:xtdf', 'image.data')
```

--------------------------------

### alias

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Enables item access via source and key aliases.

```APIDOC
## alias

Enables item access via source and key aliases.
```

--------------------------------

### Iterate Through Trains and Access Data

Source: https://github.com/european-xfel/extra-data/blob/master/docs/iterate_trains.ipynb

Iterates through trains using the `.trains()` method and accesses specific detector data. This example shows how to handle potential KeyErrors if data is missing for a source.

```python
for tid, data in sel.trains():
    print("Processing train", tid)
    print("Detector data module 0 shape:", data['SPB_DET_AGIPD1M-1/DET/0CH0:xtdf']['image.data'].shape)

    break  # Stop after the first train to keep the demo quick
```

--------------------------------

### Select Data with Source-to-Key Mapping

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Select data by providing a dictionary that maps source names to sets of key names. This method does not support glob patterns and is useful for precise selections.

```python
sel = run.select({'SPB_DET_AGIPD1M-1/DET/0CH0:xtdf': {'image.data'},
                     'SA1_XTD2_XGM/XGM/DOOCS': set()})
```

--------------------------------

### Get a single train by ID or index with DataCollection.train_from_id() / train_from_index()

Source: https://context7.com/european-xfel/extra-data/llms.txt

Retrieve all data for exactly one train, specified by its train ID or its 0-based index in the run. Can be used on a selected subset or directly from KeyData.

```python
run = extra_data.open_run(700000, 1)
sel = run.select('SA1_XTD2_XGM/XGM/DOOCS')

# By train ID
tid, data = sel.train_from_id(142844490)
print(data['SA1_XTD2_XGM/XGM/DOOCS']['beamPosition.ixPos'])  # scalar

# By index (0 = first train)
tid, data = sel.train_from_index(0)

# From KeyData directly
tid, arr = run['SA1_XTD2_XGM/XGM/DOOCS:output', 'data.intensityTD'].train_from_id(142844490)
```

--------------------------------

### Import necessary libraries

Source: https://github.com/european-xfel/extra-data/blob/master/docs/aligning_trains.ipynb

Import the numpy library for numerical operations and open_run from extra_data to access run data.

```python
import numpy as np
from extra_data import open_run
```

--------------------------------

### Get Dask Array of Detector Data

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Retrieve a labelled Dask array for detector data. Dask enables lazy, parallelized computation on large datasets, with data loading deferred until computation is triggered.

```python
dask_array = det.get_dask_array('data.adc')
```

--------------------------------

### List All Available Sources

Source: https://github.com/european-xfel/extra-data/blob/master/docs/inspection.ipynb

Access a frozenset of all unique source names available in the run. This includes both instrument and control sources.

```python
run.all_sources
```

--------------------------------

### Get Detector Data as Dask Array

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Retrieve detector data as a Dask array for lazy, parallelized computation. This is suitable for very large datasets where loading all data into memory is not feasible. Computation is triggered only when needed.

```python
from extra_data import AGIPDDetector

det = AGIPDDetector("path/to/your/data.h5")

# Get Dask array for image data
dask_image_data = det.get_dask_array('image.data')

# Get Dask array with specific fill value and cell ID indexing
dask_image_data_filled = det.get_dask_array('image.data', fill_value=0, subtrain_index='cellId')

# Trigger computation after defining the Dask array
computed_data = dask_image_data.compute()
```

--------------------------------

### Initialize DSSC with Minimum Modules

Source: https://github.com/european-xfel/extra-data/blob/master/docs/aligning_trains.ipynb

Initialize a DSSC detector component, requiring data from all modules (`min_modules=16`) in a train to be included. This filters trains where not all modules recorded data.

```python
dssc_allmod = DSSC1M(run, min_modules=16)

len(dssc_allmod.train_ids)
```

--------------------------------

### train_index_bounds

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Generate the first and last indices of trains to be used alongside data from `.ndarray()`. If `labelled` is True, returns a pandas DataFrame with 'start' and 'stop' columns. Otherwise, returns a tuple of two NumPy arrays.

```APIDOC
## train_index_bounds(labelled=False)

Generate first and last indices of trains to use alongside data
from `.ndarray()`.

If *labelled* is True, returns a pandas dataframe with columns start
and stop. Otherwise, returns a tuple of two NumPy arrays.
```

--------------------------------

### Read RUN-section snapshots with DataCollection.get_run_value() / get_run_values()

Source: https://context7.com/european-xfel/extra-data/llms.txt

Read the per-run snapshot of a control device property. This includes values recorded once at the start of the run, even if not saved to CONTROL during the run. Can retrieve single values or all run values for a source.

```python
run = extra_data.open_run(700000, 1)

# Single value
trigger_mode = run.get_run_value(
    'HED_OPT_PAM/CAM/SAMPLE_CAM_4',
    'triggerMode'
)
print(trigger_mode)   # e.g. 1

# All run values for a source
all_vals = run.get_run_values('HED_OPT_PAM/CAM/SAMPLE_CAM_4')
print(all_vals.keys())   # dict_keys(['triggerMode', 'exposureTime', ...])

# Also available on SourceData
src = run['SA1_XTD2_XGM/XGM/DOOCS']
val = src.run_value('pulseEnergy.photonFlux')
vals = src.run_values()
```

--------------------------------

### Initialize DSSC with Allowed Missing Modules

Source: https://github.com/european-xfel/extra-data/blob/master/docs/aligning_trains.ipynb

Initialize a DSSC detector component, allowing a specified number of missing modules (`min_modules=15`) per train. Missing data will be filled with 0 or NaN.

```python
dssc_mostmod = DSSC1M(run, min_modules=15)

len(dssc_mostmod.train_ids)
```

--------------------------------

### Open European XFEL Run Directory by Path

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Use this function to open a European XFEL run directory when you have the direct path to it. This is useful if the run is not in a standard location or if you are working with local copies.

```python
run = RunDirectory("/gpfs/exfel/exp/XMPL/201750/p700000/raw/r0001")
```

--------------------------------

### Get Detector Data as Labelled Array

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Retrieve detector data for a specific key, with options to select pulses, unstack them, specify fill values, and define the indexing method for frames within a train. Useful for loading data into memory for immediate analysis.

```python
from extra_data import AGIPDDetector

det = AGIPDDetector("path/to/your/data.h5")

# Get all pixel data for all pulses
image_data = det.get_array('image.data')

# Get specific pulses and unstack them
image_data_subset = det.get_array('image.data', pulses=slice(10, 20), unstack_pulses=True)

# Get data with a specific fill value and cell ID indexing
image_data_filled = det.get_array('image.data', fill_value=0, subtrain_index='cellId')

# Get data for a specific region of interest (ROI) on each module
roi_data = det.get_array('image.data', roi=np.s_[0, 100:200, 50:150])

# Get data as a different data type
image_data_float = det.get_array('image.data', astype=np.float32)
```

--------------------------------

### Print Auxiliary Source Information

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Use the `info()` method on the `run.auxiliary` object to display details about all auxiliary sources, including errata and reduction sources.

```python
run.auxiliary.info()
```

--------------------------------

### Open Detector Run with Extra-Data

Source: https://github.com/european-xfel/extra-data/blob/master/docs/dask_averaging.ipynb

Open a specific detector run using the extra-data library. This prepares the run for subsequent data access and processing.

```python
run = open_run(proposal=700000, run=2)
run.info()
```

--------------------------------

### Access KeyData for a Single Data Field

Source: https://context7.com/european-xfel/extra-data/llms.txt

Index a `DataCollection` with a `(source, key)` tuple to get a `KeyData` object. This provides direct access to time-series data and its metadata (dtype, shape, size, units). Data can be loaded as NumPy arrays, xarray DataArrays, Dask arrays, or pandas Series, with options for Region of Interest (ROI) slicing.

```python
run = extra_data.open_run(700000, 1)

kd = run['SA1_XTD2_XGM/XGM/DOOCS:output', 'data.intensityTD']

# Metadata
print(kd.dtype)          # dtype('float32')
print(kd.shape)          # (500, 2700)  — (trains, pulses)
print(kd.ndim)           # 2
print(kd.size_mb)        # 5.4
print(kd.units)          # 'μJ'  (if stored in the file)

# Load as NumPy array
arr = kd.ndarray()                # shape (500, 2700)

# Load only first 8 pulses per train (ROI)
arr_roi = kd.ndarray(roi=np.s_[:8])   # shape (500, 8)

# Load as xarray DataArray (labelled dimensions)
da = kd.xarray(extra_dims=['pulse'], name='xgm_intensity')
# <xarray.DataArray 'xgm_intensity' (trainId: 500, pulse: 2700)>

# Load as Dask array (lazy)
dask_arr = kd.dask_array()
mean_per_train = dask_arr.mean(axis=1).compute()

# Load as pandas Series (1D data only)
series = run['SA1_XTD2_XGM/XGM/DOOCS', 'beamPosition.ixPos'].series()

# Check if value is constant across the run
wavelength = run['SA3_XTD10_MONO/MDL/PHOTON_ENERGY', 'actualEnergy'].as_single_value()
# Returns the single value, or raises ValueError if it varies

# Train-ID coordinates matching the array rows
tids = kd.train_id_coordinates()  # array of train IDs, length == kd.shape[0]
```

--------------------------------

### Define human-readable aliases with DataCollection.with_aliases() / alias

Source: https://context7.com/european-xfel/extra-data/llms.txt

Define short, meaningful names (aliases) for long Karabo source and key names, then access data through them. Aliases can be defined inline or through the 'alias' attribute.

```python
run = extra_data.open_run(700000, 1)

# Define aliases inline
run_aliased = run.with_aliases({
    'xgm':      'SA1_XTD2_XGM/XGM/DOOCS',
    'mono-hv':  ('SA3_XTD10_MONO/MDL/PHOTON_ENERGY', 'actualEnergy'),
})

# Access via alias
xgm_src  = run_aliased.alias['xgm']           # → SourceData
energy_kd = run_aliased.alias['mono-hv']      # → KeyData
```

--------------------------------

### info(details_for_sources=(), counts=False, group_sources=True, with_aggregators=False, with_auxiliary=False)

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Display information about the data, with options to include details for specific sources, counts, and aggregators.

```APIDOC
## info(details_for_sources=(), counts=False, group_sources=True, with_aggregators=False, with_auxiliary=False)

Show information about the selected data.

* **Parameters:**
  * **details_for_sources** (*list* *of* *str*) – Glob patterns selecting sources to show additional information.
  * **counts** (*bool*) – Show data counts for all instrument sources.
  * **group_sources** (*bool*) – Group similar source names together if possible (on by default).
  * **with_aggregators** (*bool*) – Show which data aggregator each source was saved by.
  * **with_auxiliary** (*bool*) – Show auxiliary sources (REDUCTION & ERRATA)
```

--------------------------------

### Load Aliases from a File

Source: https://github.com/european-xfel/extra-data/blob/master/docs/reading_files.md

Aliases can be loaded from a JSON, YAML, or TOML file. Ensure the file format is correct and the file is accessible.

```yaml
sa1-xgm: SA1_XTD2_XGM/XGM/DOOCS
sa1-intensity: [SA1_XTD2_XGM/XGM/DOOCS:output, data.intensityTD]
SA3_XTD10_MONO/MDL/PHOTON_ENERGY:
  mono-central-energy: actualEnergy
```

--------------------------------

### List files in an EuXFEL run

Source: https://context7.com/european-xfel/extra-data/llms.txt

Use `lsxfel` to list files associated with a specific EuXFEL run, including auxiliary sources.

```shell
lsxfel /path/to/run --auxiliary
```

--------------------------------

### Iterate all sources by train with DataCollection.trains()

Source: https://context7.com/european-xfel/extra-data/llms.txt

Iterate over trains for multiple sources simultaneously, returning a dictionary of all data at each train. Always select data first for performance. Supports 'require_all' and 'flat_keys' options.

```python
run = extra_data.open_run(700000, 1)

sel = run.select([
    ('SA1_XTD2_XGM/XGM/DOOCS', 'beamPosition.ixPos'),
    ('SA3_XTD10_XGM/XGM/DOOCS', 'beamPosition.ixPos'),
])

for tid, data in sel.trains(require_all=True):
    x1 = data['SA1_XTD2_XGM/XGM/DOOCS']['beamPosition.ixPos']
    x2 = data['SA3_XTD10_XGM/XGM/DOOCS']['beamPosition.ixPos']
    print(f"Train {tid}: XGM1={x1:.3f}, XGM2={x2:.3f}")

# Flat keys mode: indexed by (source, key) tuples
for tid, data in sel.trains(flat_keys=True):
    val = data['SA1_XTD2_XGM/XGM/DOOCS', 'beamPosition.ixPos']
```

--------------------------------

### Create Virtual CXI File

Source: https://github.com/european-xfel/extra-data/blob/master/docs/cli.md

Generates a virtual CXI file from detector run data. Specify the input run directory and an output file name.

```shell
extra-data-make-virtual-cxi /gpfs/exfel/exp/XMPL/201750/p700000/proc/r0003 -o xmpl-3.cxi
```

--------------------------------

### JUNGFRAU Class Initialization

Source: https://github.com/european-xfel/extra-data/blob/master/docs/agipd_lpd_data.md

Initializes the JUNGFRAU detector interface. Allows specifying detector name, modules to use, and data correction preferences.

```APIDOC
## class extra_data.components.JUNGFRAU

An interface to JUNGFRAU data.

* **Parameters:**
  * **data** ([*DataCollection*](reading_files.md#extra_data.DataCollection)) – A data collection, e.g. from [`RunDirectory()`](reading_files.md#extra_data.RunDirectory).
  * **detector_name** (*str* *,* *optional*) – Name of a detector, e.g. ‘SPB_IRDA_JNGFR’. This is only needed
    if the dataset includes more than one JUNGFRAU detector.
  * **modules** (*set* *of* *ints* *,* *optional*) – Detector module numbers to use. By default, all available modules
    are used.
  * **min_modules** (*int*) – Include trains where at least n modules have data. Default is 1.
  * **n_modules** (*int*) – Number of detector modules in the experiment setup. Default is
    None, in which case it will be estimated from the available data.
  * **first_modno** (*int*) – The module number in the source name for the first detector module.
    e.g. FXE_XAD_JF500K/DET/JNGFR03:daqOutput should have first_modno = 3
  * **raw** (*bool*) – True to access raw data, False for corrected. The default is to use
    corrected if available & raw otherwise.
```