### Configuring LUTE Experiment Setup with `setup_lute` Script (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/quick_start.md

This command-line snippet illustrates the usage of the `setup_lute` script, a utility for configuring the LCLS Unified Task Executor (LUTE) for specific experiments on S3DF. It outlines the various parameters available, including experiment ID, installation options, LUTE version, and workflow selection, along with optional SLURM arguments for job submission.

```Bash
setup_lute [-h] [-d] -e EXPERIMENT [-f] [--test] [-v VERSION] [-w WORKFLOW] [SLURM_ARGS ...]
```

--------------------------------

### Cloning LUTE GitHub Repository (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/quick_start.md

This Bash command demonstrates how to clone the LUTE project's source code from its GitHub repository. This is the initial step for users who wish to install LUTE locally, outside of the S3DF environment, or for developers contributing to the project.

```Bash
git clone@github.com:slac-lcls/lute
```

--------------------------------

### Configuring LUTE Tasks with YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/quick_start.md

This YAML configuration file defines parameters for LUTE tasks. It includes general settings like experiment details, LUTE version, and timeouts, as well as specific parameters for individual tasks such as 'FindPeaksPyAlgos'. Users must modify certain parameters, like output directories and detector names, to suit their experiment's needs.

```yaml
%YAML 1.3
---
title: "LUTE Task Configuration" # Include experiment description if desired
experiment: "{{ $EXPERIMENT }}"
#run: "{{ $RUN }}"
date: "2023/10/25"
lute_version: 0.1      # Do not be change unless need to force older version
task_timeout: 6000
work_dir: ""
...
---
###########
# SFX
###########

# You must provide det_name and pv_camera_length
# pv_camera_length is the detector distance
# It can be a PV, or alternative a float for a fixed offset
# Example PVs are commented below, ask beamline staff for relevant PV
# Change outdir to an appropriate directory for CXI files.
FindPeaksPyAlgos:
    outdir: "/path/to/cxi_out"
    det_name: "epix10k2M"
    event_receiver: "evr0"
    tag: "lyso"
    event_logic: false
    psana_mask: false
    mask_file: null
    min_peaks: 10
    max_peaks: 2048
    npix_min: 2
    npix_max: 30
    amax_thr: 40
    atot_thr: 180
    son_min: 10.0
    peak_rank: 3
    r0: 3.0
    dr: 2.0
    nsigm: 10.0
    #pv_camera_length: "MFX:ROB:CONT:POS:Z"  # MFX epix10k2M
```

--------------------------------

### Cloning LUTE Repository (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/index.md

This command initiates the cloning of the LCLS Unified Task Executor (LUTE) repository from GitHub to your local machine. This is the essential first step for users who wish to run LUTE locally or contribute to its development, providing access to the source code and project files. Git must be installed and configured on your system as a prerequisite.

```bash
git clone@github.com:slac-lcls/lute
```

--------------------------------

### Example: Querying Parameters for IndexCrystFEL Task (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This example shows the output of `lute_help -t IndexCrystFEL`, illustrating how parameter details are displayed, including their type, default values, and descriptions. It highlights common parameters like `highres`, `profile`, `temp_dir`, `wait_for_file`, and `no_image_data`, and notes that `lute_config` should be ignored.

```bash
> utilities/lute_help -t IndexCrystFEL
INFO:__main__:Fetching parameter information for IndexCrystFEL.
IndexCrystFEL
-------------
Parameters for CrystFEL's `indexamajig`.

There are many parameters, and many combinations. For more information on
usage, please refer to the CrystFEL documentation, here:
https://www.desy.de/~twhite/crystfel/manual-indexamajig.html


Required Parameters:
--------------------
[...]

All Parameters:
-------------
[...]

highres (number)
	Mark all pixels greater than `x` has bad.

profile (boolean) - Default: False
	Display timing data to monitor performance.

temp_dir (string)
	Specify a path for the temp files folder.

wait_for_file (integer) - Default: 0
	Wait at most `x` seconds for a file to be created. A value of -1 means wait forever.

no_image_data (boolean) - Default: False
	Load only the metadata, no iamges. Can check indexability without high data requirements.

[...]
```

--------------------------------

### Example of Full SLURM Job Submission (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This snippet provides a comprehensive example of a `submit_slurm.sh` command, demonstrating how to specify a configuration file, a managed task name, and essential SLURM arguments like partition, account, and number of tasks. It shows a typical call for launching a LUTE task with specific resource requests.

```bash
submit_slurm.sh -c /sdf/data/lcls/ds/hutch/experiment/scratch/config.yaml -t Tester --partition=milano --account=lcls:experiment --ntasks=100 [...]
```

--------------------------------

### Example YAML Configuration with Variable Substitution

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This YAML configuration demonstrates various types of variable substitutions using Jinja-like syntax. It illustrates how to substitute values from the document header (e.g., 'experiment', 'run'), from environment variables (e.g., '$EXPERIMENT', '$RUN'), and from other sections within the same YAML document (e.g., 'OtherTask.useful_other_var'). The example also shows how to apply string formatting to substituted values and handle nested parameter references.

```YAML
%YAML 1.3
---
title: "Configuration to Test YAML Substitution"
experiment: "TestYAMLSubs"
run: 12
date: "2024/05/01"
lute_version: 0.1
task_timeout: 600
work_dir: "/sdf/scratch/users/d/dorlhiac"
...
---
OtherTask:
  useful_other_var: "USE ME!"

NonExistentTask:
  test_sub: "/path/to/{{ experiment }}/file_r{{ run:04d }}.input"         # Substitute `experiment` and `run` from header above
  test_env_sub: "/path/to/{{ $EXPERIMENT }}/file.input"                   # Substitute from the environment variable $EXPERIMENT
  test_nested:
    a: "outfile_{{ run }}_one.out"                                        # Substitute `run` from header above
    b:
      c: "outfile_{{ run }}_two.out"                                      # Also substitute `run` from header above
      d: "{{ OtherTask.useful_other_var }}"                               # Substitute `useful_other_var` from `OtherTask`
  test_fmt: "{{ run:04d }}"                                               # Subsitute `run` and format as 0012
  test_env_fmt: "{{ $RUN:04d }}"                                          # Substitute environment variable $RUN and pad to 4 w/ zeros
...
```

--------------------------------

### Configuring LUTE Tasks with YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/index.md

This YAML configuration file defines parameters for LUTE tasks, including general experiment settings and specific task-level configurations. It demonstrates how to set up output directories, detector names, and various peak-finding parameters for the `FindPeaksPyAlgos` task. Users need to modify specific parameters like `outdir` and `det_name` for their experiments. The `pv_camera_length` can be a PV or a fixed float.

```yaml
%YAML 1.3
---
title: "LUTE Task Configuration" # Include experiment description if desired
experiment: "{{ $EXPERIMENT }}"
#run: "{{ $RUN }}"
date: "2023/10/25"
lute_version: 0.1      # Do not be change unless need to force older version
task_timeout: 6000
work_dir: ""
...
---
###########
# SFX
###########

# You must provide det_name and pv_camera_length
# pv_camera_length is the detector distance
# It can be a PV, or alternative a float for a fixed offset
# Example PVs are commented below, ask beamline staff for relevant PV
# Change outdir to an appropriate directory for CXI files.
FindPeaksPyAlgos:
    outdir: "/path/to/cxi_out"
    det_name: "epix10k2M"
    event_receiver: "evr0"
    tag: "lyso"
    event_logic: false
    psana_mask: false
    mask_file: null
    min_peaks: 10
    max_peaks: 2048
    npix_min: 2
    npix_max: 30
    amax_thr: 40
    atot_thr: 180
    son_min: 10.0
    peak_rank: 3
    r0: 3.0
    dr: 2.0
    nsigm: 10.0
    #pv_camera_length: "MFX:ROB:CONT:POS:Z"  # MFX epix10k2M
```

--------------------------------

### Example Command-Line Execution for JSON User Program

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This Bash snippet illustrates a typical command-line invocation for a hypothetical `jsonuser` program. It shows how an input JSON file is passed using the `-i` flag, which is later configured within the LUTE Pydantic model.

```Bash
jsonuser -i <input_file.json>
```

--------------------------------

### Python Google Style Docstring Example

Source: https://github.com/slac-lcls/lute/blob/dev/README.md

This snippet illustrates the Google style docstring conventions for Python modules, classes, properties, and functions, including sections for arguments, returns, and exceptions. It serves as a reference for documenting code within the LUTE project, ensuring consistency and clarity.

```Python
"""This is a module level docstring.

Classes:
    Class1: Short description

Functions:
    function1(prototype): Short description

Exceptions:
    MyException: When/why it is raised.

Example usage:
    How one might use the classes and functions. Test modules may include how
        to run them.
"""

class Class1:
    """Class descriptions can go here or in __init__.

    Attributes:
        public_attr (type): Description
        # properties should be documented separately in their getter.
    """
    def __init__(self, arg1: type) -> None:
        """Description of class can go here.

        Args:
            param1 (type): Description
        """
        ...

    @property
    def my_property(self) -> type:
        """type: Property attributes are documented here."""
        ...

def function1(arg1: type) -> type:
    """Function description.

    Args:
        arg1 (type):  Description

    Returns: # Or Yields for generators
        name (type): Description # Alternatively, can just be `type: Description`

    Raises:
        MyException: Why?
    """
    ...
```

--------------------------------

### Getting Help for a Specific Lute Task's Parameters (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This command demonstrates how to use `lute_help` with the `-t` flag to retrieve a summary of available parameters for a specified Lute Task. Replace `[TaskName]` with the actual name of the task you wish to query. This is useful for understanding what parameters are available and their basic function.

```bash
> utilities/lute_help -t [TaskName]
```

--------------------------------

### Example JSON Configuration File Structure

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This JSON snippet presents a basic configuration file structure, demonstrating various data types including strings, integers, nested objects, and arrays. This serves as the base template that will be modified by Jinja for dynamic parameter substitution.

```JSON
{
    "param1": "arg1",
    "param2": 4,
    "param3": {
        "a": 1,
        "b": 2
    },
    "param4": [
        1,
        2,
        3
    ]
}
```

--------------------------------

### Example Command-Line Call for RunTask

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

Illustrates a typical command-line invocation for the `runtask` executable, showing various flag types (`-n`, `--method`, `-p`) and an optional boolean flag (`--debug`). This serves as the basis for defining the Python parameter model.

```bash
/sdf/group/lcls/ds/tools/runtask -n <nthreads> --method=<algorithm> -p <algo_param> [--debug]
```

--------------------------------

### Example LUTE Task Configuration YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This YAML snippet demonstrates the basic two-document structure for configuring LUTE Tasks. The first document contains global parameters like 'experiment' and 'work_dir', while the second defines specific parameters for individual tasks such as 'TaskOne' and 'TaskTwo'. The 'work_dir' is crucial for output storage and database location, and 'task_timeout' sets a time limit for all Task jobs.

```yaml
%YAML 1.3
---
title: "Some title."
experiment: "MYEXP123"
# run: 12 # Does not need to be provided
date: "2024/05/01"
lute_version: 0.1
task_timeout: 600
work_dir: "/sdf/scratch/users/d/dorlhiac"
...
---
TaskOne:
  param_a: 123
  param_b: 456
  param_c:
    sub_var: 3
    sub_var2: 4

TaskTwo:
  new_param1: 3
  new_param2: 4

# ...
...
```

--------------------------------

### Command-Line Usage for `run_functional.py`

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/tests.md

This snippet details the command-line arguments for `run_functional.py`, the main script for running LUTE's functional test suite. It allows users to specify the LUTE installation directory, select specific tests, and configure execution environments like Airflow or Prefect, along with options for handling output files and Git versions.

```bash
usage: run_functional.py [-h] [-a] [--test_airflow] [--use_prefect] [--git_pr_id GIT_PR_ID] [--git_tag GIT_TAG] -r RUN_DIR [--no_delete] [--run_tests RUN_TESTS] [--tests_dir TESTS_DIR] [--use_local_tests]

Run the LUTE functional test suite.

optional arguments:
  -h, --help            show this help message and exit
  -a, --admin           Run as Airflow admin. Requires permissions. Ignored if using prefect.
  --test_airflow        Use test Airflow instance. Ignored if using prefect.
  --use_prefect         Use prefect (experimental) instead of Airflow.
  --git_pr_id GIT_PR_ID
                        Check out a specific GitHub PR ID of LUTE to run (a PR branch).
  --git_tag GIT_TAG     Check out a specific git tag of LUTE to run (e.g. a release).
  -r RUN_DIR, --run_dir RUN_DIR
                        Directory to install LUTE to.
  --no_delete           If passed, do not delete output files when tests are finished.
  --run_tests RUN_TESTS
                        Provide a comma-separated string of tests to run. If provided, this script will only run those, rather than the default behaviour of running all tests. E.g: --run_these_tests
                        test2,test5. Tests that do not exist are silently ignored.
  --tests_dir TESTS_DIR
                        Specify an alternative path to tests than those from the LUTE clone. Must have the same directory structure: $DIR/test1/... $DIR/test2/... If this flag and --use_local_tests are both
                        passed, this one is used.
  --use_local_tests     Use the tests from the installation of LUTE where this script is called, rather than those from the clone of LUTE which is run against, or another directory if passed. If this flag
                        and --tests_dir are both passed, --tests_dir is used.
```

--------------------------------

### Jinja Templated JSON Configuration Example

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet shows the JSON configuration file with Jinja templating applied. Variables like `{{ str_var }}` are embedded to allow dynamic substitution of values for `param1`, `param2`, `param3.b`, and `param4` (first two elements), illustrating how LUTE uses Jinja to render the final configuration.

```JSON
{
    "param1": {{ str_var }},
    "param2": {{ int_var }},
    "param3": {
        "a": 1,
        "b": {{ p3_b }}
    },
    "param4": [
        {{ val }},
        {{ val }},
        3
    ]
}
```

--------------------------------

### Initializing an Airflow DAG for LUTE Workflow (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This snippet demonstrates the foundational code for creating an Airflow DAG within a new LUTE workflow module. It imports necessary Airflow and LUTE components, dynamically sets the DAG ID based on the filename, and defines the DAG's start date, schedule, and description, serving as the entry point for orchestrating tasks.

```Python
from datetime import datetime
import os
from airflow import DAG
from lute.operators.jidoperators import JIDSlurmOperator # Import other operators if needed

dag_id: str = f"lute_{os.path.splitext(os.path.basename(__file__))[0]}"
description: str = (
    "Run SFX processing using PyAlgos peak finding and experimental phasing"
)

dag: DAG = DAG(
    dag_id=dag_id,
    start_date=datetime(2024, 3, 18),
    schedule_interval=None,
    description=description,
)
```

--------------------------------

### Correctly Quoting YAML Substitutions - YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This example shows the correct method for defining a substitution in YAML. It is crucial to enclose the substitution expression within double quotes (e.g., `"{{ other_var:04d }}"`) to prevent `pyyaml` parsing errors, specifically the 'Found unhashable key' exception, ensuring the document is parsed correctly before substitution occurs.

```yaml
# USE THIS
MyTask:
  var_sub: "{{ other_var:04d }}"
```

--------------------------------

### Incorrect YAML Substitution Order - YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This example illustrates a common 'gotcha' where the order of definitions in a YAML document affects substitution. If `RunTaskOne` attempts to use `RunTaskTwo.path` before `RunTaskTwo` is fully defined and its own substitutions (`work_dir`, `run`) are resolved, the substitution for `input_dir` will be incorrect, resulting in an unresolved path.

```yaml
%YAML 1.3
---
title: "Configuration to Test YAML Substitution"
experiment: "TestYAMLSubs"
run: 12
date: "2024/05/01"
lute_version: 0.1
task_timeout: 600
work_dir: "/sdf/data/lcls/ds/exp/experiment/scratch"
...
---
RunTaskOne:
  input_dir: "{{ RunTaskTwo.path }}"  # Will incorrectly be "{{ work_dir }}/additional_path/{{ $RUN }}"
  # ...

RunTaskTwo:
  # Remember `work_dir` and `run` come from the header document and don't need to
  # be qualified
  path: "{{ work_dir }}/additional_path/{{ run }}"
...
```

--------------------------------

### Creating JIDSlurmOperator Instances for LUTE Tasks (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This example illustrates how to instantiate `JIDSlurmOperator` objects, which are used to launch specific LUTE managed tasks on SLURM. It shows how to link operators to the main DAG and configure task-specific parameters like `max_cores` or `custom_slurm_params` to override default SLURM settings for individual jobs.

```Python
# Using only the JIDSlurmOperator
# syntax: JIDSlurmOperator(task_id="LuteManagedTaskName", dag=dag) # optionally, max_cores=123)
peak_finder: JIDSlurmOperator = JIDSlurmOperator(task_id="PeakFinderPyAlgos", dag=dag)

# We specify a maximum number of cores for the rest of the jobs.
indexer: JIDSlurmOperator = JIDSlurmOperator(
    max_cores=120, task_id="CrystFELIndexer", dag=dag
)
# We can alternatively specify this task be only ever run with the following args.
# indexer: JIDSlurmOperator = JIDSlurmOperator(
#     custom_slurm_params="--partition=milano --ntasks=120 --account=lcls:myaccount",
#     task_id="CrystFELIndexer",
#     dag=dag,
# )

# Merge
merger: JIDSlurmOperator = JIDSlurmOperator(
    max_cores=120, task_id="PartialatorMerger", dag=dag
)
```

--------------------------------

### Initializing JIDSlurmOperator for HKLComparer in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

Initializes a `JIDSlurmOperator` instance named `hkl_comparer`. This operator is configured to use a maximum of 8 cores and is assigned the task ID 'HKLComparer', associating it with a predefined DAG object. This setup prepares a specific step within a larger workflow for execution on a SLURM cluster.

```python
hkl_comparer: JIDSlurmOperator = JIDSlurmOperator(
    max_cores=8, task_id="HKLComparer", dag=dag
)
```

--------------------------------

### Defining Linear DAG via YAML for Run-time Creation

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

Provides a YAML representation for defining a linear DAG structure dynamically at run-time. Each nested dictionary represents a task node, specifying its `task_name`, `slurm_params` (empty for defaults), and a `next` list for subsequent tasks. This example mirrors the Python linear DAG, showing `PeakFinderPyAlgos` followed by `CrystFELIndexer`, then `PartialatorMerger`, and finally `HKLComparer`.

```yaml
task_name: PeakFinderPyAlgos
slurm_params: ''
next:
- task_name: CrystFELIndexer
  slurm_params: ''
  next:
  - task_name: PartialatorMerger
    slurm_params: ''
    next:
    - task_name: HKLComparer
      slurm_params: ''
      next: []
```

--------------------------------

### Defining Nested Parameters in Python for LUTE Schema

Source: https://github.com/slac-lcls/lute/blob/dev/docs/design/database.md

This Python dictionary illustrates how nested parameters are structured before being flattened into columns in the LUTE `Task` tables. It shows an example of a dictionary with nested dictionaries and tuples, which will result in column names like `a.b[0]`, `a.b[1]`, and `a.c` after flattening.

```Python
param_dict: Dict[str, Any] = {
    "a": {               # First parameter a
        "b": (1, 2),
        "c": 1,
        # ...
    },
    "a2": 4,             # Second parameter a2
    # ...
}
```

--------------------------------

### Modifying Task Environment with Executor (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This code illustrates how to modify the execution environment for LUTE Tasks using an Executor. It shows update_environment for adding or modifying environment variables (specifically prepending to PATH), and shell_source for executing a shell script to perform more complex environment setups, such as activating conda environments or setting multiple variables.

```Python
TaskRunner: Executor = Executor("RunTask")
# update_environment(env: Dict[str,str], update_path: str = "prepend") # "append" or "overwrite"
TaskRunner.update_environment(
    { "PATH": "/sdf/group/lcls/ds/tools" }  # This entry will be prepended to the PATH available after sourcing `psconda.sh`
)

Task2Runner: Executor = Executor("RunTask2")
Task2Runner.shell_source("/sdf/group/lcls/ds/tools/new_task_setup.sh") # Will source new_task_setup.sh script
```

--------------------------------

### Implementing Dependent Pydantic Validators for Task Parameters in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This example demonstrates how to define TaskParameters with pydantic Field's and implement custom validators. It shows how to constrain a method parameter to specific values and how to dynamically set the default value of method_param1 based on the method parameter, utilizing the values dictionary and the always=True validator option for conditional default assignment.

```python
from pydantic import Field, validator, ValidationError, root_validator
class RunTaskParameters(ThirdPartyParameters):
    """Parameters for the runtask binary."""

    # [...]

    # In this case we will use the Python variable name directly when passing
    # the parameter on the command-line
    method: str = Field("algo1", description="Algorithm to use.", flag_type="--")

    # For an actual parameter we would probably have a better name. Lets assume
    # This parameter (-p) modifies the behaviour of the method above.
    method_param1: Optional[int] = Field(
        description="Modify method performance.", flag_type="-", rename_param="p"
    )

    # We will only allow method to take on one of two values
    @validator("method")
    def validate_method(cls, method: str, values: Dict[str, Any]) -> str:
        """Method validator: --method can be algo1 or algo2."""

        valid_methods: List[str] = ["algo1", "algo2"]
        if method not in valid_methods:
            raise ValueError("method must be algo1 or algo2")
        return method

    # Lets change the default value of `method_param1` depending on `method`
    # NOTE: We didn't provide a default value to the Field above and made it
    # optional. We can use this to test whether someone is purposefully
    # overriding the value of it, and if not, set the default ourselves.
    # We set `always=True` since pydantic will normally not use the validator
    # if the default is not changed
    @validator("method_param1", always=True)
    def validate_method_param1(cls, param1: Optional[int], values: Dict[str, Any]) -> int:
        """method param1 validator"""

        # If someone actively defined it, lets just return that value
        # We could instead do some additional validation to make sure that the
        # value they provided is valid...
        if param1 is not None:
            return param1

        # method_param1 comes after method, so this will be defined, or an error
```

--------------------------------

### Listing All Available Lute Tasks (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This command demonstrates using `lute_help` with the `-l` flag to retrieve a list of all currently available Lute Tasks. The output includes the task name, associated managed tasks (which are used in submission scripts), and a brief description of the task's functionality, as shown for `AnalyzeSmallDataXAS` and `AnalyzeSmallDataXES`.

```bash
> ./lute/utilities/lute_help -l
INFO:__main__:Fetching Task list.
Task List
---------

- AnalyzeSmallDataXAS
  Current Managed Tasks: SmallDataXASAnalyzer,
	TaskParameter model for AnalyzeSmallDataXAS Task.

	This Task does basic analysis of XAS data based on a SmallData HDF5 output
	file. It calculates difference absorption and signal binned by various
	scanned motors.

- AnalyzeSmallDataXES
  Current Managed Tasks: SmallDataXESAnalyzer,
	TaskParameter model for AnalyzeSmallDataXES Task.

	This Task does basic analysis of XES data based on a SmallData HDF5 output
	file. It calculates difference emission and signal binned by various
	scanned motors.
```

--------------------------------

### Executing launch_airflow.py Script with Parameters

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This command-line snippet shows the usage of the `launch_airflow.py` script. It takes required arguments for a configuration YAML path (`-c`) and a workflow name (`-w`), along with optional flags like `--debug` and `--test`, and parameters for experiment (`-e`), run number (`-r`), and additional SLURM arguments.

```Bash
launch_airflow.py -c <path_to_config_yaml> -w <workflow_name> [--debug] [--test] [-e <exp>] [-r <run>] [SLURM_ARGS]
```

--------------------------------

### Mimicking SLURM for MPI LUTE Tasks (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

Addresses issues with MPI task launch by setting the `SLURM_NPROCS` environment variable. This helps in situations where the core count determination is inconsistent, allowing the task to mimic a SLURM submission process for proper MPI initialization.

```bash
SLURM_NPROCS=2 python -B [-O] -t <ManagedTaskName> -c <path/to/config.yaml>
```

--------------------------------

### Lute Message Signals in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet defines the set of predefined string signals used within Lute `Message` objects. These signals are primarily used by base classes to trigger specific hooks on the `Executor`-side, indicating various task states like start, failure, completion, or result availability.

```Python
LUTE_SIGNALS: Set[str] = {
    "NO_PICKLE_MODE",
    "TASK_STARTED",
    "TASK_FAILED",
    "TASK_STOPPED",
    "TASK_DONE",
    "TASK_CANCELLED",
    "TASK_RESULT",
}
```

--------------------------------

### Submitting LUTE Airflow Workflows (DAGs) via Bash

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This command demonstrates how to submit a complete LUTE workflow (DAG) to Airflow using the `submit_launch_airflow.sh` script. It requires the full path to `launch_airflow.py`, a configuration YAML file, and the DAG name. Optional arguments include debug mode, test instance usage, experiment name, and run number, along with SLURM arguments for batch job submission.

```bash
> launch_scripts/submit_launch_airflow.sh /path/to/lute/launch_scripts/launch_airflow.py -c </path/to/yaml.yaml> -w <dag_name> [--debug] [--test] [-e <exp>] [-r <run>] $SLURM_ARGS
```

--------------------------------

### Minimal Lute Task Module Template

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet provides the initial boilerplate for a Lute `Task` module, including a standard docstring, the `__all__` variable to control what's imported, and an `__author__` placeholder. It sets up the file structure before the actual `Task` class definition, which must inherit from `Task` and implement a `_run` method.

```Python
"""Standard docstring..."""

__all__ = ["RunTask"]
__author__ = "" # Please include so we know who the SME is
```

--------------------------------

### Defining a Serial Airflow DAG with YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This YAML configuration defines a serial Airflow DAG for femtosecond crystallography. It specifies a sequence of tasks: `PeakFinderPyAlgos`, followed by `CrystFELIndexer`, then `PartialatorMerger`, and finally `HKLComparer`. Each task can optionally include custom SLURM parameters.

```yaml
task_name: PeakFinderPyAlgos
slurm_params: ''
next:
- task_name: CrystFELIndexer
  slurm_params: ''
  next:
  - task_name: PartialatorMerger
    slurm_params: ''
    next:
    - task_name: HKLComparer
      slurm_params: ''
      next:
```

--------------------------------

### Initializing a LUTE Executor (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet demonstrates how to initialize a standard `Executor` instance in LUTE. It creates an `Executor` named "SubmitTask", which will manage the submission and execution environment for associated tasks, handling logging and eLog communication.

```python
# Initialization: Executor("TaskName")
TaskRunner: Executor = Executor("SubmitTask")
```

--------------------------------

### Submitting LUTE Managed Task as SLURM Batch Job (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

Submits a single LUTE managed task as a batch job on S3DF using the `submit_slurm.sh` script. This method requires the task name, configuration YAML path, and additional SLURM arguments like `--partition` and `--account`.

```bash
launch_scripts/submit_slurm.sh -t <ManagedTaskName> -c </path/to/config/yaml> [--debug] $SLURM_ARGS
```

--------------------------------

### Submitting Airflow Workflow with Wrapper Script (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This snippet demonstrates the usage of `submit_launch_airflow.sh`, a wrapper script designed to submit the `launch_airflow.py` job to S3DF batch nodes. It's primarily used when launching from the eLog via ARP to bypass the 30-second timeout limit. All arguments are transparently passed to `launch_airflow.py`, with the first argument specifically being the path to `launch_airflow.py`.

```bash
submit_launch_airflow.sh /path/to/launch_airflow.py -c <path_to_config_yaml> -w <workflow_name> [--debug] [--test] [-e <exp>] [-r <run>] [SLURM_ARGS]
```

--------------------------------

### Defining a Standard Lute Task in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet defines a basic Lute `Task` class, `RunTask`, demonstrating how to inherit from `Task`, initialize with `TaskParameters`, access parameters and results, and implement the `_run` method for core logic. It also shows how to send messages and set task results.

```Python
from lute.execution.ipc import Message # Message for communication
from lute.io.models.base import *      # For TaskParameters
from lute.tasks.task import *          # For Task

class RunTask(Task): # Inherit from Task
    """Task description goes here, or in __init__"""

    def __init__(self, *, params: TaskParameters) -> None:
        super().__init__(params=params) # Sets up Task, parameters, etc.
        # Parameters will be available through:
          # self._task_parameters
          # You access with . operator: self._task_parameters.param1, etc.
        # Your result object is availble through:
          # self._result
            # self._result.payload <- Main result
            # self._result.summary <- Short summary
            # self._result.task_status <- Semi-automatic, but can be set manually

    def _run(self) -> None:
        # THIS METHOD MUST BE PROVIDED
        self.do_my_analysis()

    def do_my_analysis(self) -> None:
        # Send a message, proper way to print:
        msg: Message(contents="My message contents", signal="")
        self._report_to_executor(msg)

        # When done, set result - assume we wrote a file, e.g.
        self._result.payload = "/path/to/output_file.h5"
        # Optionally also set status - good practice but not obligatory
        self._result.task_status = TaskStatus.COMPLETED
```

--------------------------------

### Launching Airflow DAG with Run-time YAML Configuration

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

Provides a bash command for launching an Airflow DAG using a dynamically defined YAML configuration file. Instead of calling a DAG by name, the `-W` flag specifies the path to the YAML file, allowing for flexible workflow execution with custom parameters. This command includes placeholders for experiment, run, and configuration paths, along with optional test and debug flags.

```bash
/path/to/lute/launch_scripts/submit_launch_airflow.sh /path/to/lute/launch_scripts/launch_airflow.py -e <exp> -r <run> -c /path/to/config -W <path_to_dag> --test [--debug] [SLURM_ARGS]
```

--------------------------------

### Executing LUTE Functional Tests (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/tests.md

This command runs the LUTE functional test suite. It requires an existing output directory specified by --run_dir where test-related files will be stored. The optional --use_prefect argument enables integration with Prefect for workflow management.

```bash
./lute/tests/submit_run_functional.sh --run_dir <$OUTPUT_DIR> [--use_prefect]
```

--------------------------------

### Providing Experiment and Run for SLURM Submitted LUTE Task (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

Allows specifying `experiment` and `run` values directly as command-line arguments when submitting a LUTE managed task via SLURM. This provides an alternative to embedding these values in the configuration YAML for batch jobs.

```bash
launch_scripts/submit_slurm.sh -t <ManagedTaskName> -c </path/to/config/yaml> [-e EXPERIMENT] [-r RUN] [--debug] $SLURM_ARGS
```

--------------------------------

### Configuring Task Result from Parameters

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

Demonstrates how to use the `result_from_params` configuration option to dynamically specify the task result based on other parameters within the `TaskParameters` model. This option is used when `set_result` is `True` and allows for flexible result path generation.

```Python
result_from_params==f"{output_directory}/{output_filename}"
```

--------------------------------

### Correcting YAML Substitution Order - YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/configuration.md

This snippet demonstrates the correct way to structure the YAML document to ensure proper substitution. By defining `RunTaskTwo` before `RunTaskOne`, `RunTaskTwo.path` is fully resolved with its own substitutions (`work_dir`, `run`) before `RunTaskOne` attempts to use it, leading to the correct `input_dir` value.

```yaml
%YAML 1.3
---
title: "Configuration to Test YAML Substitution"
experiment: "TestYAMLSubs"
run: 12
date: "2024/05/01"
lute_version: 0.1
task_timeout: 600
work_dir: "/sdf/data/lcls/ds/exp/experiment/scratch"
...
---
RunTaskTwo:
  # Remember `work_dir` comes from the header document and doesn't need to be qualified
  path: "{{ work_dir }}/additional_path/{{ run }}"

RunTaskOne:
  input_dir: "{{ RunTaskTwo.path }}"  # Will now be /sdf/data/lcls/ds/exp/experiment/scratch/additional_path/12
  # ...
...
```

--------------------------------

### Defining Serial DAG Execution in Airflow (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This Python snippet illustrates the sequential execution order of tasks within an Airflow DAG, where `peak_finder` runs first, followed by `indexer`, then `merger`, and finally `hkl_comparer`. It represents a simplified serial femtosecond crystallography workflow.

```python
peak_finder >> indexer >> merger >> hkl_comparer
```

--------------------------------

### Running a Managed LUTE Task via Command Line - Bash

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/architecture.md

This command-line snippet demonstrates how to execute a managed analysis task within the LUTE system. It uses the `run_task.py` script, requiring the name of the managed task (`<ManagedTaskName>`) and the path to its configuration YAML file (`</path/to/config/yaml>`). The `-B` flag prevents Python from writing `.pyc` files, and the optional `-O` flag enables basic optimizations.

```bash
python -B [-O] run_task.py -t <ManagedTaskName> -c </path/to/config/yaml>
```

--------------------------------

### Launching a Custom YAML-Defined DAG in LUTE (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This Bash command demonstrates how to launch an Airflow DAG defined in a custom YAML file using the LUTE launch script. The `-W` flag specifies the path to the YAML DAG definition, allowing dynamic workflow creation instead of calling a pre-defined DAG by name.

```bash
/path/to/lute/launch_scripts/submit_launch_airflow.sh /path/to/lute/launch_scripts/launch_airflow.py -e <exp> -r <run> -c /path/to/config -W <path_to_dag> --test [--debug] [SLURM_ARGS]
```

--------------------------------

### Defining Pydantic Model for JSON User Program in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This Python snippet defines a Pydantic model, `RunJsonUserParameters`, for a third-party executable. It specifies the `executable` path and configures the `input_json` parameter to be passed via the command-line flag `-i`, setting up the basic structure for the program's arguments.

```Python
from pydantic import Field, validator

from .base import TemplateConfig

class RunJsonUserParameters:
    executable: str = Field(
        "/path/to/jsonuser", description="Executable which requires a JSON configuration file."
    )
    # Lets assume the JSON file is passed as "-i <path_to_json>"
    input_json: str = Field(
        "", description="Path to the input JSON file.", flag_type="-", rename_param="i"
    )
```

--------------------------------

### Enabling a LUTE Debug Exit Marker (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This Bash command demonstrates how to enable a `LUTE_DEBUG_EXIT` marker by setting its corresponding environment variable (e.g., `MYENVVAR=1`) before running a LUTE task. This allows developers to trigger an early exit at a predefined point for debugging purposes.

```bash
MYENVVAR=1 python -B run_task.py -t Tester -c config/test.yaml
```

--------------------------------

### Initializing MPIExecutor for SubmitTask (Python)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This snippet initializes an MPIExecutor instance, named TaskRunner, associating it with a Task identified as "SubmitTask". This is used when the Task requires MPI for its execution, providing a managed way to submit and run such tasks within the LUTE framework.

```Python
TaskRunner: MPIExecutor = MPIExecutor("SubmitTask") ## If using the MPIExecutor
```

--------------------------------

### Defining LUTE Task Parameters with Pydantic in Python

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This Python snippet defines the `RunJsonUserParameters` class using Pydantic, specifying fields for an executable, input JSON path, and LUTE template configuration. It includes a validator to automatically set the `lute_template_cfg.output_path` based on the `input_json` field, ensuring the rendered template is saved to the specified location.

```python
# from typing import Optional

from pydantic import Field, validator

from .base import TemplateConfig #, TemplateParameters

class RunJsonUserParameters:
    executable: str = Field(
        "jsonuser", description="Executable which requires a JSON configuration file."
    )
    # Lets assume the JSON file is passed as "-i <path_to_json>"
    input_json: str = Field(
        "", description="Path to the input JSON file.", flag_type="-", rename_param="i"
    )
    # Add template configuration! *MUST* be called `lute_template_cfg`
    lute_template_cfg: TemplateConfig = Field(
        TemplateConfig(
            template_name="jsonuser.json", # Only the name of the file here.
            output_path="",
        ),
        description="Template rendering configuration",
    )
    # We do not need to include these TemplateParameters, they will be added
    # automatically if provided in the YAML
    #str_var: Optional[TemplateParameters]
    #int_var: Optional[TemplateParameters]
    #p3_b: Optional[TemplateParameters]
    #val: Optional[TemplateParameters]


    # Tell LUTE to write the rendered template to the location provided with
    # `input_json`. I.e. update `lute_template_cfg.output_path`
    @validator("lute_template_cfg", always=True)
    def update_output_path(
        cls, lute_template_cfg: TemplateConfig, values: Dict[str, Any]
    ) -> TemplateConfig:
        if lute_template_cfg.output_path == "":
            lute_template_cfg.output_path = values["input_json"]
        return lute_template_cfg
```

--------------------------------

### Essential SLURM Arguments for Job Submission (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This snippet highlights the two mandatory SLURM arguments (`--partition` and `--account`) that must be provided when using `submit_slurm.sh`. These arguments specify the compute partition and the accounting group for the job, crucial for resource allocation and billing within the SLURM environment.

```bash
--partition=<...> # Usually partition=milano
--account=<...> # Usually account=lcls:$EXPERIMENT
```

--------------------------------

### Defining Parallel Airflow Tasks with YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

This YAML snippet demonstrates how to define a DAG where `Task1` executes first, followed by `Task2` and `Task3` running in parallel. The `next` field allows specifying multiple subsequent tasks that will execute concurrently.

```yaml
task_name: Task1
slurm_params: ''
next:
- task_name: Task2
  slurm_params: ''
  next: []
- task_name: Task3
  slurm_params: ''
  next: []
```

--------------------------------

### Submitting Managed LUTE Task to SLURM (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/creating_workflows.md

This snippet illustrates the basic command-line usage for `submit_slurm.sh`, a script used to launch a single managed LUTE Task on S3DF batch nodes via the SLURM scheduler. It requires a configuration YAML path and the name of the managed task, passing any additional arguments directly to SLURM.

```bash
submit_slurm.sh -c <path_to_config_yaml> -t <MANAGED_task_name> [--debug] [SLURM_ARGS ...]
```

--------------------------------

### Setting Experiment and Run Environment Variables for LUTE Task (Bash)

Source: https://github.com/slac-lcls/lute/blob/dev/docs/usage/running_lute.md

Demonstrates how to provide `experiment` and `run` values as environment variables when running a LUTE managed task. This is an alternative to including them in the configuration YAML, especially useful when tasks or external code require these values.

```bash
EXPERIMENT="..." RUN_NUM=123 python -B [-O] -t <ManagedTaskName> -c <path/to/config.yaml>
```

--------------------------------

### Configuring LUTE with Template Variables in YAML

Source: https://github.com/slac-lcls/lute/blob/dev/docs/development/new_task.md

This YAML configuration demonstrates how to provide values for template variables within a LUTE task. It maps specific variable names (`str_var`, `int_var`, `p3_b`, `val`) to their corresponding values, which LUTE will use to substitute placeholders in the `jsonuser.json` template.

```yaml
RunJsonUser:
    input_json: "/my/chosen/path.json" # We'll come back to this...
    str_var: "arg1" # Will substitute for "param1": "arg1"
    int_var: 4 # Will substitute for "param2": 4
    p3_b: 2  # Will substitute for "param3: { "b": 2 }
    val: 2 # Will substitute for "param4": [2, 2, 3] in the JSON
```