### Example Resources Configuration in JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/getting-started.md

This JSON snippet specifies the computing resources required for a job. It includes the number of nodes, CPU and GPU cores per node, the target queue name on the HPC system, and the group size for task aggregation.

```json
{
  "number_node": 1,
  "cpu_per_node": 4,
  "gpu_per_node": 1,
  "queue_name": "GPUV100",
  "group_size": 5
}
```

--------------------------------

### Example Task Configuration in JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/getting-started.md

This JSON snippet defines a single task's configuration, including the command to execute, the task's working directory, files to forward to and backward from the remote system, and the names for standard output and error log files.

```json
{
  "command": "lmp -i input.lammps",
  "task_work_path": "bct-0/",
  "forward_files": ["conf.lmp", "input.lammps"],
  "backward_files": ["log.lammps"],
  "outlog": "log",
  "errlog": "err"
}
```

--------------------------------

### Installing DPDispatcher via pip (Basic)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/README.md

This command installs the core DPDispatcher package using pip, the Python package installer. It's the standard way to get started with DPDispatcher without any specific HPC backend support beyond the default.

```bash
pip install dpdispatcher
```

--------------------------------

### Submitting Multiple Tasks with DPDispatcher in Python

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/getting-started.md

This snippet demonstrates the end-to-end process of submitting multiple tasks using DPDispatcher. It involves loading machine and resource configurations from JSON files, defining individual tasks (some from JSON, others programmatically), aggregating them into a Submission object, and executing the submission on an HPC system.

```python
from dpdispatcher import Machine, Resources, Task, Submission

machine = Machine.load_from_json("machine.json")
resources = Resources.load_from_json("resources.json")

task0 = Task.load_from_json("task.json")

task1 = Task(
    command="cat example.txt",
    task_work_path="dir1/",
    forward_files=["example.txt"],
    backward_files=["out.txt"],
    outlog="out.txt",
)
task2 = Task(
    command="cat example.txt",
    task_work_path="dir2/",
    forward_files=["example.txt"],
    backward_files=["out.txt"],
    outlog="out.txt",
)
task3 = Task(
    command="cat example.txt",
    task_work_path="dir3/",
    forward_files=["example.txt"],
    backward_files=["out.txt"],
    outlog="out.txt",
)
task4 = Task(
    command="cat example.txt",
    task_work_path="dir4/",
    forward_files=["example.txt"],
    backward_files=["out.txt"],
    outlog="out.txt",
)

task_list = [task0, task1, task2, task3, task4]

submission = Submission(
    work_base="lammps_md_300K_5GPa/",
    machine=machine,
    resources=resources,
    task_list=task_list,
    forward_common_files=["graph.pb"],
    backward_common_files=[],
)

submission.run_submission()
```

--------------------------------

### Example Machine Configuration in JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/getting-started.md

This JSON snippet defines the configuration for a computing machine, specifying the batch system type (Slurm), context type (SSHContext), local and remote root directories for work, and SSH connection details including hostname, username, port, and timeout.

```json
{
  "batch_type": "Slurm",
  "context_type": "SSHContext",
  "local_root": "/home/user123/workplace/22_new_project/",
  "remote_root": "/home/user123/dpdispatcher_work_dir/",
  "remote_profile": {
    "hostname": "39.106.xx.xxx",
    "username": "user123",
    "port": 22,
    "timeout": 10
  }
}
```

--------------------------------

### Configuring Complex Resources with DPDispatcher in Python

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/getting-started.md

This Python snippet demonstrates how to define advanced resource requirements using the `Resources` class. It includes parameters for node and core counts, queue name, custom SLURM flags, CUDA device handling, module loading/unloading, environment sourcing, and explicit environment variable exports.

```python
resources = Resources(
    number_node=1,
    cpu_per_node=4,
    gpu_per_node=2,
    queue_name="GPU_2080Ti",
    group_size=4,
    custom_flags=["#SBATCH --nice=100", "#SBATCH --time=24:00:00"],
    strategy={
        # used when you want to add CUDA_VISIBLE_DIVECES automatically
        "if_cuda_multi_devices": True
    },
    para_deg=1,
    # will unload these modules before running tasks
    module_unload_list=["singularity"],
    # will load these modules before running tasks
    module_list=["singularity/3.0.0"],
    # will source the environment files before running tasks
    source_list=["./slurm_test.env"],
    # the envs option is used to export environment variables
    # And it will generate a line like below.
    # export DP_DISPATCHER_EXPORT=test_foo_bar_baz
    envs={"DP_DISPATCHER_EXPORT": "test_foo_bar_baz"},
)
```

--------------------------------

### Installing DPDispatcher with Bohrium Support (Bash)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/install.md

This command installs DPDispatcher along with its specific dependencies required for Bohrium integration. The `[bohrium]` extra ensures that all necessary components for Bohrium functionality are included.

```Bash
pip install dpdispatcher[bohrium]
```

--------------------------------

### Installing DPDispatcher with Bohrium Support via pip

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/README.md

This command installs DPDispatcher along with additional dependencies required for Bohrium support. Bohrium is a specific HPC platform, and this 'extra' installation ensures compatibility and necessary modules are included.

```bash
pip install dpdispatcher[bohrium]
```

--------------------------------

### Installing DPDispatcher Core Library (Bash)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/install.md

This command installs the base DPDispatcher library using the pip package manager. It retrieves the latest stable version from PyPI, providing the core functionalities.

```Bash
pip install dpdispatcher
```

--------------------------------

### Example Yarn Container Execution Script (Bash)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/dpdispatcher_on_yarn.md

This Bash script is an example of what is executed within a Yarn container. It sets environment variables, downloads and extracts forward files from HDFS, executes the main job (e.g., mpirun vasp_std), and uploads the result files back to HDFS. It also includes logic to check for job completion tags on HDFS.

```Bash
#!/bin/bash

## set envionment variables
source /opt/intel/oneapi/setvars.sh

## download the tar file from hdfs which contains forward files
if ! ls uuid_upload_*.tgz 1>/dev/null 2>&1; then
    hadoop fs -get /root/uuid/uuid_upload_*.tgz .
fi
for tgz_file in `ls *.tgz`; do tar xvf $tgz_file; done

## check whether the task has finished successfully
hadoop fs -test -e /root/uuid/sys-0001-0015/tag_0_finished
{ if [ ! $? -eq 0 ] ;then
  cur_dir=`pwd`
  cd t sys-0001-0015
  test $? -ne 0 && exit 1

  ## do your job here
  mpirun -n 32 vasp_std  1>> log 2>> err

  if test $? -ne 0; then
      exit 1
  else
      hadoop fs -touchz /root/uuid/sys-0001-0015/tag_0_finished
  fi
  cd $cur_dir
  test $? -ne 0 && exit 1
fi }&

wait

## upload result files to hdfs
tar czf uuid_download.tar.gz sys-0001-0015
hadoop fs -put -f uuid_download.tar.gz /root/uuid/sys-0001-0015
```

--------------------------------

### Example Python Script with PEP 723 Metadata (Python)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/run.md

This Python script illustrates the required structure for scripts intended to be run by `dpdisp run`. It includes inline script metadata compliant with PEP 723, specifically defining entries under `tool.dpdispatcher` to configure project name, task name, and the command to be executed.

```Python
# /// script
# requires-python = ">=3.8"
# dependencies = [
#   "dpdispatcher",
# ]
# ///
# /// script
# [tool.dpdispatcher]
# project_name = "my_dp_project"
# task_name = "my_dp_task"
# command = "python my_main_logic.py"
# # Additional DPDispatcher configurations can be added here
# # For example:
# # group_size = 10
# # resource_type = "cpu"
# # num_nodes = 1
# # num_cores = 4
# # queue = "normal"
# # ///

import sys

def my_main_logic():
    print(f"Hello from DPDispatcher script! Python version: {sys.version}")
    # Your main script logic goes here

if __name__ == "__main__":
    my_main_logic()
```

--------------------------------

### Configuring DistributedShell Job with HDFSContext (JSON)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/dpdispatcher_on_yarn.md

This JSON snippet provides an example configuration for a distributed job. It specifies the command to run, defines the machine's batch and context types (DistributedShell and HDFSContext), and details resource requirements such as node count, CPU/GPU per node, queue name, and environment variables for a Hadoop/YARN-based execution.

```JSON
  "fp": [
    {
      "command": "mpirun -n 32 vasp_std",
      "machine": {
        "batch_type": "DistributedShell",
        "context_type": "HDFSContext",
        "local_root": "./",
        "remote_root": "hdfs://path/to/remote/root"
      },
      "resources": {
        "number_node": 1,
        "cpu_per_node": 32,
        "gpu_per_node": 0,
        "queue_name": "queue_name",
        "group_size": 1,
        "source_list": ["/opt/intel/oneapi/setvars.sh"],
        "kwargs": {
          "img_name": "",
          "mem_limit": 32,
          "yarn_path": "/path/to/yarn/jars"
        },
        "envs" : {
          "HADOOP_HOME" : "${HADOOP_HOME:/path/to/hadoop/bin}",
          "CLASSPATH": "`${HADOOP_HOME}/bin/hadoop classpath --glob}`",
          "PATH": "${HADOOP_HOME}/bin:${PATH}"}
        }
      }
    }
  ]
```

--------------------------------

### Launching DP-GUI via CLI (Shell)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/machine.rst

This command launches the DP-GUI web-based tool from the command line, allowing users to load, modify, and export machine parameter input files. It provides an alternative to the online version of DP-GUI and enables local management of configuration files.

```Shell
dpdisp gui
```

--------------------------------

### Executing Python Script with DPDispatcher (Shell)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/run.md

This shell command demonstrates how to run a Python script named `script.py` using the `dpdisp run` command. The script must contain PEP 723 compliant inline metadata for DPDispatcher to process it correctly, allowing for direct execution without a separate configuration file.

```Shell
dpdisp run script.py
```

--------------------------------

### Configuring DeePMD-kit Task Parameters

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/expanse.md

This snippet outlines the parameters for a DeePMD-kit task. It defines the command to execute, specifies 'input.json' as a file to be forwarded to the remote machine, and 'graph.pb' and 'log.out' as files to be backwarded (retrieved). A common dataset file, 'data.npy', is included in 'forward_common_files' to ensure it's available across all tasks without being repeatedly forwarded.

```json
{
  "command": "dp train input.json",
  "forward_files": [
    "input.json"
  ],
  "backward_files": [
    "graph.pb",
    "log.out"
  ],
  "forward_common_files": [
    "data.npy"
  ]
}
```

--------------------------------

### Configuring Machine Batch Type (Shell) - JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/shell.md

This snippet configures the machine settings for `dpdispatcher`, specifically setting the `batch_type` to `Shell`. This is suitable for workstations without a dedicated job scheduling system, allowing tasks to be executed directly via shell commands.

```json
{
  "batch_type": "Shell"
}
```

--------------------------------

### Defining a Custom Slurm Header Template (Slurm/Shell)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/template.md

This Slurm template file defines the structure of the submission script header. It utilizes Python's `str.format` syntax (e.g., `{job_name}`) to dynamically insert resource parameters provided by DPDispatcher, such as job name, nodes, tasks per node, time, partition, output, and error paths.

```shell
#!/bin/bash
#SBATCH --job-name={job_name}
#SBATCH --nodes={nodes}
#SBATCH --ntasks-per-node={ntasks_per_node}
#SBATCH --time={time}
#SBATCH --partition={partition}
#SBATCH --output={output}
#SBATCH --error={error}
```

--------------------------------

### Configuring Gaussian 16 Task with Allowed Failure in JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/g16.md

This JSON snippet defines a `dpdispatcher` task for running Gaussian 16 single-point calculations. It demonstrates how to allow non-zero exit codes by appending `||:` to the command, preventing task retries on SCF errors. This is useful for large batches where some failures are acceptable, ensuring the task always reports a zero exit code to `dpdispatcher`.

```json
{
  "command": "g16 input.com output.log ||:",
  "task_name": "g16_sp_task",
  "input_files": [
    "input.com"
  ],
  "output_files": [
    "output.log"
  ],
  "backward_files": [
    "output.log"
  ]
}
```

--------------------------------

### Configuring Expanse Machine for DeePMD-kit

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/expanse.md

This snippet defines the machine configuration for connecting to the Expanse cluster. It specifies the hostname, port, username, and uses SSH key authentication for enhanced security. The remote root directory and work path for job execution are also defined, along with the SLURM scheduler type.

```json
{
  "hostname": "expanse.sdsc.edu",
  "port": 22,
  "username": "your_username",
  "auth": {
    "type": "ssh_key",
    "ssh_key_file": "~/.ssh/id_rsa"
  },
  "remote_root": "/expanse/lustre/projects/your_project/dpdispatcher_work",
  "work_path": "dpdispatcher_jobs",
  "scheduler": {
    "type": "slurm"
  }
}
```

--------------------------------

### Configuring Resource Allocation for GPU Workstation - JSON

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/shell.md

This snippet defines resource allocation parameters for a GPU workstation. It enables multi-device CUDA parallelism (`if_cuda_multi_devices: true`), sets the parallelism degree (`para_deg`) to 6 tasks per GPU, and ensures all tasks run within a single job by setting `group_size` to 0 (infinity).

```json
{
  "if_cuda_multi_devices": true,
  "para_deg": 6,
  "group_size": 0
}
```

--------------------------------

### Configuring Custom Slurm Header Template in DPDispatcher (JSON)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/template.md

This JSON snippet demonstrates how to configure DPDispatcher to use a custom Slurm submission script header template. The `customized_script_header_template_file` parameter within the `strategy` section points to the `template.slurm` file, which contains the actual Slurm header template.

```json
{
  "strategy": {
    "customized_script_header_template_file": "template.slurm"
  }
}
```

--------------------------------

### Defining HDFSContext Class for File Operations (Python)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/dpdispatcher_on_yarn.md

This Python class, HDFSContext, extends BaseContext to manage file operations on HDFS. It provides methods for uploading forward files and command files, downloading backward files, and checking the existence of files on HDFS, which is often used to verify job completion.

```Python
class HDFSContext(BaseContext) :
    def upload(self, job_dirs, local_up_files):
    """ upload forward files and forward command files to HDFS root dir

    Parameters
    ----------
    job_dirs : list
        the dictionary which contains the upload files
    local_up_files: list
        the file names which will be uploaded

    Returns
    -------
    none
    """
        pass

    def download(self, submission):
    """ download backward files from HDFS root dir

    Parameters
    ----------
    submission : Submission class instance
        represents a collection of tasks, such as backward file names

    Returns
    -------
    none
    """
        pass

   def check_file_exists(self, fname):
   """ check whether the given file exists, often used in checking whether the belonging job has finished

    Parameters
    ----------
    fname : string
        file name to be checked

    Returns
    -------
    status: boolean
    """
        pass
```

--------------------------------

### Defining CPU Resources for Expanse SLURM Jobs

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/examples/expanse.md

This configuration snippet specifies the computational resources to be requested from the Expanse cluster's SLURM scheduler for a CPU-only job. It requests one node with 32 CPU cores and 16 GB of memory from the 'shared' partition. The 'custom_gpu_line' is set to an empty string to explicitly avoid requesting GPU resources, as Expanse does not support the '--gres=gpu:0' command.

```json
{
  "number_node": 1,
  "cpu_per_node": 32,
  "memory_per_node": "16G",
  "partition": "shared",
  "custom_gpu_line": ""
}
```

--------------------------------

### Marking Job as Finished in HDFS (Bash)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/dpdispatcher_on_yarn.md

This Bash command creates an empty file in HDFS, serving as a flag to indicate that a specific job or process has successfully completed. It's a common pattern for signaling job status within a distributed file system.

```Bash
hadoop fs -touchz /root/uuid/uuid_tag_finished
```

--------------------------------

### Defining DistributedShell Class for Yarn Job Management (Python)

Source: https://github.com/deepmodeling/dpdispatcher/blob/master/doc/dpdispatcher_on_yarn.md

This Python class, DistributedShell, extends Machine to handle job submission and status checking on Yarn using DistributedShell. It includes methods to submit a job, check its status (returning a JobStatus), and generate the necessary shell script command for execution within a DistributedShell container.

```Python
class DistributedShell(Machine):
    def do_submit(self, job):
    """ submit th job to yarn using distributed shell

    Parameters
    ----------
    job : Job class instance
        job to be submitted

    Returns
    -------
    job_id: string
        usually a yarn application id
    """
        pass

    def check_status(self, job):
    """ check the yarn job status

    Parameters
    ----------
    job : Job class instance
        the submitted job

    Returns
    -------
    status: JobStatus
    """
        pass

    def gen_script_command(self, job):
    """ Generate the shell script to be executed in DistibutedShell container

    Parameters
    ----------
    job : Job class instance
        the submitted job

    Returns
    -------
    script: string
        script command string
    """
        pass
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.