### Examples of Input Value and Schema Referencing Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Illustrates various ways to reference input values and schema properties using 'inputs', bracket notation, and relative paths ('../', '.'). ```yaml on: execute: inputs: field_1: type: number field_2: type: string default: "${{ inputs.field_1 }}" # The value of field 1 field_3: type: string default: "${{ inputs[field_1] }}" # The field_1 schema object field_4: type: string default: "${{ inputs.field_1[type] }}" # The string "number" field_5: type: string default: "${{ ../field_1 }}" # The field_1 schema object field_6: type: string default: "${{ ../field_1.type }}" # The string "number" field_7: type: string default: "${{ .type }}" # The string "string" ``` -------------------------------- ### Tooltip Usage Example Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Demonstrates how to use the 'tooltip' property to provide additional information for input fields. This assists users by offering more context or instructions when they hover over an input field. ```yaml tooltip: 'Provide a detailed description of the job' ``` -------------------------------- ### Serve Go Application on a Port Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions This snippet demonstrates how to serve a simple Go HTTP server. It defines a basic 'hello' handler and starts the server on a specified port. This is often used in conjunction with agent actions to expose services. ```go package main import ( "fmt" "net/http" ) func hello(w http.ResponseWriter, req *http.Request) { fmt.Fprintf(w, "Hello from slurm agent server!\n") } func main() { http.HandleFunc("/", hello) http.ListenAndServe(":${{ needs.main.outputs.sessionPort }}", nil) } ``` -------------------------------- ### Install Parallel Works Operator using Helm Source: https://parallelworks.com/docs/self-hosting/kubernetes-deployment This command adds the Parallel Works Helm repository and installs the operator. Ensure you replace `` with the actual URL of the Helm repository. The operator is installed into a dedicated namespace, which is created if it doesn't exist. ```bash helm repo add corerepo helm upgrade --install parallelworks-operator corerepo/operator \ --namespace pw-operator-system \ --create-namespace ``` -------------------------------- ### Displaying Job Output in Bash Script Source: https://parallelworks.com/docs/run/workflows-and-apps/running-workflows This snippet shows an example of output from a simple bash script executed as part of a workflow. It displays the start time, input arguments, the hostname of the running resource, and a confirmation message. This output is typically found in the Job logs module. ```bash Starting test workflow at Wed Jan 25 17:12:21 UTC 2023 INPUT ARGUMENTS: Running on the following computer: pw-user-demo This is a NO-OP workflow - nothing got launched remotely. Done! ``` -------------------------------- ### Scheduler Agent API Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions Provisions a node and starts an SSH server for remote command execution. Requires a cluster with a partition. ```APIDOC ## POST /websites/parallelworks/scheduler-agent ### Description You can use the `scheduler-agent` action to provision a node and start an ssh server so that you can run commands on it through ssh. Note that you must use a cluster with a partition for this to properly work. See Building Sessions for example usage. ### Method POST ### Endpoint /websites/parallelworks/scheduler-agent ### Parameters #### Inputs - **wait** (boolean) - Optional - Whether to wait for the node to provision. Defaults to true. - **scheduler-type** (string) - Optional - Use this scheduler. Defaults to slurm, options are slurm and pbs. - **scheduler-flags** (object) - Optional - Use these flags for the sbatch command. Defaults to none. You can find a list of flags here under job submission for slurm and here for pbs. - **script-headers** (string) - Optional - Additional lines to add to the top of the bash script being submitted via sbatch or qsub (depending on your scheduler). Useful if you want to allow users to add their own `#SBATCH/PBS` comment flags at will. ### Request Example ```json { "wait": false, "scheduler-type": "slurm", "scheduler-flags": { "--time": "1:00:00" } } ``` ### Response #### Success Response (200) - **agentId** (string) - The ID of the provisioned agent. - **remoteHost** (string) - The remote host address for the agent. - **sshPort** (int) - The SSH port for the agent. #### Response Example ```json { "agentId": "agent-12345", "remoteHost": "192.168.1.100", "sshPort": 22 } ``` ``` -------------------------------- ### Provision Node and Start SSH Server with Scheduler Agent Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions Provisions a compute node and initiates an SSH server, enabling command execution via SSH. This action requires a cluster with a partition to function correctly. It offers options to wait for node provisioning and specify scheduler types and flags. ```yaml uses: parallelworks/scheduler-agent id: slurmstep with: wait: false ``` -------------------------------- ### Sample ParallelWorks Custom Resource YAML Configuration Source: https://parallelworks.com/docs/self-hosting/kubernetes-deployment This YAML configuration defines a custom resource for the ParallelWorks platform. It specifies settings for the platform's image, default scaling, ingress host, TLS certificate, and database configurations (PostgreSQL and MongoDB). This serves as a template for deploying the ParallelWorks application within a Kubernetes environment. ```yaml --- apiVersion: v1 kind: Namespace metadata: name: pw-system --- apiVersion: platform.parallelworks.com/v1alpha1 kind: ParallelWorks metadata: name: customer namespace: pw-system spec: image: registry: ghcr.io repositoryPrefix: parallelworks tag: v5.203 pullPolicy: IfNotPresent pullSecrets: - name: image-pull-secret # Default scale for all services that don't specify their own replicas # Services default to enabled=true, so you only need to specify services # if you want to customize them or disable them defaultScale: 1 # Host is the ingress hostname for the platform host: customer.parallel.works # can be your own domain # TLSSecretName for TLS certificate tlsSecretName: pw-tls-secret # Database configuration database: # PostgreSQL configuration postgres: endpoint: 'pw-pgsql-service.postgres.svc.cluster.local:5432' database: 'pworks' username: 'postgres' urlSecretRef: name: 'postgres-secret' key: 'password' # MongoDB configuration (no authentication configured) mongodb: replicaSetName: 'rs0' endpoints: - 'mongodb-0-external.mongo.svc.cluster.local:27017' - 'mongodb-1-external.mongo.svc.cluster.local:27017' - 'mongodb-2-external.mongo.svc.cluster.local:27017' ``` -------------------------------- ### Access Bucket Input Value in Expressions Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions This example demonstrates how to access specific fields of a 'bucket' input type using dot notation within Parallel Works expressions. The 'bucket' input provides an object with fields like 'uri' and 'csp' at runtime. ```expression ${{ inputs.mybucket.uri }} ${{ inputs.mybucket.csp }} ``` -------------------------------- ### Fetch Bucket Credentials for AWS Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions When `generateCredentials: true` is set for a bucket input, temporary credentials are fetched from the vault. This example shows how to access and use AWS credentials (accessKeyId, secretAccessKey, sessionToken) and bucket details (bucketName, region) within a job to sync data using the AWS CLI. ```yaml on: execute: inputs: data_bucket: type: bucket label: Data Bucket csp: aws generateCredentials: true jobs: sync: steps: - name: Sync data from bucket run: | export AWS_ACCESS_KEY_ID=${{ inputs.data_bucket.credentials.accessKeyId }} export AWS_SECRET_ACCESS_KEY=${{ inputs.data_bucket.credentials.secretAccessKey }} export AWS_SESSION_TOKEN=${{ inputs.data_bucket.credentials.sessionToken }} aws s3 sync s3://${{ inputs.data_bucket.bucketName }}/data ./local-data --region ${{ inputs.data_bucket.region }} ``` -------------------------------- ### Writing Key-Value Outputs to a File Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Shows how to create accessible outputs by writing key-value pairs to the file path specified by the `$OUTPUTS` environment variable. Each output must be on a new line in the `key=value` format. ```yaml jobs: example: steps: - name: Create outputs run: | echo PORT=3001 >> $OUTPUTS echo DATABASE_URL=postgres://localhost:5432/mydb >> $OUTPUTS echo STATUS=success >> $OUTPUTS ``` -------------------------------- ### Define Job Dependencies with 'needs' (YAML) Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields The 'needs' field specifies a list of jobs that must complete successfully before the current job can start. If a preceding job fails, dependent jobs will not execute. ```yaml jobs: first: steps: - run: echo runs first second: needs: - first steps: - run: echo runs second ``` -------------------------------- ### Verify Storage Mounts using df -h Source: https://parallelworks.com/docs/storage/attaching-storage This command verifies that storage resources are mounted correctly on a cluster. It lists all mounted filesystems and their usage. Ensure you have logged into your cluster to run this command. ```shell df -h ``` -------------------------------- ### Execute Commands Remotely via SSH Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Configures the job or specific steps to execute commands on a remote host using SSH. This includes specifying the remote host, user, and an optional jump node for more complex network setups. ```yaml jobs: echo_on_remote: ssh: remoteHost: "${{ inputs.resource1.ip }}" jumpNodeHost: "${{ inputs.resource2.ip }}" steps: - run: echo "This is executed on the remote host!" - run: echo "This is executed on the jump node!" ssh: remoteHost: "${{ inputs.resource2.ip }}" - run: echo "This is executed in the user workspace!" ssh: null on: execute: inputs: resource1: label: Resource 1 type: compute-clusters optional: false resource2: label: Resource 2 type: compute-clusters optional: false ``` -------------------------------- ### Job and Step Configuration Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields This section covers the configuration of individual steps within a job, including cleanup commands, conditional execution, retry logic, environment variables, error handling, working directories, and early cancellation. ```APIDOC ## `jobs..steps[*].cleanup` ### Description Defines a cleanup command to be run after step execution finishes. This is used to delete temporary files, terminate connections, remove credentials, or perform any other cleanup necessary when a job is shutting down. Cleanup commands always run at the end of a job in reverse order of definition, regardless of the job's success, failure, or cancellation. If a step did not run, its cleanup step will not run. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: steps: - run: echo "Running main task" cleanup: echo "Cleaning up temporary files" ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..steps[*].retry` ### Description Configures retry logic for a step that has a chance of failure but is necessary for the workflow. It allows specifying the maximum number of retries, the interval between retries, and the timeout for each attempt. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: steps: - run: potentially_failing_command retry: max-retries: 5 interval: 1m timeout: 10m ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..steps[*].ignore-errors` ### Description If set to `true`, non-zero exit codes from a step will not be counted as failures. By default, this attribute is set to `false`. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: steps: - run: command_that_might_fail_but_is_ok ignore-errors: true ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..steps[*].working-directory` ### Description Specifies the working directory for a step's execution. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: steps: - run: echo "Hello" working-directory: /app/src ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..steps[*].early-cancel` ### Description Defines conditions for early cancellation of a step. Currently, only `early-cancel: any-job-failed` is supported, which cancels the step if any job fails before the step finishes running. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: steps: - run: long_running_task early-cancel: any-job-failed ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..if` ### Description Prevents a job or step from running unless a specified condition evaluates to true. It accepts expressions like `${{ inputs.input-name }}` or `${{ always }}`. `${{ always }}` ensures execution even if dependencies fail. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: conditional_job: if: ${{ inputs.run-this-job }} steps: - run: echo "This step runs only if the condition is met." if: ${{ always }} ``` ### Response #### Success Response (200) N/A #### Response Example N/A ## `jobs..env` ### Description Sets environment variables for a job or step. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: example_job: env: MY_VARIABLE: my_value steps: - run: echo "$MY_VARIABLE" ``` ### Response #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### Publishing Job Outputs Using the Outputs Map Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Illustrates how to explicitly define and publish specific job outputs using the `outputs` map. This allows other jobs to consume these outputs by referencing them via `needs..outputs.`. This method ensures outputs are reliably available to dependent jobs. ```yaml jobs: build: outputs: version: ${{ needs.build.steps.get_version.outputs.VERSION }} artifact_url: ${{ needs.build.steps.upload.outputs.URL }} steps: - name: Get version id: get_version run: echo VERSION=1.2.3 >> $OUTPUTS - name: Upload artifact id: upload run: echo URL=https://example.com/artifact.zip >> $OUTPUTS ``` -------------------------------- ### Accessing Outputs from Named Steps Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Demonstrates how to define named steps using the `id` field and access their outputs in subsequent steps using the `needs..steps..outputs.` syntax. Outputs from named steps are not automatically published to job outputs and require explicit mapping. ```yaml jobs: main: steps: - name: Create step output id: test run: echo FOO=BAR >> $OUTPUTS - name: Access output from named step run: echo ${{ needs.main.steps.test.outputs.FOO }} # Prints BAR ``` -------------------------------- ### Expression Syntax and Evaluation Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Illustrates the fundamental syntax for expressions in Parallelworks, which must be wrapped in `${{ }}`. It shows how expressions are evaluated and the importance of whitespace for correct parsing. ```text `${{ true && 'truthy value' }}` `${{ 1 + 1.5 }}` `${{ 'a' + 'b' }}` `${{ 1 - 1.5 }}` `${{ 4 * 4 }}` `${{ 9 / 4 }}` `${{ 2 ** 4 }}` `${{ 4 // 3 }}` `${{ 'hello' ?? 'default' }}` `${{ undefined ?? 'default' }}` `${{ ['a', 'b'] get 1 }}` `${{ {"a":"c", "b":"d"} get a }}` `${{ 'ab' get 0 }}` `${{ 'a' in ['a', 'b'] }}` `${{ 'b' in {"a":"c", "b":"d"} }}` `${{ 'd' in {"a":"c", "b":"d"} }}` `${{ !false }}` `${{ 'hello' !in ['hello', 'goodbye'] }}` `${{ true && (true || true) && false }}` `${{ 10 / (2 + 8) }}` `${{ true && false ? 'a' : 'b' }}` `${{ 'truthy value' ? 'a' : 'b' }}` ``` -------------------------------- ### Wait For Agent API Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions Waits for an agent to become ready, typically used after initiating an agent with `wait: false`. ```APIDOC ## POST /websites/parallelworks/wait-for-agent ### Description This action should only be used if you already used scheduler-agent with `wait: false` passed as an input to the `with` property, which might be a good idea if you want the workflow job to perform other operations while the scheduler-agent node is configuring. ### Method POST ### Endpoint /websites/parallelworks/wait-for-agent ### Parameters #### Inputs - **agentId** (string) - Required - The id of the agent to wait for. The output from the scheduler-agent action includes this field (see usage in example above). ### Request Example ```json { "agentId": "agent-12345" } ``` ### Response #### Success Response (200) - **remoteHost** (string) - The remote host address for the agent. - **sshPort** (int) - The SSH port for the agent. #### Response Example ```json { "remoteHost": "192.168.1.100", "sshPort": 22 } ``` ``` -------------------------------- ### Update Session with Information Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions Updates an existing session with specified details. This action is crucial for initializing session information, especially when sessions lack initial display data. It supports various session types like 'link' and 'tunnel' and allows configuration of target compute clusters and session status. ```yaml uses: parallelworks/update-session with: status: running name: ${{ sessions.session }} target: ${{ inputs.resource.id }} remoteHost: ${{ needs.main.steps.waitstep.outputs.remoteHost }} remotePort: ${{ needs.main.outputs.sessionPort }} ``` -------------------------------- ### Project Configuration API Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields This section details the configuration parameters available for a ParallelWorks project, including job definitions, environment variables, and execution timeouts. ```APIDOC ## Project Configuration Details ### Description This endpoint provides access to the configuration schema for a ParallelWorks project, outlining all possible inputs for job execution, session management, and general project settings. ### Method GET ### Endpoint /websites/parallelworks ### Parameters #### Query Parameters None #### Request Body None ### Response #### Success Response (200) - **jobs** (object) - Defines the jobs that can be executed within the project. - **jobs..needs** (array) - Specifies dependencies for a job. - **jobs..steps** (array) - A list of steps to be executed within a job. - **jobs..steps[*].name** (string) - The name of a step. - **jobs..steps[*].run** (string) - The command to run for a step. - **jobs..steps[*].uses** (string) - A reusable action or script to use for a step. - **jobs..steps[*].with** (object) - Input parameters for a reusable action. - **jobs..steps[*].ssh** (boolean) - Enables SSH access for a step. - **jobs..steps[*].cleanup** (boolean) - Enables cleanup after a step. - **jobs..steps[*].if** (string) - Conditional execution for a step. - **jobs..steps[*].retry** (integer) - Number of times to retry a step. - **jobs..steps[*].env** (object) - Environment variables for a step. - **jobs..steps[*].ignore-errors** (boolean) - Ignores errors for a step. - **jobs..steps[*].working-directory** (string) - The working directory for a step. - **jobs..steps[*].early-cancel** (boolean) - Enables early cancellation for a step. - **jobs..steps[*].timeout** (integer) - Timeout for a step in seconds. - **jobs..ssh** (boolean) - Enables SSH access for a job. - **jobs..cleanup** (boolean) - Enables cleanup after a job. - **jobs..if** (string) - Conditional execution for a job. - **jobs..env** (object) - Environment variables for a job. - **jobs..working-directory** (string) - The working directory for a job. - **jobs..timeout** (integer) - Timeout for a job in seconds. - **sessions** (object) - Defines session configurations. - **sessions..type** (string) - The type of session. - **sessions..openAI** (object) - OpenAI specific configurations for a session. - **sessions..prompt-for-name** (boolean) - Prompts for session name. - **sessions..redirect** (string) - Session redirect URL. - **sessions..useTLS** (boolean) - Enables TLS for the session. - **sessions..useCustomDomain** (boolean) - Uses a custom domain for the session. - **permissions** (object) - Defines permissions for the project. - **configurations** (object) - General project configurations. - **env** (object) - Global environment variables for the project. - **timeout** (integer) - Default timeout for the project in seconds. - **on** (object) - Defines triggers for project execution. - **on.execute.inputs** (object) - Input parameters for the execute trigger. #### Response Example ```json { "jobs": {}, "sessions": {}, "permissions": {}, "configurations": {}, "env": {}, "timeout": 3600, "on": { "execute": { "inputs": {} } } } ``` ``` -------------------------------- ### Checkout Git Repository using Parallel Works Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions The 'checkout' action clones a git repository. It supports specifying the repository URL, branch, and optionally a sparse checkout for specific files or directories. It can also clone to a cluster's file system using SSH. ```yaml jobs: job-name: steps: - uses: parallelworks/checkout with: repo: https://github.com/parallelworks/interactive_session.git branch: main ``` ```yaml jobs: job-name: steps: - uses: parallelworks/checkout with: repo: https://github.com/parallelworks/interactive_session.git branch: main sparse_checkout: - utils - platforms ``` ```yaml jobs: job-name: steps: - uses: parallelworks/checkout with: repo: https://github.com/parallelworks/interactive_session.git branch: main ssh: remoteHost: ${{ inputs.cluster.ip }} on: execute: inputs: cluster: type: compute-clusters optional: false ``` -------------------------------- ### Access Input Schema Using Bracket Notation Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions An alternative method to conditionally hide an input using the 'inputs' keyword and bracket notation to access schema properties. ```yaml on: execute: inputs: ... option_1_input: type: string hidden: "${{ some long and complicated boolean expression }}" option_2_input: type: string hidden: "${{ !inputs.option_1_input[hidden] }}" ``` -------------------------------- ### Define Workflow Configurations Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Save and reuse inputs for workflows using the `configurations` field. This allows for consistent and time-saving input management. Configurations can be defined in the workflow YAML or saved by users. ```yaml configurations: config_1: variables: input_1: hello! jobs: echo: steps: - run: echo ${{ inputs.input_1 }} on: execute: inputs: input_1: type: string ``` -------------------------------- ### Reference External Workflows or Actions with 'uses' (YAML) Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields The 'uses' field allows a step to execute another workflow from the current repository, a personal workflow, or a workflow from the Marketplace. It cannot be used with the 'run' field. Versions can be specified for Marketplace workflows. ```yaml jobs: job-name: steps: - name: Sample Step uses: marketplace/default-local-workflow with: resource: sample-cluster ``` -------------------------------- ### Execute Multi-line Commands in a Step (YAML) Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields The 'run' field within a step allows for the execution of shell commands. Multiple bash commands can be combined into a single step using the multi-line format. ```yaml jobs: main: steps: - run: | echo part 1 echo part 2 ``` -------------------------------- ### Wait for Agent to be Ready Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/actions Waits for an agent to become ready, typically used after initiating the scheduler-agent with 'wait: false'. This allows the workflow to perform other tasks concurrently while the agent configures. It requires the agentId obtained from the scheduler-agent action. ```yaml uses: parallelworks/wait-for-agent id: waitstep with: agentId: ${{ needs.main.steps.slurmstep.outputs.agentId }} ``` -------------------------------- ### Dynamic Input Visibility Based on Boolean Toggle Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Configures conditional visibility for 'advanced_setting' and 'advanced_setting2' inputs. 'advanced_setting' is hidden based on the boolean value of 'show_advanced', while 'advanced_setting2' is hidden based on the 'hidden' attribute of 'advanced_setting'. ```yaml on: execute: inputs: header: type: header text: Starter Local Workflow size: 20 show_advanced: type: boolean label: Show Advanced Settings default: false advanced_setting: type: string label: Advanced Setting hidden: "${{ !inputs.show_advanced }}" advanced_setting2: type: string label: Advanced Setting 2 hidden: "${{ inputs.advanced_setting[hidden] }}" ``` -------------------------------- ### Accessing Job Outputs in Parallelworks Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Demonstrates how to access job outputs using the `needs` context. This allows jobs to share information, either with themselves or with dependent jobs, by referencing the outputs of previous steps or jobs. ```yaml jobs: job1: steps: - name: fake output run: echo PORT=3001 >> $OUTPUTS - name: Echo output from self run: echo ${{ needs.job1.outputs.PORT }} # Prints 3001 job2: needs: - job1 steps: - name: Echo output from job 1 run: echo ${{ needs.job1.outputs.PORT }} # Prints 3001 ``` -------------------------------- ### Define Cleanup Commands for Job Steps Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Specifies a command to execute after a job step finishes, regardless of its success or failure. This is useful for removing temporary files or releasing resources. Cleanup commands run in reverse order of their definition. ```yaml jobs: my_job: steps: - run: echo "Running main step" cleanup: rm -rf /tmp/myapp ``` -------------------------------- ### Define Workflow Inputs in YAML Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions This snippet shows the YAML structure for defining workflow inputs. Inputs are configured under the 'on.execute.inputs' key. Each input has a name, a type, and potentially other type-specific attributes. Input names must consist of alphanumeric characters, dashes, and underscores. ```yaml on: execute: inputs: name_of_input_1: type: type_of_input_1 ... other fields (potentially type specific) ... name-of-input-2: type: type_of_input_2 ... other fields (potentially type specific) ... ``` -------------------------------- ### Implement Early Cancellation for Steps Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Allows a step to be cancelled prematurely based on certain conditions. Currently, the only supported condition is 'any-job-failed', which cancels the step if any preceding job fails. ```yaml jobs: my_job: steps: - run: potentially_long_running_task early_cancel: any_job_failed ``` -------------------------------- ### Specify Working Directory for Steps Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Sets the directory in which a job step will be executed. This is useful for navigating to specific project directories before running commands. ```yaml jobs: my_job: steps: - run: echo "Running in specific directory" working_directory: /path/to/my/project ``` -------------------------------- ### Configure Step Retries Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields Enables automatic retries for steps that have a chance of failure. You can configure the maximum number of retries, the interval between retries, and the timeout for each attempt. ```yaml jobs: my_job: steps: - run: potentially_failing_command retry: max_retries: 5 interval: 1m timeout: 1h ``` -------------------------------- ### SSH Configuration for Remote Execution Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields This section details how to configure SSH access for executing commands on remote hosts, including specifying remote host details, usernames, and jump nodes. ```APIDOC ## `jobs..ssh` ### Description Configures SSH access for executing commands on a remote host. It allows specifying the remote host's IP address, the username for SSH connection, and optionally, the IP address and username for a jump node. ### Method N/A (Configuration Property) ### Endpoint N/A ### Parameters #### Path Parameters N/A #### Query Parameters N/A #### Request Body N/A ### Request Example ```yaml jobs: echo_on_remote: ssh: remoteHost: ${{ inputs.resource1.ip }} jumpNodeHost: ${{ inputs.resource2.ip }} steps: - run: echo "This is executed on the remote host!" - run: echo "This is executed on the jump node!" ssh: remoteHost: ${{ inputs.resource2.ip }} - run: echo "This is executed in the user workspace!" ssh: null on: execute: inputs: resource1: label: Resource 1 type: compute-clusters optional: false resource2: label: Resource 2 type: compute-clusters optional: false ``` ### Response #### Success Response (200) N/A #### Response Example N/A ``` -------------------------------- ### Define Workflow Jobs and Steps (YAML) Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/yaml-fields The 'jobs' field is essential for defining workflow execution. Each job must have 'steps', and by default, jobs run in parallel. Dependencies can be managed using the 'needs' field. ```yaml jobs: job-name: steps: - name: Step 1 run: echo step 1 - name: Step 2 run: echo step 2 ``` -------------------------------- ### Set Default Input Value Using Other Inputs Source: https://parallelworks.com/docs/run/workflows-and-apps/building-workflows/inputs-and-expressions Demonstrates setting a default value for the 'workdir' input based on the 'username' input. This uses string concatenation within the default value expression. ```yaml on: execute: inputs: username: type: string workdir: type: string default: "${{ '~/path/to/directory/' + inputs.username }}" ```