### Install sbt on Linux

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Installs sbt on Linux by downloading the Scala package, adding the sbt repository, and then installing sbt via apt-get.

```bash
wget www.scala-lang.org/files/archive/scala-2.13.7.deb
sudo dpkg -i scala-2.13.7.deb
echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt
```

--------------------------------

### Install sbt on macOS

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Installs sbt using Homebrew. The `--ignore-dependencies` flag can be used if you do not want to install the newest JDK.

```bash
brew install sbt
```

```bash
brew install --ignore-dependencies sbt
```

--------------------------------

### Dockerfile for dx-toolkit Integration

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Prepare Docker images for WDL tasks that need to access platform resources by including dx-toolkit. This example uses Ubuntu 22.04 and installs necessary packages including dxpy.

```dockerfile
FROM ubuntu:22.04
ENV DEBIAN_FRONTEND noninteractive
RUN apt-get update && \
    apt-get install -y ... jq default-jdk vim && \
    pip3 install dxpy
```

--------------------------------

### WDL Scatter Syntax Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/CWL_v1.2.0_to_WDL_v1.md

Demonstrates the WDL syntax for scattering over a list of inputs, mapping them to a called step. This corresponds to CWL's 'scatter' functionality.

```wdl
scatter(cwl_scatter_list[0] as scatter0, .. cwl_scatter_list(N) as scatterN) {

  call step_id { input: non_scattered_inputs_spec, cwl_scatter_list[0] = scatter0, .. 
                         
                         cwl_scatter_list[N] = scatterN}
}

```

--------------------------------

### Install dx-cwl-runner with Poetry

Source: https://github.com/dnanexus/dxcompiler/blob/develop/cwl_runner/README.md

Use this command to install the dx-cwl-runner package within the cwl_runner folder using Poetry.

```bash
$ poetry install
```

--------------------------------

### Install OpenJDK 11 on Linux

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Installs the JRE headless version of OpenJDK 11 on Ubuntu 16.04.

```bash
sudo apt install openjdk-11-jre-headless
```

--------------------------------

### Install OpenJDK 11 on macOS

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Installs OpenJDK 11 using Homebrew and sets the JAVA_HOME environment variable. Ensure you use the correct path output by `/usr/libexec/java_home -V`.

```bash
brew install openjdk@11
# Use java_home to find the location of JAVA_HOME to set
/usr/libexec/java_home -V
export JAVA_HOME=/Library/Java/...
```

--------------------------------

### Example dx build and dx publish for Global Workflow

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Demonstrates building a global workflow with a specified resource project in a particular region and then publishing it. Replace placeholders with actual IDs and versions.

```shell
dx build --globalworkflow \
  --from project-xxxx:workflow-xxxx --version 0.0.2 \
  --bill-to org-dnanexus \
  --extra-args '{"regionalOptions": {"aws:us-east-1": { "resources": "project-xxxx"}}}'
dx publish test_workflow/0.0.2
```

--------------------------------

### WDL Directory Input Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Specifies a directory input for a WDL task using a standard JSON input file. The URI must include a trailing slash.

```json
{
  "mytask.dir": "dx://project-xxx:/path/to/folder/"
}
```

--------------------------------

### Install test dependencies with pip and poetry

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Installs dependencies for running tests, including cwltool and dependencies for integration tests, using pip and poetry.

```bash
cd cwl_runner && pip3 install poetry && poetry install && cd ..
```

```bash
cd dxcint && pip3 install poetry && poetry install && cd ..
```

```bash
cd scripts && pip3 install -r requirements.txt && cd ..
```

--------------------------------

### WDL Workflow Definition

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Example WDL workflow definition used to illustrate manifest creation.

```wdl
version 1.1

task t1 {
  input {
    File f
  }
  command <<<
    echo "t1: "  >>out
    cat "~{f}" >>out
  >>>
  output {
     File t1_out = "out"
  }
}

workflow test {
  input {
    String s
    File f
  }
  call t1 {    input:      f = f  }
  output {
    File wf_out = t1.t1_out
  }
}
```

--------------------------------

### Intermediate Manifest Output Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Format of an intermediate manifest file passed between workflow stages. This is for informational purposes only.

```json
{
  "encoded": false,
  "id": "stage-common",
  "values": {
    "s": "hello",
    "f": "dx://project-aaa:file-xxx"
  }
}
```

--------------------------------

### Compile WDL with Extra Options

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Use the -extras flag to provide a JSON file that overrides metadata and runtime attributes during compilation. This example adds a default Docker image to all tasks.

```json
{
    "defaultRuntimeAttributes" : {
      "docker" : "quay.io/encode-dcc/atac-seq-pipeline:v1"
    }
}
```

```bash
java -jar dxCompiler-xxx.jar compile test/files.wdl -project project-xxxx -defaults test/files_input.json -extras extraOptions.json
```

--------------------------------

### Unsupported Instance Override Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Illustrates dynamic calculation of memory, which is not supported for instance overrides when the workflow is compiled with `-instanceTypeSelection dynamic`. Use explicit instance names instead.

```wdl
...
Int calc_mem = 30+15
runtime {
  memory: "~{calc_mem} GB"
}
...
```

--------------------------------

### Install sbpack and Pack CWL Workflow

Source: https://github.com/dnanexus/dxcompiler/blob/develop/README.md

Installs the sbpack package and uses the cwlpack command to combine a top-level CWL workflow and its dependencies into a single JSON file. The --add-ids flag ensures all components have unique identifiers.

```bash
$ pip3 install sbpack
$ cwlpack --add-ids --json bam_chrom_counter.cwl > bam_chrom_counter.cwl.json
```

--------------------------------

### WDL Workflow with Top-Level Calls as Stages

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Example WDL workflow demonstrating how top-level calls without subexpressions are compiled directly into dx:workflow stages. Missing arguments for these calls become compulsory stage inputs.

```wdl
workflow foo {
    String username
    Boolean flag

    call add
    call concat {input: x="hello", y="_" + username }

    if (flag) {
        call check {input:  factor = 1 }
    }
}

task add {
    Int a
    Int b
    command {}
    output { Int result = a + b }
}

task concat {
   String s1
   String s2
   command {}
   output { String result = s1 + s2 }
}

task check {
   Int factor = 3
   ...
}
```

--------------------------------

### Complex CWL File Input Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

A more complex CWL file input example, specifying class, basename, and contents directly. This is transformed into a DNAnexus input with the '___' key.

```json
{
  "myapp.myfile": {
    "class": "File",
    "basename": "foo.txt",
    "contents": "This goes into the file"
  }
}
```

--------------------------------

### Workflow Output Definition (Correct)

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This example demonstrates the correct way to define workflow outputs by assigning the result to a variable. This avoids the issue of partial definitions being discarded.

```wdl
workflow foo {
  call add { ... }
  output {
     Int r = add.result
  }
}
```

--------------------------------

### WDL JSON Input Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Specifies file inputs for a WDL workflow using Cromwell's JSON format. Ensure project IDs are included in dx URIs for faster execution.

```json
{
  "files.f": "dx://project-aaaa:file-wwww",
  "files.f1": "dx://project-aaaa:file-xxxx",
  "files.f2": "dx://project-aaaa:file-yyyy",
  "files.fruit_list": "dx://project-aaaa:file-zzzz"
}
```

--------------------------------

### Manifest JSON Input Format

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Example of a manifest JSON file for providing inputs to a workflow. This format is recommended for workflow inputs.

```json
{
  "test.input_manifest___": {
    "s": "hello",
    "f": "dx://project-aaa:file-xxx"
  }
}
```

--------------------------------

### Compiling with dxWDL Docker Script

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This command demonstrates how to use the dxwdl script, which runs the compiler within a Docker container, to compile a WDL file. This is useful for running the compiler without installing dependencies locally.

```bash
$ runc.sh compile /path/to/foo.wdl -project project-xxxx
```

--------------------------------

### WDL Struct Field Access Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Demonstrates correct usage of struct fields as call inputs to avoid runtime errors. Ensure the struct definition and field access are correctly formatted.

```wdl
struct MyStruct {
  String s
}
workflow wf {
  input {
    MyStruct my
  }
  call mytask { input: s = my.s }
}
```

--------------------------------

### Run Local Bearer-Auth Import Test

Source: https://github.com/dnanexus/dxcompiler/blob/develop/dxcint/README.md

This command runs the local-only bearer-auth import test without requiring a full DX platform setup.

```bash
Usage: dxcint local-bearer-auth [OPTIONS] DXC_REPOSITORY_ROOT

  Run the local-only bearer-auth import test without DX platform setup.
```

--------------------------------

### WDL Task and Workflow Definition

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Defines a simple WDL task 'add' and a workflow 'math' that calls this task. This example illustrates how to define tasks and workflows in WDL.

```wdl
task add {
    Int a
    Int b
    output {
        Int result = a + b
    }
}

workflow math {
    call add { input: a = 3 }
    output {
       add.result
    }
}
```

--------------------------------

### Simple CWL File Input Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

A simple CWL file input in standard JSON format, which is transformed into a DNAnexus input with a special '___' key for CWL File types.

```json
{
  "myapp.myfile": "dx://project-xxx:/path/to/file"
}
```

--------------------------------

### Extras Configuration (extras.json)

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/CustomReorgAppletExample.md

An example of an 'extras.json' file, which is provided to the DNAnexus CLI during applet or job execution. It maps a custom configuration name ('custom-reorg') to an applet ID and a configuration file path.

```json
{
  "custom-reorg" : {
    "app_id" : "applet-12345678910",
    "conf" : "dx://file-xxxxxxxx"
  }
}
```

--------------------------------

### Handle Unexecuted Calls in dxWDL Workflows

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This example demonstrates a fix for a bug where accessing results of unexecuted calls in dxWDL workflows, such as in a 'path_not_taken' scenario, would incorrectly raise an exception.

```wdl
version 1.0

workflow path_not_taken {
    if (false) {
        call compare
    }
    output {
        Boolean? equality = compare.equality
    }
}

task compare {
    command {}
    output {
        Boolean equality = true
    }
}
```

--------------------------------

### Workflow Output Definition (Incorrect)

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This example shows an incorrect way to define workflow outputs, where a partial definition is used. Partial definitions are discarded during compilation.

```wdl
workflow foo {
  call add { ... }
  output {
     add.result
  }
}
```

--------------------------------

### AWS Credentials for ECR

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Example AWS credentials file format used for ECR registry configuration. Ensure the 'dx://myproj:/aws_credentials' path points to a file containing these details.

```ini
[default]
aws_access_key_id: AKI123ABCDEFT1234567
aws_secret_access_key: ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789/ABC
```

--------------------------------

### dxWDL Warning Messages for Unset Arguments

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This example shows the warning messages that were removed in dxWDL version 1.12 for variables that cannot be set from the inputs file. Top-level call arguments can now be set.

```text
Argument unify.contig_shards, is not treated as an input, it cannot be set
Argument etl.shards, is not treated as an input, it cannot be set
Argument seq.iter_compare, is not treated as an input, it cannot be set
```

--------------------------------

### Configure Network Access for Symbolic Links

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

When using DNAx symbolic links as input files, allow network access in the `extras.json` file. This example sets the timeout policy to 8 hours and enables network access for all URLs.

```json
{
  "default_task_dx_attributes" : {
    "runSpec": {
      "timeoutPolicy": {
        "*": {
          "hours": 8
        }
      },
      "access" : {
        "network": [
          "*"
        ]
      }
    }
  }
}
```

--------------------------------

### Set Runtime Memory in WDL Task

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Example of setting the runtime memory for a WDL task. This addresses an issue with type coercion when memory is specified as a string in the development version of WDL.

```wdl
task {
  ...
  runtime {
    memory: "1 GiB"
  }
}
```

--------------------------------

### Workflow and Task Input Type Mismatch Example

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This WDL code illustrates a scenario where a workflow calls a task, and both have an input with the same name but different types. This was a bug fixed in version 0.69.

```wdl
workflow w {
    Array[String] fruit = ["Banana", "Apple"]
    scatter (index in indices) {
        call PTAsays {
            input: fruit = fruit[index], y = " is good to eat"
        }
        call Add { input:  a = 2, b = 4 }
    }
}

task PTAsays {
    String fruit
    String y
     ...
}
```

--------------------------------

### Run Hello World Workflow

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/demo/demo.md

Execute the compiled 'hello_world' workflow on the DNAnexus platform. Provide input parameters 'ia' and 'ib'.

```bash
dx run hello_world -ia=1 -ib=2
```

--------------------------------

### Display Docker Run Help

Source: https://github.com/dnanexus/dxcompiler/blob/develop/scripts/docker_image/README.md

Run the docker_run.sh script with the -help flag to see available options and usage instructions.

```bash
./docker_run.sh -help
```

--------------------------------

### Compile Hello World WDL

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/demo/demo.md

Compile a simple 'hello_world' WDL file. Use the -verbose flag for detailed compiler output.

```bash
java -jar dxCompiler-2.0.0.jar compile hello_world -verbose
```

```text
Preprocessing pass
simplifying scatters
simplifying workflow top level
Wrote simplified WDL to /tmp/dxCompiler_Compile/hello_world.simplified.wdl
FrontEnd pass
Accessed tasks = Set(AddNumbers)
FrontEnd: compiling tasks into dx:applets
Compiling task AddNumbers
FrontEnd: compiling workflow
Compiling common applet hello_world_common
Wrote intermediate representation to /tmp/dxCompiler_Compile/hello_world.ir.yaml
Backend pass
Compiling applet AddNumbers
Building applet AddNumbers
dx build /tmp/dxCompiler_Compile/AddNumbers --destination project-xxxx:/AddNumbers
Applet AddNumbers = applet-yyyy
Compiling applet hello_world_common
Building applet hello_world_common
dx build /tmp/dxCompiler_Compile/hello_world_common --destination project-xxxx:/hello_world_common
Applet hello_world_common = applet-zzzz
Stage hello_world_common = stage-uuuu
Stage AddNumbers = stage-vvvv
workflow-xxxx
```

--------------------------------

### Run dxCompiler Unit Tests (Full)

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Execute `sbt test` to run the complete suite of unit tests. This is recommended after checking out code or before pushing changes.

```shell
sbt test
```

--------------------------------

### Install and Run cwl-upgrader

Source: https://github.com/dnanexus/dxcompiler/blob/develop/README.md

Installs the cwl-upgrader Python package and uses it to upgrade CWL files from v1.0 to v1.2. Ensure you are in the correct directory before running the upgrade command.

```bash
$ pip3 install cwl-upgrader

# upgrade all dependent CWL files, which will be saved in the current working directory
$ cd contrib/beginner_example
$ cwl-upgrader cwl_v1.0/bam_chrom_counter.cwl cwl_v1.0/slice_bam.cwl cwl_v1.0/count_bam.cwl
```

--------------------------------

### Call Native App using WDL `development` Hints

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

In WDL version `development` (2.0), use the `hints` section to specify native apps instead of the `runtime` section. This is the recommended approach for newer WDL versions.

```wdl
version development
  
task concat {
  ... 
    
  hints {
    dnanexus: {
      "app": {
        "name": "concat_native/1.0.0",
        "type": "app"
      }
    }
  }
}
```

--------------------------------

### Running a Workflow with Integer Input

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Demonstrates the simplified command-line syntax for providing integer inputs to a workflow. This allows for easy parameterization of workflow runs.

```bash
dx run foo -iN=19
```

--------------------------------

### Download dxCompiler JAR

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/TestCompilerInYourEnvironment.md

Download the dxCompiler JAR file to your mini environment. Ensure the version matches the asset version you uploaded.

```bash
DXCOMPILER_VERSION=2.10.0
wget https://github.com/dnanexus/dxCompiler/releases/download/$DXCOMPILER_VERSION/dxCompiler-$DXCOMPILER_VERSION.jar
```

--------------------------------

### Get Applet Sources Without Resources

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Use the 'dx get --omit-resources' command to download applet source code without its associated large resource files. This is useful for inspecting the applet's structure.

```shell
dx get --omit-resources <applet path>
```

--------------------------------

### Run dxCompiler Unit Tests (Quick)

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Execute `sbt testQuick` to run only the tests that failed previously or tests for recently modified code. This is generally faster than running all tests.

```shell
sbt testQuick
```

--------------------------------

### WDL Task Definition

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Internals.md

Example of a WDL task definition for counting BAM file entries.

```wdl
version 1.0

task count_bam {
  input {
    File bam
  }
  command <<< 
    samtools view -c ${bam}
  >>>
  runtime {
    docker: "quay.io/ucsc_cgl/samtools"
  }
  output {
    Int count = read_int(stdout())
  }
}
```

--------------------------------

### Build and run Docker container

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Builds the Docker image using the provided Dockerfile and runs a Docker container. This should be executed from the ./docker directory.

```bash
make
```

--------------------------------

### Running dxWDL Docker Compiler

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

This command shows how to set the dxWDL version and then use the run-dxwdl-docker script to compile a WDL file. Ensure Docker is runnable without sudo if your system is configured that way.

```bash
$ export DXWDL_VERSION=0.79.1
$ sudo run-dxwdl-docker compile /path/to/foo.wdl -project project-xxx
```

--------------------------------

### Export GITHUB_TOKEN environment variable

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Sets the GITHUB_TOKEN environment variable, which is required by the sbt-github-packages plugin. This example shows how to add it to your .profile.

```bash
export GITHUB_TOKEN=<your personal access token>
```

--------------------------------

### List Available Suites

Source: https://github.com/dnanexus/dxcompiler/blob/develop/dxcint/README.md

Retrieve a hash of available test suites. The output shows suite names and their corresponding configuration file names within dxcint/config.

```bash
Usage: dxcint suites [OPTIONS]
```

```bash
Available suites are:
{"Suite name": "config file name in the dxcint/config where test names are registered within categories"}
```

--------------------------------

### Run DNAnexus Workflow

Source: https://github.com/dnanexus/dxcompiler/blob/develop/README.md

Executes a compiled DNAnexus workflow. This command assumes the workflow has already been compiled and provides an example of how to link an input file.

```bash
$ dx run bam_chrom_counter -istage-common.bam=project-BQbJpBj0bvygyQxgQ1800Jkk:file-FpQKQk00FgkGV3Vb3jJ8xqGV
```

--------------------------------

### Create New Project for dxCompiler Assets

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Add_New_Region.md

Use this command to create a new project in a specified region, intended for storing WDL and CWL assets. Ensure you replace `<PROJECT_NAME>` and `<REGION>` with appropriate values.

```bash
dx new project <PROJECT_NAME> --region <REGION> --bill-to=org-dnanexus_apps
```

--------------------------------

### Extract Source Code Script

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Internals.md

Use this script to extract workflow source code from compiled workflow or task applets. Ensure 'base64', 'jq', and 'gzip' are installed.

```bash
scripts/extract_source_code.sh
```

--------------------------------

### Configure Proxy with User and Password for dxWDL

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Configure HTTPS proxy settings by exporting the HTTPS_PROXY environment variable with user credentials and proxy details.

```bash
$ export HTTPS_PROXY = https://john_smith:welcome1@proxy.acme.com:8080
```

--------------------------------

### Run local bearer-auth import test

Source: https://github.com/dnanexus/dxcompiler/blob/develop/dxcint/README.md

Execute the bearer-auth import test without requiring DX authentication or project setup. This is useful for isolated testing of authentication mechanisms.

```bash
dxcint local-bearer-auth ${ROOT_DIR_OF_DXCOMPILER_REPO}
```

--------------------------------

### Build and Publish Global Workflow

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Build a global workflow from a specified project and workflow ID, then publish it with a version. Ensure you specify the correct bill-to account.

```shell
dx build --globalworkflow --from <project id>:<workflow id> --version <version> --bill-to <user-xxxx | org-yyyy>
```

```shell
dx publish globalworkflow-<workflow name>/<version>
```

--------------------------------

### Make dx-download-agent executable

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Makes the dx-download-agent binary executable for the current user. This is necessary after downloading and renaming the binary.

```bash
sudo chmod +x dx-download-agent
```

--------------------------------

### Runtime Configuration for GPU Instances

Source: https://github.com/dnanexus/dxcompiler/blob/develop/RELEASE_NOTES.md

Configure runtime attributes to request instances with GPU chipsets. Set the 'gpu' attribute to true within the runtime block.

```wdl
runtime {
   memory: "4 GB"
   cpu : 4
   gpu : true
}
```

--------------------------------

### Compile WDL Workflow with Docker

Source: https://github.com/dnanexus/dxcompiler/blob/develop/scripts/docker_image/README.md

Use the docker_run.sh script to compile a WDL source file. The local working directory is mounted into the container for this operation. Provide the 'compile' command followed by the path to your WDL source file.

```bash
# General example
<path to docker_run.sh> compile <path to WDL source>

# From dxCompiler repo directory
./scripts/docker_image/docker_run.sh compile test/single_tasks/add3.wdl
```

--------------------------------

### Running Workflow with Manifest Input

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/ExpertOptions.md

Command to run a workflow using a generated dxCompiler manifest file as input.

```bash
dx run workflow-yyy -f mymanifest.dx.json
```

--------------------------------

### Upload Runtime Executor Asset to Mini Environment

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/TestCompilerInYourEnvironment.md

Upload the downloaded dxCompiler runtime executor asset to your dedicated DNAnexus mini environment. This involves creating a project, a folder, uploading the asset, and creating a record pointing to it.

```bash
#!/usr/bin/env bash

set -e

# Set the same environment
LANGUAGE=WDL
PROJECT_NAME=dxCompiler

# Create project and folder
echo "Creating the project for dxCompiler $DXCOMPILER_VERSION asset"
DXCOMPILER_PROJECT_ID=$(dx new project --region aws:us-east-1 ${PROJECT_NAME} --brief)
dx mkdir -p ${DXCOMPILER_PROJECT_ID}:/releases/$DXCOMPILER_VERSION

# Upload the asset
echo "Uplading the asset dx${LANGUAGE}rt.tar.gz.."
ASSET_FILE_ID=$(dx upload dx${LANGUAGE}rt.tar.gz --brief --visibility hidden --destination ${PROJECT_NAME}:/releases/$DXCOMPILER_VERSION/dx${LANGUAGE}rt.tar.gz)

# Create a record pointing to the asset
echo "Creating the record.."
dx new record \
     --type AssetBundle \
     --details '{"archiveFileId": {"$dnanexus_link": "'"${ASSET_FILE_ID}"ímav"}}' \
     --property title="dx${LANGUAGE} asset" \
     --property description="Prerequisites for running ${LANGUAGE} workflows compiled to the platform" \
     --property version="${DXCOMPILER_VERSION}" \
     --property distribution="Ubuntu" \
     --property release="20.04" \
     --property runSpecVersion="0" \
     --close \
     ${PROJECT_NAME}:/releases/${DXCOMPILER_VERSION}/dx${LANGUAGE}rt

echo "Done."
```

--------------------------------

### Compile GATK Workflow with Inputs

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/demo/demo.md

Compile a GATK best practices WDL workflow, providing both the WDL file and an input JSON file.

```bash
java -jar dxCompiler-2.0.0.jar compile test/gatk_170412.wdl -inputs test/gatk_170412_exome_input.json
```

--------------------------------

### Run dxcint integration tests

Source: https://github.com/dnanexus/dxcompiler/blob/develop/dxcint/README.md

Execute integration tests by specifying the root directory of the dxcompiler repository and the name of a single test or a test suite.

```bash
dxcint integration ${ROOT_DIR_OF_DXCOMPILER_REPO} --test_name ${TEST_NAME}
```

--------------------------------

### Workflow with Toplevel Calls as Stages

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/MissingCallArguments.md

This WDL workflow demonstrates a common use case where leaving task arguments unbound is desirable. It shows how a complex workflow can be structured to avoid an unreadable script and potential argument name collisions by compiling toplevel calls as stages.

```wdl
workflow detect_virus {
  File raw_reads_unmapped_bam

  call deplete_taxa {
    input:
      raw_reads_unmapped_bam = raw_reads_unmapped_bam,
      bmtaggerDbs = bmtaggerDbs,
      blastDbs = blastDbs,
      bwaDbs = bwaDbs
  }

  call filter_to_taxon {
    input:
      reads_unmapped_bam = deplete_taxa.cleaned_bam
  }

  call assembly.assemble {
   input:
     reads_unmapped_bam = filter_to_taxon.taxfilt_bam,
     trim_clip_db = trim_clip_db
  }

  ...
}

task deplete_taxa {
  File         raw_reads_unmapped_bam
  Array[File]? bmtaggerDbs  # .tar.gz, .tgz, .tar.bz2, .tar.lz4, .fasta, or .fasta.gz
  Array[File]? blastDbs  # .tar.gz, .tgz, .tar.bz2, .tar.lz4, .fasta, or .fasta.gz
  Array[File]? bwaDbs  # .tar.gz, .tgz, .tar.bz2, .tar.lz4, .fasta, or .fasta.gz
  Int?         query_chunk_size
  Boolean?     clear_tags = false
  String? tags_to_clear_space_separated = "XT X0 X1 XA AM SM BQ CT XN OP"

  ...
}

task filter_to_taxon {
  File reads_unmapped_bam
  File lastal_db_fasta
  String bam_basename = basename(basename(reads_unmapped_bam, ".bam"), ".cleaned")
  ...
}

task assemble {
  File    reads_unmapped_bam
  File    trim_clip_db
  Int?    trinity_n_reads=250000
  Int?    spades_n_reads=10000000
  String? assembler="trinity"  # trinity, spades, or trinity-spades
  String  cleaned_assembler = select_first([assembler, ""])
  String  sample_name = basename(basename(reads_unmapped_bam, ".bam"), ".taxfilt")
  ...
}

```

--------------------------------

### Workflow Demonstrating Reduced Expression Evaluation Batching

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/MissingCallArguments.md

This WDL workflow highlights a downside of compiling toplevel calls as stages: reduced opportunities for batching expression evaluation. In this example, the calculation of 'sz' and the call to 'allele_freq' would normally be combined into one job, but with the feature enabled, they require separate jobs.

```wdl
workflow bar {
   File config
   Float overhead_factor

   call preamble

   Int sz = round(size(config) * overhead_factor)
   call allele_freq { disk_space = sz }
}

```

--------------------------------

### Login to DNAnexus Staging Environment

Source: https://github.com/dnanexus/dxcompiler/blob/develop/doc/Developing.md

Log in to the DNAnexus staging environment without selecting a default project. Use your staging token and save the credentials.

```shell
dx login --staging --noprojects --save --token=<your_staging_token_value>
```