### Install Docker Prerequisites on Ubuntu

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Installs necessary packages like ca-certificates and curl, sets up the Docker GPG key, and adds the Docker repository to apt sources. This prepares the system for Docker installation.

```bash
sudo apt-get update && \
    sudo apt-get install ca-certificates curl && \
    sudo install -m 0755 -d /etc/apt/keyrings && \
    sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc && \
    sudo chmod a+r /etc/apt/keyrings/docker.asc && \
    echo \
      "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
      $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
      sudo tee /etc/apt/sources.list.d/docker.list > /dev/null && \
    sudo apt-get update
```

--------------------------------

### Define Example Arguments for GATK-SV Docker Builds (Python)

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/docker/build_docker_template.ipynb

Provides example argument lists for various Docker build scenarios in the GATK-SV project. These examples demonstrate how to configure builds from local files, remote Git repositories, and how to handle dependencies and Git protection.

```python
# example args for testing different scenarios
EXAMPLE_BUILD_TAG = "my.example"

# build from local files, and push to both Dockerhub and GCR
EXAMPLE_ARGS_LOCAL = ['--targets', 'all', 
                      '--image-tag', EXAMPLE_BUILD_TAG, 
                      '--dockerhub-root', 'shuangbroad',
                      '--gcr-project', 'broad-dsde-methods']

# build from remote (Github) files, but not pushing
EXAMPLE_ARGS_REMOTE = ['--targets', 'all', 
                       '--image-tag', EXAMPLE_BUILD_TAG, 
                       '--remote-git-hash', '72acbdce2660e7cc360aebcd71f47c4665a7002a', 
                       '--staging-dir', '/Users/shuang/Desktop/tmp',
                       '--use-ssh']

# build a single image, but
# it has dependencies that must be built first
EXAMPLE_ARGS_DEPENDENCY = ['--targets', 'sv-pipeline-rdtest', 
                           '--image-tag', EXAMPLE_BUILD_TAG]

# tells git to ignore untracked files and uncommitted changes
# (applies only when building from local files)
EXAMPLE_ARGS_TURNOFF_GIT_PROTECT = ['--targets', 'sv-pipeline-rdtest', 
                                    '--image-tag', EXAMPLE_BUILD_TAG, 
                                    '--disable-git-protect']
```

--------------------------------

### Install SVTK using pip

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/svtk/README.md

This snippet shows how to clone the gatk-sv repository and install the SVTK package using pip. It assumes you have git and pip installed.

```bash
git clone https://github.com/broadinstitute/gatk-sv.git
cd gatk-sv
pip install -e ./src/svtk
```

--------------------------------

### Install Docker Engine and Plugins on Ubuntu

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Installs the Docker CE, CLI, containerd.io, and buildx/compose plugins. It also adds the current user to the 'docker' group for easier access and applies the group changes.

```bash
sudo apt-get -y install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin && \
    sudo usermod -aG docker ${USER} && \
    newgrp docker
```

--------------------------------

### Apply SL Filter Script Example

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/modules/score_genotypes.md

Example of arguments for the apply_sl_filter.py script, which is used to adjust SL cutoff thresholds based on data quality and desired accuracy.

```python
# Example arguments for apply_sl_filter.py (refer to script for full options)
# python apply_sl_filter.py \
#   --input filtered_vcf.vcf \
#   --output filtered_vcf_filtered.vcf \
#   --sl_cutoff_table cutoff_table.txt
```

--------------------------------

### Site-Level Comparison Dataset Example

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/modules/main_vcf_qc.md

Specifies a site-level comparison dataset for benchmarking. It's an array of two-element arrays, where each inner array contains a prefix and a URI path to a directory of BED files stratified by subpopulation. The BED files should be gzipped and indexed.

```text
["gnomAD_v2_Collins", "gs://gatk-sv-resources-public/hg38/v0/sv-resources/resources/v1/gnomAD_v2_Collins"]

```

--------------------------------

### Sample-Level Comparison Dataset Example

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/modules/main_vcf_qc.md

Specifies a sample-level comparison dataset for benchmarking. It's an array of two-element arrays, where each inner array contains a prefix and a URI path to a tarball containing sample-level benchmarking data. The tarball should contain relevant files for per-sample analysis.

```text
[["HGSV_ByrskaBishop", "gs://gatk-sv-resources-public/hg38/v0/sv-resources/resources/v1/HGSV_ByrskaBishop_GATKSV_perSample.tar.gz"]]

```

--------------------------------

### Install SVtools from GitHub

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Installs the 'svtk' Python package directly from its GitHub repository using pip. This is necessary as 'svtk' is currently only available via GitHub.

```bash
$ git clone git@github.com:talkowski-lab/svtk.git
$ cd svtk
$ pip install -e .
```

--------------------------------

### Install Python Package for GATK-SV Notebook

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

Installs the 'upsetplot' Python package, which is required for generating upset plots used in data visualization within this notebook. This is a one-time installation.

```python
! pip install upsetplot
```

--------------------------------

### Install svtk Python Package

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Installs the 'svtk' Python package, which is a requirement for the SV-Adjudicator pipeline. This involves navigating to the 'svtk' directory and running the setup script.

```bash
cd svtk
python setup.py install --user
```

--------------------------------

### Install pybedtools from GitHub

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Installs the 'pybedtools' Python package from GitHub, specifically targeting the master branch or a specific commit. This is required for per-chromosome parallelization in the pipeline.

```bash
$ pip install git+git://github.com/daler/pybedtools.git@master
```

```bash
$ pip install git+git://github.com/daler/pybedtools.git@b1e0ce0
```

--------------------------------

### Python: Get Ordered Build Chain for Single and List Targets

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/docker/test_build_docker.ipynb

This snippet demonstrates how to retrieve the ordered build chain for single or multiple targets using the `Project_Build` class. It's useful for understanding build dependencies and ensuring correct build order. No external dependencies are required beyond the `build_docker` module.

```python
from build_docker import *
import unittest
import pprint

DEBUG_MODE = True

this_script_path = os.path.realpath('__file__')

TEST_BUILD_TAG = 'mytest'

# test if build chain can be constructed in desired order
print( Project_Build.get_ordered_build_chain_single('manta') )
print( Project_Build.get_ordered_build_chain_single('delly') )
print( Project_Build.get_ordered_build_chain_single('melt') )
print( Project_Build.get_ordered_build_chain_single('wham') )
print( Project_Build.get_ordered_build_chain_single('sv-base-mini') )
print()
print( Project_Build.get_ordered_build_chain_single('sv-base') )
print( Project_Build.get_ordered_build_chain_single('samtools-cloud') )
print( Project_Build.get_ordered_build_chain_single('sv-pipeline') )
print( Project_Build.get_ordered_build_chain_single('cnmops') )
print()
print( Project_Build.get_ordered_build_chain_single('sv-pipeline-rdtest') )
print( Project_Build.get_ordered_build_chain_single('sv-pipeline-qc') )


print( Project_Build.get_ordered_build_chain_list(['manta']) )
print( Project_Build.get_ordered_build_chain_list(['delly']) )
print( Project_Build.get_ordered_build_chain_list(['melt']) )
print( Project_Build.get_ordered_build_chain_list(['wham']) )
print( Project_Build.get_ordered_build_chain_list(['sv-base-mini']) )
print()
print( Project_Build.get_ordered_build_chain_list(['sv-base']) )
print( Project_Build.get_ordered_build_chain_list(['samtools-cloud']) )
print( Project_Build.get_ordered_build_chain_list(['sv-pipeline']) )
print( Project_Build.get_ordered_build_chain_list(['cnmops']) )
print()
print( Project_Build.get_ordered_build_chain_list(['sv-pipeline-rdtest']) )
print( Project_Build.get_ordered_build_chain_list(['sv-pipeline-qc']) )


print( Project_Build.get_ordered_build_chain_list(['manta','delly','melt','wham','sv-base-mini']) )
print( Project_Build.get_ordered_build_chain_list(['sv-base','samtools-cloud','sv-pipeline','cnmops']) )
print( Project_Build.get_ordered_build_chain_list(['sv-pipeline-rdtest','sv-pipeline-qc']) )
print( Project_Build.get_ordered_build_chain_list(['manta', 'cnmops', 'sv-pipeline-rdtest','sv-pipeline-qc']) )
```

--------------------------------

### Clone GATK-SV Repository

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/build_inputs.md

Clones the GATK-SV repository from GitHub. This is a prerequisite for running subsequent commands and assumes Git is installed on the system.

```shell
git clone https://github.com/broadinstitute/gatk-sv && cd gatk-sv
```

--------------------------------

### Python Imports and Attribute Maps for GATK-SV Reference Panel

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/CreateReferencePanel.ipynb

This Python code snippet imports necessary libraries and defines attribute maps for GATK-SV. The SAMPLE_KEYS_MAP maps sample table columns to GATK-SV resource identifiers, while SAMPLE_EXAMPLE_MAP provides example inputs for single-sample workflows.

```python
#####################
###### Imports ######
#####################

import argparse
import io
import json
import os
import sys
import subprocess
import zipfile

from google.cloud import storage
import firecloud.api as fapi
import pandas as pd

############################
###### Attribute maps ######
############################

# Map from sample table columns to the resource identifiers
SAMPLE_KEYS_MAP = {
    "entity:sample_id": "samples",
    "bam_or_cram_file": "bam_or_cram_files",
    "coverage_counts": "counts",
    "manta_vcf": "manta_vcfs",
    "manta_index" : "manta_vcfs_index",
    "pesr_disc": "PE_files",
    "pesr_disc_index": "PE_files_index",
    "pesr_sd": "SD_files",
    "pesr_sd_index": "SD_files_index",
    "pesr_split": "SR_files",
    "pesr_split_index": "SR_files_index",
    "scramble_vcf": "scramble_vcfs",
    "scramble_index": "scramble_vcfs_index",
    "wham_vcf": "wham_vcfs",
    "wham_index": "wham_vcfs_index"
}

# Maps arrays to special "example" inputs used for workflows that run on one sample at a time
SAMPLE_EXAMPLE_INDEX = 0  # Index of the sample to use
SAMPLE_EXAMPLE_MAP = {
    "bam_or_cram_file": "bam_or_cram_example",
    "entity:sample_id": "sample_example"
}

```

--------------------------------

### Create Ubuntu VM Instance on GCP

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Creates a new Ubuntu 23.10 (amd64) VM instance on Google Cloud Platform (GCP) using the gcloud command-line tool. This command specifies the project, zone, machine type, and disk configuration, including a 100 GiB disk size suitable for GATK-SV Docker images. It ensures the disk is automatically deleted when the instance is deleted.

```bash
gcloud compute instances create $INSTANCE_NAMES \
  --project=$PROJECT_ID \
  --zone=$ZONE_ID \
  --machine-type=e2-standard-2 \
  --create-disk=auto-delete=yes,boot=yes,device-name=$INSTANCE_NAMES,image=projects/ubuntu-os-cloud/global/images/ubuntu-2310-mantic-amd64-v20240213,mode=rw,size=100
```

--------------------------------

### Terra CLI: Install firecloud-cli

Source: https://context7.com/broadinstitute/gatk-sv/llms.txt

Installs the firecloud-cli Python package, a command-line tool for interacting with the Terra platform.

```bash
# Using firecloud-cli (fissfc) to submit workflows
pip install firecloud
```

--------------------------------

### Build Single-Sample Terra Workspace Configuration with Python

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/build_ref_panel.md

This script builds the necessary input files for a single-sample Terra workspace configuration. It takes input directories, an output directory, and a JSON string for specific parameters. Ensure the reference panel name matches the corresponding JSON file.

```shell
python scripts/inputs/build_inputs.py \
  inputs/values \
  inputs/templates/terra_workspaces/single_sample \
  inputs/build/NA12878/MY_TERRA_CONFIG \
  -a '{ "single_sample" : "test_single_sample_NA12878", "ref_panel" : "REF_PANEL_NAME" }'
```

--------------------------------

### Download Reference Panel Resources JSON

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/build_ref_panel.md

Downloads the generated reference panel resources JSON file from Google Cloud Storage to the local GATK-SV repository. This JSON file is required for building inputs for the single-sample calling pipeline. Ensure the `gsutil` command uses the correct file path provided after running the notebook.

```shell
gsutil cp gs://fc-7dd8986b-d916-46b0-ba1a-8b09f80f7b83/json/test_panel.json ./inputs/values/
```

--------------------------------

### Build and Publish GATK-SV Docker Images

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Builds and publishes GATK-SV Docker images using a Python script. Requires specifying target images, a tag, and the container registry. Replace placeholders with actual values.

```shell
python3 scripts/docker/build_docker.py \
    --targets <IMAGES> \
    --image-tag <TAG> \
    --docker-repo <CONTAINER_REGISTRY>
```

--------------------------------

### Install Snakemake using Pip or Conda

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Provides commands to install the Snakemake workflow management system using either pip or Conda. Snakemake is the core tool used to build and run the SV detection pipeline.

```bash
$ pip install snakemake
```

```bash
$ conda install -c bioconda snakemake
```

--------------------------------

### Set GCP Environment Variables for VM Creation

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Sets essential environment variables for creating a Google Cloud Platform (GCP) virtual machine. These variables include the project ID, zone ID, and the desired name for the VM instance. Ensure these are correctly configured before proceeding with VM creation.

```bash
export PROJECT_ID="<GOOGLE PROJECT ID>"
export ZONE_ID="<ZONE ID>"

# Make sure no machine with the following name exist,
# and you follow VM naming conventions, e.g., all lower-case characters.
export INSTANCE_NAMES="<VM NAME>"
```

--------------------------------

### svtk: Install and Cluster VCF Records

Source: https://context7.com/broadinstitute/gatk-sv/llms.txt

Installs the svtk Python package and demonstrates how to cluster VCF records based on genomic overlap. Requires input VCF file and specifies clustering parameters like distance and fraction.

```bash
# Install svtk
pip install svtk

# Cluster VCF records by genomic overlap
svtk vcfcluster input.vcf.gz clustered.vcf.gz \
    --dist 300 \
    --frac 0.1 \
    --sample-overlap 0.5 \
    --preserve-ids
```

--------------------------------

### Load and Display Sample Metadata

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/Batching.ipynb

This Python code snippet loads sample metadata from a TSV file into a pandas DataFrame and then displays the DataFrame. It uses `os.path.join` for path construction and `pd.read_table` for file reading.

```python
# Display metadata table, which was generated as part of EvidenceQC
PASS_METADATA = os.path.join(WS_BUCKET, "evidence_qc/filtering/passing_samples_metadata.tsv")
pass_df = pd.read_table(PASS_METADATA)

pass_df
```

--------------------------------

### Clone SV Detection Pipeline Repository

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Clones the SV-Adjudicator repository from GitHub to start a new project. This is the initial step to obtain the pipeline's code.

```bash
git clone https://github.com/talkowski-lab/SV-Adjudicator.git MySVDiscoveryProject
```

--------------------------------

### Execute GATK-SV Pipeline on Cromwell

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/cromwell/quick_start.md

This shell script demonstrates the steps to set up the working directory, copy necessary WDL files, zip dependencies, build default inputs, and submit the GATK-SV pipeline to a Cromwell server. It requires a configured Cromwell server and Cromshell, and uses a JSON file for workflow options.

```shell
> mkdir gatksv_run && cd gatksv_run
> mkdir wdl && cd wdl
> cp $GATK_SV_ROOT/wdl/*.wdl .
> zip dep.zip *.wdl
> cd ..
> bash scripts/inputs/build_default_inputs.sh -d $GATK_SV_ROOT
> cp $GATK_SV_ROOT/inputs/build/ref_panel_1kg/test/GATKSVPipelineBatch/GATKSVPipelineBatch.json GATKSVPipelineBatch.my_run.json
> cromshell submit wdl/GATKSVPipelineBatch.wdl GATKSVPipelineBatch.my_run.json cromwell_config.json wdl/dep.zip
```

--------------------------------

### Import Libraries and Configure Settings for GATK-SV Notebook

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

Imports essential Python libraries for data manipulation, plotting, and interacting with Google Cloud services. It also configures plotting settings for higher resolution and sets up logging.

```python
# Package imports
import os
import io
import re
import logging
import shutil
import subprocess
import zipfile
from collections import defaultdict
from logging import INFO

# Aliased imports
import pandas as pd
import numpy as np
import seaborn as sns
import firecloud.api as fapi
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import matplotlib_inline.backend_inline

# Plotting imports
from upsetplot import UpSet
from upsetplot import from_memberships
from IPython.display import IFrame
from matplotlib.lines import Line2D
from matplotlib.collections import LineCollection
import matplotlib.ticker
from PIL import Image

# Plotting settings 
default_dpi = plt.rcParams['figure.dpi']  
plt.rcParams['figure.dpi'] = 200
matplotlib_inline.backend_inline.set_matplotlib_formats('svg')

# Logger settings
logger = logging.getLogger()
logger.setLevel(INFO)
```

--------------------------------

### Check and Install R Packages for RdTest

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/02_evidence_assessment/02a_rdtest/README.md

This R script verifies the presence of necessary R packages required for the RdTest analysis. It ensures that all dependencies are met before proceeding with the main RdTest execution.

```r
Rscript scripts/Rpackage_check.R
```

--------------------------------

### Create and Activate Conda Environment

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/README.md

Creates a new Anaconda environment named 'sv_pipeline' using the 'environment.yaml' file and then activates it. This ensures all necessary dependencies are installed in an isolated environment.

```bash
cd MySVDiscoveryProject
conda env create -f environment.yaml
source activate sv_pipeline
```

--------------------------------

### Prepare RdTest Bin Coverage Metrics

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/sv-pipeline/02_evidence_assessment/02a_rdtest/README.md

This section describes the preparation of required Rd metrics, specifically `{batch}.binCov.bed.gz` and `{batch}.binCov.median`. These files are crucial for the RdTest process and involve applying bincov, concatenating calls, and then compressing and indexing the resulting BED file.

```bash
bgzip {batch}.binCov.bed
tabix -p bed {batch}.binCov.bed.gz
```

--------------------------------

### Define Configuration Parameters in Python

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/CreateReferencePanel.ipynb

This code defines configuration parameters for the GATK-SV pipeline, including sample set IDs, file paths, and bucket information. These parameters are used to configure the workflow and specify the location of input and output files. It also includes an optional map for manual data inputs.

```python
SAMPLE_SET_ID = "all_samples"
SAMPLE_SET_SET_ID = "all_batches"
REF_PANEL_NAME = "test_panel"
FILE_LISTS_BUCKET = os.environ['WORKSPACE_BUCKET'].replace('gs://', '')
FILE_LISTS_DESTINATION_PATH = "lists"
JSON_BUCKET = FILE_LISTS_BUCKET
JSON_DESTINATION_PATH = "json"
MANUAL_DATA_MAP = {}
```

--------------------------------

### Get Batch for Sample ID (Python)

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

Retrieves the batch ID associated with a given sample ID. It iterates through a sample set table, comparing the provided sample ID against the samples within each batch to find a match and return the corresponding batch ID.

```python
def get_batch_for_sample_id(sample_id):
    """
    Function to retrieve the batch associated with a given sample ID.

    Args:
        sample_id (str): The sample ID to get the batch for.
        
    Returns:
        str: Batch ID corresponding to searched sample.
    """
    for batch, samples in zip(sample_set_tbl['entity:sample_set_id'].values, sample_set_tbl.samples.values):
        if sample_id in samples:
            return batch
```

--------------------------------

### Specify Docker Repository for Pushing Images

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

The `--docker-repo` argument is used to specify the container registry where Docker images should be pushed. This is necessary for WDL testing or hosting images on custom registries. Ensure you are logged into Docker with appropriate credentials.

```shell
myregistry.azurecr.io/gatk-sv
```

```shell
us.gcr.io/my-repository/gatk-sv
```

--------------------------------

### Specify Docker Images to Rebuild with --targets

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

The `--targets` argument allows manual specification of Docker images to rebuild. You can provide a single image name or a space-separated list of multiple image names. This is mutually exclusive with the automatic commit SHA detection method.

```shell
python scripts/docker/build_docker.py \
    --targets sv-pipeline
```

```shell
python scripts/docker/build_docker.py \
    --targets sv-pipeline str
```

--------------------------------

### Run Mean Insert Size Analysis

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

This snippet runs the analysis for Mean Insert Size. It utilizes the 'run_analysis' function, taking the samples QC table and metric name as primary arguments, along with parameters for line deviations, line styles, and log scaling.

```python
run_analysis(samples_qc_table, 'mean_insert_size', LINE_DEVIATIONS, LINE_STYLES, log_scale=LOG_SCALE)
```

--------------------------------

### SVTK Command Line Interface Overview

Source: https://github.com/broadinstitute/gatk-sv/blob/main/src/svtk/README.md

This displays the general usage and available subcommands for the SVTK command-line tool. It categorizes commands for preprocessing, algorithm integration, statistics, read-depth analysis, PE/SR analysis, and variant analysis.

```bash
SVTK: A toolkit for manipulating structural variation

usage: svtk [-h] <subcommand> [options]

[ Preprocessing ]
    standardize    Convert SV calls to a standardized format.
    rdtest2vcf     Convert an RdTest-formatted bed to a standardized VCF.

[ Algorithm integration ]
    vcfcluster     Cluster SV calls from a list of VCFs. (Generally PE/SR.)
    bedcluster     Cluster SV calls from a BED. (Generally depth.)

[ Statistics ]
    count-svtypes  Count instances of each svtype in each sample in a VCF

[ Read-depth analysis ]
    bincov         Calculate normalized genome-wide depth of coverage.
    rdtest*        Calculate comparative coverage statistics at CNV sites.

[ PE/SR analysis ]
    collect-pesr   Count clipped reads and extract discordant pairs genomewide.
    sr-test        Calculate enrichment of clipped reads at SV breakpoints.
    pe-test        Calculate enrichment of discordant pairs at SV breakpoints.

[ Variant analysis ]
    resolve        Resolve complex variants from VCF of breakpoints.
    annotate       Annotate genic effects and ovelrap with noncoding elements.

* Not yet implemented

optional arguments:
  -h, --help  show this help message and exit
```

--------------------------------

### Verify Single-Sample Terra Workspace Configuration Build

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/build_ref_panel.md

This command verifies if the build process for the single-sample Terra workspace configuration was successful by listing the generated files. If the directory or expected files are not present, the build failed.

```shell
ls inputs/build/NA12878/MY_TERRA_CONFIG
```

--------------------------------

### Map Sample Set Settable Columns to Resource IDs (Python)

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/CreateReferencePanel.ipynb

This Python dictionary maps settable column names from a sample set to their corresponding resource identifiers. This mapping is essential for configuring and updating sample set resources.

```python
SAMPLE_SET_SET_KEYS_MAP = {
    "annotated_vcf": "annotated_vcf",
    "annotated_vcf_index": "annotated_vcf_index",
    "breakpoint_overlap_dropped_record_vcfs": "breakpoint_overlap_dropped_record_vcfs",
    "breakpoint_overlap_dropped_record_vcf_indexes": "breakpoint_overlap_dropped_record_vcf_indexes",
    "cleaned_vcf": "clean_vcf",
    "cleaned_vcf_index": "clean_vcf_index",
    "cluster_background_fail_lists": "cluster_background_fail_lists",
    "cluster_bothside_pass_lists": "cluster_bothside_pass_lists",
    "combined_vcfs": "combined_vcfs",
    "combined_vcf_indexes": "combined_vcf_indexes",
    "complex_genotype_vcfs": "complex_genotype_vcfs",
    "complex_genotype_vcf_indexes": "complex_genotype_vcf_indexes",
    "complex_resolve_background_fail_list": "complex_resolve_background_fail_list",
    "complex_resolve_bothside_pass_list": "complex_resolve_bothside_pass_list",
    "complex_resolve_vcfs": "complex_resolve_vcfs",
    "complex_resolve_vcf_indexes": "complex_resolve_vcf_indexes",
    "concordance_vcf": "concordance_vcf",
    "concordance_vcf_index": "concordance_vcf_index",
    "cpx_evidences": "cpx_evidences",
    "cpx_refined_vcf": "complex_refined_vcf",
    "cpx_refined_vcf_index": "complex_refined_vcf_index",
    "filtered_vcf": "genotype_filtered_vcf",
    "filtered_vcf_index": "genotype_filtered_vcf_index",
    "joined_raw_calls_vcf": "joined_raw_calls_vcf",
    "joined_raw_calls_vcf_index": "joined_raw_calls_vcf_index",
    "number_regenotyped_file": "number_regenotyped_file",
    "number_regenotyped_filtered_file": "number_regenotyped_filtered_file",
    "ploidy_table": "ploidy_table",
    "regenotyped_depth_vcfs": "regenotyped_depth_vcf",
    "regenotyped_depth_vcf_indexes": "regenotyped_depth_vcf_index",
    "main_vcf_qc_tarball": "genotype_filtered_qc_tarball",
    "unfiltered_recalibrated_vcf": "gq_filtered_vcf",
    "unfiltered_recalibrated_vcf_index": "gq_filtered_vcf_index"
}
```

--------------------------------

### Map Sample Set Columns to Resource IDs (Python)

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/CreateReferencePanel.ipynb

This Python dictionary maps column names from a sample set table to their corresponding resource identifiers within the GATK-SV project. It is used to standardize resource naming and access.

```python
SAMPLE_SET_KEYS_MAP = {
    "clustered_depth_vcf": "merged_depth_vcf",
    "clustered_depth_vcf_index": "merged_depth_vcf_index",
    "clustered_manta_vcf": "merged_manta_vcf",
    "clustered_manta_vcf_index": "merged_manta_vcf_index",
    "clustered_scramble_vcf": "merged_scramble_vcf",
    "clustered_scramble_vcf_index": "merged_scramble_vcf_index",
    "clustered_wham_vcf": "merged_wham_vcf",
    "clustered_wham_vcf_index": "merged_wham_vcf_index",
    "contig_ploidy_model_tar": "contig_ploidy_model_tar",
    "cutoffs": "cutoffs",
    "filtered_batch_samples_file": "final_sample_list",
    "gcnv_model_tars": "gcnv_model_tars",
    "genotyped_depth_vcf": "genotyped_depth_vcf",
    "genotyped_depth_vcf_index": "genotyped_depth_vcf_index",
    "genotyped_pesr_vcf": "genotyped_pesr_vcf",
    "genotyped_pesr_vcf_index": "genotyped_pesr_vcf_index",
    "median_cov": "medianfile",
    "merged_BAF": "merged_baf_file",
    "merged_BAF_index": "merged_baf_file_index",
    "merged_PE": "merged_pe_file",
    "merged_PE_index": "merged_pe_file_index",
    "merged_SR": "merged_sr_file",
    "merged_SR_index": "merged_sr_file_index",
    "merged_bincov": "merged_rd_file",
    "merged_bincov_index": "merged_rd_file_index",
    "merged_dels": "del_bed",
    "merged_dups": "dup_bed",
    "metrics": "evidence_metrics",
    "outlier_filtered_depth_vcf": "filtered_depth_vcf",
    "outlier_filtered_depth_vcf_index": "filtered_depth_vcf_index",
    "outlier_filtered_pesr_vcf": "filtered_pesr_vcf",
    "outlier_filtered_pesr_vcf_index": "filtered_pesr_vcf_index",
    "regeno_coverage_medians": "regeno_coverage_medians",
    "sites_filtered_depth_vcf": "sites_filtered_depth_vcf",
    "sites_filtered_manta_vcf": "sites_filtered_manta_vcf",
    "sites_filtered_scramble_vcf": "sites_filtered_scramble_vcf",
    "sites_filtered_wham_vcf": "sites_filtered_wham_vcf",
    "sr_background_fail": "raw_sr_background_fail_file",
    "sr_bothside_pass": "raw_sr_bothside_pass_file",
    "std_manta_vcf_tar": "std_manta_vcf_tar",
    "std_scramble_vcf_tar": "std_scramble_vcf_tar",
    "std_wham_vcf_tar": "std_wham_vcf_tar",
    "trained_PE_metrics": "PE_metrics",
    "trained_SR_metrics": "SR_metrics",
    "trained_genotype_depth_depth_sepcutoff": "genotype_depth_depth_sepcutoff",
    "trained_genotype_depth_pesr_sepcutoff": "genotype_depth_pesr_sepcutoff",
    "trained_genotype_pesr_depth_sepcutoff": "genotype_pesr_depth_sepcutoff",
    "trained_genotype_pesr_pesr_sepcutoff": "genotype_pesr_pesr_sepcutoff"
}
```

--------------------------------

### Process Ploidy Data - Python

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

Localizes files, extracts tarballs, and creates directories for ploidy data. It takes a sample set table, a column name for file paths, and a destination path as input, returning paths to created ploidy directories. Dependencies include 'os', 'subprocess', and 'shutil'.

```python
import os
import subprocess
import shutil

def process_ploidy_data(sample_set_tbl, column_name, dest_path):
    """
    Localize files, extract tarballs, and create directories for ploidy data.
    
    Args:
        sample_set_tbl (pandas.DataFrame): The sample set table.
        column_name (str): The name of the column containing file paths.
        dest_path (str): The destination path for localized files.
    
    Returns:
        list: Paths to the created ploidy directories.
    """
    os.makedirs(dest_path, exist_ok=True)
    ploidy_dirs = []
    
    for i, (batch, file) in enumerate(zip(sample_set_tbl['entity:sample_set_id'], sample_set_tbl[column_name])):
        # Localize file
        local_file = os.path.join(dest_path, os.path.basename(file))
        subprocess.run(["gsutil", "cp", file, local_file], check=True)
        
        # Create and extract to subdirectory
        subdir = os.path.join(dest_path, f"{batch}_ploidy")
        if os.path.exists(subdir):
            shutil.rmtree(subdir)
        os.mkdir(subdir)
        
        # Extract contents of tarball into new directory
        os.system(f'tar -xf {local_file} -C {subdir}')
        ploidy_dirs.append(subdir)
    
    return ploidy_dirs

```

--------------------------------

### Build Inputs for Batched Workflows

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/build_inputs.md

Uses a Python script to construct input JSON files for batched GATK-SV workflows. This command takes input values, a template, an output directory, and an optional argument for batch configuration.

```python
python scripts/inputs/build_inputs.py \
  inputs/values \
  inputs/templates/test/GATKSVPipelineSingleSample \
  inputs/build/NA19240/test \
  -a '{ "test_batch" : "ref_panel_1kg" }'
```

--------------------------------

### Execute GATK-SV Docker Build with Parsed Arguments (Python)

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/docker/build_docker_template.ipynb

Parses command-line arguments and initiates the Docker build process for the GATK-SV project. It utilizes a predefined argument parser and the script's path for context.

```python
# you can use the example args above, or provide your own args
actual_args = ['--targets', 'all', 
               '--image-tag', EXAMPLE_BUILD_TAG]

parse_and_build(CMD_line_args_parser(actual_args).project_args,
                this_script_path)
```

--------------------------------

### Configure VM Resources with Runtime Attributes

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/troubleshooting/faq.md

This JSON snippet demonstrates how to override default VM resource allocations for disk and memory within a workflow. It's useful for handling larger sample batches or cohorts that require more resources than the default settings provide. The `RuntimeAttr` inputs, such as `disk_gb` and `mem_gb`, can be specified in an input JSON file.

```json
{
  "MyWorkflow.runtime_attr_override": {
    "disk_gb": 100,
    "mem_gb": 16
  }
}
```

--------------------------------

### Run Mean Insert Size Filtering

Source: https://github.com/broadinstitute/gatk-sv/blob/main/scripts/notebooks/SampleQC.ipynb

This snippet executes the filtering process for Mean Insert Size. It uses the 'run_filtering' function, requiring the samples QC table, the chosen method, and corresponding cutoff values. Log scaling is also an available option.

```python
run_filtering(samples_qc_table, 'mean_insert_size', METHOD, lower_cutoff=LOWER_CUTOFF, 
              upper_cutoff=UPPER_CUTOFF, mad_cutoff=MAD_CUTOFF, log_scale=LOG_SCALE)
```

--------------------------------

### Connect to GCP Ubuntu VM via SSH

Source: https://github.com/broadinstitute/gatk-sv/blob/main/website/docs/advanced/docker/manual.md

Establishes an SSH connection to a Google Cloud Platform (GCP) Ubuntu virtual machine instance. This command uses the gcloud compute ssh utility, requiring the project ID and instance name to be previously defined. Follow on-screen prompts for SSH key authorization.

```bash
gcloud compute ssh $INSTANCE_NAMES --project $PROJECT_ID
```