### Setup Python virtual environment for Apache Beam

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Installs the necessary tools to create a Python virtual environment, activates it, and installs the required Apache Beam and TensorFlow dependencies.

```bash
sudo apt install -y python3.10-venv
python3 -m venv beam
. beam/bin/activate
sudo apt -y update && sudo apt -y install python3-pip
pip3 install --upgrade pip
pip3 install "setuptools<82" "protobuf==4.21.9" "tensorflow==2.16.1"
pip3 install apache_beam[gcp]==2.71.0
```

--------------------------------

### Install Software with Apt

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-vg-case-study.md

Installs essential software packages including aria2, docker.io, and samtools using apt-get. This is a prerequisite for subsequent steps.

```bash
sudo apt update -y
sudo apt-get -y install aria2 docker.io samtools
```

--------------------------------

### Prepare Data and Install Dependencies

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Creates the local workspace directory structure, downloads reference data from GCS, and installs required system packages including NVIDIA Docker support.

```bash
mkdir -p "${OUTPUT_DIR}"
mkdir -p "${BIN_DIR}"
mkdir -p "${DATA_DIR}"
mkdir -p "${LOG_DIR}"

gsutil -m cp ${DATA_BUCKET}/BGISEQ_PE100_NA12878.sorted.chr*.bam* "${DATA_DIR}"
gsutil -m cp -r "${DATA_BUCKET}/ucsc_hg19.fa*" "${DATA_DIR}"
gsutil -m cp -r "${DATA_BUCKET}/HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_*" "${DATA_DIR}"

sudo apt -y update
sudo apt -y install parallel
curl -O https://raw.githubusercontent.com/google/deepvariant/r1.10/scripts/install_nvidia_docker.sh
bash -x install_nvidia_docker.sh
```

--------------------------------

### Install DeepVariant and Dependencies

Source: https://github.com/google/deepvariant/blob/r1.10/docs/visualizing_examples.ipynb

Installs the DeepVariant library from GitHub using pip and sets up necessary build tools like protoc. This is a prerequisite for running the visualization examples.

```bash
! pip install git+https://github.com/google/deepvariant.git@r1.10
```

```bash
#@title Install protoc to be used later.
PROTOC_VERSION = "26.1" # Example: "26.1", "25.3", etc. Check releases page for the latest
PROTOC_ZIP = f"protoc-{PROTOC_VERSION}-linux-x86_64.zip"
DOWNLOAD_URL = f"https://github.com/protocolbuffers/protobuf/releases/download/v{PROTOC_VERSION}/{PROTOC_ZIP}"

# Download the protoc zip file
!curl -LO {DOWNLOAD_URL}

# Unzip the archive
!unzip -o {PROTOC_ZIP} -d protoc_install

# Move the protoc binary to a directory in your PATH
!sudo mv protoc_install/bin/protoc /usr/local/bin/

# Move the include files to a standard location
!sudo mv protoc_install/include/* /usr/local/include/

# Optional: Clean up the downloaded zip and extracted folder
!rm {PROTOC_ZIP}
!rm -rf protoc_install

# Verify the installation
!protoc --version
```

```bash
!git clone https://github.com/google/deepvariant.git
%cd deepvariant
! git checkout r1.10
! sudo python3 setup.py build_proto
```

--------------------------------

### Install Prerequisites and Build DeepVariant Binaries (Shell)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-build-test.md

Installs system packages and dependencies required for DeepVariant, downloads and builds TensorFlow and CLIF from source, and compiles the DeepVariant binaries. This script is intended to be run after installing the Google Cloud SDK and executing 'sudo su'.

```shell
./build-prereq.sh

./build_and_test.sh
```

--------------------------------

### Execute show_examples to Generate Pileup Images

Source: https://github.com/google/deepvariant/blob/r1.10/docs/show-examples.md

Run the show_examples tool via Docker to convert tfrecord files into PNG images. This command requires the path to the examples tfrecord and the corresponding JSON info file.

```bash
INPUT_DIR="${PWD}/quickstart-testdata"
OUTPUT_DIR="${PWD}/quickstart-output"
BIN_VERSION="1.10.0"
sudo docker run \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}":"/output" \
  google/deepvariant:"${BIN_VERSION}" /opt/deepvariant/bin/show_examples \
  --examples=/output/intermediate_results_dir/make_examples.tfrecord-00000-of-00001.gz \
  --example_info_json=/output/intermediate_results_dir/make_examples.tfrecord-00000-of-00001.gz.example_info.json \
  --output=/output/pileup \
  --num_records=20 \
  --curate

ls "${OUTPUT_DIR}"/pileup*.png
```

--------------------------------

### Download DeepVariant Examples

Source: https://github.com/google/deepvariant/blob/r1.10/docs/visualizing_examples.ipynb

Downloads a TFRecord gzipped file containing DeepVariant examples from Google Cloud Storage to the local environment. This file is used for subsequent visualization.

```bash
!gsutil -q cp gs://deepvariant/datalab-testdata/make_examples_datalab.tfrecord.gz /tmp/make_examples_colab.tfrecord.gz
```

--------------------------------

### Download VG and KMC Binaries

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-vg-case-study.md

Downloads and installs the vg and kmc binaries from their respective GitHub releases. It also makes them executable.

```bash
KMC_VERSION=3.2.4
VG_VERSION=1.67.0
wget https://github.com/refresh-bio/KMC/releases/download/v${KMC_VERSION}/KMC${KMC_VERSION}.linux.x64.tar.gz
tar zxf KMC${KMC_VERSION}.linux.x64.tar.gz bin/kmc
mv bin/kmc ${DATA_DIR}/
wget https://github.com/vgteam/vg/releases/download/v${VG_VERSION}/vg -O ${DATA_DIR}/vg
chmod +x ${DATA_DIR}/vg ${DATA_DIR}/kmc
```

--------------------------------

### Running postprocess_variants for gVCF Output (Python/Bash)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-gvcf-support.md

This example shows the command to run `postprocess_variants` for gVCF generation. It requires the `--nonvariant_site_tfrecord_path` flag pointing to the output from `make_examples` and the `--gvcf_outfile` flag for the final gVCF output.

```bash
OUTPUT_GVCF="${OUTPUT_DIR}/HG002.output.g.vcf.gz"

( time python "${BIN_DIR}"/postprocess_variants.zip \
    --ref "${REF}" \
    --infile "${CALL_VARIANTS_OUTPUT}" \
    --outfile "${OUTPUT_VCF}" \
    --nonvariant_site_tfrecord_path "${GVCF_TFRECORDS}" \
    --gvcf_outfile "${OUTPUT_GVCF}" \
) >"${LOG_DIR}/postprocess_variants.log" 2>&1
```

--------------------------------

### Install and Pull DeepVariant Docker Image

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-quick-start.md

Installs Docker if not present and pulls the specified version of the DeepVariant Docker image. This image contains the DeepVariant programs and models.

```bash
BIN_VERSION="1.10.0"

sudo apt -y update
sudo apt-get -y install docker.io
sudo docker pull google/deepvariant:"${BIN_VERSION}"
```

--------------------------------

### Generate DeepVariant Training and Calling Examples

Source: https://context7.com/google/deepvariant/llms.txt

Uses the make_examples binary within a Docker container to generate TFRecord files for either model training or variant calling inference. It supports parallel processing across multiple shards to optimize throughput.

```bash
seq 0 $((N_SHARDS-1)) | parallel --halt 2 --line-buffer sudo docker run -v ${HOME}:${HOME} google/deepvariant:${BIN_VERSION} make_examples --mode training --ref "${REF}" --reads "${BAM_CHR1}" --examples "${OUTPUT_DIR}/training_set.with_label.tfrecord@${N_SHARDS}.gz" --truth_variants "${TRUTH_VCF}" --confident_regions "${TRUTH_BED}" --task {} --regions "'chr1'" --channel_list "BASE_CHANNELS,insert_size"

seq 0 $((N_SHARDS-1)) | parallel --halt 2 --line-buffer sudo docker run -v ${HOME}:${HOME} google/deepvariant:${BIN_VERSION} make_examples --mode calling --ref "${REF}" --reads "${BAM}" --examples "${OUTPUT_DIR}/calling.tfrecord@${N_SHARDS}.gz" --gvcf "${OUTPUT_DIR}/gvcf.tfrecord@${N_SHARDS}.gz" --task {} --checkpoint "/opt/models/wgs" --channel_list "BASE_CHANNELS,insert_size"
```

--------------------------------

### Run DeepVariant - Main Entry Point

Source: https://context7.com/google/deepvariant/llms.txt

Orchestrates the entire DeepVariant pipeline, automatically configuring and running all three stages (make_examples, call_variants, postprocess_variants). This example demonstrates a complete workflow using Docker.

```APIDOC
## POST /run_deepvariant

### Description
This endpoint runs the complete DeepVariant pipeline using Docker. It takes aligned reads and reference data, processes them through the DeepVariant model, and outputs variant calls in VCF and gVCF formats.

### Method
POST

### Endpoint
/run_deepvariant

### Parameters
#### Path Parameters
None

#### Query Parameters
- **model_type** (string) - Required - The type of pre-trained model to use (e.g., WGS, WES, PACBIO, ONT_R104, HYBRID_PACBIO_ILLUMINA).
- **ref** (string) - Required - Path to the reference genome FASTA file.
- **reads** (string) - Required - Path to the aligned reads BAM or CRAM file.
- **regions** (string) - Optional - Comma-separated list of genomic regions to process (e.g., "chr20:10,000,000-10,010,000").
- **output_vcf** (string) - Required - Path for the output VCF file.
- **output_gvcf** (string) - Optional - Path for the output gVCF file.
- **intermediate_results_dir** (string) - Optional - Directory to store intermediate results.
- **num_shards** (integer) - Optional - Number of shards to use for parallel processing. Defaults to the number of available processors if not specified.
- **vcf_stats_report** (boolean) - Optional - Whether to generate VCF statistics report. Defaults to false.

### Request Example
```bash
sudo docker run \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}":"/output" \
  google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref=/input/ucsc.hg19.chr20.unittest.fasta \
  --reads=/input/NA12878_S1.chr20.10_10p1mb.bam \
  --regions "chr20:10,000,000-10,010,000" \
  --output_vcf=/output/output.vcf.gz \
  --output_gvcf=/output/output.g.vcf.gz \
  --intermediate_results_dir /output/intermediate_results_dir \
  --num_shards=1 \
  --vcf_stats_report=true
```

### Response
#### Success Response (200)
- **output.vcf.gz** (file) - Variant calls in VCF format.
- **output.vcf.gz.tbi** (file) - VCF index.
- **output.g.vcf.gz** (file) - Genomic VCF with non-variant sites.
- **output.g.vcf.gz.tbi** (file) - gVCF index.
- **output.visual_report.html** (file) - VCF statistics report.

#### Response Example
(Files are generated in the specified output directory)
```

--------------------------------

### Generate Training Examples with make_examples

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Executes the make_examples tool in training mode using GNU Parallel to process data across multiple shards. This generates TFRecord files containing labels for specific genomic regions.

```bash
( time seq 0 $((N_SHARDS-1)) | \
  parallel --halt 2 --line-buffer \
    sudo docker run \
      -v ${HOME}:${HOME} \
      ${DOCKER_IMAGE} \
      make_examples \
      --mode training \
      --ref "${REF}" \
      --reads "${BAM_CHR1}" \
      --examples "${OUTPUT_DIR}/training_set.with_label.tfrecord@${N_SHARDS}.gz" \
      --truth_variants "${TRUTH_VCF}" \
      --confident_regions "${TRUTH_BED}" \
      --task {} \
      --regions "'chr1'" \
      --channel_list "BASE_CHANNELS,insert_size" \
) 2>&1 | tee "${LOG_DIR}/training_set.with_label.make_examples.log"
```

--------------------------------

### Generate Validation Examples with make_examples

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Similar to the training set generation, this command creates validation TFRecord files for specific genomic regions (e.g., chr21) to evaluate model performance.

```bash
( time seq 0 $((N_SHARDS-1)) | \
  parallel --halt 2 --line-buffer \
    sudo docker run \
      -v /home/${USER}:/home/${USER} \
      ${DOCKER_IMAGE} \
      make_examples \
      --mode training \
      --ref "${REF}" \
      --reads "${BAM_CHR21}" \
      --examples "${OUTPUT_DIR}/validation_set.with_label.tfrecord@${N_SHARDS}.gz" \
      --truth_variants "${TRUTH_VCF}" \
      --confident_regions "${TRUTH_BED}" \
      --task {} \
      --regions "'chr21'" \
      --channel_list "BASE_CHANNELS,insert_size" \
) 2>&1 | tee "${LOG_DIR}/validation_set.with_label.make_examples.log"
```

--------------------------------

### Initialize Environment and Download Data

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-pacbio-model-case-study.md

Sets up the local directory structure and downloads the GRCh38 reference genome and PacBio HiFi BAM files required for variant calling.

```bash
BASE="${HOME}/pacbio-case-study"
INPUT_DIR="${BASE}/input/data"
OUTPUT_DIR="${BASE}/output"
mkdir -p "${INPUT_DIR}"
mkdir -p "${OUTPUT_DIR}"
FTPDIR=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz | gunzip > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta.fai
HTTPDIR=https://storage.googleapis.com/deepvariant/pacbio-case-study-testdata
curl ${HTTPDIR}/HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam > ${INPUT_DIR}/HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam
curl ${HTTPDIR}/HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam.bai > ${INPUT_DIR}/HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam.bai
REF="GRCh38_no_alt_analysis_set.fasta"
BAM="HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam"
THREADS=$(nproc)
REGION="chr20"
OUTPUT_VCF="HG003_PACBIO_SPRQ_GRCh38.chr20.output.vcf.gz"
OUTPUT_GVCF="HG003_PACBIO_SPRQ_GRCh38.chr20.output.g.vcf.gz"
INTERMEDIATE_DIRECTORY="intermediate_results_dir"
mkdir -p "${OUTPUT_DIR}/${INTERMEDIATE_DIRECTORY}"
```

--------------------------------

### Download DeepVariant Test Data

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-quick-start.md

Downloads a small bundle of test data required for the DeepVariant quick start guide. This includes BAM, VCF, and FASTA files along with their respective index files.

```bash
INPUT_DIR="${PWD}/quickstart-testdata"
DATA_HTTP_DIR="https://storage.googleapis.com/deepvariant/quickstart-testdata"

mkdir -p ${INPUT_DIR}
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/NA12878_S1.chr20.10_10p1mb.bam.bai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.bed
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/test_nist.b37_chr20_100kbp_at_10mb.vcf.gz.tbi
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.fai
wget -P ${INPUT_DIR} "${DATA_HTTP_DIR}"/ucsc.hg19.chr20.unittest.fasta.gz.gzi
```

--------------------------------

### Running make_examples for gVCF Output (Python/Bash)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-gvcf-support.md

This snippet demonstrates how to execute the `make_examples` program with the necessary flags to generate gVCF output. It requires the `--gvcf` flag to specify the output TFRecord file and must be run in 'calling' mode.

```bash
GVCF_TFRECORDS="${OUTPUT_DIR}/HG002.gvcf.tfrecord@${N_SHARDS}.gz"

( time seq 0 $((N_SHARDS-1)) | \
  parallel --halt 2 --joblog "${LOG_DIR}/log" --res "${LOG_DIR}" \
    python "${BIN_DIR}"/make_examples.zip \
    --mode calling \
    --ref "${REF}" \
    --reads "${BAM}" \
    --examples "${EXAMPLES}" \
    --gvcf "${GVCF_TFRECORDS}" \
    --task {} \
) >"${LOG_DIR}/make_examples.log" 2>&1
```

--------------------------------

### Initialize Analysis Environment

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-masseq-case-study.md

Creates the necessary directory structure to organize input files, reference genomes, benchmark data, and output results.

```bash
mkdir -p input benchmark reference output happy
```

--------------------------------

### Install DeepVariant Runtime Dependencies (Shell)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-build-test.md

Installs runtime dependencies for DeepVariant on any machine where it will be executed. This includes essential Python packages like numpy and Tensorflow.

```shell
./run-prereq.sh
```

--------------------------------

### DeepVariant make_examples Configuration

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-fast-pipeline-case-study.md

This bash script creates a configuration file (`make_examples.ini`) for the `make_examples` binary in DeepVariant. It specifies input TFRecords, GVCF output, processing mode, input BAM file, reference genome, and checkpoint path. The `@14.gz` notation indicates sharding.

```bash
mkdir -p config
FILE=config/make_examples.ini

cat <<EOM >$FILE
--examples=/tmp/examples.tfrecords@14.gz
--gvcf=/tmp/examples.gvcf.tfrecord@14.gz
--mode=calling
--reads=/input/HG003.SPRQ.pacbio.GRCh38.nov2024.chr20.bam
--ref=/reference/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz
--output_phase_info
--checkpoint=/opt/models/pacbio
--regions=chr20
EOM
```

--------------------------------

### Save DeepVariant Example Images

Source: https://github.com/google/deepvariant/blob/r1.10/docs/visualizing_examples.ipynb

Iterates through multiple DeepVariant examples, generates pileup images for each, and saves them as PNG files. The filenames include the locus ID and truth label.

```python
for e in dataset.take(10):
  example = tf.train.Example()
  example.ParseFromString(e.numpy())

  # For example, in chr20:19503712_T_C, T is the reference allele and C is the
  # alternate allele that is proposed in this pileup image.
  filename = 'pileup_{}_truth={}.png'.format(vis.locus_id_with_alt(example), vis.label_from_example(example))
  vis.draw_deepvariant_pileup(example, path=filename, show=False)
```

--------------------------------

### Visualize a Single DeepVariant Example

Source: https://github.com/google/deepvariant/blob/r1.10/docs/visualizing_examples.ipynb

Loads a single DeepVariant example from a TFRecord file and visualizes its pileup image using the `vis.draw_deepvariant_pileup` function. It also prints the locus ID and truth label.

```python
examples_path = '/tmp/make_examples_colab.tfrecord.gz'
dataset = tf.data.TFRecordDataset(examples_path, compression_type="GZIP")

# Take a single example and show the deepvariant pileup image for it.
for e in dataset.take(1):
  example = tf.train.Example()
  example.ParseFromString(e.numpy())

  # For example, in chr20:19503712_T_C, T is the reference allele and C is the
  # alternate allele that is proposed in this pileup image.
  print("Locus ID with alt:", vis.locus_id_with_alt(example))
  print("Draw the channels:")
  vis.draw_deepvariant_pileup(example)

  print("Truth label:", vis.label_from_example(example))
```

--------------------------------

### Prepare Environment and Download Data for DeepVariant

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-xy-calling-case-study.md

Sets up the local directory structure and downloads the reference genome, BAM files, and PAR BED files required for variant calling on chrX and chrY.

```bash
BASE="${HOME}/XY-walkthrough"
INPUT_DIR="${BASE}/input"
OUTPUT_DIR="${BASE}/output"
mkdir -p "${INPUT_DIR}"
mkdir -p "${OUTPUT_DIR}/data"
FTPDIR=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz | gunzip > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta.fai
HTTPDIR=https://storage.googleapis.com/deepvariant/xy-case-study-testdata
curl ${HTTPDIR}/HG002.pfda_challenge.grch38.chrXY.bam > ${INPUT_DIR}/HG002.pfda_challenge.grch38.chrXY.bam
curl ${HTTPDIR}/HG002.pfda_challenge.grch38.chrXY.bam.bai > ${INPUT_DIR}/HG002.pfda_challenge.grch38.chrXY.bam.bai
HTTPDIR=https://storage.googleapis.com/deepvariant/case-study-testdata
curl ${HTTPDIR}/GRCh38_PAR.bed > ${INPUT_DIR}/GRCh38_PAR.bed
REF="GRCh38_no_alt_analysis_set.fasta"
BAM="HG002.pfda_challenge.grch38.chrXY.bam"
THREADS=$(nproc)
REGION="chrX chrY"
HAPLOID_CONTIGS="chrX,chrY"
PAR_BED="GRCh38_PAR.bed"
OUTPUT_VCF="HG002_pacbio_hifi.chrXY.output.vcf.gz"
OUTPUT_GVCF="HG002_pacbio_hifi.chrXY.output.g.vcf.gz"
INTERMEDIATE_DIRECTORY="intermediate_results_dir"
mkdir -p "${OUTPUT_DIR}/${INTERMEDIATE_DIRECTORY}"
```

--------------------------------

### gVCF Format Example (Bash)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-gvcf-support.md

An example illustrating the structure of a Genomic VCF (gVCF) file, which includes variant calls and non-variant sites. It shows how non-variant regions are represented and how adjacent records with similar qualities are merged.

```bash
#CHROM  POS ID  REF  ALT     QUAL  FILTER  INFO       FORMAT   Sample
1      4370  .   G   <*>       .     .     END=4383   GT:GQ    0/0:37
1      4384  .   C   <*>       .     .     END=4388   GT:GQ    0/0:41
1      4389  .   T   TC,<*>   50     .     .          GT:GQ    0/1:50
1      4390  .   C   <*>       .     .     END=4390   GT:GQ    0/0:3
```

--------------------------------

### POST /call_variants

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-details.md

Evaluates the deep learning model on TFRecord examples generated by the make_examples stage.

```APIDOC
## POST /call_variants

### Description
Consumes TFRecord files containing tf.Examples protos and a model checkpoint to perform variant calling. This process is computationally intensive and optimized for CPU or single-GPU environments.

### Method
POST

### Endpoint
/call_variants

### Parameters
#### Request Body
- **examples** (string) - Required - Path to input TFRecord file(s) or glob pattern.
- **checkpoint** (string) - Required - Path to the deep learning model checkpoint.
- **mode** (string) - Optional - Execution mode (e.g., CPU or GPU).

### Request Example
{
  "examples": "/path/to/examples/*.tfrecord",
  "checkpoint": "/path/to/model.ckpt",
  "mode": "GPU"
}

### Response
#### Success Response (200)
- **output** (string) - Path to the generated TFRecord of CallVariantsOutput protos.

#### Response Example
{
  "status": "success",
  "output": "/path/to/output.tfrecord"
}
```

--------------------------------

### Prepare FASTQ Path List

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-vg-case-study.md

Creates a file named HG003.fq.paths containing the paths to the input FASTQ files (R1 and R2). This file is used by subsequent commands.

```bash
cat > HG003.fq.paths <<-"EOM"
${DATA_DIR}/HG003.novaseq.pcr-free.35x.R1.fastq.gz
${DATA_DIR}/HG003.novaseq.pcr-free.35x.R2.fastq.gz
EOM
```

--------------------------------

### Import DeepVariant and TensorFlow

Source: https://github.com/google/deepvariant/blob/r1.10/docs/visualizing_examples.ipynb

Imports the necessary DeepVariant library components and TensorFlow. It also prints the TensorFlow version to confirm successful setup.

```python
import deepvariant
import tensorflow as tf
from third_party.nucleus.util import vis
print(tf.__version__)
```

--------------------------------

### Model Types Configuration

Source: https://context7.com/google/deepvariant/llms.txt

Demonstrates how to configure DeepVariant for different sequencing platforms using various pre-trained model types. Each example uses Docker.

```APIDOC
## POST /run_deepvariant/{model_type}

### Description
This endpoint runs the DeepVariant pipeline with a specific pre-trained model type, optimized for different sequencing platforms. It automatically configures parameters for optimal accuracy with the chosen data type.

### Method
POST

### Endpoint
/run_deepvariant

### Parameters
#### Path Parameters
None

#### Query Parameters
- **model_type** (string) - Required - The type of pre-trained model to use. Supported types include:
    - WGS: Illumina Whole Genome Sequencing
    - WES: Whole Exome Sequencing
    - PACBIO: PacBio HiFi data (includes phasing output)
    - ONT_R104: Oxford Nanopore R10.4.1 simplex data
    - HYBRID_PACBIO_ILLUMINA: Combined PacBio HiFi and Illumina data
- **ref** (string) - Required - Path to the reference genome FASTA file.
- **reads** (string) - Required - Path to the aligned reads BAM or CRAM file.
- **reads_parent1** (string) - Optional - Path to a second BAM file, used for hybrid models (e.g., HYBRID_PACBIO_ILLUMINA).
- **output_vcf** (string) - Required - Path for the output VCF file.
- **num_shards** (integer) - Optional - Number of shards to use for parallel processing. Defaults to the number of available processors if not specified.

### Request Example (WGS Model)
```bash
sudo docker run \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}":"/output" \
  google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref=/input/reference.fasta \
  --reads=/input/sample.bam \
  --output_vcf=/output/sample.vcf.gz \
  --num_shards=$(nproc)
```

### Request Example (HYBRID_PACBIO_ILLUMINA Model)
```bash
sudo docker run \
  -v "${INPUT_DIR}":"/input" \
  -v "${OUTPUT_DIR}":"/output" \
  google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=HYBRID_PACBIO_ILLUMINA \
  --ref=/input/reference.fasta \
  --reads=/input/pacbio.bam \
  --reads_parent1=/input/illumina.bam \
  --output_vcf=/output/hybrid.vcf.gz \
  --num_shards=$(nproc)
```

### Response
#### Success Response (200)
- **output.vcf.gz** (file) - Variant calls in VCF format.
- **output.vcf.gz.tbi** (file) - VCF index.

#### Response Example
(Files are generated in the specified output directory)
```

--------------------------------

### Verify Google Cloud Authentication

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-gcp-info.md

This command verifies that the Google Cloud SDK is correctly installed and that the user is authenticated. It outputs the active account email address associated with the current configuration.

```shell
gcloud auth list
```

--------------------------------

### Download FASTQ Input Files

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-vg-case-study.md

Downloads FASTQ files from a Google Cloud Storage bucket for the HG003 sample. These files are essential for the variant calling process.

```bash
DATA_DIR=${PWD}/data
mkdir -p ${DATA_DIR}
gcloud storage cp gs://brain-genomics-public/research/sequencing/fastq/novaseq/wgs_pcr_free/35x/HG003.novaseq.pcr-free.35x.R?.fastq.gz ${DATA_DIR}/
```

--------------------------------

### Prepare Environment and Download Inputs for DeepVariant

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-ont-r104-simplex-case-study.md

This script sets up the necessary directories and downloads reference genomes and BAM files required for running DeepVariant. It defines input and output paths and downloads data using curl.

```bash
BASE="${HOME}/ont-case-study"

# Set up input and output directory data
INPUT_DIR="${BASE}/input/data"
OUTPUT_DIR="${BASE}/output"

## Create local directory structure
mkdir -p "${INPUT_DIR}"
mkdir -p "${OUTPUT_DIR}"

# Download reference to input directory
FTPDIR=ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz | gunzip > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta
curl ${FTPDIR}/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.fai > ${INPUT_DIR}/GRCh38_no_alt_analysis_set.fasta.fai

# Download HG003 Ultra-long chr20 bam file to input directory
HTTPDIR=https://storage.googleapis.com/deepvariant/ont-case-study-testdata
curl ${HTTPDIR}/HG003_R104_sup_merged.80x.chr20.bam > ${INPUT_DIR}/HG003_R104_sup_merged.80x.chr20.bam
curl ${HTTPDIR}/HG003_R104_sup_merged.80x.chr20.bam.bai > ${INPUT_DIR}/HG003_R104_sup_merged.80x.chr20.bam.bai

# Set up input variables
REF="GRCh38_no_alt_analysis_set.fasta"
BAM="HG003_R104_sup_merged.80x.chr20.bam"
THREADS=$(nproc)
REGION="chr20"

# Set up output variable
OUTPUT_VCF="HG003_UL_R1041_Guppy6_sup_2_GRCh38.chr20.output.vcf.gz"
OUTPUT_GVCF="HG003_UL_R1041_Guppy6_sup_2_GRCh38.output.g.vcf.gz"
INTERMEDIATE_DIRECTORY="intermediate_results_dir"

mkdir -p "${OUTPUT_DIR}/${INTERMEDIATE_DIRECTORY}"
```

--------------------------------

### Run DeepVariant with Singularity

Source: https://context7.com/google/deepvariant/llms.txt

Provides commands for running DeepVariant in HPC environments using Singularity. Includes examples for CPU-only execution, GPU-enabled execution with --nv, and handling environment compatibility with --cleanenv.

```bash
BIN_VERSION="1.10.0"
singularity pull docker://google/deepvariant:"${BIN_VERSION}"
singularity run -B /usr/lib/locale/:/usr/lib/locale/ \
  docker://google/deepvariant:"${BIN_VERSION}" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref="${INPUT_DIR}/reference.fasta" \
  --reads="${INPUT_DIR}/sample.bam" \
  --regions "chr20:10,000,000-10,010,000" \
  --output_vcf="${OUTPUT_DIR}/output.vcf.gz" \
  --output_gvcf="${OUTPUT_DIR}/output.g.vcf.gz" \
  --intermediate_results_dir "${OUTPUT_DIR}/intermediate_results_dir" \
  --num_shards=$(nproc) \
  --vcf_stats_report=true
singularity run --nv -B /usr/lib/locale/:/usr/lib/locale/ \
  docker://google/deepvariant:"${BIN_VERSION}-gpu" \
  /opt/deepvariant/bin/run_deepvariant \
  --model_type=WGS \
  --ref="${INPUT_DIR}/reference.fasta" \
  --reads="${INPUT_DIR}/sample.bam" \
  --output_vcf="${OUTPUT_DIR}/output.vcf.gz" \
  --num_shards=$(nproc)
```

--------------------------------

### Provision and Access GPU VM for DeepVariant Training

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Commands to create a Google Cloud Compute Engine instance with an NVIDIA Tesla P100 GPU and establish an SSH connection to the new environment.

```bash
host="${USER}-deepvariant-vm"
zone="us-west1-b"

gcloud compute instances create ${host} \
  --scopes "compute-rw,storage-full,cloud-platform" \
  --maintenance-policy "TERMINATE" \
  --accelerator=type=nvidia-tesla-p100,count=1 \
  --image-family="ubuntu-2204-lts" \
  --image-project="ubuntu-os-cloud" \
  --machine-type="n1-standard-16" \
  --boot-disk-size="300" \
  --zone="${zone}"

gcloud compute ssh ${host} --zone ${zone}
```

--------------------------------

### Copy Model Metadata

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Copies the required model.example_info.json file from a GCS bucket to the local training checkpoint directory.

```bash
gsutil cp \
  gs://deepvariant/models/DeepVariant/1.10.0/checkpoints/wgs/model.example_info.json \
  ${TRAINING_DIR}/checkpoints/ema/
```

--------------------------------

### Configure Environment Variables for DeepVariant Training

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Sets up the necessary directory paths, bucket locations, and file references required for the DeepVariant training pipeline.

```bash
YOUR_PROJECT=REPLACE_WITH_YOUR_PROJECT
OUTPUT_GCS_BUCKET=REPLACE_WITH_YOUR_GCS_BUCKET

BUCKET="gs://deepvariant"
VERSION="1.10.0"
DOCKER_IMAGE="google/deepvariant:${VERSION}"

GCS_PRETRAINED_WGS_MODEL="${BUCKET}/models/DeepVariant/1.10.0/checkpoints/wgs/deepvariant.wgs.ckpt"

OUTPUT_BUCKET="${OUTPUT_GCS_BUCKET}/customized_training"
TRAINING_DIR="${OUTPUT_BUCKET}/training_dir"

BASE="${HOME}/training-case-study"
DATA_BUCKET=gs://deepvariant/training-case-study/BGISEQ-HG001

INPUT_DIR="${BASE}/input"
BIN_DIR="${INPUT_DIR}/bin"
DATA_DIR="${INPUT_DIR}/data"
OUTPUT_DIR="${BASE}/output"
LOG_DIR="${OUTPUT_DIR}/logs"
SHUFFLE_SCRIPT_DIR="${HOME}/deepvariant/tools"

REF="${DATA_DIR}/ucsc_hg19.fa"
BAM_CHR1="${DATA_DIR}/BGISEQ_PE100_NA12878.sorted.chr1.bam"
BAM_CHR20="${DATA_DIR}/BGISEQ_PE100_NA12878.sorted.chr20.bam"
BAM_CHR21="${DATA_DIR}/BGISEQ_PE100_NA12878.sorted.chr21.bam"
TRUTH_VCF="${DATA_DIR}/HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_PGandRTGphasetransfer_chrs_FIXED.vcf.gz"
TRUTH_BED="${DATA_DIR}/HG001_GRCh37_GIAB_highconf_CG-IllFB-IllGATKHC-Ion-10X-SOLID_CHROM1-X_v.3.3.2_highconf_nosomaticdel_chr.bed"

N_SHARDS=$(nproc)
```

--------------------------------

### Downsample BAM file for training augmentation

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

This command demonstrates how to use the --downsample_fraction parameter within the DeepVariant training pipeline. By setting this value to 0.5, the tool creates training examples with reduced coverage, which helps in training a more robust model.

```bash
seq_pipeline.py --downsample_fraction=0.5 --bam=input.bam --output=output_downsampled.tfrecord
```

--------------------------------

### Create Compute Engine Instance via gcloud

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-gcp-info.md

Provisions a new Compute Engine instance with specified CPU, disk, and GPU resources. This command uses the gcloud CLI to set up an Ubuntu 22.04 environment suitable for DeepVariant.

```shell
gcloud beta compute instances create "${USER}-deepvariant-quickstart" \
  --scopes "compute-rw,storage-full,cloud-platform"  \
  --image-family ubuntu-2204-lts --image-project ubuntu-os-cloud \
  --machine-type n1-standard-8  \
  --boot-disk-size=200GB \
  --zone us-west1-b \
  --accelerator type=nvidia-tesla-k80,count=1 --maintenance-policy TERMINATE --restart-on-failure
```

--------------------------------

### Download GIAB Benchmarks (Bash)

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-ont-r104-simplex-case-study.md

Downloads the necessary Genome in a Bottle (GIAB) benchmark files for HG003, including a VCF file, its index, and a BED file. These files are essential for evaluating variant calling accuracy.

```bash
FTPDIR=ftp://ftp-trace.ncbi.nlm.nih.gov/giab/ftp/release/AshkenazimTrio/HG003_NA24149_father/NISTv4.2.1/GRCh38

curl ${FTPDIR}/HG003_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed > ${INPUT_DIR}/HG003_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed
curl ${FTPDIR}/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz > ${INPUT_DIR}/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz
curl ${FTPDIR}/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi > ${INPUT_DIR}/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz.tbi

TRUTH_VCF="HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz"
TRUTH_BED="HG003_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed"
```

--------------------------------

### Run Customized DeepVariant Model

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-training-case-study.md

Executes the DeepVariant pipeline using a specific customized model checkpoint. This command processes a BAM file for a defined region and outputs a VCF file, with the small model disabled to ensure all examples pass through the custom model.

```bash
sudo docker run --gpus 1 \
  -v /home/${USER}:/home/${USER} \
  "${DOCKER_IMAGE}-gpu" \
  run_deepvariant \
  --model_type WGS \
  --customized_model "${BEST_CHECKPOINT}" \
  --ref "${REF}" \
  --reads "${BAM_CHR20}" \
  --regions "chr20" \
  --output_vcf "${OUTPUT_DIR}/test_set.vcf.gz" \
  --num_shards=${N_SHARDS} \
  --disable_small_model
```

--------------------------------

### Configuring Realigned Reads Output in DeepVariant

Source: https://github.com/google/deepvariant/blob/r1.10/docs/FAQ.md

This example demonstrates how to configure DeepVariant to output realigned reads for debugging purposes. It involves setting specific arguments to emit realigned reads and specify a diagnostic output directory. This is intended for debugging and can generate a large number of small BAM files, so it's recommended to use with the `--regions` flag.

```bash
--make_examples_extra_args="emit_realigned_reads=true,realigner_diagnostics=/output/realigned_reads"
```

--------------------------------

### Run hap.py Docker Container for Benchmarking

Source: https://github.com/google/deepvariant/blob/r1.10/docs/deepvariant-exome-case-study.md

This command executes the hap.py benchmarking tool within a Docker container. It mounts local directories for input, output, and reference data, and specifies various parameters for the benchmark, including truth VCF, output VCF, BED files for filtering, reference FASTA, and the benchmarking engine.

```bash
mkdir -p happy

sudo docker pull jmcdani20/hap.py:v0.3.12

sudo docker run \
  -v "${PWD}/benchmark":"/benchmark" \
  -v "${PWD}/input":"/input" \
  -v "${PWD}/output":"/output" \
  -v "${PWD}/reference":"/reference" \
  -v "${PWD}/happy:/happy" \
  jmcdani20/hap.py:v0.3.12 /opt/hap.py/bin/hap.py \
  /benchmark/HG003_GRCh38_1_22_v4.2.1_benchmark.vcf.gz \
  /output/HG003.output.vcf.gz \
  -f /benchmark/HG003_GRCh38_1_22_v4.2.1_benchmark_noinconsistent.bed \
  -T /input/idt_capture_novogene.grch38.bed \
  -r /reference/GRCh38_no_alt_analysis_set.fasta \
  -o /happy/happy.output \
  --engine=vcfeval \
  --pass-only
```