PanPop

https://github.com/starskyzheng/panpop
Admin
PanPop is a sequence-aware structural variation (SV) merging and processing pipeline designed for...
Tokens:5,477
Snippets:26
Trust Score:7.3
License:MIT
Update:11 months ago
Show doc for...
Tokens:
Raw
### Install PanPop Manually

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

Installs PanPop by manually installing required software and Python packages. It involves adding software paths to a configuration file and installing packages using pip and cpanm.

```sh
pip3 install --user snakemake
curl -L https://cpanmin.us | perl - App::cpanminus
cpanm Data::Dumper MCE::Flow MCE::Candy MCE::Channel MCE::Shared Getopt::Long List::Util Carp File::Spec YAML Tie::CharArray IPC::Open2 File::Temp
```

--------------------------------

### Install PanPop with Conda

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

Installs PanPop and its dependencies using conda, creating a dedicated environment. This is the recommended installation method. It also installs other necessary Perl modules.

```sh
conda env create -f conda.panpop.yaml
conda activate panpop
conda install -c bioconda svim-asm
cpanm MCE::Channel Tie::CharArray
```

--------------------------------

### Run PanPop Pipeline for TGS and NGS Data

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

This command initiates the PanPop pipeline for either TGS or NGS datasets using Snakemake. It requires cloning the repository, navigating to the directory, and executing Snakemake with specified parameters and Snakefile. The results are generated in the example directories. Ensure the correct config file is used for each data type.

```shell
git clone https://github.com/StarSkyZheng/panpop.git
cd panpop
snakemake -j 3 --reason --printshellcmds -s Snakefile_TGS # For TGS dataset, using configs/config.TGS.yaml
snakemake -j 3 --reason --printshellcmds -s Snakefile_NGS # For NGS dataset, using configs/config.NGS.yaml
```

--------------------------------

### Execute TGS Pipeline with Snakemake

Source: https://context7.com/starskyzheng/panpop/llms.txt

This snippet outlines the steps to set up and execute the PanPop pipeline for third-generation sequencing (TGS) data. It involves cloning the repository, installing dependencies via Conda, preparing sample read lists, configuring the pipeline by editing `configs/config.TGS.yaml`, and finally running the Snakemake workflow. The output includes per-sample SVs and population-merged SVs.

```bash
# Clone repository
git clone https://github.com/StarSkyZheng/panpop.git
cd panpop

# Install dependencies via conda
conda env create -f conda.panpop.yaml
conda activate panpop
conda install -c bioconda svim-asm
cpanm MCE::Channel Tie::CharArray

# Prepare sample reads list (tab-separated: sample, platform, fq, AssmbFasta)
cat > example/example_long_reads/1.sample.reads.tsv << EOF
sample	platform	fq	AssmbFasta
s1	hifi	s1.fq.gz	s1.fa
s2	hifi	s1.fq.gz	-
s3	pb	s1.fq.gz	s1.fa
s4	ont	s1.fq.gz	s2.fa
EOF

# Configure pipeline (edit configs/config.TGS.yaml)
# workdir: "example/example_long_reads"
# sample_reads_list_file: "1.sample.reads.tsv"
# ref: 'Ref.fa'
# mode: "enhanced"  # or "simple"
# MIN_SUPPORT_CALLER: 2
# SV_CALLER_LIST: [sniffles, cuteSV, svim, pbsv, SVIM_asm]
# MAPPER: 'minimap2'  # or 'ngmlr'

# Execute TGS pipeline with 16 threads
snakemake -j 16 --reason --printshellcmds -s Snakefile_TGS --configfile configs/config.TGS.yaml

# Output VCFs located in:
# - 04_consensus_vcf/{sample}/10.concat_inv.split.sv.vcf.gz (per-sample SVs)
# - 05_merge_samples/15.thin3.sv.vcf.gz (population-merged SVs)
```

--------------------------------

### Run PART Algorithm Standalone (Perl)

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

This command executes the PART algorithm standalone using Perl scripts. It requires an input VCF file, an output directory, a reference FASTA file, and the number of threads to use. The `--tmpdir` option specifies a temporary directory. The `not_first_merge` option should be used for subsequent runs after the initial merge.

```perl
perl panpop/bin/PART_run.pl --in_vcf INPUT.vcf -o OUTDIR_RUN1 -r REFERENCE.FASTA  -t 16 --tmpdir TMPDIR
perl panpop/bin/PART_run.pl --in_vcf OUTDIR_RUN1/3.final.vcf.gz -o OUTDIR_RUN2 -r REFERENCE.FASTA -t 16 --tmpdir TMPDIR -not_first_merge
```

--------------------------------

### Run Fill-depth-information Algorithm Standalone (Perl)

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

This command runs the Fill-depth-information algorithm standalone using a Perl script. It requires an input VCF, an output directory, a reference FASTA file, the number of threads, and a sample information TSV file. The sample information file should contain sample name, BAM path, and VCF path in tab-separated format without a header.

```perl
perl panpop/bin/Fill_DP.pl --in_vcf INPUT.vcf --outdir OUTDIR --ref_fasta_file REFERENCE.FASTA --threads 16 --sid2files SAMPLE_INFORMATION.tsv
```

--------------------------------

### Construct GFA File using VG (Command Line)

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

This command constructs a GFA file from a reference FASTA and a processed VCF file (e.g., after removing asterisks). It utilizes the vg construct tool and outputs the result in VG format, which is then converted to GFA format using vg view. This is a crucial step in preparing graph-genome data for certain analyses.

```bash
vg construct -r REF.fa -v OUT1.vcf.gz -t 4 > OUT2.vg; vg view --vg-in --gfa OUT2.vg > OUT3.gfa
```

--------------------------------

### Run PART Algorithm for SV Merging (Perl)

Source: https://context7.com/starskyzheng/panpop/llms.txt

This snippet shows how to use the standalone PART algorithm script (`PART_run.pl`) for sequence-aware realignment and merging of structural variations from any VCF file. It requires a complete VCF with REF, ALT, and GT fields. The script is typically run twice for optimal merging. Inputs include the input VCF, reference genome, and desired threads. Temporary directory can also be specified.

```perl
# PART requires complete VCF with REF, ALT, and GT fields filled
# Run PART algorithm twice for optimal merging (recommended)

# First merge pass
perl panpop/bin/PART_run.pl \
  --in_vcf input_variants.vcf.gz \
  -o output_run1 \
  -r reference_genome.fasta \
  -t 16 \
  --tmpdir /tmp/panpop_temp

# Second merge pass (improves accuracy)
perl panpop/bin/PART_run.pl \
  --in_vcf output_run1/3.final.vcf.gz \
  -o output_run2 \
  -r reference_genome.fasta \
  -t 16 \
  --tmpdir /tmp/panpop_temp \
  -not_first_merge

# Final optimized VCF output: output_run2/3.final.vcf.gz

# PART performs three stages:
# 1. Realignment: extends and realigns nearby variants (1.realign0.sorted.vcf.gz)
# 2. Thin1: merges similar alleles by sequence identity (2.thin1.sorted.vcf.gz)
```

--------------------------------

### Execute NGS Pipeline with Snakemake

Source: https://context7.com/starskyzheng/panpop/llms.txt

This snippet demonstrates how to run the PanPop pipeline for next-generation sequencing (NGS) data, focusing on genotyping or augmenting SVs from graph-based genomes using VG toolkit. It requires preparing a GFA file for the graph genome, a sample read list, and configuring the pipeline by editing `configs/config.NGS.yaml`. The execution uses Snakemake, producing PAV and standard format SVs in genotype mode, or InDels and novel SVs in augment mode.

```bash
# Prepare graph genome in GFA format
# The GFA file must contain P lines for chromosomes (numeric names only)
# Place as: example/example_short_reads/Ref.gfa

# Prepare sample reads list (tab-separated: sampleid, read1_path, read2_path)
cat > example/example_short_reads/1.sample.reads.list << EOF
s1 r1.fq.gz r2.fq.gz
s2 r1.fq.gz r2.fq.gz
EOF

# Configure pipeline (edit configs/config.NGS.yaml)
# workdir: "example/example_short_reads"
# sample_reads_list_file: "1.sample.reads.list"
# split_chr: True  # or False for single-node
# mode: "genotype"  # or "augment" for novel SVs
# graph: 'Ref'  # filename without .gfa extension
# mapper: 'gaffe'  # or 'map'

# Execute NGS pipeline with 16 threads
snakemake -j 16 --reason --printshellcmds -s Snakefile_NGS --configfile configs/config.NGS.yaml

# Genotype mode outputs:
# - 5.final_result/1.final_mergechr.pav.sv.vcf.gz (PAV format SVs)
# - 5.final_result/2.final_mergechr.sv.vcf.gz (standard format SVs)

# Augment mode outputs:
# - 9.aug_final_result/5.thin3.final_mergechr.all.vcf.gz (InDels)
# - 9.aug_final_result/6.filter_maf1.final_mergechr.all.sv.vcf.gz (novel SVs)
```

--------------------------------

### Configure Advanced Filtering Parameters

Source: https://context7.com/starskyzheng/panpop/llms.txt

This section outlines advanced filtering parameters configurable in `configs/base.yaml` for quality control. It includes thresholds for mapping quality, allele frequency, missing data, depth, and SV classification.

```yaml
# Quality and depth filters
MAP_MINQ: 5  # Minimum mapping quality
MAF: 0.01  # Minor allele frequency threshold (remove rarer alleles)
max_missing_rate: 0.3  # Maximum missing rate (0=no missing, 1=no filter)
dp_min_fold: 0.333  # Minimum depth = mean_depth * 0.333
dp_max_fold: 3  # Maximum depth = mean_depth * 3
mad_min_fold: 0.111  # Minimum allele depth = mean_depth * (1/3 * 1/3)
mad_min_abs: 1  # Absolute minimum allele depth

# Realignment parameters
realign_extend_bp_max: 10  # Maximum extension for nearby variant merging
realign_extend_bp_min: 1  # Minimum extension for realignment

# SV classification
SV_min_length: 50  # Variants >= 50bp are SVs, else InDels

# Augment mode thresholds (NGS)
aug_nonmut_min_cov: 0.8  # Minimum coverage to call reference allele
aug_nomut_min_dp: 3  # Minimum depth for reference allele

# Thinning parameters
sv2pav_merge_identity_threshold: 0.8  # Sequence similarity for merging
ssindel_maxlen: 10  # Small indel maximum length
ssindel_maxallele: 3  # Small indel maximum allele count
```

--------------------------------

### Build Graph Genome in GFA Format

Source: https://context7.com/starskyzheng/panpop/llms.txt

Constructs GFA format graph genomes from FASTA files, suitable for Next-Generation Sequencing (NGS) pipeline processing. It supports building graphs from backbone and additional genome assemblies, converting rGFA, and removing non-backbone paths.

```bash
# Build graph from backbone + additional genome assemblies
# Backbone chromosomes must be numeric only
# Non-backbone genomes must NOT be numeric only

perl scripts/build_graph.pl \
  -b backbone_genome.fa \
  -o output_graph_dir \
  -t 16 \
  additional_genome1.fa \
  additional_genome2.fa \
  additional_genome3.fa

# Output: output_graph_dir/3.final.gfa

# Convert rGFA from minigraph to compatible GFA
perl scripts/build_graph.pl \
  --rgfa minigraph_output.rgfa \
  -o output_dir \
  -t 16

# Remove non-backbone paths from existing GFA
perl scripts/build_graph.pl \
  --gfa input_genome.gfa \
  -o output_dir \
  -t 16

# Chromosome naming requirements:
# - Backbone: 1, 2, 3, ... (numeric only, no leading zeros)
# - Non-backbone: chr1_sample1, scaffold_A, etc. (must contain non-numeric characters)
```

--------------------------------

### Convert Complex SVs to PAV Format

Source: https://context7.com/starskyzheng/panpop/llms.txt

This section describes the conversion of complex structural variants (SVs) to the PAV format using the `thin2` script. It also includes a pre-merging step for multiple input VCFs using `bcftools`.

```bash
bcftools merge sample1.vcf.gz sample2.vcf.gz -o merged_input.vcf.gz
perl panpop/bin/PART_run.pl --in_vcf merged_input.vcf.gz -o output -r ref.fa -t 16
```

--------------------------------

### Build Graph from FASTA Files (Perl)

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

The `build_graph.pl` script is a utility for generating or modifying GFA format files from genome FASTA files. It supports creating suitable GFA formats and can also parse rGFA format generated by minigraph. Additionally, it can be used to remove non-backbone paths from existing GFA files.

```bash
scripts/build_graph.pl genome.fasta
```

```bash
scripts/build_graph.pl --rgfa minigraph_output.rgfa
```

```bash
scripts/build_graph.pl --gfa input.gfa --remove_non_backbone
```

--------------------------------

### Fill Missing Depth Information for SVs

Source: https://context7.com/starskyzheng/panpop/llms.txt

Recalculates and fills missing depth information for structural variants to reduce false negatives. It requires a sample information TSV file mapping sample names to BAM and VCF paths.

```bash
# Prepare sample information TSV (no header, tab-separated: sample_name, BAM_path, VCF_path)
cat > sample_info.tsv << EOF
sample1	/path/to/sample1.sorted.bam	/path/to/sample1.vcf.gz
sample2	/path/to/sample2.sorted.bam	/path/to/sample2.vcf.gz
sample3	/path/to/sample3.sorted.bam	/path/to/sample3.vcf.gz
EOF

# Fill depth information for merged VCF with missing genotypes (./.)
perl panpop/bin/Fill_DP.pl \
  --in_vcf population_merged.vcf.gz \
  --outdir depth_filled_output \
  --ref_fasta_file reference_genome.fasta \
  --threads 16 \
  --sid2files sample_info.tsv

# Process stages:
# 1. Extract variant positions (1.poss)
# 2. Calculate depth at each position from BAMs (2.cal_aug_dp/*.depth.txt.gz)
# 3. Fill VCF with recalculated depths (3.fill_dp/*.sort.vcf.gz)
# 4. Merge all samples (4.bcftools_merge/2.merged.vcf.gz)

# Output: depth_filled_output/4.bcftools_merge/2.merged.vcf.gz
# Post-processing: Filter by depth thresholds to remove false positives
bcftools filter -i 'FORMAT/DP >= 3' depth_filled_output/4.bcftools_merge/2.merged.vcf.gz \
  -o filtered_final.vcf.gz
```

--------------------------------

### Pipeline Cleanup

Source: https://context7.com/starskyzheng/panpop/llms.txt

Provides commands to remove all intermediate files and outputs generated by the TGS (long reads) and NGS (short reads) pipelines. This ensures a clean working environment after analysis.

```bash
# Clean all TGS pipeline outputs
cd example/example_long_reads
snakemake cleanall -s ../../Snakefile_TGS

# Clean all NGS pipeline outputs
cd example/example_short_reads
snakemake cleanall -s ../../Snakefile_NGS

# Removed items include:
# - SV caller outputs (2.callSV/)
# - Merged VCFs (3.merge_rawvcf/, 7.aug_merge_rawvcf/)
```

--------------------------------

### Convert TGS SVs for NGS Processing

Source: https://context7.com/starskyzheng/panpop/llms.txt

Removes asterisk ALT alleles from TGS-derived VCF files to enable VG graph construction for NGS analysis. It then constructs a VG graph and converts it to GFA format.

```bash
# Process TGS-derived VCF for NGS pipeline compatibility
perl scripts/prase_vcf_rm_asterisk.pl \
  --infile tgs_merged_svs.vcf.gz \
  -r reference_genome.fasta \
  -o asterisk_removed.vcf.gz

# Construct VG graph from processed VCF
vg construct \
  -r reference_genome.fasta \
  -v asterisk_removed.vcf.gz \
  -t 4 \
  > graph_genome.vg

# Convert to GFA format for NGS pipeline
vg view --vg-in --gfa graph_genome.vg > graph_genome.gfa

# Move to NGS working directory
mv graph_genome.gfa example/example_short_reads/Ref.gfa

# Now run NGS pipeline in genotype mode
snakemake -j 16 -s Snakefile_NGS --configfile configs/config.NGS.yaml
```

--------------------------------

### Process VCF for TGS SVs to NGS SVs Conversion (Perl)

Source: https://github.com/starskyzheng/panpop/blob/main/README.md

This script processes VCF files to remove asterisks from ALT columns, which is a necessary step before using VG to construct a GFA file. It is particularly useful when converting TGS SVs generated by PanPop into a format suitable for NGS data processing. The script takes an input VCF file, a reference FASTA file, and an output VCF file as arguments.

```bash
perl scripts/prase_vcf_rm_asterisk.pl --infile IN.vcf.gz -r REF.fa -o OUT1.vcf.gz
```
Tokens:
Raw