### Install PanPop Manually Source: https://github.com/starskyzheng/panpop/blob/main/README.md Installs PanPop by manually installing required software and Python packages. It involves adding software paths to a configuration file and installing packages using pip and cpanm. ```sh pip3 install --user snakemake curl -L https://cpanmin.us | perl - App::cpanminus cpanm Data::Dumper MCE::Flow MCE::Candy MCE::Channel MCE::Shared Getopt::Long List::Util Carp File::Spec YAML Tie::CharArray IPC::Open2 File::Temp ``` -------------------------------- ### Install PanPop with Conda Source: https://github.com/starskyzheng/panpop/blob/main/README.md Installs PanPop and its dependencies using conda, creating a dedicated environment. This is the recommended installation method. It also installs other necessary Perl modules. ```sh conda env create -f conda.panpop.yaml conda activate panpop conda install -c bioconda svim-asm cpanm MCE::Channel Tie::CharArray ``` -------------------------------- ### Run PanPop Pipeline for TGS and NGS Data Source: https://github.com/starskyzheng/panpop/blob/main/README.md This command initiates the PanPop pipeline for either TGS or NGS datasets using Snakemake. It requires cloning the repository, navigating to the directory, and executing Snakemake with specified parameters and Snakefile. The results are generated in the example directories. Ensure the correct config file is used for each data type. ```shell git clone https://github.com/StarSkyZheng/panpop.git cd panpop snakemake -j 3 --reason --printshellcmds -s Snakefile_TGS # For TGS dataset, using configs/config.TGS.yaml snakemake -j 3 --reason --printshellcmds -s Snakefile_NGS # For NGS dataset, using configs/config.NGS.yaml ``` -------------------------------- ### Execute TGS Pipeline with Snakemake Source: https://context7.com/starskyzheng/panpop/llms.txt This snippet outlines the steps to set up and execute the PanPop pipeline for third-generation sequencing (TGS) data. It involves cloning the repository, installing dependencies via Conda, preparing sample read lists, configuring the pipeline by editing `configs/config.TGS.yaml`, and finally running the Snakemake workflow. The output includes per-sample SVs and population-merged SVs. ```bash # Clone repository git clone https://github.com/StarSkyZheng/panpop.git cd panpop # Install dependencies via conda conda env create -f conda.panpop.yaml conda activate panpop conda install -c bioconda svim-asm cpanm MCE::Channel Tie::CharArray # Prepare sample reads list (tab-separated: sample, platform, fq, AssmbFasta) cat > example/example_long_reads/1.sample.reads.tsv << EOF sample platform fq AssmbFasta s1 hifi s1.fq.gz s1.fa s2 hifi s1.fq.gz - s3 pb s1.fq.gz s1.fa s4 ont s1.fq.gz s2.fa EOF # Configure pipeline (edit configs/config.TGS.yaml) # workdir: "example/example_long_reads" # sample_reads_list_file: "1.sample.reads.tsv" # ref: 'Ref.fa' # mode: "enhanced" # or "simple" # MIN_SUPPORT_CALLER: 2 # SV_CALLER_LIST: [sniffles, cuteSV, svim, pbsv, SVIM_asm] # MAPPER: 'minimap2' # or 'ngmlr' # Execute TGS pipeline with 16 threads snakemake -j 16 --reason --printshellcmds -s Snakefile_TGS --configfile configs/config.TGS.yaml # Output VCFs located in: # - 04_consensus_vcf/{sample}/10.concat_inv.split.sv.vcf.gz (per-sample SVs) # - 05_merge_samples/15.thin3.sv.vcf.gz (population-merged SVs) ``` -------------------------------- ### Run PART Algorithm Standalone (Perl) Source: https://github.com/starskyzheng/panpop/blob/main/README.md This command executes the PART algorithm standalone using Perl scripts. It requires an input VCF file, an output directory, a reference FASTA file, and the number of threads to use. The `--tmpdir` option specifies a temporary directory. The `not_first_merge` option should be used for subsequent runs after the initial merge. ```perl perl panpop/bin/PART_run.pl --in_vcf INPUT.vcf -o OUTDIR_RUN1 -r REFERENCE.FASTA -t 16 --tmpdir TMPDIR perl panpop/bin/PART_run.pl --in_vcf OUTDIR_RUN1/3.final.vcf.gz -o OUTDIR_RUN2 -r REFERENCE.FASTA -t 16 --tmpdir TMPDIR -not_first_merge ``` -------------------------------- ### Run Fill-depth-information Algorithm Standalone (Perl) Source: https://github.com/starskyzheng/panpop/blob/main/README.md This command runs the Fill-depth-information algorithm standalone using a Perl script. It requires an input VCF, an output directory, a reference FASTA file, the number of threads, and a sample information TSV file. The sample information file should contain sample name, BAM path, and VCF path in tab-separated format without a header. ```perl perl panpop/bin/Fill_DP.pl --in_vcf INPUT.vcf --outdir OUTDIR --ref_fasta_file REFERENCE.FASTA --threads 16 --sid2files SAMPLE_INFORMATION.tsv ``` -------------------------------- ### Construct GFA File using VG (Command Line) Source: https://github.com/starskyzheng/panpop/blob/main/README.md This command constructs a GFA file from a reference FASTA and a processed VCF file (e.g., after removing asterisks). It utilizes the vg construct tool and outputs the result in VG format, which is then converted to GFA format using vg view. This is a crucial step in preparing graph-genome data for certain analyses. ```bash vg construct -r REF.fa -v OUT1.vcf.gz -t 4 > OUT2.vg; vg view --vg-in --gfa OUT2.vg > OUT3.gfa ``` -------------------------------- ### Run PART Algorithm for SV Merging (Perl) Source: https://context7.com/starskyzheng/panpop/llms.txt This snippet shows how to use the standalone PART algorithm script (`PART_run.pl`) for sequence-aware realignment and merging of structural variations from any VCF file. It requires a complete VCF with REF, ALT, and GT fields. The script is typically run twice for optimal merging. Inputs include the input VCF, reference genome, and desired threads. Temporary directory can also be specified. ```perl # PART requires complete VCF with REF, ALT, and GT fields filled # Run PART algorithm twice for optimal merging (recommended) # First merge pass perl panpop/bin/PART_run.pl \ --in_vcf input_variants.vcf.gz \ -o output_run1 \ -r reference_genome.fasta \ -t 16 \ --tmpdir /tmp/panpop_temp # Second merge pass (improves accuracy) perl panpop/bin/PART_run.pl \ --in_vcf output_run1/3.final.vcf.gz \ -o output_run2 \ -r reference_genome.fasta \ -t 16 \ --tmpdir /tmp/panpop_temp \ -not_first_merge # Final optimized VCF output: output_run2/3.final.vcf.gz # PART performs three stages: # 1. Realignment: extends and realigns nearby variants (1.realign0.sorted.vcf.gz) # 2. Thin1: merges similar alleles by sequence identity (2.thin1.sorted.vcf.gz) ``` -------------------------------- ### Execute NGS Pipeline with Snakemake Source: https://context7.com/starskyzheng/panpop/llms.txt This snippet demonstrates how to run the PanPop pipeline for next-generation sequencing (NGS) data, focusing on genotyping or augmenting SVs from graph-based genomes using VG toolkit. It requires preparing a GFA file for the graph genome, a sample read list, and configuring the pipeline by editing `configs/config.NGS.yaml`. The execution uses Snakemake, producing PAV and standard format SVs in genotype mode, or InDels and novel SVs in augment mode. ```bash # Prepare graph genome in GFA format # The GFA file must contain P lines for chromosomes (numeric names only) # Place as: example/example_short_reads/Ref.gfa # Prepare sample reads list (tab-separated: sampleid, read1_path, read2_path) cat > example/example_short_reads/1.sample.reads.list << EOF s1 r1.fq.gz r2.fq.gz s2 r1.fq.gz r2.fq.gz EOF # Configure pipeline (edit configs/config.NGS.yaml) # workdir: "example/example_short_reads" # sample_reads_list_file: "1.sample.reads.list" # split_chr: True # or False for single-node # mode: "genotype" # or "augment" for novel SVs # graph: 'Ref' # filename without .gfa extension # mapper: 'gaffe' # or 'map' # Execute NGS pipeline with 16 threads snakemake -j 16 --reason --printshellcmds -s Snakefile_NGS --configfile configs/config.NGS.yaml # Genotype mode outputs: # - 5.final_result/1.final_mergechr.pav.sv.vcf.gz (PAV format SVs) # - 5.final_result/2.final_mergechr.sv.vcf.gz (standard format SVs) # Augment mode outputs: # - 9.aug_final_result/5.thin3.final_mergechr.all.vcf.gz (InDels) # - 9.aug_final_result/6.filter_maf1.final_mergechr.all.sv.vcf.gz (novel SVs) ``` -------------------------------- ### Configure Advanced Filtering Parameters Source: https://context7.com/starskyzheng/panpop/llms.txt This section outlines advanced filtering parameters configurable in `configs/base.yaml` for quality control. It includes thresholds for mapping quality, allele frequency, missing data, depth, and SV classification. ```yaml # Quality and depth filters MAP_MINQ: 5 # Minimum mapping quality MAF: 0.01 # Minor allele frequency threshold (remove rarer alleles) max_missing_rate: 0.3 # Maximum missing rate (0=no missing, 1=no filter) dp_min_fold: 0.333 # Minimum depth = mean_depth * 0.333 dp_max_fold: 3 # Maximum depth = mean_depth * 3 mad_min_fold: 0.111 # Minimum allele depth = mean_depth * (1/3 * 1/3) mad_min_abs: 1 # Absolute minimum allele depth # Realignment parameters realign_extend_bp_max: 10 # Maximum extension for nearby variant merging realign_extend_bp_min: 1 # Minimum extension for realignment # SV classification SV_min_length: 50 # Variants >= 50bp are SVs, else InDels # Augment mode thresholds (NGS) aug_nonmut_min_cov: 0.8 # Minimum coverage to call reference allele aug_nomut_min_dp: 3 # Minimum depth for reference allele # Thinning parameters sv2pav_merge_identity_threshold: 0.8 # Sequence similarity for merging ssindel_maxlen: 10 # Small indel maximum length ssindel_maxallele: 3 # Small indel maximum allele count ``` -------------------------------- ### Build Graph Genome in GFA Format Source: https://context7.com/starskyzheng/panpop/llms.txt Constructs GFA format graph genomes from FASTA files, suitable for Next-Generation Sequencing (NGS) pipeline processing. It supports building graphs from backbone and additional genome assemblies, converting rGFA, and removing non-backbone paths. ```bash # Build graph from backbone + additional genome assemblies # Backbone chromosomes must be numeric only # Non-backbone genomes must NOT be numeric only perl scripts/build_graph.pl \ -b backbone_genome.fa \ -o output_graph_dir \ -t 16 \ additional_genome1.fa \ additional_genome2.fa \ additional_genome3.fa # Output: output_graph_dir/3.final.gfa # Convert rGFA from minigraph to compatible GFA perl scripts/build_graph.pl \ --rgfa minigraph_output.rgfa \ -o output_dir \ -t 16 # Remove non-backbone paths from existing GFA perl scripts/build_graph.pl \ --gfa input_genome.gfa \ -o output_dir \ -t 16 # Chromosome naming requirements: # - Backbone: 1, 2, 3, ... (numeric only, no leading zeros) # - Non-backbone: chr1_sample1, scaffold_A, etc. (must contain non-numeric characters) ``` -------------------------------- ### Convert Complex SVs to PAV Format Source: https://context7.com/starskyzheng/panpop/llms.txt This section describes the conversion of complex structural variants (SVs) to the PAV format using the `thin2` script. It also includes a pre-merging step for multiple input VCFs using `bcftools`. ```bash bcftools merge sample1.vcf.gz sample2.vcf.gz -o merged_input.vcf.gz perl panpop/bin/PART_run.pl --in_vcf merged_input.vcf.gz -o output -r ref.fa -t 16 ``` -------------------------------- ### Build Graph from FASTA Files (Perl) Source: https://github.com/starskyzheng/panpop/blob/main/README.md The `build_graph.pl` script is a utility for generating or modifying GFA format files from genome FASTA files. It supports creating suitable GFA formats and can also parse rGFA format generated by minigraph. Additionally, it can be used to remove non-backbone paths from existing GFA files. ```bash scripts/build_graph.pl genome.fasta ``` ```bash scripts/build_graph.pl --rgfa minigraph_output.rgfa ``` ```bash scripts/build_graph.pl --gfa input.gfa --remove_non_backbone ``` -------------------------------- ### Fill Missing Depth Information for SVs Source: https://context7.com/starskyzheng/panpop/llms.txt Recalculates and fills missing depth information for structural variants to reduce false negatives. It requires a sample information TSV file mapping sample names to BAM and VCF paths. ```bash # Prepare sample information TSV (no header, tab-separated: sample_name, BAM_path, VCF_path) cat > sample_info.tsv << EOF sample1 /path/to/sample1.sorted.bam /path/to/sample1.vcf.gz sample2 /path/to/sample2.sorted.bam /path/to/sample2.vcf.gz sample3 /path/to/sample3.sorted.bam /path/to/sample3.vcf.gz EOF # Fill depth information for merged VCF with missing genotypes (./.) perl panpop/bin/Fill_DP.pl \ --in_vcf population_merged.vcf.gz \ --outdir depth_filled_output \ --ref_fasta_file reference_genome.fasta \ --threads 16 \ --sid2files sample_info.tsv # Process stages: # 1. Extract variant positions (1.poss) # 2. Calculate depth at each position from BAMs (2.cal_aug_dp/*.depth.txt.gz) # 3. Fill VCF with recalculated depths (3.fill_dp/*.sort.vcf.gz) # 4. Merge all samples (4.bcftools_merge/2.merged.vcf.gz) # Output: depth_filled_output/4.bcftools_merge/2.merged.vcf.gz # Post-processing: Filter by depth thresholds to remove false positives bcftools filter -i 'FORMAT/DP >= 3' depth_filled_output/4.bcftools_merge/2.merged.vcf.gz \ -o filtered_final.vcf.gz ``` -------------------------------- ### Pipeline Cleanup Source: https://context7.com/starskyzheng/panpop/llms.txt Provides commands to remove all intermediate files and outputs generated by the TGS (long reads) and NGS (short reads) pipelines. This ensures a clean working environment after analysis. ```bash # Clean all TGS pipeline outputs cd example/example_long_reads snakemake cleanall -s ../../Snakefile_TGS # Clean all NGS pipeline outputs cd example/example_short_reads snakemake cleanall -s ../../Snakefile_NGS # Removed items include: # - SV caller outputs (2.callSV/) # - Merged VCFs (3.merge_rawvcf/, 7.aug_merge_rawvcf/) ``` -------------------------------- ### Convert TGS SVs for NGS Processing Source: https://context7.com/starskyzheng/panpop/llms.txt Removes asterisk ALT alleles from TGS-derived VCF files to enable VG graph construction for NGS analysis. It then constructs a VG graph and converts it to GFA format. ```bash # Process TGS-derived VCF for NGS pipeline compatibility perl scripts/prase_vcf_rm_asterisk.pl \ --infile tgs_merged_svs.vcf.gz \ -r reference_genome.fasta \ -o asterisk_removed.vcf.gz # Construct VG graph from processed VCF vg construct \ -r reference_genome.fasta \ -v asterisk_removed.vcf.gz \ -t 4 \ > graph_genome.vg # Convert to GFA format for NGS pipeline vg view --vg-in --gfa graph_genome.vg > graph_genome.gfa # Move to NGS working directory mv graph_genome.gfa example/example_short_reads/Ref.gfa # Now run NGS pipeline in genotype mode snakemake -j 16 -s Snakefile_NGS --configfile configs/config.NGS.yaml ``` -------------------------------- ### Process VCF for TGS SVs to NGS SVs Conversion (Perl) Source: https://github.com/starskyzheng/panpop/blob/main/README.md This script processes VCF files to remove asterisks from ALT columns, which is a necessary step before using VG to construct a GFA file. It is particularly useful when converting TGS SVs generated by PanPop into a format suitable for NGS data processing. The script takes an input VCF file, a reference FASTA file, and an output VCF file as arguments. ```bash perl scripts/prase_vcf_rm_asterisk.pl --infile IN.vcf.gz -r REF.fa -o OUT1.vcf.gz ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.