Snakemake

https://snakemake.readthedocs.io/en/stable/
Admin
Snakemake is a workflow management system that enables the creation of reproducible and scalable...
Tokens:185,351
Snippets:1,539
Trust Score:8.3
Update:3 hours ago
Show doc for...
Tokens:
Raw
### Download example data with curl

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Downloads the example data tarball from the snakemake-tutorial-data GitHub repository. The file is saved as `snakemake-tutorial-data.tar.gz` in the current directory.

```bash
$ curl -L https://api.github.com/repos/snakemake/snakemake-tutorial-data/tarball -o snakemake-tutorial-data.tar.gz
```

--------------------------------

### Import environment with Pixi

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Initializes a Pixi project and imports the `environment.yaml` file to install all required workflow dependencies.

```bash
$ pixi init --import environment.yaml
```

--------------------------------

### Activate environment with Conda

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Activates the `snakemake-tutorial` Conda environment so you can use the installed tools.

```bash
$ conda activate snakemake-tutorial
```

--------------------------------

### Activate Snakemake environment and verify

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Activates the 'snakemake' environment and runs 'snakemake --help' to verify the installation. This is the activation step after creating the full installation environment.

```bash
$ conda activate snakemake
$ snakemake --help
```

--------------------------------

### choose_tmp examples

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Chooses a temporary folder from a list; returns the system temporary folder if none is valid. Four examples are shown.

```python
choose_tmp(["foo/bar", "/tmp"], read=True, write=True, open=True, creatable=True)  # "foo/bar"
```

```python
choose_tmp(["/foo/bar", "/tmp", "/tmp/jobid"], read=True, write=True, open=True, creatable=None)  # "/tmp"
```

```python
choose_tmp(["/foo/bar", "/tmp/$USER", "/tmp/"], read=True, write=True, open=True, creatable=True  # "/tmp/$USER"
```

```python
choose_tmp(["/foo", "/bar"], read=True, write=True, open=True, creatable=True)  # "system_tmpdir"
```

--------------------------------

### Complete toy example with from_queue

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

A complete example showing a thread filling a queue with input files, a target rule using from_queue, and a generate rule that creates the requested files. The finish sentinel is an object() and the queue is a queue.Queue().

```python
import threading, queue, time

# the finish sentinel
finish_sentinel = object()
# a synchronized queue for the input files
all_results = queue.Queue()

# a thread that fills the queue with input files to be considered
def update_results():
    try:
        for i in range(10):
            all_results.put(f"test{i}.txt")
            time.sleep(1)
        all_results.put(finish_sentinel)
        all_results.join()
    except (KeyboardInterrupt, SystemExit):
        return

update_thread = threading.Thread(target=update_results)
update_thread.start()


# target rule which will be continuously updated until the queue is finished
rule all:
    input:
        from_queue(all_results, finish_sentinel=finish_sentinel)


# job that generates the requested input files
rule generate:
    output:
        "test{i}.txt"
    shell:
        "echo {wildcards.i} > {output}"
```

--------------------------------

### Serve documentation with live reload

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Use this command to start a local server that automatically reloads when changes are detected.

```bash
$ pixi run docs
[sphinx-autobuild] Starting initial build
[sphinx-autobuild] > python -m sphinx build docs/ docs/_build/html
...
The HTML pages are in docs/_build/html.
[sphinx-autobuild] Serving on http://0.0.0.0:8000
[sphinx-autobuild] Waiting to detect changes...
```

--------------------------------

### Install pixi environment

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Run this command after cloning the Snakemake repository to set up the development environment with pixi.

```bash
$ pixi install
```

--------------------------------

### Example global profile configuration (profile.yaml)

Source: https://snakemake.readthedocs.io/en/stable/executing/cli.html

Default options for a global profile, including the executor, jobs, cores, and default resources. Comments recommend motivating configuration choices. This example uses the slurm executor and oversubscribes jobs and cores.

```yaml
# This cluster uses the slurm job submission system, for details on how to
# configure the respective executor plugin for snakemake, see
# https://snakemake.github.io/snakemake-plugin-catalog/plugins/executor/slurm.html
executor: slurm
# This cluster allows users to run 100 jobs concurrently. As the slurm
# executor plugin only checks for completed jobs with a lag, we slightly
# oversubmit jobs to always have jobs available in the queue.
jobs: 110
# This cluster allows the use of 200 cores per user. As with the number of
# jobs, we slightly oversubmit.
cores: 220
# If a rule doesn't have the following resources specified, it will default
# to requesting the resources specified here.
default-resources:
  mem_mb: 1024
```

--------------------------------

### Recommended post-deploy script for installing R packages

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html

Recommended pattern for installing R packages not available via conda. Uses `--no-environ` to avoid inheriting user library paths (e.g., from `$R_LIBS_USER` in `.Renviron`) and sets `R_REMOTES_NO_ERRORS_FROM_WARNINGS` to make `remotes` fail on errors instead of warnings.

```bash
#!env bash
set -o pipefail
Rscript --no-environ -e 'Sys.setenv(R_REMOTES_NO_ERRORS_FROM_WARNINGS="false"); remotes::install_github("some-org/RPackageNotOnConda", ref = "v1.3.2", upgrade = "never")'
```

--------------------------------

### Install a development version of Snakemake with conda and pip

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Create a conda environment with Snakemake dependencies, activate it, and install the current main branch from the GitHub repository. Useful for testing unreleased bug fixes.

```bash
$ conda create --only-deps -n snakemake-dev -c conda-forge -c bioconda -c nodefaults snakemake
$ conda activate snakemake-dev
$ pip install git+https://github.com/snakemake/snakemake
```

--------------------------------

### Example workflow execution output

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Shows the expected Snakemake execution log for the example workflow with the given params.tsv. Demonstrates how wildcards are expanded for each parameter combination.

```text
[Fri Nov 27 20:57:27 2020]
rule simulate:
    output: results/simulations/alpha~2.0/beta~0.0/gamma~3.9.tsv
    jobid: 4
    wildcards: alpha=2.0, beta=0.0, gamma=3.9

[Fri Nov 27 20:57:27 2020]
rule simulate:
    output: results/simulations/alpha~1.0/beta~0.1/gamma~0.99.tsv
    jobid: 2
    wildcards: alpha=1.0, beta=0.1, gamma=0.99

[Fri Nov 27 20:57:27 2020]
rule plot:
    input: results/simulations/alpha~2.0/beta~0.0/gamma~3.9.tsv
    output: results/plots/alpha~2.0/beta~0.0/gamma~3.9.pdf
    jobid: 3
    wildcards: alpha=2.0, beta=0.0, gamma=3.9


[Fri Nov 27 20:57:27 2020]
rule plot:
    input: results/simulations/alpha~1.0/beta~0.1/gamma~0.99.tsv
    output: results/plots/alpha~1.0/beta~0.1/gamma~0.99.pdf
    jobid: 1
    wildcards: alpha=1.0, beta=0.1, gamma=0.99


[Fri Nov 27 20:57:27 2020]
localrule all:
    input: results/plots/alpha~1.0/beta~0.1/gamma~0.99.pdf, results/plots/alpha~2.0/beta~0.0/gamma~3.9.pdf
    jobid: 0
```

--------------------------------

### Install cluster-generic executor plugin

Source: https://snakemake.readthedocs.io/en/stable/getting_started/migration.html

Required before using the new --executor cluster-generic option. Install via pip.

```bash
pip install snakemake-executor-plugin-cluster-generic
```

--------------------------------

### Set up a directory-specific Pixi environment for Snakemake

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Initializes a Pixi environment in the current directory, adds the conda-forge and bioconda channels, and adds Snakemake as a dependency. Run these commands in the directory where you want to install Snakemake.

```bash
$ pixi init
$ pixi workspace channel add conda-forge
$ pixi workspace channel add bioconda
$ pixi add snakemake
```

--------------------------------

### choose_folder example

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Chooses a folder from a list where the user has read/write/open access; creatable condition is ignored.

```python
choose_folder(["foo/bar", "/proc", "~"], read=True, write=True, open=True, creatable=None)  # "~"
```

--------------------------------

### choose_file example

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Chooses a file from a list where the user has read/write access; execute and creatable conditions are ignored.

```python
choose_file(["foo", "~/.bashrc"], read=True, write=True, execute=None, creatable=None)  # "~/.bashrc"
```

--------------------------------

### Build documentation with pixi

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Execute this command to build the documentation locally.

```bash
$ pixi run build-docs
```

--------------------------------

### Using lookup with collect

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Example of using lookup with collect to get sample names from a dataframe.

```python
collect("results/{item.sample}.txt", item=lookup("someval > 2", within=samples))
```

--------------------------------

### Create and enter the snakemake-tutorial working directory

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Creates a new directory named `snakemake-tutorial` and changes into it. Run these commands in your terminal before setting up the workflow.

```bash
$ mkdir snakemake-tutorial
$ cd snakemake-tutorial
```

--------------------------------

### Masking a wildcard expression in expand()

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Use double curly braces to mask a wildcard expression so it is kept in the output. This example creates strings with all values for ext but starting with the wildcard "{dataset}".

```python
expand("{{dataset}}/a.{ext}", ext=FORMATS)
```

--------------------------------

### Snakefile with report() flags for linking

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/reporting.html

Defines a rule with report() flags on input and output files, including a directory with a pattern. This setup is required for the subsequent report_href() examples to link to these report items.

```snakemake
rule a:
    input:
        report("test.html"),
        report(
            "subdir",
            patterns=["{name}.html"],
        )
    output:
        report(
            "test2.html",
        )
    script:
        "test_script.py"
```

--------------------------------

### Run snakemake with --edit-notebook

Source: https://snakemake.readthedocs.io/en/stable/tutorial/interaction_visualization_reporting/tutorial.html

Command to create a skeleton notebook and start a Jupyter server for interactive editing. Use Ctrl-Click on one of the server output options to open the notebook in the browser.

```bash
$ snakemake --sdm conda --cores 1 --edit-notebook resources/data/cars.tsv
```

--------------------------------

### Use a callable priority based on input size

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html

Define a priority as a lambda that receives `wildcards` and optionally `input`, `attempt`, and `rulename` keyword arguments. The callable must return an integer or float (will be rounded). This example dynamically prioritizes jobs by input file size so larger jobs start first.

```snakemake
rule sort:
    input: "{dataset}.txt"
    output: "{dataset}.sorted"
    priority: lambda wildcards, input: input.size_mb
    shell: "sort {input} > {output}"
```

--------------------------------

### Install Snakemake globally with pixi global install

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Installs Snakemake and conda globally using Pixi, pulling packages from conda-forge and bioconda channels. Does not require root/administrator access. The conda installation enables rule-specific software management via the conda directive; omit it if you deploy software differently.

```bash
$ pixi global install snakemake conda -c conda-forge -c bioconda
```

--------------------------------

### Install Snakemake via pip

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Use pip, the default Python package manager, to install Snakemake. Run this command in your terminal.

```bash
$ pip install snakemake
```

--------------------------------

### List pixi documentation tasks

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Run this command to view available tasks for managing the documentation environment.

```bash
$ ➜ pixi task list --environment docs
Tasks that can run on this machine:
-----------------------------------
build-apidocs, build-docs, docs

- build-apidocs   Build the API documentation in the apidocs/ directory
- build-docs      Build the documentation in the docs/ directory
- docs            Serve the documentation on http://localhost:8000 with live reload
```

--------------------------------

### Install Snakemake on RedHat/Fedora via yum

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Install Snakemake using the official package manager on RedHat or Fedora systems. Requires root or administrator access. These packages are not maintained by the snakemake organization and may be older versions.

```bash
$ sudo yum install snakemake
```

--------------------------------

### Install Snakemake on Ubuntu/Debian via apt

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Install Snakemake using the official package manager on Ubuntu or Debian systems. Requires root or administrator access. These packages are not maintained by the snakemake organization and may be older versions.

```bash
$ sudo apt-get update
$ sudo apt-get install snakemake
```

--------------------------------

### Create Conda environment with mamba

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Activates the base Conda environment and creates a new environment named `snakemake-tutorial` from the `environment.yaml` file using mamba.

```bash
$ conda activate base
$ mamba env create --name snakemake-tutorial --file environment.yaml
```

--------------------------------

### Directory layout for post-deploy script

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html

Shows the required file structure: the post-deploy script must be placed next to its corresponding environment file and have the `.post-deploy.sh` suffix.

```text
├── Snakefile
└── envs
    ├── interproscan.yaml
    └── interproscan.post-deploy.sh
```

--------------------------------

### Post-deploy script with bash shebang

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html

Example of a post-deploy script that sets a bash-specific option. If no shebang line like `#!env bash` is provided, the script is executed with the `sh` command.

```bash
#!env bash
set -o pipefail
# ...
```

--------------------------------

### Install Snakemake from Bioconda channel explicitly

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Activates the base environment and creates a new environment 'some-env' with Snakemake from the Bioconda channel using the 'bioconda::snakemake' prefix. This is useful when combining Snakemake with other channels.

```bash
$ conda activate base
$ conda create -n some-env -c conda-forge -c nodefaults bioconda::snakemake ...
```

--------------------------------

### Run simple test suite

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Runs the main tests located in tests/tests.py.

```bash
$ pixi run test-simple
```

--------------------------------

### Create minimal Snakemake conda environment

Source: https://snakemake.readthedocs.io/en/stable/getting_started/installation.html

Creates an isolated conda environment named 'snakemake' with only the necessary requirements using the 'snakemake-minimal' package. Use this when you need a minimal installation without extra features.

```bash
$ conda create -c conda-forge -c bioconda -c nodefaults -n snakemake snakemake-minimal
```

--------------------------------

### Complete Snakemake workflow with SAMPLES and rules all, bwa_map, samtools_sort, samtools_index, bcftools_call, plot_quals

Source: https://snakemake.readthedocs.io/en/stable/tutorial/basics.html

Final workflow from the tutorial summary. Defines SAMPLES list and six rules: all (final target plots/quals.svg), bwa_map (bwa mem + samtools view), samtools_sort (samtools sort), samtools_index (samtools index), bcftools_call (bcftools mpileup + call), and plot_quals (script scripts/plot-quals.py). Uses wildcards {sample} and expand() for BAM/BAI inputs.

```snakemake
SAMPLES = ["A", "B"]


rule all:
    input:
        "plots/quals.svg"


rule bwa_map:
    input:
        "data/genome.fa",
        "data/samples/{sample}.fastq"
    output:
        "mapped_reads/{sample}.bam"
    shell:
        "bwa mem {input} | samtools view -Sb - > {output}"


rule samtools_sort:
    input:
        "mapped_reads/{sample}.bam"
    output:
        "sorted_reads/{sample}.bam"
    shell:
        "samtools sort -T sorted_reads/{wildcards.sample} "
        "-O bam {input} > {output}"


rule samtools_index:
    input:
        "sorted_reads/{sample}.bam"
    output:
        "sorted_reads/{sample}.bam.bai"
    shell:
        "samtools index {input}"


rule bcftools_call:
    input:
        fa="data/genome.fa",
        bam=expand("sorted_reads/{sample}.bam", sample=SAMPLES),
        bai=expand("sorted_reads/{sample}.bam.bai", sample=SAMPLES)
    output:
        "calls/all.vcf"
    shell:
        "bcftools mpileup -f {input.fa} {input.bam} | "
        "bcftools call -mv - > {output}"


rule plot_quals:
    input:
        "calls/all.vcf"
    output:
        "plots/quals.svg"
    script:
        "scripts/plot-quals.py"
```

--------------------------------

### Activate dev environment

Source: https://snakemake.readthedocs.io/en/stable/project_info/contributing.html

Activates the 'dev' environment, which includes docs and style features for building documentation and running black.

```bash
$ pixi shell -e dev
```

--------------------------------

### Example YTE metadata template for report

Source: https://snakemake.readthedocs.io/en/stable/snakefiles/reporting.html

Example metadata yaml template that contains information about the work directory in which the workflow was run. It defines a Workflow name, the current working directory via ?os.getcwd(), and a list of contributors.

```yaml
__definitions__:
- import os

Workflow name: Test Workflow
Workdir: ?os.getcwd()
Contributors:
  - Test Contributor
  - Another Contributor
```

--------------------------------

### Exit Pixi shell

Source: https://snakemake.readthedocs.io/en/stable/tutorial/setup.html

Exits the Pixi shell to return to your normal shell. Do not run this now if you want to continue with the tutorial.

```bash
$ exit
```

--------------------------------

### expand() with single wildcard sample

Source: https://snakemake.readthedocs.io/en/stable/tutorial/basics.html

Shows how expand() formats a pattern with a list of sample names. This example is for explanation only and should not be added to the Snakefile yet.

```python
expand("sorted_reads/{sample}.bam", sample=SAMPLES)
```
Tokens:
Raw