Snakemake

Alternatives

See Nextflow and Cylc for other possible choices of workflow engine.

The Snakemake workflow management system is a tool to create reproducible and scalable data analyses. snakemake Homepage

Available Modules¶

module load snakemake/7.32.3-gimkl-2022a-Python-3.11.3

Job groups¶

Please use Snakemake's job grouping functionality to avoid Snakemake submitting many short jobs to the queue. Each job submitted to Slurm should take at least 30 minutes.

Running in an interactive Slurm session¶

Request an interactive Slurm session with the resources required to run all sub-jobs labeled as localrules:

srun --account nesi12345

Once the interactive session has launched, load the Snakemake module (see above) and you should be able to launch your workflow.

Submitting a workflow as a Slurm job¶

As with an interactive session, the main batch job should request the resources needed to run any sub-jobs marked as localrules.

#!/bin/bash -e

#SBATCH --job-name        my job
#SBATCH --account         nesi12345
#SBATCH --time            01:00:00
#SBATCH --mem             2G

# load snakemake
module load snakemake/7.32.3-gimkl-2022a-Python-3.11.3

# launch snakemake workflow
snakemake -pr --keep-going -j $SLURM_CPUS_PER_TASK all

Slurm plugin¶

Snakemake maintains a plugin to assist with running Snakemake with Slurm as the executor. This plugin supports the use of MPI jobs.

Community resources¶

There are two main repositories for reusable Snakemake rules and workflows. Snakemake Wrappers provide version controlled rules to conduct common steps in workflows. Snakemake Workflows are entire Snakemake workflows for common pipelines.

Other Resources