GDC HPC Euler

https://www.gdc-docs.ethz.ch/EulerManual/site
Admin
GDC HPC Euler is a high-performance computing system provided by GDC, with this manual serving as a...
Tokens:21,136
Snippets:181
Trust Score:9.7
Update:10 months ago
Show doc for...
Tokens:
Raw
### Install R Packages

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Provides commands for installing R packages from different repositories: CRAN, Bioconductor, and GitHub. It covers installing single packages, multiple packages, and using BiocManager for Bioconductor packages.

```r
#### CRAN Repository

install.packages("package")
install.packages(c("packageA", "packageB"))

#### Bioconductor Repository

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")
BiocManager::install("package")

#### GitHub
library(devtools)
devtools::install_github("link/to/package")

```

--------------------------------

### View available software modules with module avail

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Lists all available software modules that can be loaded on the system. This command helps in discovering the software packages installed on the cluster.

```bash
module avail
```

--------------------------------

### Example: Download and Alias Samtools Apptainer Container

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Demonstrates downloading a Samtools Singularity container from a remote repository, renaming it, and creating an alias 'sm' for easy execution. This allows running Samtools commands within the containerized environment.

```bash
cd ${SCRATCH}
#Let's download the container from galaxy to our Scratch.
wget https://depot.galaxyproject.org/singularity/samtools:0.1.19--3
mv samtools:0.1.19--3 samtools-1.9.sif
#Create alias
alias "sm=apptainer exec \
       --bind ${SCRATCH} \
       samtools-1.9.sif \
       samtools"
#run samtools view
sm view -h

```

--------------------------------

### Create and Activate Python Virtual Environment

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Steps to create and activate a Python virtual environment for managing project-specific dependencies. This involves creating the environment using 'venv', activating it, and then installing packages using 'pip'.

```bash
cd ${HOME}

module load python/3.11.6

python -m venv cutadapt

source ${HOME}/cutadapt/bin/activate

pip install cutadapt

```

--------------------------------

### Get resource usage summary with jefflow

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Retrieves a summary of your resource usage over the past few days.

```bash
jefflow
```

--------------------------------

### Submitting Slurm Jobs

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Demonstrates how to submit a Slurm job using the `sbatch` command with a script file. This is the standard method for queuing and running jobs on the Euler cluster.

```bash
sbatch < submit.map.slurm.sh
```

--------------------------------

### Submit Simple Command with sbatch

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Submits a simple command using sbatch, specifying CPU count and time limit. It's recommended to use submission scripts for reproducibility.

```bash
sbatch --cpus-per-task=1 --time=04:00:00 --wrap="my_command"
```

--------------------------------

### Chaining Slurm Jobs with Dependencies

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Illustrates how to chain Slurm jobs, ensuring that a subsequent job only starts after a preceding job has completed successfully. This is achieved using the `--dependency=afterok:<job_id>` option.

```bash
sbatch < jobA.slurm.sh
sbatch --dependency=afterok:"9999" < jobB.slurm.sh
```

--------------------------------

### Start Interactive Job with srun

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Initiates an interactive job session using srun, providing a bash shell on a compute node. This is useful for testing scripts or running short, demanding commands without a full submission script.

```bash
srun --pty bash
```

--------------------------------

### Create Apptainer Alias for Tool Execution

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Defines an alias to simplify the execution of containerized tools using Apptainer. This example creates an alias 'toolX' that binds the $SCRATCH directory and executes 'command_to_call_toolX' within the 'container.sif' image.

```bash
alias "toolX=apptainer exec \
      --bind ${SCRATCH} \
      container.sif \
      command_to_call_toolX"

```

--------------------------------

### Submit SLURM Script

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Submits a SLURM script for execution using the sbatch command. This is the standard method for queuing and running jobs on the cluster.

```bash
sbatch < submit.bcf.slurm.sh
```

--------------------------------

### Archive folder with tar

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Creates a compressed archive (tar.gz) of a specified folder. It also includes an example of how to delete the original folder after archiving.

```bash
tar cvzf data1.tar.gz data1
#delete the folder
rm -rf data1
```

--------------------------------

### Submitting R Script with sbatch

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This command submits an R script (`Reformat.R`) to the Slurm scheduler using `sbatch`. It specifies resource requirements (CPUs, memory, time) and uses the `--wrap` option to execute the Rscript command with specific arguments, including `--vanilla` for a clean R environment.

```bash
sbatch --cpus-per-task=1 --mem-per-cpu=2G --time 4:00:00 --wrap="Rscript --vanilla Reformat.R Fst_chromosomes1.txt Fst_chromosomes1_reduced"
```

--------------------------------

### Slurm Job Management Commands

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This section details commands for monitoring and managing Slurm jobs. `jeffrun -j` provides a summary of a specific job, `jview` offers an overview of all submitted jobs, and `jeff24` gives a summary of finished jobs. It also explains Slurm job state codes.

```bash
jeffrun -j ${SLURM_JOB_ID}
```

```bash
jview
```

```bash
jeff24
```

--------------------------------

### Chaining Slurm Job Arrays

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

Explains how to chain Slurm job arrays, allowing specific parts of an array job to complete before initiating dependent jobs. The syntax `--dependency=afterok:"<job_id>_<start>:<job_id>_<end>"` is used for this purpose.

```bash
sbatch < job_array.slurm.sh
sbatch --dependency=afterok:"6666_1:6666_10" < jobB.slurm.sh
```

--------------------------------

### Making R Script Executable

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This command uses `chmod` to grant execute permissions to the R script file (`Reformat.R`), allowing it to be run directly from the command line.

```bash
chmod +x Reformat.R
```

--------------------------------

### Samtools BAM File Processing and Conversion

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This snippet demonstrates using samtools to generate statistics (flagstat, coverage) from a BAM file and then converting the BAM file to CRAM format for increased compression. It utilizes shell scripting with variables for output paths, sample names, and reference files.

```bash
samtools flagstat ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.bam  > ${OUT}/statsQ20/${SAMPLE}
samtools coverage ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.bam  > ${OUT}/statsQ20/${SAMPLE}.cov

echo "Convert bam file to cram to increase compression"
samtools view -@ ${CPU} -T ${REF} -C -o ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.cram ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.bam
```

--------------------------------

### SLURM Script for BWA-MEM2 Read Mapping Job Array

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This script outlines a SLURM job array for parallel read mapping using `bwa-mem2`. It processes samples listed in `sample.lst`, utilizes local scratch space (`--tmp`), and performs steps including alignment, sorting, quality filtering, and duplicate marking with `samtools`. Resource allocation for CPUs, memory, and time are specified.

```bash
#!/bin/bash
#SBATCH --job-name=map         #Name of the job
#SBATCH --array=1-20%5         #Array with 20 Jobs, always 5 running in parallel
#SBATCH --ntasks=1             #Requesting 1 node for each job (always 1)
#SBATCH --cpus-per-task=4      #Requesting 4 CPU for each job, 20 CPUs in total
#SBATCH --mem-per-cpu=2G       #Requesting 8 Gb memory per job, 40G in total
#SBATCH --time=4:00:00         #4 hours run-time per job
#SBATCH --tmp=20G              #Request 20G local scratch
#SBATCH --output=map_%a.log    #Log files


##########################################
##Nik Zemp, GDC, 02/10/23
echo "$(date) start ${SLURM_JOB_ID}"
##########################################

#Source the GDC stack
source /cluster/project/gdc/shared/stack/GDCstack.sh

#Load the needed modules
module load samtools/1.22.1 bwa-mem2/2.2.3

#Running variable 
IDX=${SLURM_ARRAY_TASK_ID}
SAMPLE=$(sed -n ${IDX}p sample.lst)

##provide path infos
IN=raw
OUT=mapping
REF=Ref/Ref.fas

#Let's extract the number of requested CPUs and save it as variable cpu. 
CPU=${SLURM_CPUS_ON_NODE}
#for the pip we need twice memory as requested per CPU. Let's use a bit less then 0.5 of the requested 2 Gb. 
MEM=800M 

##generate output folder if not present
if [ ! -e ${OUT} ]  ; then mkdir ${OUT} ; fi
if [ ! -e ${OUT}/stats ]  ; then mkdir ${OUT}/stats ; fi
if [ ! -e ${OUT}/statsQ20 ]  ; then mkdir ${OUT}/statsQ20 ; fi
if [ ! -e ${OUT}/statsDup ]  ; then mkdir ${OUT}/statsDup ; fi

echo "Start processing ${SAMPLE}" 

echo "Run bwa-mem2"
bwa-mem2 mem ${REF} ${IN}/${SAMPLE}_R1.fq.gz ${IN}/${SAMPLE}_R2.fq.gz -t ${CPU} > ${TMPDIR}/${SAMPLE}.sam

echo "Sam2bam and sort"
samtools sort ${TMPDIR}/${SAMPLE}.sam -T ${TMPDIR} -@ ${CPU} -m ${MEM} -o ${TMPDIR}/${SAMPLE}_sort.bam

rm ${TMPDIR}/${SAMPLE}.sam

echo "Get MappingStats"
samtools flagstat ${TMPDIR}/${SAMPLE}_sort.bam > ${OUT}/stats/${SAMPLE}

echo "Clean mappings" 
samtools view -hb -q 20 -F 0x800 -F 0x100 -@ ${CPU} -o ${TMPDIR}/${SAMPLE}_sort_Q20.bam ${TMPDIR}/${SAMPLE}_sort.bam
samtools index ${TMPDIR}/${SAMPLE}_sort_Q20.bam

rm ${TMPDIR}/${SAMPLE}_sort.bam

echo "Remove PCRdups"
samtools collate -@ ${CPU} -O -u ${TMPDIR}/${SAMPLE}_sort_Q20.bam | samtools fixmate -@ ${CPU} -m -u - - | samtools sort -@ ${CPU} -m ${MEM} -T ${TMPDIR}  -u - | samtools markdup -@ ${CPU} -T ${TMPDIR} -r -f ${OUT}/statsDup/${SAMPLE} - ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.bam
samtools index ${OUT}/${SAMPLE}_sort_Q20_fix_nodup.bam

echo "Get MappingStats"

```

--------------------------------

### Set Java Memory Options for Picard Tools

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This snippet demonstrates how to set the JAVA_TOOL_OPTIONS environment variable to allocate memory for Java-based tools like Picard. It specifies a heap size of 4GB. Ensure the picard command is then executed.

```bash
export JAVA_TOOL_OPTIONS=-Xmx4G
picard FixMateInformation -h
```

--------------------------------

### SLURM Script for bcftools SNP Calling Job Array

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This script demonstrates how to run `bcftools mpileup` and `bcftools call` as a SLURM job array. It splits the task by chromosome, allowing for parallel SNP calling on subsets of the data. The script includes directives for job name, array definition, resource allocation, and output logging.

```bash
#!/bin/bash
#SBATCH --job-name=bcf       #Name of the job
#SBATCH --array=1-20%10      #Array with 20 Jobs, always 10 running in parallel
#SBATCH --ntasks=1           #Requesting 1 node for each job (always 1)
#SBATCH --cpus-per-task=1    #Requesting 1 CPU for each job
#SBATCH --mem-per-cpu=2G     #Requesting 2 Gb memory per core and job
#SBATCH --time=4:00:00       #4 hours run-time per job
#SBATCH --output=bcf_%a.log  #Log files


##########################################
##Nik Zemp, GDC, 02/10/23
echo "$(date) start ${SLURM_JOB_ID}"
##########################################

#Source the GDC stack
source /cluster/project/gdc/shared/stack/GDCstack.sh

#Load the modules needed
module load bcftools/1.22

#define input and outputs
OUT=SNPs
if [ ! -e ${OUT} ]  ; then mkdir ${OUT} ; fi

##The internal variable of slurm (1-20 in our case; see header slurm) can be used to extract the names of the chromosomes. 
IDX=${SLURM_ARRAY_TASK_ID}
CHROM=$(sed -n ${IDX}p chrom.lst)
MAXCOV=80

REF=Ref/Ref.fasta

#The bcftools command
bcftools mpileup -f ${REF} --skip-indels -b bam.lst -r ${CHROM} -a 'FORMAT/AD,FORMAT/DP' -d ${MAXCOV}  | \
bcftools call -mv -Ob -o ${OUT}/raw.${CHROM}.bcf

##############################################
##Get a summary of the job
jeffrun -j ${SLURM_JOB_ID}
##############################################

```

--------------------------------

### Submit bcftools Job with SLURM Script

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

A SLURM submission script for running the bcftools SNP-caller. It configures job parameters like name, tasks, CPUs, memory, time, and output logging. It also sources a GDC stack, loads the bcftools module, and defines input/output paths before executing the bcftools command.

```bash
#!/bin/bash   
#SBATCH --job-name=bcf           #Name of the job   
#SBATCH --ntasks=1               #Requesting 1 node (is always 1)
#SBATCH --cpus-per-task=1        #Requesting 1 CPU
#SBATCH --mem-per-cpu=1G         #Requesting 2 Gb memory per core 
#SBATCH --time=4:00:00           #Requesting 4 hours running time 
#SBATCH --output bcf.log         #Log

##########################################
##Nik Zemp, GDC, 02/10/23
echo "$(date) start ${SLURM_JOB_ID}"
##########################################

#Source the GDC stack
source /cluster/project/gdc/shared/stack/GDCstack.sh

#Load the needed modules
module load bcftools/1.22

#define in and outputs
OUT=SNPs
if [ ! -e ${OUT} ]  ; then mkdir ${OUT} ; fi
REF=Ref/Ref.fasta
MAXCOV=80

#The bcftools command, for a list of bam files
echo "Run bcf"
bcftools mpileup -f ${REF} --skip-indels -b bam.lst -a 'FORMAT/AD,FORMAT/DP'  -d ${MAXCOV} | \
bcftools call -mv -Ob -o ${OUT}/raw.bcf

##############################################
##Get a summary of the job 
jeffrun -j ${SLURM_JOB_ID}
##############################################

```

--------------------------------

### Manage Software Modules (List and Load)

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Commands to manage software modules on the Euler cluster. 'module avail' lists all available modules, 'module avail <tool>' searches for a specific tool, and 'module load <tool>' loads a specific version of a tool into the current environment.

```bash
module avail

```

```bash
module avail samtools

```

```bash
module load samtools/1.22.1

```

```bash
module load stack

```

```bash
module load r/4.3.2

```

```bash
module load r/4.4.1

```

```bash
module load python/3.11.6

```

--------------------------------

### Set Java Memory Options for GATK

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This example shows how to limit memory for GATK using the --java-options flag. It allocates 4GB of heap space for the HaplotypeCaller function. Adjust the memory as needed based on your job's requirements.

```bash
gatk --java-options "-Xmx4G" HaplotypeCaller -h
```

--------------------------------

### Load GDC Software Stack

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

This command sources the GDC software stack initialization script. It's essential for accessing GDC-specific tools and environments on the cluster. This should be added to submission scripts or run interactively.

```bash
source /cluster/project/gdc/shared/stack/GDCstack.sh

```

--------------------------------

### Load GDC software stack with source

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Loads the GDC software stack environment variables and paths. This command is essential for making GDC-specific software and tools available in your shell session.

```bash
source /cluster/project/gdc/shared/stack/GDCstack.sh
```

--------------------------------

### Create Symbolic Links for Directories

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/transfer

These commands create symbolic links to frequently accessed directories like your home directory, scratch space, and shared projects. This simplifies navigation within the Euler cluster.

```bash
ln -s /cluster/work/gdc/people/<USER>  gdc_home
ln -s /cluster/scratch/<USER> scratch
ln -s /cluster/work/gdc/shared/p999 p999

```

--------------------------------

### Load ETH Proxy Module

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/transfer

This command loads the necessary module to enable connections through the ETH proxy server. This is required for accessing external services from the compute nodes.

```bash
module load eth_proxy

```

--------------------------------

### Get folder size with du

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Calculates and displays the size of a specified folder in a human-readable format (e.g., KB, MB, GB). The '--si' flag uses powers of 1000.

```bash
du -sh --si  mapping
```

--------------------------------

### Converting CRAM to BAM with GATK

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/data

Illustrates the process of converting CRAM files to BAM format, a necessary step before running certain analysis tools like GATK for variant calling. This is typically performed on the scratch directory.

```bash
# Example command structure, specific GATK command may vary
# gatk --java-options "-Xmx4g" SamToFastq -I input.cram -O output.fastq
```

--------------------------------

### Configure Apptainer Environment Variables

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

These commands set environment variables for Apptainer, specifying cache and temporary directories. Setting APPTAINER_CACHEDIR to a location on $SCRATCH and APPTAINER_TMPDIR to $TMPDIR (or /tmp) is recommended for optimal performance and resource management.

```bash
export APPTAINER_CACHEDIR="$SCRATCH/.apptainer"
export APPTAINER_TMPDIR="${TMPDIR:-/tmp}"

```

--------------------------------

### Set Java Memory Options for BBMap

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

This command illustrates how to set the maximum heap size for BBMap using the -Xmx option. It allocates 4GB of memory. This is useful for optimizing memory usage of the bbmap.sh script.

```bash
bbmap.sh -Xmx4G -h
```

--------------------------------

### Search for a specific software module with module avail

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Searches for a specific software tool within the available modules. This is useful for quickly finding the module name for a desired tool.

```bash
module avail toolXY
```

--------------------------------

### R Script for Data Reformatting

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/jobs

An R script designed to read a CSV file, process it using the `tidyverse` library, and save the result as an RData file. It accepts input and output file names as command-line arguments.

```r
#!/usr/bin/env Rscript

## Use this argument to provide file names for the commands below, order must be consistent
args <-   commandArgs(trailingOnly=TRUE)

####load package, compare to the base functions very fast
library(tidyverse)

## Read table
samples <- read_csv(args[1], header = F)

## Save it as a RData file
name<- paste(args[2], "RData")
save(samples, name)
```

--------------------------------

### SLURM Job Script for Initial Resource Testing

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/monitoring

This SLURM job script requests resources for an initial test run. It specifies job name, array task range, number of tasks, CPUs per task, memory per CPU, and maximum runtime. This is used to assess performance with a limited set of representative chunks.

```bash
#SBATCH --job-name=fb                #Name of the job 
#SBATCH --array=1,10,15,20,50%15   
#SBATCH --ntasks=1                   #Requesting 1 node (is always 1)
#SBATCH --cpus-per-task=4            #Requesting 4 CPU
#SBATCH --mem-per-cpu=500            #Requesting 0.5 Gb memory per core, 2 Gb in total 
#SBATCH --time=24:00:00              #Requesting 24 hours run time

```

--------------------------------

### Upload Files to ENA using LFTP Command Line Client (LFTP)

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/data

This snippet demonstrates how to use the LFTP client to upload gzipped FASTQ files and their corresponding MD5 checksum files to the ENA FTP server. It includes commands for connecting, logging in, and transferring multiple files using wildcard patterns.

```lftp
lftp 
open webin.ebi.ac.uk
user username 
mput A*gz
put A*md5

# Help:
# Type `ls` command to check the content of your drop box.
# Use `mput` command to upload files.
# Use `bye` command to exit the ftp client.

```

--------------------------------

### Use Python Virtual Environment in Submission Script

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

Illustrates how to set up a Python virtual environment within a submission script. This ensures that the correct Python version and activated environment are used when running your application.

```bash
module load python/3.11.6
source ${HOME}/cutadapt/bin/activate
cutadapt -h

```

--------------------------------

### Check disk usage with lquota

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Displays the disk usage for your personal home directory and scratch space. This command helps in managing your storage quota on the cluster.

```bash
lquota
```

--------------------------------

### Submit a job using sbatch

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Submits a batch job to the SLURM workload manager. This command is used to schedule and run scripts on the cluster. It requires a SLURM script file as input.

```bash
sbatch < submit.tool.slurm.sh
```

--------------------------------

### Switch to Old Software Stack (Shell)

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/news

Instructions for users needing to revert to the old GDC software stack. This involves using the `lmod2env` command, or setting it permanently with `set_software_stack.sh old`, or sourcing a specific script in submission files.

```bash
lmod2env
source /cluster/apps/local/lmod2env.sh
set_software_stack.sh old
```

--------------------------------

### Interactively connect to a job with srun

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Connects to a running job interactively using a pseudo-terminal (pty). This is useful for real-time monitoring of CPU usage or inspecting local scratch space during job execution.

```bash
srun --interactive --jobid <job-ID> --pty bash
```

--------------------------------

### On-the-fly Unzipping with zcat

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/data

Demonstrates how to process gzipped files without writing unzipped content to disk, saving space and reducing I/O. This is useful for intermediate processing steps where the uncompressed file is not needed permanently.

```bash
gunzip -c sample.fq.gz > ${SCRATCH}/<USER>/sample.fq
```

--------------------------------

### SLURM Job Script for Optimized Resource Allocation

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/monitoring

This SLURM job script configures resources based on test run results, aiming for efficiency. It sets a lower CPU count and moderate memory per CPU for a reduced runtime. The script includes job name, an expanded array range, and optimized resource requests.

```bash
#SBATCH --job-name=fb       #Name of the job   
#SBATCH --array=1-120%15   
#SBATCH --ntasks=1          #Requesting 1 node (is always 1)
#SBATCH --cpus-per-task=1   #Requesting 1 CPU
#SBATCH --mem-per-cpu=3G    #Requesting 3 Gb memory per core
#SBATCH --time=4:00:00      #Requesting 4 hours runtime

```

--------------------------------

### Set File Permissions (Read for Group, Owner Write)

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/file

Changes ownership of fastq files and sets permissions to allow the owner to read and write, while group members can only read.

```bash
chown <USER>.USYS-IBZ-GDC-EULER-<p999> raw/*.fq.fz
chmod 640 raw/*.fq.gz

```

--------------------------------

### View efficiency of jobs in last 24 hours with jeff24

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Provides an overview of the efficiency of all jobs that have run within the last 24 hours.

```bash
jeff24
```

--------------------------------

### Count files and directories with find and wc

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Counts the total number of files and directories within a specified directory. It uses 'find' to list all entries and 'wc -l' to count the lines.

```bash
find raw_data | wc -l
```

--------------------------------

### View job status with jview

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Displays an overview of submitted jobs, including those that are currently running or pending. This command helps in monitoring the status of your jobs on the cluster.

```bash
jview
```

--------------------------------

### Archiving and Compressing Folders with tar

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/data

Shows how to archive and compress a folder into a single .tar.gz file. This is recommended for managing many small files to reduce inode usage and simplify storage.

```bash
tar cfzv folder.tar.gz folder
```

```bash
tar -zcvf simulations.tar.gz simulations
```

--------------------------------

### Connect to Euler via SSH

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/access

This command is used to establish an SSH connection to the Euler cluster. Replace `<USER>` with your ETH username. The first connection will prompt for verification.

```bash
ssh <USER>@euler.ethz.ch (e.g. ssh f.muster@euler.ethz.ch)
```

--------------------------------

### Extract archive with tar

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/commands

Extracts files from a tar.gz archive. It shows how to view the archive's contents, extract the entire archive, or extract specific folders.

```bash
#get overview about the archive
tar -ztvf data1.tar.gz
#extract entire archive
tar xvf data1.tar.gz
#extract specific folder of the archive
tar xvf data1.tar.gz data1/raw
```

--------------------------------

### Blast Against NCBI NT Database using blastn

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/software

This command demonstrates how to perform a nucleotide BLAST search against the NCBI NT database. It requires the `blastn` executable and specifies the query file, the path to the NT database, the output file, and the desired output format. The output format is set to '6' for tabular output, including various alignment statistics and sequence identifiers.

```bash
blastn -task blastn -query query.fata \
-db /cluster/project/clcgenomics/CLC_BLAST_DB/nt -out query_nt.tab  \
-outfmt '6 qseqid sseqid pident length mismatch gapopen qstart qend sstart send evalue bitscore stitle sscinames sskingdoms'
```

--------------------------------

### SCP: Transfer Files to Euler

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/transfer

Uses the secure copy protocol (scp) to transfer a local file to a specified directory on the Euler server. Replace placeholders with your user ID and the correct path.

```bash
scp  local-file  <USER-ID>@euler.ethz.ch:/cluster/scratch/<USER-ID>/folder

```

--------------------------------

### SCP: Transfer Files from Euler

Source: https://www.gdc-docs.ethz.ch/EulerManual/site/transfer

Uses the secure copy protocol (scp) to transfer a file from a specified directory on the Euler server to your local machine. Replace placeholders with your user ID and the correct path.

```bash
scp <USER-ID>@euler.ethz.ch:/cluster/scratch/<USER-ID>/folder  local-file

```
Tokens:
Raw