{"id":20153489,"url":"https://github.com/nextflow-io/crg-course-nov16","last_synced_at":"2025-03-03T01:23:13.940Z","repository":{"id":68691154,"uuid":"71648453","full_name":"nextflow-io/crg-course-nov16","owner":"nextflow-io","description":"Nextflow + Docker tutorial material","archived":false,"fork":false,"pushed_at":"2017-10-24T11:23:16.000Z","size":5550,"stargazers_count":23,"open_issues_count":0,"forks_count":6,"subscribers_count":3,"default_branch":"master","last_synced_at":"2025-01-13T12:23:01.583Z","etag":null,"topics":["nextflow"],"latest_commit_sha":null,"homepage":"","language":null,"has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/nextflow-io.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2016-10-22T15:38:35.000Z","updated_at":"2024-05-27T09:02:26.000Z","dependencies_parsed_at":"2023-03-15T10:32:19.668Z","dependency_job_id":null,"html_url":"https://github.com/nextflow-io/crg-course-nov16","commit_stats":null,"previous_names":[],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextflow-io%2Fcrg-course-nov16","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextflow-io%2Fcrg-course-nov16/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextflow-io%2Fcrg-course-nov16/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/nextflow-io%2Fcrg-course-nov16/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/nextflow-io","download_url":"https://codeload.github.com/nextflow-io/crg-course-nov16/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":241592984,"owners_count":19987467,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["nextflow"],"created_at":"2024-11-13T23:19:14.726Z","updated_at":"2025-03-03T01:23:13.919Z","avatar_url":"https://github.com/nextflow-io.png","language":null,"funding_links":[],"categories":[],"sub_categories":[],"readme":"# Nextflow + Docker tutorial \n\nThis repository contains the tutorial material for the *Parallel distributed computational workflows\nwith Nextflow and Docker containers* course. \n\n## Prerequisite\n\n* Java 7 or 8 \n* Docker engine 1.10.x (or higher) \n\n## Installation \n\nInstall Nextflow by using the following command: \n\n```\ncurl -fsSL get.nextflow.io | bash\n```\n    \nThe above snippet creates the `nextflow` launcher in the current directory. \nComplete the installation moving it into a directory on your `PATH` eg: \n\n```\nmv nextflow $HOME/bin\n``` \n   \nFinally, clone this repository with the following command: \n\n```\ngit clone https://github.com/nextflow-io/crg-course-nov16.git \u0026\u0026 cd crg-course-nov16\n```\n\n## Nextflow hands-on \n\nDuring this tutorial you will implement a proof of concept of a RNA-Seq pipeline which: \n\n1. Indexes a genome file.\n2. Maps read pairs against the genome.\n3. Performs quantification.\n\n### Step 1 - Command line parameters\n\nThe script `rna-ex1.nf` defines the pipeline input parameters. Run it by using the \nfollowing command: \n\n```\nnextflow run rna-ex1.nf\n```\n\nTry to specify a different input parameter, for example: \n\n```\nnextflow run rna-ex1.nf --genome this/and/that\n```\n\n### Step 2 - Build genome index\n\nThe second example adds the `buildIndex` process. It takes the genome file as \ninput and creates the genome index by using the `bowtie-build` tool. \n\nTry to run it by using the command: \n\n```\nnextflow run rna-ex2.nf\n```\n\nThe execution will fail because Bowtie is not installed in the test environment. \n\nAdd the command line option `-with-docker` to launch the execution through a Docker container\nas shown below: \n\n```\nnextflow run rna-ex2.nf -with-docker\n```\n\nThis time it works because it uses the Docker container `nextflow/rnatoy:1.3` defined in the \n`nextflow.config` file. \n\nIn order to avoid to add the option `-with-docker` add the following line in the `nextflow.config` file: \n\n```\ndocker.enabled = true\n```\n\n### Step 3 - Collect read files by pairs\n\nThis step shows how to match *read* files into pairs, so thay can be mapped by *TopHat*. \n\nEdit the script `rna-ex3.nf` and add the following statement as the last line: \n\n```\nread_pairs.println()\n```\n\nSave it and execute it with the following command: \n\n```\nnextflow run rna-ex3.nf\n```\n\nTry it again specifying different read files by using a glob pattern:\n\n```\nnextflow run rna-ex3.nf --reads 'data/ggal/reads/*_{1,2}.fq'\n```\n\nIt shows how read files matching the pattern specified are grouped in pairs having \nthe same prefix.\n\n\n### Step 4 - Map sequence reads \n\nThe script `rna-ex4.nf` adds the `mapping` process. Note how it declares three inputs: \nthe genome fasta file, the genome index file produced by the `buildIndex` process and \nthe read pairs. Also note as the last input is defined as a `set` ie. it's composed by \ndifferent elements: the pair ID, the first read file and the second read file. \n\nExecute it by using the following command: \n\n```\nnextflow run rna-ex4.nf -resume\n```\n\nThe `-resume` option skips the execution of any step that has been processed in a previous \nexecution. \n\nTry to execute it with more read files as shown below: \n\n```\nnextflow run rna-ex4.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq'\n```\n\n\n### Step 5 - Perform reads quantification \n\nThis step adds the quantification step to the example script. It takes the \nannotation file and the *bam* files produced by *TopHat* and outputs the transcripts \n*gtf* files. \n\nYou can run it by using the following command: \n\n```\nnextflow run rna-ex5.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq' \n```\n\n### Step 6 - Define the pipeline output\n\nThis step shows how produce the pipeline output to a folder of your choice by using the \n`publishDir` directive. \n\nRun the example by using the following command: \n\n```\nnextflow run rna-ex6.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq' \n```\n\nThen you will find the quantification files in the folder `results`. \n\nModify the `rna-ex6.nf` script by adding the following line at the beginning of the file: \n\n```\nparams.outdir = 'results'\n```\n\nThen, look for the `publishDir` directive in the `makeTranscript` process, and \nreplace the `'results'` string with the `params.outdir` parameter. \n\nFinally run it again with the following command: \n\n```\nnextflow run rna-ex6.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq' --outdir my_transcripts\n```\n\nYou will find the transcripts produced by the pipeline in the `my_transcripts` folder.\n\n\n### Step 7 - Handle completion event\n\nThis step shows how to execute an action when the pipeline completes the execution. \n\nNote that Nextflow processes define the execution of *asynchronous* tasks i.e. they are not \nexecuted one after another as they are written in the pipeline script as it would happen in a \ncommon *iperative* programming language.\n\nThe script uses the `workflow.onComplete` event handler to print a confirmation message \nwhen the script completes. \n\nTry to run it by using the following command: \n\n```\nnextflow run rna-ex7.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq'\n``` \n \n### Step 8 - Manage custom scripts\n\nReal world pipelines use a lot of custom user scripts (BASH, R, Python, etc). Nextflow \nallows you to use and manage all these scripts in consistent manner. Simply put them \nin a directory named `bin` in the pipeline project root. They will be automatically added \nto the pipeline execution `PATH`. \n\nFor example, create a file named `quantify.sh` with the following content: \n\n```\n#!/bin/bash \nset -e \nset -u\n\nannot=${1}\nbam_file=${2}\npair_id=${3}\n\ncufflinks --no-update-check -q -G $annot ${bam_file}\nmv transcripts.gtf transcript_${pair_id}.gtf\n```\n\nSave it, grant the execute permission and move it under the `bin` directory as shown below: \n\n```\nchmod +x quantify.sh\nmkdir -p bin \nmv quantify.sh bin\n```\n\nThen, open the `rna-ex7.nf` file and replace the `makeTranscript` process with \nthe following code: \n\n```\nprocess makeTranscript {\n    tag \"$pair_id\"\n    publishDir params.outdir, mode: 'copy'  \n       \n    input:\n    file annot from annotation_file \n    set pair_id, file(bam_file) from bam\n     \n    output:\n    set pair_id, file('transcript_*.gtf') into transcripts\n \n    \"\"\"\n    quantify.sh $annot $bam_file $pair_id\n    \"\"\"\n}\n\n```\n\nFor the sake of simplicity of this example, the *cpus* parameter it's ignored. \n\nRun it as before: \n\n```\nnextflow run rna-ex7.nf -resume --reads 'data/ggal/reads/*_{1,2}.fq'\n```\n\n\n### Step 9 - Publish to GitHub (bonus)  \n\nHere you will lean how to publish your pipeline on [GitHub](https://github.com) and share \nit with other people and allowing you to track all the project \ndependencies and changes with ease. \n \nSetup your `git` credentials: \n\n```\ngit config --config user.name \"your name\"\ngit config --config user.email your@email.com \n```\n\nCreate a new empty project folder that will contain the files you \nwant to upload to GitHub eg. \n\n```\nmkdir $HOME/rnaseq-demo\ncd $HOME/rnaseq-demo\n``` \n\nCreate a new project on GitHub to host your pipeline, and follow \nthe instruction provided by it to publish the project in the project in \nthe folder `$HOME/rnaseq-demo/` in that repository. \n\nNote: make sure to use the same email address you have defined in your \n`git` configuration setup with the previous commands.  \n\nFinally, copy the pipeline files and data and upload to the GitHub repository \n\n```\ncp $HOME/crg-course-nov16/rna-ex6.nf $HOME/rnaseq-demo/main.nf\ncp $HOME/crg-course-nov16/nextflow.config $HOME/rnaseq-demo/\ncp -r $HOME/crg-course-nov16/bin $HOME/rnaseq-demo/\ncp -r $HOME/crg-course-nov16/data $HOME/rnaseq-demo/\n\ngit add bin/ data/ main.nf nextflow.config \ngit commit -m 'Added pipeline files'\ngit push \n```\n\nWhen done, you will be able to run your pipeline by using the following \ncommand: \n\n```\nnextflow run \u003cyour-github-user-name\u003e/rnaseq-demo\n```\n\n\n### Manage revisions (bonus)\n\nGit and GitHub are tools specifically designed to track project changes and versions. \nYou can use Git tags, branches or commit IDs to maintain an history revision of your \npipeline projects.\n\nNextflow integrates these tools making possible to run any revision of your pipeline \nby simply specifying it on the run command line by using the `-revision` option, as shown \nbelow: \n\n```\nnextflow run \u003cproject name\u003e -r \u003crevision name\u003e\n```\n\nThe list of available revision can be list by using the following command: \n\n```\nnextflow info \u003cproject-name\u003e\n```\n\n\n## Docker hands-on \n\nGet practice with basic Docker commands to pull, run and build your own containers.\n \nA container is a ready-to-run Linux environment which can be executed in an isolated \nmanner from the hosting system. It has own copy of the file system, processes space,\nmemory management, etc. \n \nContainers are a Linux feature known as *Control Groups* or [Ccgroups](https://en.wikipedia.org/wiki/Cgroups)\nintroduced with kernel 2.6. \n\nDocker adds to this concept an handy management tool to build, run and share container images. \n\nThese images can be uploaded and published in a centralised repository know as \n[Docker Hub](https://hub.docker.com), or hosted by other parties like for example [Quay](https://quay.io).\n\n\n### Step 1 - Run a container \n\nRun a container is easy as using the following command: \n\n```\ndocker run \u003ccontainer-name\u003e \n```\n\nFor example: \n\n```\ndocker run hello-world  \n```\n\n### Step 2 - Pull a container \n\nThe pull command allows you to download a Docker image without running it. For example: \n\n```\ndocker pull debian:wheezy \n```\n\nThe above command download a Debian Linux image.\n\n\n### Step 3 - Run a container in interactive mode \n\nLaunching a BASH shell in the container allows you to operate in an interactive mode \nin the containerised operating system. For example: \n\n```\ndocker run -it debian:wheezy bash \n``` \n\nOnce launched the container you wil noticed that's running as root (!). \nUse the usual commands to navigate in the file system.\n\nTo exit from the container, stop the BASH session with the exit command.\n\n### Step 4 - Your first Dockerfile\n\nDocker images are created by using a so called `Dockerfile` i.e. a simple text file \ncontaining a list of commands to be executed to assemble and configure the image\nwith the software packages required.    \n\nIn this step you will create a Docker image containing the Samtools and Bowtie2 tools.\n\nIn order to build a Docker image, start creating an empty directory eg. \n`~/docker-tutorial` and change to it: \n\n```\nmkdir -p ~/docker-tutorial \u0026\u0026 cd ~/docker-tutorial \n```\n\nWarning: the Docker build process automatically copies all files that are located in the \ncurrent directory to the Docker daemon in order to create the image. This can take \na lot of time when big/many files exist. For this reason it's important to *always* work in \na directory containing only the files you really need to include in your Docker image. \nAlternatively you can use the `.dockerignore` file to select the path to exclude from the build. \n\nThen use your favourite editor eg. `vim` to create a file named `Dockerfile` and copy the \nfollowing content: \n\n```\nFROM debian:wheezy \n\nMAINTAINER \u003cyour name\u003e\n\nRUN apt-get update --fix-missing \u0026\u0026 \\\n  apt-get install -q -y python wget unzip samtools\n```\n\nWhen done save the file. \n\n\n### Step 5 - Build the image  \n\nBuild the Docker image by using the following command: \n\n```\ndocker build -t my-image .\n```\n\nNote: don't miss the dot in the above command. When it completes, verify that the image \nhas been created listing all available images: \n\n```\ndocker images\n```\n\n### Step 6 - Add a software package to the image\n\nAdd the Bowtie package to the Docker image by adding to the `Dockerfile` the following snippet: \n\n```\nRUN wget --no-check-certificate -O bowtie.zip https://sourceforge.net/projects/bowtie-bio/files/bowtie2/2.2.7/bowtie2-2.2.7-linux-x86_64.zip/download \u0026\u0026 \\\n  unzip bowtie.zip -d /opt/ \u0026\u0026 \\\n  ln -s /opt/bowtie2-2.2.7/ /opt/bowtie \u0026\u0026 \\\n  rm bowtie.zip \n\nENV PATH $PATH:/opt/bowtie2-2.2.7/\n```\n\nSave the file and build again the image with the same command as before: \n\n```\ndocker build -t my-image .\n```\n\nYou will notice that it creates a new Docker image with the same name *but* with a \ndifferent image ID. \n\n### Step 7 - Run Bowtie in the container \n\nCheck that everything is fine running Bowtie in the container as shown below: \n\n```\ndocker run my-image bowtie2 --version\n```\n\nYou can even launch a container in an interactive mode by using the following command: \n\n```\ndocker run -it my-image bash\n```\n\n\n### Step 8 - File system mounts\n\nCreate an genome index file by running Bowtie in the container. \n\nTry to run Bowtie in the container with the following command: \n\n```\ndocker run my-image \\\n  bowtie2-build ~/crg-course-nov16/data/ggal/genome.fa genome.index\n```\n\nThe above command fails because Bowtie cannot access the input file.\n\nThis happens because the container runs in a complete separate file system and \nit cannot access the hosting file system by default. \n\nYou will need to use the `--volume` command line option to mount the input file(s) eg. \n\n```\ndocker run --volume ~/crg-course-nov16/data/ggal/genome.fa:/genome.fa my-image \\\n  bowtie2-build /genome.fa genome.index\n```\n\nAn easier way is to mount a parent directory to an identical one in the container, \nthis allows you to use the same path when running it in the container eg. \n\n```\ndocker run --volume $HOME:$HOME --workdir $PWD my-image \\\n  bowtie2-build ~/crg-course-nov16/data/ggal/genome.fa genome.index\n```\n\n### Step 9 - Upload the container in the Docker Hub (bonus)\n\nPublish your container in the Docker Hub to share it with other people. \n\nCreate an account in the https://hub.docker.com web site. Then from your shell terminal run \nthe following command, entering the user name and password you specified registering in the Hub: \n\n```\ndocker login \n``` \n\nTag the image with your Docker user name account: \n\n```\ndocker tag my-image \u003cuser-name\u003e/my-image \n```\n\nFinally push it to the Docker Hub:\n\n```\ndocker push \u003cuser-name\u003e/my-image \n```\n\nAfter that anyone will be able to download it by using the command: \n\n```\ndocker pull \u003cuser-name\u003e/my-image \n```\n\n\n## Deploy a NF pipeline in the CRG cluster \n\nNextflow supports different execution platforms. This means that your script \ncan be executed in a single computer, a cluster or a cloud by simply providing a configuration\nfile that specify what computational platform you want to use. \n\nFor the sake of this tutorial you will run the [RNA-Toy](https://github.com/nextflow-io/rnatoy) \npipeline in the CRG cluster. \n\nLog-in the CRG cluster by using the following cluster: \n\n```\nssh \u003csitXX\u003e@ant-login.linux.crg.es\n```\n\n* Replace the `\u003csitXX\u003e` string with the user name that you have been assigned. \n\nCreate a project directory eg. `rnatoy` and create a file named `nextflow.config` with\nthe following content: \n\n```\nprocess.executor = 'crg' \nprocess.queue = 'course'\nprocess.scratch = true\nprocess.time = '1h'\nprocess.memory = '1G'\ndocker.enabled = true\n```\n\nThen launch the execution of the pipeline by using the following command: \n\n```\nnextflow run rnatoy\n```\n\nWhen completed you will find the pipeline output in the `results` folder.\n\n\n### Run the pipeline against a real dataset \n\nCreate a new folder to run the pipeline against the mouse genome dataset eg: \n\n```\nmkdir -p $HOME/mouse-run\ncd $HOME/mouse-run\n```\n\nThen create the `nextflow.config` file with the following content: \n\n```\nparams.reads = \"/software/rg/rnaseq/data/*_{1,2}.fastq.gz\"\nparams.annot = \"/software/rg/rnaseq/refs/mm65.long.ok.sorted.gtf\"\nparams.genome = \"/users/cn/ptommaso/projects/nf-course/mouse_genome_mm9_chr1.fa\" \n\nprocess.executor = 'crg' \nprocess.queue = 'course'\nprocess.scratch = true\nprocess.time = '1h'\nprocess.memory = '8G'\nprocess.cpus = 4 \nprocess.$buildIndex.cpus = 8 \n\ndocker.enabled = true\ntrace.enabled = true\n```\n\nWhen done, launch the execution by using this command: \n\n```\nnextflow run rnatoy -bg \u003e log\n```\n\nThe `-bg` will launch NF in the background, to check the execution status you can \nfollow the `log` as shown below: \n\n```\ntail -f log\n```\n \n\n### Automatic errors fail over \n\nWhen running large scale pipelines launching thousands of jobs on many \ndifferent computing nodes errors are not a remote event. \n\nNextflow allows failing tasks to be automatically re-executed, in this way it's possible \nto address temporary failures such as failing hardware or network hiccups. In order to enable \nautomatic jobs re-execution add the following setting in the `nextflow.config` file: \n\n```\nprocess.errorStrategy = 'retry'\n```\n\nA more common source of errors in computational pipeline are peaks in computing resources, \nallocated by a jobs exceeding the original resource request. In this context automatically \nre-executing the failed task is useless because it would simply replicate the same error condition. \n\nA common solution consists of increasing the resource request for the needs of the most consuming job, \neven though this will result in a suboptimal allocation of most of the jobs that are less resource hungry.\n\nNextlow allows resources to be defined in a dynamic manner. In this way it is possible to \nincrease the memory request when rescheduling a failing task execution. For example: \n\n```\nprocess.memory = { 1.GB * task.attempt }\nprocess.errorStrategy { task.exitStatus == 140 ? 'retry' : 'terminate' }\n```\n\nBy using the above settings pipeline a task will initially request one GB of memory. \nIn case of an error it will be rescheduled requesting 2 GB and so on, until it is executed \nsuccessfully or the limit of times a task can be retried is reached, forcing the termination \nof the pipeline.\n\n\n## Deploy a NF pipeline in the AWS cloud (bonus)\n\nNextflow pipelines can be seamlessly executed in the Amazon cloud. All you need is an AWS \nuser account a base Amazon VM image (AMI) that will be used to setup the computing cluster \nin the cloud. \n\nThe following screen cast shows how to configure, setup the cluster and launch the pipeline \nexecution in the AWS cloud in a few commands:  \n\n[![asciicast](https://asciinema.org/a/9vupd4d72ivaz6h56pajjjkop.png)](https://asciinema.org/a/9vupd4d72ivaz6h56pajjjkop)\n\n\n## Assignment \n\nCreate a two steps pipeline that given any number of protein sequence FASTA files creates \na phylogenetic tree for each or them. Bonus: use a Docker container to isolate and deploy \nthe binary dependencies.  \n\n#### Tip \n\nUse [Clustalw2](http://www.clustal.org/clustal2/) to align the protein sequences. Example \ncommand line: \n\n    clustalw2 -infile=sample.fa -output=phylip -outfile=aln.phy\n    \nUse [RAxML](https://github.com/stamatak/standard-RAxML) to create the phylogenetic tree. \nExample command line: \n\n    raxmlHPC -f d -j -p 9 -T 2 -m PROTGAMMALG -s aln.phy -n aln       \n\nUse the input protein sequence FASTA files in the following folder: \n\n    $HOME/crg-course-nov16/data/prot\n    \n    \nPossible implementation: https://github.com/nextflow-io/phytoy-nf    \n\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextflow-io%2Fcrg-course-nov16","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fnextflow-io%2Fcrg-course-nov16","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fnextflow-io%2Fcrg-course-nov16/lists"}