Pipeline release v2.0

.github/workflows/ci.yml

@@ -1,14 +1,15 @@
 name: splicing-pipelines-nf CI
 # This workflow is triggered on pushes and PRs to the repository.
-on: [push, pull_request]
+on: [pull_request]
 
 jobs:
-  test:
+  docker:
     runs-on: ubuntu-latest
     strategy:
+      fail-fast: false
       matrix:
-        nxf_ver: ['19.04.0', '']
-        max_retries: [1, 10]
+        nxf_ver: ['20.01.0', '']
+        test_type: ['ultra_quick_test', 'sra_test']
     steps:
       - uses: actions/checkout@v1
       - name: Install Nextflow
@@ -18,4 +19,27 @@ jobs:
           sudo mv nextflow /usr/local/bin/
       - name: Basic workflow tests
         run: |
-          nextflow run ${GITHUB_WORKSPACE} --max_retries ${{ matrix.max_retries }} -profile base,ultra_quick_test,docker
+          nextflow run ${GITHUB_WORKSPACE} -profile base,${{ matrix.test_type }},docker
+          echo "Results tree view:" ; tree -a results
+  singularity:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        singularity_version: ['3.6.4']
+        nxf_ver: ['20.01.0', '']
+        test_type: ['ultra_quick_test']
+    steps:
+      - uses: actions/checkout@v1
+      - uses: eWaterCycle/setup-singularity@v6
+        with:
+          singularity-version: ${{ matrix.singularity_version }}
+      - name: Install Nextflow
+        run: |
+          export NXF_VER=${{ matrix.nxf_ver }}
+          wget -qO- get.nextflow.io | bash
+          sudo mv nextflow /usr/local/bin/
+      - name: Basic workflow tests
+        run: |
+          nextflow run ${GITHUB_WORKSPACE} -profile base,${{ matrix.test_type }},singularity --echo true
+          echo "Results tree view:" ; tree -a results

README.md

@@ -76,6 +76,40 @@ Documentation about the pipeline, found in the [`docs/`](docs) directory:
 3. [Running the pipeline](docs/usage.md)
     * [Running on Sumner](docs/run_on_sumner.md)
     * [Running on CloudOS](docs/run_on_cloudos.md)
+    * [Running locally](docs/run_locally.md)
 
 ## Pipeline DAG
-<img src="https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/blob/c23ee9552eb033dec087fb3b6fb01fe26716ce29/DAG.png" alt="splicing_pip_dag" align = "left" width="600"/>
+<img src="https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/blob/c23ee9552eb033dec087fb3b6fb01fe26716ce29/DAG.png" alt="splicing_pip_dag" align = "center" width="600"/>
+
+
+## Changelog
+
+### v 1.1 - Pipeline improvements
+#### Fixes:
+ - Adds missing trimmomatic logs to the multiqc report
+ - Implemented correct support for input strandness in star process when `--stranded` is `second-strand` (was hardcoded to `strType=2` and only supported `first-strand` or `false` before)
+#### Updates:
+ - Updates the following tools:
+   - **STAR** `2.7.3` -> `2.7.9a` NOTE: Requires a new index! (updated in test profile)
+   - **Samtools** `1.10` -> `1.13`
+   - **StringTie** `2.1.3b` -> `2.1.7`
+   - **Gffread** `0.11.7` -> `0.12.7`
+   - multiqc `1.8` -> `1.11`
+   - deeptools `3.4.0` -> `3.5.1`
+   - bioconductor-rtracklayer `1.46.0` -> `1.52.0`
+   - gffcompare `0.11.2` -> `0.12.6`
+   - bedtools `2.29.2` -> `2.30.0`
+   - sra-tools `2.10.8` -> `2.11.0`
+   - pigz `2.3.4` -> `2.6.0`
+   - gdc-client `1.5.0` -> `1.6.1`
+ - Moves all containers to https://hub.docker.com/u/anczukowlab
+
+#### Maintenance:
+ - Consideably reduces number of basic redundant CI tests by removing completely the `max_retries` matrix and `push` from `on: [push, pull_request]`
+ - Adds CI test for sra-downloading pipeline pathway (only supported with docker profile for now)
+#### QOL:
+ - Adds saving of all the process .command* log files to results/process-logs folder
+ - Adds pipeline workdir `--cleanup` option to clear all intermediate files on pipeline successful completion
+ - Adds pipeline `--error_strategy` parameter to be able to specify pipeline error strategy directly from command line (doesn't work if specified in config linked by `-c` or `-config` nextflow params)
+
+### v 1.0 - Initial pipeline release

assets/sra-user-settings.mkfg

@@ -0,0 +1,2 @@
+/LIBS/IMAGE_GUID = "aee5f45c-f469-45f1-95f2-b2d2b1c59163"
+/libs/cloud/report_instance_identity = "true"

conf/examples/sra_test.config

@@ -0,0 +1,31 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running ultra quick tests
+ * -------------------------------------------------
+ * Defines bundled input files and everything required
+ * to run a fast and simple test. Use as follows:
+ *   nextflow run jacksonlabs/splicing-pipelines-nf -profile ultra_quick_test
+ * 
+ */
+
+params {
+  // Input data
+  singleEnd = true
+  reads = "$baseDir/examples/testdata/sra/sra.csv"
+  download_from = 'SRA'
+
+  // Genome references
+  gtf = 'https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/projects/jax/splicing-pipelines-nf/genes.gtf'
+  star_index = 'https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/projects/jax/splicing-pipelines-nf/star_2.7.9a_yeast_chr_I.tar.gz'
+
+  // Other
+  test = true
+  readlength = 500
+  // This doesn't make biological sense but prevents all reads being removed during trimming
+  overhang = 100
+
+  // Limit resources
+  max_cpus = 2
+  max_memory = 6.GB
+  max_time = 48.h
+}

conf/examples/ultra_quick_test.config

@@ -15,7 +15,7 @@ params {
 
   // Genome references
   gtf = 'https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/projects/jax/splicing-pipelines-nf/genes.gtf'
-  star_index = 'https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/projects/jax/splicing-pipelines-nf/star.tar.gz'
+  star_index = 'https://lifebit-featured-datasets.s3-eu-west-1.amazonaws.com/projects/jax/splicing-pipelines-nf/star_2.7.9a_yeast_chr_I.tar.gz'
 
   // Other
   test = true

conf/executors/singularity.config

@@ -0,0 +1,17 @@
+/*
+ * -------------------------------------------------
+ *  Nextflow config file for running pipeline with Singularity locally
+ * -------------------------------------------------
+ * Base config needed for running with -profile singularity
+ */
+
+params {
+  singularity_cache = "local_singularity_cache"
+}
+
+singularity {
+  enabled = true
+  cacheDir = params.singularity_cache
+  autoMounts = true
+}
+

containers/download_reads/environment.yml

@@ -5,8 +5,8 @@ channels:
   - defaults
   - anaconda
 dependencies:
-  - sra-tools=2.10.8
-  - pigz=2.3.4
-  - gdc-client=1.5.0
-  - samtools=1.10
-  - bedtools=2.29.2
+  - sra-tools=2.11.0
+  - pigz=2.6.0
+  - gdc-client=1.6.1
+  - samtools=1.13
+  - bedtools=2.30.0

containers/fasp/Dockerfile

@@ -17,7 +17,7 @@ RUN apt-get update \
   && cd fasp-scripts \
   && python setup.py install \
   && chmod +x fasp/scripts/* \
-  && conda install samtools=1.11 -c bioconda -c conda-forge \
+  && conda install samtools=1.13 -c bioconda -c conda-forge \
   && conda clean -a
 
 ENV PATH /fasp-scripts/fasp/scripts:$PATH

containers/splicing-pipelines-nf/Dockerfile

@@ -9,12 +9,15 @@ ENV PATH /opt/conda/envs/splicing-pipelines-nf/bin:$PATH
 COPY ./tagXSstrandedData.awk /usr/local/bin/
 RUN chmod +x /usr/local/bin/tagXSstrandedData.awk
 
-# Install the latest stringtie & gffcompare
-RUN wget http://ccb.jhu.edu/software/stringtie/dl/stringtie-2.1.3b.Linux_x86_64.tar.gz -O stringtie.tar.gz && \
-    tar xvzf stringtie.tar.gz && mv stringtie-2.1.3b.Linux_x86_64 stringtie && \
+# Install the latest stringtie, gffread & gffcompare
+RUN wget http://ccb.jhu.edu/software/stringtie/dl/stringtie-2.1.7.Linux_x86_64.tar.gz -O stringtie.tar.gz && \
+    tar xvzf stringtie.tar.gz && mv stringtie-2.1.7.Linux_x86_64 stringtie && \
     rm stringtie.tar.gz && mv stringtie/prepDE.py stringtie/stringtie /usr/local/bin && \
-    wget http://ccb.jhu.edu/software/stringtie/dl/gffcompare-0.11.6.Linux_x86_64.tar.gz -O gffcompare.tar.gz && \
-    tar xvzf gffcompare.tar.gz && mv gffcompare-0.11.6.Linux_x86_64 gffcompare && \
+    wget http://ccb.jhu.edu/software/stringtie/dl/gffread-0.12.7.Linux_x86_64.tar.gz -O gffread.tar.gz && \
+    tar xvzf gffread.tar.gz && mv gffread-0.12.7.Linux_x86_64 gffread && \
+    rm gffread.tar.gz && mv gffread/gffread /usr/local/bin/ && \
+    wget http://ccb.jhu.edu/software/stringtie/dl/gffcompare-0.12.6.Linux_x86_64.tar.gz -O gffcompare.tar.gz && \
+    tar xvzf gffcompare.tar.gz && mv gffcompare-0.12.6.Linux_x86_64 gffcompare && \
     rm gffcompare.tar.gz && mv gffcompare/gffcompare gffcompare/trmap /usr/local/bin/
 # Install gawk to fix https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/issues/120
 RUN apt-get install -y gawk

containers/splicing-pipelines-nf/environment.yml

@@ -6,13 +6,13 @@ channels:
 dependencies:
   - fastqc=0.11.9
   - trimmomatic=0.39
-  - star=2.7.3
-  - samtools=1.10
-  - deeptools=3.4.0
-  - multiqc=1.8
-  - gffread=0.11.7
-  - bioconductor-rtracklayer=1.46.0
+  - star=2.7.9a
+  - samtools=1.13
+  - deeptools=3.5.1
+  - multiqc=1.11
+  - bioconductor-rtracklayer=1.52.0
 
 # Now installed via executable in `Dockerfile`
-  # - stringtie=2.1.2
-  # - gffcompare=0.11.2
+  # - stringtie=2.1.7
+  # - gffread=0.12.7 (version not available on conda)
+  # - gffcompare=0.12.6

docs/run_locally.md

@@ -0,0 +1,73 @@
+# Run locally
+
+This guide provides information needed to run the pipeline in a local linux environment.
+
+## 0) Install Nextflow, Docker and/or Singularity
+
+### 0.1) Install Nextflow
+```bash
+wget -qO- https://get.nextflow.io | bash && sudo mv nextflow /usr/bin/
+```
+
+### 0.1) Install Docker
+See https://docs.docker.com/engine/install/ubuntu/
+
+
+### 0.2) Install Singularity
+See https://sylabs.io/guides/3.8/user-guide/quick_start.html and https://sylabs.io/guides/3.0/user-guide/installation.html
+
+Commands to install on CentOS:
+```bash
+sudo yum update -y && \
+sudo yum groupinstall -y 'Development Tools' && \
+sudo yum install -y \
+    openssl-devel \
+    libuuid-devel \
+    libseccomp-devel \
+    wget \
+    squashfs-tools
+
+export VERSION=1.16.6 OS=linux ARCH=amd64 && # adjust this as necessary \
+    wget https://dl.google.com/go/go$VERSION.$OS-$ARCH.tar.gz && \
+    sudo tar -C /usr/local -xzvf go$VERSION.$OS-$ARCH.tar.gz && \
+    rm go$VERSION.$OS-$ARCH.tar.gz
+
+echo 'export GOPATH=${HOME}/go' >> ~/.bashrc && \
+    echo 'export PATH=/usr/local/go/bin:${PATH}:${GOPATH}/bin' >> ~/.bashrc && \
+    source ~/.bashrc
+
+go get -u github.com/golang/dep/cmd/dep
+
+
+export VERSION=3.8.0 && # adjust this as necessary \
+    wget https://github.com/sylabs/singularity/releases/download/v${VERSION}/singularity-ce-${VERSION}.tar.gz && \
+    tar -xzf singularity-ce-${VERSION}.tar.gz && \
+    cd singularity-ce-${VERSION}
+
+./mconfig && \
+    make -C builddir && \
+    sudo make -C builddir install
+
+singularity help
+
+```
+
+## 1) Clone the git repository to local machine
+
+```bash
+git clone https://github.com/TheJacksonLaboratory/splicing-pipelines-nf
+cd splicing-pipelines-nf
+
+```
+
+## 2) Run the pipeline
+
+### 2.1) Quick test with Docker container engine
+```bash
+nextflow run . -profile ultra_quick_test,docker --cleanup
+```
+
+### 2.2) Quick test with Docker container engine
+```bash
+nextflow run . -profile ultra_quick_test,singularity --cleanup
+```