Pipeline release v2.0 #245
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Adds --cleanup option to clean work directory * Adds explanatory comment to config for cleanup option * Adds "Workdir cleanup" param to pipeline summary * Adds --cleanup option description to help message * Adds --cleanup option description to usage.md * Adds singularity check in ci.yml (#240) * Adds local singularity profile (#239) * Adds profile to run with singularity locally * Adds docks on how to run the pipeline locally * Adds link to new docs to README.md * Renames profile to singularity * Adds singularity check in ci.yml (#240) * Renames singularity_local -> singularity everywhere Co-authored-by: cgpu <[email protected]> * Adds workflow.onComplete notifications * Reduces ci tests (#241) * Reduces ci tests to only run on pull_request, not push * Reduces ci tests to only run with max_retries: 1 * Updates ci test nextflow version 19.04.0 -> 20.01.0 * Removes max_retries matrix option from ci tests Co-authored-by: cgpu <[email protected]>
* Adds trimmomatic logs to multiqc report * Puts Trimmomatic above STAR in MultiQC report
Best solution to #246 is to update documentation.
Adding some clarification to address #247
Adding the description to run pipeline with bams.csv to documentation
* Implements .command.* save in results/ * Cherry-pick residue removal * Updates ci strategy to fail-fast: false * Fix for "Unknown method `optional` on FileInParam type" The mistake was I pasted in the input instead of the output directive The failed ci is here: https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/pull/251/checks?check_run_id=3419423845#step:5:51 * Adds ${task.process} in command-log results naming * Adds publishDir pattern: negation for logs in all results * Adds tree view step for verifying results * Removes install tree [already available] [docker] * Removes install tree [already available] [singularity] * Improves folder structure per sample for logs * Keeps only .command.log, .command.sh, .command.err * Adds config param (CloudOS configs) * Removes redundancy of .bw files in star_mapped folder After feedback from @angarb who spotted this redundancy, we are removing the .bw file from the ${SRR}/<all-files> folder and keeping it only in ${SRR}/<all-files>/all_bigwig only ├── star_mapped │ ├── SRR4238351 │ │ ├── SRR4238351.Aligned.sortedByCoord.out.bam │ │ ├── SRR4238351.Aligned.sortedByCoord.out.bam.bai │ │ ├── SRR4238351.Log.final.out │ │ ├── SRR4238351.Log.out │ │ ├── SRR4238351.Log.progress.out │ │ ├── SRR4238351.ReadsPerGene.out.tab │ │ ├── SRR4238351.SJ.out.tab │ │ ├── SRR4238351.Unmapped.out.mate1 │ │ └── SRR4238351.bw │ └── all_bigwig │ ├── SRR4238351.bw * Sets fail-fast strategy to false in singularity CI Complementary commit of 09b8787 * Fix for indentation
* Fixes *_data/* files not being saved for multiqcs step * Fixes sample_lst.txt not being saved for prep_de step * Fixes no files being saved for stringtie_merge step * Fix prep_de step input * Saves tmp/*_read_outcomes_by_bam.txt in both rmats proc
* Adds SRA test profile * Adds sra_test to ci tests * Changes CI strategy to fail-fast:false * CI syntax fix [previous commit] * Parameterises echo in process scope * Adds echo true for ci debugging * Fixes sra-toolkit run with singularity * Change sra example file to a really small one * Revert the main container version to the newer one * Removes commented unnecessary docker.runOptions line * Removes failing sra_test ci test for singularity * Fix star step in ci test * Makes more robust star issue solution * Returns errorStrategy = 'finish' Co-authored-by: cgpu <[email protected]>
Accidentally edited this file.
* Parametrizes error strategy * Adds error_strategy parameter to usage.md docs * Update log.info to show actual errorStrategy value * Fix typo
cgpu
reviewed
Sep 28, 2021
cgpu
reviewed
Sep 28, 2021
* Fixes containers being overwritten by google.config * Parametrize google options
|
First successful big run in non-ctrl env: |
|
Successful runs by @angarb of tests in ctrl env: |
* [DEL 3039] Implement ftp download for SRA accessions (#2) * add option for read download through FTP * fix ftp path * update information on download_from param * Update run_on_sumner.md * Fix FTP link generation; add test configs for both pair and single end data * add catch for when single end run ends in _1 * Update main.nf Co-authored-by: Vlad-Dembrovskyi <[email protected]> * Changes http to ftp in get_ftp_accession Because this works now!! * Make sra example data smaller * Re-enable sra_test for singularity, now with ftp Co-authored-by: Vlad-Dembrovskyi <[email protected]> Co-authored-by: Vlad-Dembrovskyi <[email protected]> * Update docs/run_on_sumner.md Co-authored-by: cgpu <[email protected]> Co-authored-by: imendes93 <[email protected]> Co-authored-by: cgpu <[email protected]>
These are instructions on how to copy files from the JAX HPC Sumner to the cloud. Addresses #139
* [DEL 3039] Implement ftp download for SRA accessions (#2) * add option for read download through FTP * fix ftp path * update information on download_from param * Update run_on_sumner.md * Fix FTP link generation; add test configs for both pair and single end data * add catch for when single end run ends in _1 * Update main.nf Co-authored-by: Vlad-Dembrovskyi <[email protected]> * Changes http to ftp in get_ftp_accession Because this works now!! * Make sra example data smaller * Re-enable sra_test for singularity, now with ftp Co-authored-by: Vlad-Dembrovskyi <[email protected]> Co-authored-by: Vlad-Dembrovskyi <[email protected]> * Update docs/run_on_sumner.md Co-authored-by: cgpu <[email protected]> * Add #274 , Set cleanup default as true (#3) Co-authored-by: imendes93 <[email protected]> Co-authored-by: cgpu <[email protected]>
Updating config info
Example NF Config
Updating to match usage.md parameter descriptions
Moving NF tips from "running pipeline on Sumner"
Moved NF tips to usage.md
|
Latest test from Britanny on CloudOS shows we are good to go for now. GTEX issues and improvements will be addressed in next release. https://cloudos.lifebit.ai/public/jobs/619bbaa4fb26cf01dcee8446 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
v 2.0 - Pipeline improvements
Improvements:
--cleanupoption to clear all intermediate files on pipeline successful completion (true by default, false for CloudOS) (Adds workdir cleanup option #238, Makes saving of unmapped files optional, cleanup true by default #284, 089d6e3)--error_strategyparameter to be able to specify pipeline error strategy directly from command line (doesn't work if specified in config linked by-cor-confignextflow params) (Parametrize error strategy (clean pr) #267)--download_fromoptionFTPmode to download SRA samples from EBI FTP ([DEL 3039] Implement ftp download for SRA accessions (#2) #283)-save_unmappedthat makes saving of STAR unmapped files optional (false by default) (Makes saving of unmapped files optional, cleanup true by default #284)Fixes:
--strandedissecond-strand(was hardcoded tostrType=2and only supportedfirst-strandorfalsebefore) (Adds xstag strType parameter (clean pr) #264)params.strandedwas set tofalse(Fix strType issue when stranded=false #276)Updates:
2.7.3->2.7.9aNOTE: Requires a new index! (updated in test profile)1.10->1.132.1.3b->2.1.70.11.7->0.12.71.8->1.113.4.0->3.5.11.46.0->1.52.00.11.2->0.12.62.29.2->2.30.02.10.8->2.11.02.3.4->2.6.01.5.0->1.6.1Maintenance:
max_retriesmatrix andpushfromon: [push, pull_request]To test
Pull the repository
Run the test profile with docker
Tested on sumner (JAX HPC) and approved by @angarb here: link