Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Adds sra test #253

Merged
merged 17 commits into from
Sep 15, 2021
Merged

Adds sra test #253

merged 17 commits into from
Sep 15, 2021

Conversation

Vlad-Dembrovskyi
Copy link
Contributor

@Vlad-Dembrovskyi Vlad-Dembrovskyi commented Aug 26, 2021 •
edited
Loading

Description

Adds a new test profile that tests the SRA input download option.
Closes #252

Purpose

Pipeline is designed to also support providing input fastq files as SRA accessions and download them with SRA toolkit, but this way of running the pipeline hasn't been covered yet by the CI tests.

Changes

  • Adds sra.csv test data and sra_test.config to be run with new profile sra_test
  • Removes the docker.runOptions in nextflow.config because sra-toolkit can't be run as non-root user with current container
  • Changes the ci tests to run also with sra_test profile (only for docker, as for singularity in ci it always fails)

To test

git clone https://github.com/TheJacksonLaboratory/splicing-pipelines-nf
cd splicing-pipelines-nf
git checkout add-sra-test

NXF_VER=20.01.0 nextflow run . -profile base,sra_test,docker

To test on sumner

git clone https://github.com/TheJacksonLaboratory/splicing-pipelines-nf
cd splicing-pipelines-nf
git checkout add-sra-test

NXF_VER=20.01.0 nextflow run . -profile base,sra_test,sumner

@Vlad-Dembrovskyi
Copy link
Contributor Author

I can't undesrstand why both ultra quick test and singularity tests are failing now.

Locally ultra_quick test is running fine:
nextflow run . -profile base,ultra_quick_test,docker
image

@Vlad-Dembrovskyi
Copy link
Contributor Author

Locally the singularity one is failing too, although with a different error
I think it may be due to how sra-toolkit was installed in docker container. sra-toolkit needs some kind of configuration to be run first before using it, and this configuration is user-specific. In the used docker container the sra-toolkit was installed for root user only. So if you are running the container with singularity with not a root user - you don't have the configuration set and that's why it fails. I tried to see how could one set this configuration for new user non-interactively, and I couldn't find a solution so far:

@Vlad-Dembrovskyi Vlad-Dembrovskyi linked an issue Aug 27, 2021 that may be closed by this pull request
Add small SRA dataset to CI [SRR15346687] #252
Closed
@Vlad-Dembrovskyi Vlad-Dembrovskyi mentioned this pull request Sep 1, 2021
Open
@Vlad-Dembrovskyi
Copy link
Contributor Author

The remaining SRA tests are finally not failing, @angarb and @cgpu approved the merge for closest release and to leave the singularity<>SRA issue for next release. Merging.

@Vlad-Dembrovskyi Vlad-Dembrovskyi merged commit 265a243 into dev Sep 15, 2021
@JoseEspinosa JoseEspinosa mentioned this pull request Sep 23, 2021
Merged
16 tasks
@Vlad-Dembrovskyi Vlad-Dembrovskyi mentioned this pull request Nov 24, 2021
Merged
Vlad-Dembrovskyi added a commit that referenced this pull request Nov 24, 2021
@Vlad-Dembrovskyi @cgpu
@angarb @imendes93
Pipeline release v2.0 (#245)
a330894 
* Adds workdir cleanup option (#238)

* Adds --cleanup option to clean work directory

* Adds explanatory comment to config for cleanup option

* Adds "Workdir cleanup" param to pipeline summary

* Adds --cleanup option description to help message

* Adds --cleanup option description to usage.md

* Adds singularity check in ci.yml (#240)

* Adds local singularity profile (#239)

* Adds profile to run with singularity locally

* Adds docks on how to run the pipeline locally

* Adds link to new docs to README.md

* Renames profile to singularity

* Adds singularity check in ci.yml (#240)

* Renames singularity_local -> singularity everywhere

Co-authored-by: cgpu <[email protected]>

* Adds workflow.onComplete notifications

* Reduces ci tests (#241)

* Reduces ci tests to only run on pull_request, not push

* Reduces ci tests to only run with max_retries: 1

* Updates ci test nextflow version 19.04.0 -> 20.01.0

* Removes max_retries matrix option from ci tests

Co-authored-by: cgpu <[email protected]>

* Adds trimmomatic logs to multiqc report (#244)

* Adds trimmomatic logs to multiqc report

* Puts Trimmomatic above STAR in MultiQC report

* Updates all tools and moves docker containers to anczukowlab (#248)

* Update usage.md

Best solution to #246 is to update documentation.

* Update usage.md

* Update run_on_sumner.md

Adding some clarification to address #247

* Update run_on_sumner.md

Adding the description to run pipeline with bams.csv to documentation

* Implements .command.* save in results/ (#251)

* Implements .command.* save in results/

* Cherry-pick residue removal

* Updates ci strategy to fail-fast: false

* Fix for "Unknown method `optional` on FileInParam type"

The mistake was I pasted in the input instead of the output directive
The failed ci is here:
https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/pull/251/checks?check_run_id=3419423845#step:5:51

* Adds ${task.process} in command-log results naming

* Adds publishDir pattern: negation for logs in all results

* Adds tree view step for verifying results

* Removes install tree [already available] [docker]

* Removes install tree [already available] [singularity]

* Improves folder structure per sample for logs

* Keeps only .command.log, .command.sh, .command.err

* Adds config param (CloudOS configs)

* Removes redundancy of .bw files in  star_mapped folder

After feedback from @angarb who spotted this redundancy,
we are removing the .bw file from the ${SRR}/<all-files> folder
and keeping it only in ${SRR}/<all-files>/all_bigwig only

├── star_mapped
│   ├── SRR4238351
│   │   ├── SRR4238351.Aligned.sortedByCoord.out.bam
│   │   ├── SRR4238351.Aligned.sortedByCoord.out.bam.bai
│   │   ├── SRR4238351.Log.final.out
│   │   ├── SRR4238351.Log.out
│   │   ├── SRR4238351.Log.progress.out
│   │   ├── SRR4238351.ReadsPerGene.out.tab
│   │   ├── SRR4238351.SJ.out.tab
│   │   ├── SRR4238351.Unmapped.out.mate1
│   │   └── SRR4238351.bw
│   └── all_bigwig
│       ├── SRR4238351.bw

* Sets fail-fast strategy to false in singularity CI

Complementary commit of 09b8787

* Fix for indentation

* Fix saving files (#263)

* Fixes *_data/* files not being saved for multiqcs step

* Fixes sample_lst.txt not being saved for prep_de step

* Fixes no files being saved for stringtie_merge step

* Fix prep_de step input

* Saves tmp/*_read_outcomes_by_bam.txt in both rmats proc

* Adds xstag strType parameter (#264)

* Adds sra test (#253)

* Adds SRA test profile

* Adds sra_test to ci tests

* Changes CI strategy to fail-fast:false

* CI syntax fix [previous commit]

* Parameterises echo in process scope

* Adds echo true for ci debugging

* Fixes sra-toolkit run with singularity

* Change sra example file to a really small one

* Revert the main container version to the newer one

* Removes commented unnecessary docker.runOptions line

* Removes failing sra_test ci test for singularity

* Fix star step in ci test

* Makes more robust star issue solution

* Returns errorStrategy = 'finish'

Co-authored-by: cgpu <[email protected]>

* Add b1 - control and b2- case to docs

* permissions changes

* Fix strType issue when stranded=false (#276)

* Update Dockerfile

Accidentally edited this file.

* Parametrize error strategy (clean pr) (#267)

* Parametrizes error strategy

* Adds error_strategy parameter to usage.md docs

* Update log.info to show actual errorStrategy value

* Fix typo

* Adds Changelog to README.md

* Outsources changelog into a separate file

* Fixes containers and parametrizes options in google.config (#281)

* Fixes containers being overwritten by google.config

* Parametrize google options

* [DEL 3039] Implement ftp download for SRA accessions (#2) (#283)

* [DEL 3039] Implement ftp download for SRA accessions (#2)

* add option for read download through FTP

* fix ftp path

* update information on download_from param

* Update run_on_sumner.md

* Fix FTP link generation; add test configs for both pair and single end data

* add catch for when single end run ends in _1

* Update main.nf

Co-authored-by: Vlad-Dembrovskyi <[email protected]>

* Changes http to ftp in get_ftp_accession

Because this works now!!

* Make sra example data smaller

* Re-enable sra_test for singularity, now with ftp

Co-authored-by: Vlad-Dembrovskyi <[email protected]>
Co-authored-by: Vlad-Dembrovskyi <[email protected]>

* Update docs/run_on_sumner.md

Co-authored-by: cgpu <[email protected]>

Co-authored-by: imendes93 <[email protected]>
Co-authored-by: cgpu <[email protected]>

* Create Copying_Files_From_Sumner_to_Cloud

These are instructions on how to copy files from the JAX HPC Sumner to the cloud. Addresses #139

* Rename Copying_Files_From_Sumner_to_Cloud to Copying_Files_From_Sumner_to_Cloud.md

* Makes saving of unmapped files optional, cleanup true by default  (#284)

* [DEL 3039] Implement ftp download for SRA accessions (#2)

* add option for read download through FTP

* fix ftp path

* update information on download_from param

* Update run_on_sumner.md

* Fix FTP link generation; add test configs for both pair and single end data

* add catch for when single end run ends in _1

* Update main.nf

Co-authored-by: Vlad-Dembrovskyi <[email protected]>

* Changes http to ftp in get_ftp_accession

Because this works now!!

* Make sra example data smaller

* Re-enable sra_test for singularity, now with ftp

Co-authored-by: Vlad-Dembrovskyi <[email protected]>
Co-authored-by: Vlad-Dembrovskyi <[email protected]>

* Update docs/run_on_sumner.md

Co-authored-by: cgpu <[email protected]>

* Add #274 , Set cleanup default as true (#3)

Co-authored-by: imendes93 <[email protected]>
Co-authored-by: cgpu <[email protected]>

* Sets default --cleanup false for google.config

* Adds --save_unmapped to usage.md

* Update usage.md

Updating config info

* Create NF_splicing_pipeline.config

Example NF Config

* Update run_on_sumner.md

* Update run_on_sumner.md

* Update usage.md

* Update main.nf

Updating to match usage.md parameter descriptions

* Update main.nf

* set errorStrategy to finish in base.config for sumner

* Update run_on_sumner.md

* Update help message

* Update usage.md to match main.nf help message

* Update usage.md to fix indentations

* Update usage.md

Moving NF tips from "running pipeline on Sumner"

* Update run_on_sumner.md

Moved NF tips to usage.md

* Update Copying_Files_From_Sumner_to_Cloud.md

* Fix: Removes rmats container from google.config

* Update changelog.md for v2.0

Co-authored-by: cgpu <[email protected]>
Co-authored-by: angarb <[email protected]>
Co-authored-by: Brittany Angarola <[email protected]>
Co-authored-by: imendes93 <[email protected]>
@Vlad-Dembrovskyi Vlad-Dembrovskyi deleted the add-sra-test branch November 24, 2021 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add small SRA dataset to CI [SRR15346687]
2 participants
@Vlad-Dembrovskyi @cgpu