Methylation #543

dennishendriksen · 2023-12-18T09:06:11Z

End-to-end tests are not executed by Travis CI, please execute manually:

APPTAINER_BIND=$PWD bash test/test.sh passes
Updated documentation

A module to run dorado basecalling. A module to run modkit after sorting bam files Added a template file for modkit tool.

Updated modules to be more generic. Config file need path to data and run id. added dorado module for shell command.

Merge branch 'main' into PoC/Methylation

…p 7.1.0

Updated requirements

…com/molgenis/vip into PoC/Methylation

dennishendriksen · 2024-02-08T14:30:27Z

modules/pod5/templates/dorado.sh

+
+mod_basecaller() {
+  # Command for Dorado tool
+  echo "working"


can be removed

dennishendriksen · 2024-02-08T14:31:18Z

modules/pod5/templates/dorado.sh

+mod_basecaller() {
+  # Command for Dorado tool
+  echo "working"
+  ${CMD_DORADO} basecaller !{params.dorado_model} ./ --modified-bases 5mCG_5hmCG --reference !{reference} > !{bam}


command fails if variables contain characters such as a space:

Suggested change

${CMD_DORADO} basecaller !{params.dorado_model} ./ --modified-bases 5mCG_5hmCG --reference !{reference} > !{bam}

${CMD_DORADO} basecaller "!{params.dorado_model}" ./ --modified-bases 5mCG_5hmCG --reference "!{reference}" > "!{bam}"

dennishendriksen · 2024-02-08T14:31:45Z

modules/pod5/templates/modkit.sh

+
+summary() {
+  # Use modkit tool to summarize bam files
+  ${CMD_MODKIT} summary !{sorted_bam} > !{summary_modkit}


Suggested change

${CMD_MODKIT} summary !{sorted_bam} > !{summary_modkit}

${CMD_MODKIT} summary "!{sorted_bam}" > "!{summary_modkit}"

dennishendriksen · 2024-02-08T14:31:59Z

modules/pod5/templates/modkit.sh

+}
+
+adjust_mod() {
+  ${CMD_MODKIT} adjust-mods !{sorted_bam} !{converted_bam} --convert h m


Suggested change

${CMD_MODKIT} adjust-mods !{sorted_bam} !{converted_bam} --convert h m

${CMD_MODKIT} adjust-mods "!{sorted_bam}" "!{converted_bam}" --convert h m

dennishendriksen · 2024-02-08T14:32:08Z

modules/pod5/templates/modkit.sh

+
+adjust_mod() {
+  ${CMD_MODKIT} adjust-mods !{sorted_bam} !{converted_bam} --convert h m
+  ${CMD_SAMTOOLS} index -c !{converted_bam}


Suggested change

${CMD_SAMTOOLS} index -c !{converted_bam}

${CMD_SAMTOOLS} index -c "!{converted_bam}"

dennishendriksen · 2024-02-08T14:57:25Z

config/nxf_pod5.config

+	}
+
+	withLabel: 'modkit'{
+		executor = 'slurm'


dennishendriksen · 2024-02-08T15:00:34Z

docs/get_started/requirements.md

@@ -5,7 +5,7 @@ Before installing VIP please check whether your system meets the following requi
 - Bash ≥ 3.2
 - Java ≥ 11
 - [Apptainer](https://apptainer.org/docs/admin/main/installation.html#install-from-pre-built-packages) (setuid installation)
- 8GB RAM <sup>1</sup>
+- 100GB RAM <sup>1</sup>


note to self: update when config ram is updated

dennishendriksen · 2024-02-08T15:01:52Z

docs/usage/workflow.md

-VIP consists of four workflows depending on the type of input data: fastq, bam/cram, gvcf or vcf.
-The `fastq` workflow is an extension of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.
+VIP consists of five workflows depending on the type of input data: pod5, fastq, bam/cram, gvcf or vcf.
+The `fastq` and `pod5` workflows RE an extension of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.


Suggested change

The `fastq` and `pod5` workflows RE an extension of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.

The `fastq` and `pod5` workflows are extensions of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.

dennishendriksen · 2024-02-08T15:02:34Z

docs/usage/workflow.md

+1. Parallelize sample sheet per sample and for each sample
+2. Modified basecalling and alignment using [Dorado](https://github.com/nanoporetech/dorado) producing a `bam` file per sample
+3. Sorting the `bam` file per sample and create an index and stats file using [Samtools](http://samtools.github.io/)
+4. Perform pileup with [Modkit](https://github.com/nanoporetech/modkit) to construct a bedMethyl table per sample


Suggested change

4. Perform pileup with [Modkit](https://github.com/nanoporetech/modkit) to construct a bedMethyl table per sample

4. Perform pileup with [Modkit](https://github.com/nanoporetech/modkit) to construct a bedMethyl file per sample

dennishendriksen · 2024-02-08T15:07:03Z

docs/examples/pod5.md

+cd vip
+vip --workflow pod5 --input path/to/samplesheet.tsv --output path/to/output/folder
+```
+


This example would benefit from an actual report that displays methylation in a report. Admittedly other sections of the VIP documentation could benefit from more examples as well. As a user it is hard to understand how methylation is beneficial.

dennishendriksen

A new or modified test case in test/suites/vcf using bedmethyl would have been nice

dennishendriksen · 2024-02-09T05:56:55Z

test/suites/pod5/hg001_giab_2023.05.sh

+
+vip "${args[@]}" 1> /dev/null
+
+compare expected to actual output and store result


test currently fails due to syntax error:

Suggested change

compare expected to actual output and store result

# compare expected to actual output and store result

dennishendriksen · 2024-02-09T06:08:34Z

test/suites/pod5/hg001_giab_2023.05.sh

+
+compare expected to actual output and store result
+if [ "$(zcat "${OUTPUT_DIR}/vip.vcf.gz" | grep -vc "^#")" -gt 0 ]; then
+  result="0"


the output report contains 1 record and is 5.4MB, this might indicate an issue for larger number of records. possible cause: the genome browser displays many "white" reads possibly related to " In IGV, reads with a mapping quality = 0 are displayed in white while reads with a positive mapping quality are displayed in grey".

KloostermanJoukje and others added 30 commits October 26, 2023 10:04

feat: accept base modification workflow and input

7f341d2

feat: add first draft vip base modification workflow

7bc46f6

feat: added first draft base modification config file

b83599b

fix: input only tsv file not pod5 dir

ab020d6

feat: added working dorado tool

36bf8e7

feat: added config for dorado tool

a8e4baf

fix: no process named samtools fixed

47a37b5

Merge branch 'main' into PoC/Methylation, use vip relase 7.0.0

3d9ca87

Feat: added modules for pipeline

44fb4a6

A module to run dorado basecalling. A module to run modkit after sorting bam files Added a template file for modkit tool.

feat: updated workflow for modkit

d76589e

feat: updated config for modkit

9aa3601

fix: updated modules and added template dorado tool

e4028d0

Updated modules to be more generic. Config file need path to data and run id. added dorado module for shell command.

Update: newest vip version

da3d75f

Merge branch 'main' into PoC/Methylation

docs: updated usage in README.md

8ef202f

fix: improved workflow and added comments

ddd2212

fix: improved dorado process and added comments

32520ca

fix: improved process and added comments

29b1611

fix: improved process samtools and added comments

3778998

fix: updated config mod file and added comments

bc5cade

fix: added label to samtools workflow

a06be6b

feat: change to reference genome build 38

6017920

feat: add tool methplotlib for methylation frequency per region

3ae3dc9

feat: add toolconfig for tool methplotlib

3ee0bab

bug: methplotlib tool does not yet work

a0aeedb

feat: add conversion bam to cram to continue with vip pipeline

ebdd11e

fix: modified base pipeline until making of cram file

de6c4fd

Merge branch 'main' into PoC/Methylation, update to newest version vi…

dda383e

…p 7.1.0

Merge branch 'main' into PoC/Methylation, update to vip 7.2.0

ac46037

fix: undefined parameter

15299e7

fix: undefined parameter

8325265

KloostermanJoukje and others added 26 commits January 22, 2024 15:51

Feat: fix bed to bedmethyl bug, add h to m conversion

9117441

Refactor: delete unessecary code and update variables

3ee31d8

Refactor: change workflow name from mod to pod5

2369eb5

refactor: change module mod directory to pod5

8fa5388

Refactor: delete old mod workflow files

da010a8

Update docs

93e19e4

Fix: pod5 to cram workflow variables found

6f0bf24

Feat: added first modkit "draft" def file

45f5f91

Fix: cargo not found

591a3c9

Refactor: other method for mdokit def file

5dd56d4

Add extension to bed file

9892628

Refactor: commit added to modkit.def

a10fa30

Feat: use docker images for Dorado and Modkit

5c01e15

Feat: add docker images to pod5 workflow

f5edd79

Feat: add a pod5 workflow test

300b8e6

Fix: fix typo in template parameter

229fc8c

Feat: validate dorado model parameter

e9e9012

Add modkit and dorado image to install.sh

a96f6eb

Refactor: cleanup code and add test to docs

fe06d5f

Fix: download Dorado correctly

51080df

Feat: Add dorado Fast, HAC and SUP models to resources

0a5a4ff

Update requirements.md

3d9f831

Updated requirements

Add: add installation of tool to README.md

bac6588

Update docs lead to merge branch 'PoC/Methylation' of https://github.…

b0fc308

…com/molgenis/vip into PoC/Methylation

README.md: need to load awscli to execute pod5 test

c45f66f

README.md: add path to output test pod5

7ba380c

dennishendriksen commented Feb 8, 2024

View reviewed changes

dennishendriksen commented Feb 9, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Methylation #543

Methylation #543

dennishendriksen commented Dec 18, 2023

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen Feb 8, 2024

dennishendriksen left a comment

dennishendriksen Feb 9, 2024

dennishendriksen Feb 9, 2024

	${CMD_DORADO} basecaller !{params.dorado_model} ./ --modified-bases 5mCG_5hmCG --reference !{reference} > !{bam}
	${CMD_DORADO} basecaller "!{params.dorado_model}" ./ --modified-bases 5mCG_5hmCG --reference "!{reference}" > "!{bam}"

	${CMD_MODKIT} summary !{sorted_bam} > !{summary_modkit}
	${CMD_MODKIT} summary "!{sorted_bam}" > "!{summary_modkit}"

	${CMD_MODKIT} adjust-mods !{sorted_bam} !{converted_bam} --convert h m
	${CMD_MODKIT} adjust-mods "!{sorted_bam}" "!{converted_bam}" --convert h m

	${CMD_SAMTOOLS} index -c !{converted_bam}
	${CMD_SAMTOOLS} index -c "!{converted_bam}"

	The `fastq` and `pod5` workflows RE an extension of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.
	The `fastq` and `pod5` workflows are extensions of the `cram` workflow. The `cram` and `gvcf` workflows are extensions of the `vcf` workflow.

	4. Perform pileup with [Modkit](https://github.com/nanoporetech/modkit) to construct a bedMethyl table per sample
	4. Perform pileup with [Modkit](https://github.com/nanoporetech/modkit) to construct a bedMethyl file per sample


		vip "${args[@]}" 1> /dev/null

		compare expected to actual output and store result

	compare expected to actual output and store result
	# compare expected to actual output and store result

Methylation #543

Are you sure you want to change the base?

Methylation #543

Conversation

dennishendriksen commented Dec 18, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dennishendriksen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment