Add Mummer4 tools #1953

nickeener · 2018-06-28T20:21:41Z

Hey my name's Nick and this is my first project for Dr. Nekrutenko. It's not quite done and there are still a few problems that I will list, but Nekrutenko suggested making a pull request so more people could offer help.

This is my attempt at wrapping the 16 commands of the mummer4 package, which contains sequence alignment and visualization tools. Of the 16 commands, I was able to able to get 10 of them to pass the test run by planemo (nucmer, delta-filter, exact-tandems, mgaps, mummer, repeat-match, show-aligns, show-coords, show-diff, show-snps). Another 3 fail the test but that problem should be fixed by the following pull request by Dave B: mummer4/mummer#68 (promer, dnadiff, mummerplot).

Problems:
mummer - needs to incorporate data manager created by Dave B (https://github.com/davebx/tools-iuc/tree/mummer_dm/data_managers/data_manager_mummer4_index_builder). Not sure how to go about this.

dnadiff - First test gives error: "Syntax error: Unbound variable input". Not sure what's going on here. Second test fails due to foundation.pm error that's fixed by aforementioned pull request.

show-tiling - When running the command, the output is blank. I have have the dependencies and optional utilities installed so I don't know why it's not working.

combineMUMs - When I run this command I get a seg fault. Also no idea why.

annotate - Confused about the input for this command. Man page says it's the output from the gaps command except there is no gaps command and the mgaps output doesn't seem to work.

I'd appreciate any suggestions on fixing these problems or any concerning any of the other commands. I look forward to working with you all.

Nick Keener

peterjc · 2018-06-28T20:45:37Z

For reference, I wrapped a sub-set of the suite here:
http://toolshed.g2.bx.psu.edu/view/peterjc/mummer
https://github.com/peterjc/pico_galaxy/tree/master/tools/mummer

What you are doing here will be far more flexible and powerful - good plan 👍

bgruening · 2018-06-28T20:55:21Z

Wonderful @nickeener! Many of us are currently at the GCC ... so sorry if we take a little bit longer to review it. Thanks!

bgruening

please add a .shed.yml file

bgruening · 2018-06-28T21:09:37Z

tools/mummer4/annotate.xml

+    <command detect_errors="exit_code">
+    	<![CDATA[
+    	annotate
+    		$mgaps


single quotes for all data-params and text-params

bgruening · 2018-06-28T21:15:52Z

tools/mummer4/combineMUMs.xml

+    			<option value="no">NO</option>
+    		</param>
+    		<when value="yes" >
+    			<param name="qlabel" type="text" argument="-q" value="query" label="Query match label" />


you could remove this conditional and simply make the text parameter optional="True"

bgruening · 2018-06-28T21:16:21Z

tools/mummer4/combineMUMs.xml

+    	<param name="strings" type="boolean" argument="-s" truevalue="-s" falsevalue="" label="Output all differences in strings" />
+    	<param name="qheader" type="boolean" argument="-t" truevalue="-t" falsevalue="" label="Label query matches with query fasta header" />
+    	<param name="verbose" type="select" label="Set verbose level for extra output" >
+    		<option value="0">0</option>


please use indentation of 4 spaces everywhere.

I just used a single tab for all my indentation, does that mean I need to go through and replace every tab with 4 spaces?

Yes, this would be cool. please also see here: http://galaxy-iuc-standards.readthedocs.io/en/latest/best_practices/tool_xml.html

bgruening · 2018-06-28T21:16:39Z

tools/mummer4/combineMUMs.xml

+    </outputs>
+    <tests>
+    	<test>
+    		<param name="reference_sequence" ftype="fasta" value="K12.fasta" />


indentation

bgruening · 2018-06-28T21:17:36Z

tools/mummer4/dnadiff.xml

+                <param name="query_sequence" type="data" format="fasta" multiple="True" label="Query Sequence(s)" />
+            </when>
+            <when value="delta">
+                <param name="delta_data" type="data" format="txt" label="Alignment file from nucmer" />


general question to all of these txt formats. Are they really undefined, or are these maybe tabular etc...?

How would I tell if they were tabular? The original file in this case had the suffix ".delta" which wasn't an accepted format so I changed it to .txt.

nickeener · 2018-06-30T17:47:39Z

I have a weird problem with the mgaps command. I initially installed mummer4 on my laptop several months ago and mgaps worked fine. But then just recently I got a new laptop and installed mummer4 on it and now the mgaps command does not exist. Not sure why it didn't get installed the 2nd time or how to get it now.

Also, the nucmer command is now failing even though it passed before making this pull request and I have not updated it since.

bgruening · 2018-07-01T15:05:56Z

How did you installed it? With conda create -n mummer mummer4?

nickeener · 2018-07-02T03:18:04Z

I used: conda install -c bioconda mummer4
I have all the other commands, just not mgaps.

nekrut · 2018-10-16T21:09:15Z

Great job! Some notes:

Let's restrict the toolkit to mummer, mummerplot, nucmer, nucplot, filter-delta, show-coord
Combine nucmer and nucplot into one tool with an additional option to output the plot. Same thing for mummer

nickeener · 2018-10-30T20:33:03Z

@bgruening Could you please take a look at my modified code? Travis-CI tests do not pass yet due to some problem with git but we are working on that.

bgruening

@nickeener thanks, this looks very good!

please check all filetypes if you can not use tabular instead of txt
please check your requirements, ideally you have just mummer4 as the requirement
you could save a lot of lines if you use macros ... e.g. requirements, citations, version numbers and so on can all be moved into a macro file and shared across the tools

bgruening · 2018-11-01T08:24:46Z

tools/mummer4/delta-filter.xml

+<tool id="delta-filter" name="Delta-Filter" version="0.1.0">
+    <requirements>
+        <requirement type="package" version="4.0.0beta2">mummer4</requirement>
+        <requirement type="package" version="1.12.14">sh</requirement>


@nickeener why are all the other dependency needed? You just need mummer4 isn't it? And what is sh?

bgruening · 2018-11-01T08:25:22Z

tools/mummer4/mummer.xml

@@ -0,0 +1,176 @@
+<tool id="mummer" name="Mummer" version="0.1.0">


I would change all tool versions to the mummer version.

bgruening · 2018-11-01T08:27:44Z

tools/mummer4/mummer.xml

+        </conditional>
+    </inputs>
+    <outputs>
+        <data name="output" format="txt" from_work_dir="mummer.txt" label="mummer" />


I would remove all label from the outputs ... or use something like label="${tool.name} on ${on_string}: mumplot"

For all tools :)

bgruening · 2018-11-01T08:30:42Z

tools/mummer4/nucmer.xml

+    </requirements>
+    <command detect_errors="exit_code">
+        <![CDATA[
+        #set $query=""+"".join(str($query_sequence).split(','))+""


can you also use "'+'" here? Is the leading +"" needed?

bgruening · 2018-11-01T08:32:45Z

tools/mummer4/show-coords.xml

+    <tests>
+        <test>
+            <param name="delta" ftype="txt" value="nucmer.txt" />
+            <output name="output" ftype="txt" compare="diff" value="show-coords.txt" />


We could, if you like, attach the column names actually as metadata.
See here: https://github.com/galaxyproject/tools-iuc/blob/master/tools/deseq2/deseq2.xml#L170

And it should probably be tabular isn't it?

nickeener · 2018-11-05T17:25:10Z

@bgruening Thanks for the feedback. I believe I've fixed everything you mentioned. Are there any other changes I should make?

… plus some other minor fixes

… 'from_work_dir' flag to reflect their correct extension, added nucmerplot and promerplot (combo tools of nucmer and mummerplot and promer and mummerplot), completely re-did the test data using smaller sequences, added labels to outputs, plus several other minor fixes

…axis option for mummerplot, and some other minor changes

…quirements, added macro for citation and mummerplot input, fixed version number, fixed output labels, added column names for certain outputs, and a few other fixes

…ti-fasta is much simpler)

bgruening · 2018-11-06T08:32:11Z

@nickeener I rebased against master for you, but a few tests are still failing. Sorry.

…names to some dnadiff outputs

…m tools can find original input files

bgruening · 2018-11-30T23:28:28Z

@nekrut can you have a final look at this?
@nickeener I added a few things, can you maybe add a <description> to all tools?

nekrut · 2018-12-03T21:51:52Z

@nickeener add help section as we discussed. Once done, @bgruening I think we are ready!

bgruening

Anyone else?

Thanks @nickeener!

nickeener · 2018-12-04T17:53:13Z

Thanks for all the help @bgruening!

bgruening reviewed Jun 28, 2018

View reviewed changes

nsoranzo changed the title ~~Initial commit~~ Add Mummer4 tools Jun 29, 2018

bgruening reviewed Nov 1, 2018

View reviewed changes

Nicholas Keener and others added 14 commits November 5, 2018 21:13

Initial commit

f390a7c

Added single quotes around data and text params and fixed indentation…

0f7a93c

… plus some other minor fixes

Updated test-data file headers to reflect new path

e88e641

Made some fixes to combineMUMs

b76ea88

Fixed a few typos that were causing some tools to fail the test

5c1ba92

Fixed error in combineMUMs that caused it to fail linting

8c5f8b1

Modified/expanded help sections and some other minor fixes

de49b2e

fixed header option

e7be16b

Deleted extra tools, added mummerplot option to mummer, fixed ID and …

4ec5211

…axis option for mummerplot, and some other minor changes

Minor fixes

a72cbd6

Update .shed.yml

60cf3bd

Changed appropriate txt data types to tabular, removed unnecessary re…

19c2cd4

…quirements, added macro for citation and mummerplot input, fixed version number, fixed output labels, added column names for certain outputs, and a few other fixes

Removed option to have multiple query seqs as input (using single mul…

6dff371

…ti-fasta is much simpler)

nickeener and others added 4 commits November 7, 2018 12:43

Changed header in nucmer.txt file to relative paths and added column …

e1546f8

…names to some dnadiff outputs

fixed stupid mistake

de9b794

changed absolute paths in output files to symbolic links so downstrea…

d54bf42

…m tools can find original input files

use more macros

8da939e

Added <description> tag to each xml

a7c36a1

Nicholas Keener and others added 2 commits December 3, 2018 23:08

Added command line options descriptions in the help section

7c698a5

enable suite in the shed.yml

ab9c3b9

bgruening approved these changes Dec 4, 2018

View reviewed changes

bgruening merged commit 9c835e6 into galaxyproject:master Dec 4, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Mummer4 tools #1953

Add Mummer4 tools #1953

nickeener commented Jun 28, 2018

peterjc commented Jun 28, 2018

bgruening commented Jun 28, 2018

bgruening left a comment

bgruening Jun 28, 2018

bgruening Jun 28, 2018

bgruening Jun 28, 2018

nickeener Jun 29, 2018

bgruening Jun 29, 2018

bgruening Jun 28, 2018

bgruening Jun 28, 2018

nickeener Jun 29, 2018

nickeener commented Jun 30, 2018

bgruening commented Jul 1, 2018

nickeener commented Jul 2, 2018

nekrut commented Oct 16, 2018

nickeener commented Oct 30, 2018

bgruening left a comment

bgruening Nov 1, 2018

bgruening Nov 1, 2018

bgruening Nov 1, 2018

bgruening Nov 1, 2018

bgruening Nov 1, 2018

bgruening Nov 1, 2018

bgruening Nov 1, 2018

nickeener commented Nov 5, 2018

bgruening commented Nov 6, 2018

bgruening commented Nov 30, 2018

nekrut commented Dec 3, 2018

bgruening left a comment

nickeener commented Dec 4, 2018

		@@ -0,0 +1,176 @@
		<tool id="mummer" name="Mummer" version="0.1.0">

Add Mummer4 tools #1953

Add Mummer4 tools #1953

Conversation

nickeener commented Jun 28, 2018

peterjc commented Jun 28, 2018

bgruening commented Jun 28, 2018

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nickeener commented Jun 30, 2018

bgruening commented Jul 1, 2018

nickeener commented Jul 2, 2018

nekrut commented Oct 16, 2018

nickeener commented Oct 30, 2018

bgruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nickeener commented Nov 5, 2018

bgruening commented Nov 6, 2018

bgruening commented Nov 30, 2018

nekrut commented Dec 3, 2018

bgruening left a comment

Choose a reason for hiding this comment

nickeener commented Dec 4, 2018