Finish Release-0.17.3

Updated documentation.

Fix to handling large introns in markov changes

Fix to creating output directories consistently.

AUTHORS

@@ -1,3 +1,4 @@
-Daniel Mapleson <[email protected]>
-Luca Venturini <[email protected]>
-David Swarbreck <[email protected]>
+Daniel Mapleson <[email protected]>
+David Swarbreck <[email protected]>
+Luca Venturini <[email protected]>
+

README.md

@@ -1,16 +1,18 @@
+![alt text](doc/source/images/portcullis_logo.png "Portcullis")
 
 #Portcullis
 
-Portcullis predicts junctions from aligned RNA-seq data.  We expect
-the user to have already generated a BAM file using a splice aware aligner of their
-choice.  For example, Tophat, Gsnap, STAR or HISAT will work fine.  Portcullis is designed to
-be as portable as possible so where possible does not rely on esoteric SAM tags and other
-artifacts that are not consistently present in all SAM/BAMs.  Portcullis
-will then analyse the BAM file to look for alignments containing gaps (REFSKIP 'N' cigar ops)
-and create a detailed analysis of all distinct gaps found in the BAM file, these
-are considered as potential junctions.  Portcullis provides various means (both 
-manual and/or automatic) of filtering these potential junctions in order to remove 
-false positives.  Portcullis can also filter the original BAM file removing alignments 
+Portcullis stands for PORTable CULLing of Invalid Splice junctions from pre-aligned 
+RNA-seq data.  It is known that RNAseq mapping tools generate many invalid junction
+predictions, particularly in deep datasets with high coverage over splice sites.
+In order to address this, instead for creating a new RNAseq mapper, with a focus 
+on SJ accuracy we created a tool that takes in a BAM 
+file generated by an RNAseq mapper of the user's own choice (e.g. Tophat2, Gsnap, 
+STAR2 or HISAT2) as input (i.e. it's portable).  It then, analyses and quantifies 
+all splice junctions in the file before, filtering (culling) those which are unlikely 
+to be genuine.  Portcullis output's junctions in a variety of formats making it
+suitable for downstream analysis (such as differential splicing analysis and gene
+modelling) without additional work.  Portcullis can also filter the original BAM file removing alignments 
 associated with `bad` junctions.  Both the filtered junctions and BAM files are cleaner
 and more usable resources which can more effectively be used to assist in downstream 
 analyses such as gene prediction and genome annotation. 

configure.ac

@@ -4,7 +4,7 @@
 
 # Autoconf setup
 AC_PREREQ([2.68])
-AC_INIT([portcullis],[0.16.1],[[email protected]],[portcullis],[http://www.tgac.ac.uk])
+AC_INIT([portcullis],[0.17.3],[[email protected]],[portcullis],[http://www.tgac.ac.uk])
 AC_CONFIG_SRCDIR([src/portcullis.cc])
 AC_CONFIG_AUX_DIR([build-aux])
 AC_CONFIG_MACRO_DIR([m4])

doc/source/conf.py

@@ -29,7 +29,7 @@
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.pngmath',
+    'sphinx.ext.mathjax',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -53,9 +53,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '0.16.1'
+version = '0.17.3'
 # The full version, including alpha/beta/rc tags.
-release = '0.16.1'
+release = '0.17.3'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.
@@ -100,12 +100,14 @@
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'alabaster'
+html_theme = 'nature'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
-#html_theme_options = {}
+#html_theme_options = {
+#    'page_width': 'auto',
+#}
 
 # Add any paths that contain custom themes here, relative to this directory.
 #html_theme_path = []
@@ -119,7 +121,7 @@
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+html_logo = 'images/portcullis_logo.png'
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -145,7 +147,8 @@
 #html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
+html_sidebars = { '**': ['globaltoc.html', 'relations.html', 'sourcelink.html', 'searchbox.html'], }
+
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.

doc/source/conf.py.in

@@ -29,7 +29,7 @@ needs_sphinx = '1.3'
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.pngmath',
+    'sphinx.ext.mathjax',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -100,12 +100,14 @@ pygments_style = 'sphinx'
 
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
-html_theme = 'alabaster'
+html_theme = 'nature'
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the
 # documentation.
-#html_theme_options = {}
+#html_theme_options = {
+#    'page_width': 'auto',
+#}
 
 # Add any paths that contain custom themes here, relative to this directory.
 #html_theme_path = []
@@ -119,7 +121,7 @@ html_theme = 'alabaster'
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+html_logo = 'images/portcullis_logo.png'
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -145,7 +147,8 @@ html_static_path = ['_static']
 #html_use_smartypants = True
 
 # Custom sidebar templates, maps document names to template names.
-#html_sidebars = {}
+html_sidebars = { '**': ['globaltoc.html', 'relations.html', 'sourcelink.html', 'searchbox.html'], }
+
 
 # Additional templates that should be rendered to pages, maps page names to
 # template names.

doc/source/index.rst

@@ -3,26 +3,22 @@
    You can adapt this file completely to your liking, but it should at least
    contain the root `toctree` directive.
 
-Welcome to portcullis' documentation!
-======================================
-
-#.. image:: images/spectre.png
-#    :scale: 50%
-
-Portcullis predicts junctions from aligned RNA-seq data.  We expect
-the user to have already generated a BAM file using a splice aware aligner of their
-choice.  For example, Tophat, Gsnap, STAR or HISAT will work fine.  Portcullis is designed to
-be as portable as possible so where possible does not rely on esoteric SAM tags and other
-artifacts that are not consistently present in all SAM/BAMs.  Portcullis
-will then analyse the BAM file to look for alignments containing gaps (REFSKIP 'N' cigar ops)
-and create a detailed analysis of all distinct gaps found in the BAM file, these
-are considered as potential junctions.  Portcullis provides various means (both 
-manual and/or automatic) of filtering these potential junctions in order to remove 
-false positives.  Portcullis can also filter the original BAM file removing alignments 
-associated with `bad` junctions.  Both the filtered junctions and BAM files are cleaner
-and more usable resources which can more effectively be used to assist in downstream 
-analyses such as gene prediction and genome annotation.  
+.. image:: images/portcullis_logo.png
 
+Welcome to Portcullis' documentation
+====================================
+
+Portcullis stands for PORTable CULLing of Invalid Splice junctions from pre-aligned 
+RNA-seq data.  It is known that RNAseq mapping tools generate many invalid junction
+predictions, particularly in deep datasets with high coverage over splice sites.
+In order to address this, instead for creating a new RNAseq mapper, with a focus 
+on SJ accuracy we created a tool that takes in a BAM 
+file generated by an RNAseq mapper of the user's own choice (e.g. Tophat2, Gsnap, 
+STAR2 or HISAT2) as input (i.e. it's portable).  It then, analyses and quantifies 
+all splice junctions in the file before, filtering (culling) those which are unlikely 
+to be genuine.  Portcullis output's junctions in a variety of formats making it
+suitable for downstream analysis (such as differential splicing analysis and gene
+modelling) without additional work.
 
 Contents:
 
@@ -31,30 +27,26 @@ Contents:
     :maxdepth: 2
 
     installation
+    quickstart
     using
     metrics
+    scripts
+    requirements
 
 
-.. _system:
-System requirements
-===================
-
-Portcullis supports Unix, linux or Mac systems.  Windows may work but hasn't been
-tested.  A minimum of 8GB RAM, which will enable you to process small - medium sized datasets.  
-Large datasets will require more RAM (potentially a lot more), the actual amount of
-memory required depends on how many spliced alignments are present in your BAM file.
-Portcullis does not need to store the whole spliced alignment, just a hashcode from the name.
-
 
 .. _issues:
+
 Issues
 ======
 
-Should you discover any issues with spectre, or wish to request a new feature please raise a ticket at https://github.com/maplesond/portcullis/issues.
-Alternatively, contact Daniel Mapleson at: [email protected]
+Should you discover any issues with portcullis, or wish to request a new feature 
+please raise a ticket at https://github.com/maplesond/portcullis/issues.  Alternatively, 
+contact Daniel Mapleson at: [email protected]
 
 
 .. _availability:
+
 Availability and License
 ========================
 
@@ -63,10 +55,22 @@ Open source code available on github: https://github.com/maplesond/portcullis.gi
 Spectre is available under GNU GLP V3: http://www.gnu.org/licenses/gpl.txt
 
 
-Indices and tables
-==================
+Authors and Acknowledgements
+============================
+
+**Daniel Mapleson** - Project lead, software developer
+
+**David Swarbreck** came up with the original plan and has helped design and
+guide the project from day 1.
+
+**Luca Venturini** made the logic for the rule-based filtering and has been constantly
+testing the software helping to find bugs, improve runtime performance at every
+stage.
 
-* :ref:`genindex`
-* :ref:`modindex`
-* :ref:`search`
+Thanks to **Gemy Kaithokattil** and **Shabonham Caim** for hunting bugs and providing 
+valuable feedback and **Sarah Bastkowski** for helping me understanding 
+the various machine learning algorithms and the maths behind it.  Also thanks to 
+**Chris Bennett** for making the logo.
 
+Finally, thanks to the Earlham Insitute and the NBI computing services for supporting
+the work.

doc/source/installation.rst

@@ -4,14 +4,14 @@ Installation
 ============
 
 Portcullis is primarily a C++ application with some python scripts.  We use the 
-GNU build system ``Autotools`` to assist with package management and to make the 
+GNU build system *Autotools* to assist with package management and to make the 
 software portable across UNIX type operating systems.  Installation of portcullis
 therefore follows a similar incantation to other autotools based projects::
 
-  ```./configure; make; sudo make install;```
+  ./configure && make && sudo make install
 
 However, there are a few caveats.  If you cloned the software directly from the 
-git repository you must first run "./autogen.sh" to create the configure and make 
+git repository you must first run ``./autogen.sh`` to create the configure and make 
 files for your project.  If you downloaded a source code distribution tarball those
 scripts are already present so you can skip this step.
 
@@ -23,56 +23,57 @@ Portcullis depends on some external software:
  * samtools
  * pthreads
  * zlib
- * sphinx (optional)
+ * sphinx (optional - for building documentation)
+ * python3 (optional - for running additional scripts)
 
 Please make sure these programs are correctly configured and installed 
 on your system prior to building portcullis.  Consult the each program's installation
 guide separately for instructions on how to do this.  Should you install these dependencies
 into non-standard locations you can direct portcullis to them by using the following
 options when running the configure script.
 
-  - ```--with-boost``` - for specifying a custom boost installation directory
-  - ```--with-zlib``` - for specifying a custom zlib installation directory
+  - ``--with-boost`` - for specifying a custom boost installation directory
+  - ``--with-zlib`` - for specifying a custom zlib installation directory
 
-Note there is not option for specifying a custom pthreads or samtools location.  
-We assume pthreads is correctly installed and configured for your system already.  In most cases
-it will be.  For samtools, we just require the executable to be on the path.
+Note there is not option for specifying a custom *pthreads* or *samtools* location.  
+We assume *pthreads* is correctly installed and configured for your system already.  In most cases
+it will be.  For *samtools*, we just require the executable to be on the path.
 
-If the user has sphinx installed then documentation will also be built along with
-the software.  If sphinx is not detected then the documentation building stage is
+If the user has *sphinx* installed then documentation will also be built along with
+the software.  If *sphinx* is not detected then the documentation building stage is
 skipped and documentation won't be available locally, although it can still be 
-found at: https://kat.readthedocs.org/en/latest/
+found at: https://portcullis.readthedocs.org/en/latest/
 
-Boost is statically linked and doesn't need to be available at runtime.  zlib and pthreads are 
-dynamically linked so will need to be on your LD_LIBRARY_PATH,
+Boost is statically linked and doesn't need to be available at runtime.  *zlib* and *pthreads* are 
+dynamically linked so will need to be on your ``LD_LIBRARY_PATH``,
 or in one of the automatically searched lib directories in order for portcullis 
 to dynamically link them at runtime.  No other non-system libraries need linking at runtime.
 
 
 Internal Dependencies
 ---------------------
 
-Portcullis contains HTSlib in the source tree.  The user does
-not need to do anything special to handle htslib as it is automatically
+Portcullis contains *HTSlib* and *Ranger* (a random forest implementation)  in the source tree.  The user does
+not need to do anything special to handle *htslib* and *ranger* as these are automatically
 built and managed inside portcullis.
 
 
 Compilation and Installation
 ----------------------------
 
-First change into the portcullis root directory and run ```./configure```, providing
-any options you feel are appropriate.  By default the installation directory is "/usr/local", 
+First change into the portcullis root directory and run ``./configure``, providing
+any options you feel are appropriate.  By default the installation directory is ``/usr/local``, 
 so the portcullis executable would be found at "/usr/local/bin" by default.  If you
 want to change this use the ``--prefix`` option as previously described.  For a full
-list of configuration options type ```./configure --help```.
+list of configuration options type ``./configure --help``.
 
-Next compile the software.  This can be done by typing ```make```.  The compiles
+Next compile the software.  This can be done by typing ``make``.  The compiles
 all internal dependencies and portcullis itself.
 
 To check the code compiled correct and is operating as expected you can optionally
-type  ```make check``` to runs some tests.  This includes unit tests for HTSlib, 
+type  ``make check`` to runs some tests.  This includes unit tests for HTSlib, 
 which are embedded in the portcullis source tree.  To run only portcullis 
-unit tests go into the ``tests`` subdirectory and run ```make check``` there.
+unit tests go into the ``tests`` subdirectory and run ``make check`` there.
 
 Finally to install the compiled code to the specified (or default) installation
-directory type ```make install```.
+directory type ``make install``.