Merge pull request #1295 from deeptools/develop

v3.5.5

.github/CONTRIBUTING.md

@@ -17,4 +17,4 @@ for contributing to the repository :**
 
 ## Testing
 
-* Please make sure that travis tests are passing
+* Please make sure that github actions are passing

.github/test_and_build.yml

@@ -2,7 +2,7 @@ channels:
   - conda-forge
   - bioconda
 dependencies:
-  - python >= 3.7
+  - python > 3.7
   - numpy
   - scipy
   - flake8

.github/workflows/planemo.yml

@@ -49,7 +49,7 @@ jobs:
     runs-on: ubuntu-latest
     strategy:
       matrix:
-        python-version: ['3.7', '3.11']
+        python-version: ['3.8', '3.11']
     steps:
     - uses: actions/download-artifact@v3
       with:

.github/workflows/test.yml

@@ -63,22 +63,22 @@ jobs:
         micromamba activate test_and_build
         rm -f dist/*
         python -m build
-    - uses: actions/upload-artifact@master
+    - uses: actions/upload-artifact@v3
       with:
-        name: "Dist files"
+        name: "distfiles"
         path: "dist"
   test-wheels:
     name: test wheel
     runs-on: ubuntu-latest
     needs: build-linux
     strategy:
       matrix:
-        python-version: ['3.7','3.8','3.9','3.10', '3.11']
+        python-version: ['3.8','3.9','3.10', '3.11', '3.12']
     steps:
       - uses: actions/checkout@v3
       - uses: actions/download-artifact@v3
         with:
-          name: "Dist files"
+          name: "distfiles"
           path: ~/dist/
       - uses: actions/setup-python@v4
         with:

.readthedocs.yaml

@@ -3,11 +3,13 @@ version: 2
 build:
   os: ubuntu-22.04
   tools:
-    python: "3.11"
+    python: "3.12"
 
 sphinx:
   configuration: docs/conf.py
 
 python:
   install:
+  - method: pip
+    path: .
   - requirements: docs/requirements.txt

CHANGES.txt

@@ -1,3 +1,19 @@
+3.5.5
+* drop support for python 3.7
+* doc fixes (argparse properly displayed, minor changes in installation instructions)
+* deepblue support stops
+* initiate deprecation of tight_layout in plotheatmap, in favor of constrained_layout. Minor changes in paddings, etc can occur (but for the better).
+* documentation changes to improve ESS tab, table constraints have been lifted & sphinx_rtd_theme to v2.0.0
+* upload artifact in gh test runner pinned to 3
+* Try to get the number of processors from sched_getaffinity, to avoid using to many in job submissions for example. #1199
+* Fix typo in estimateScaleFactor that fixes broken argparsing. #1286
+
+3.5.4
+* error handling and cases for bwAverage with >2 samples
+* Tick.label deprecation for mpl 3.8
+* minimal mpl version is 3.5
+* cicd update for pypi push
+
 3.5.3
 * requirement cap for matplotlib lifted (changes in plotting can occur)
 * nose has been deprecated in favor of pytests

README.md

@@ -4,7 +4,6 @@
 [![install with bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg?style=flat)](http://bioconda.github.io/recipes/deeptools/README.html)
 [![European Galaxy server](https://img.shields.io/badge/usegalaxy-.eu-brightgreen?logo=data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABgAAAASCAYAAABB7B6eAAAABGdBTUEAALGPC/xhBQAAACBjSFJNAAB6JgAAgIQAAPoAAACA6AAAdTAAAOpgAAA6mAAAF3CculE8AAAACXBIWXMAAAsTAAALEwEAmpwYAAACC2lUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyI+CiAgICAgICAgIDx0aWZmOlJlc29sdXRpb25Vbml0PjI8L3RpZmY6UmVzb2x1dGlvblVuaXQ+CiAgICAgICAgIDx0aWZmOkNvbXByZXNzaW9uPjE8L3RpZmY6Q29tcHJlc3Npb24+CiAgICAgICAgIDx0aWZmOk9yaWVudGF0aW9uPjE8L3RpZmY6T3JpZW50YXRpb24+CiAgICAgICAgIDx0aWZmOlBob3RvbWV0cmljSW50ZXJwcmV0YXRpb24+MjwvdGlmZjpQaG90b21ldHJpY0ludGVycHJldGF0aW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KD0UqkwAAAn9JREFUOBGlVEuLE0EQruqZiftwDz4QYT1IYM8eFkHFw/4HYX+GB3/B4l/YP+CP8OBNTwpCwFMQXAQPKtnsg5nJZpKdni6/6kzHvAYDFtRUT71f3UwAEbkLch9ogQxcBwRKMfAnM1/CBwgrbxkgPAYqlBOy1jfovlaPsEiWPROZmqmZKKzOYCJb/AbdYLso9/9B6GppBRqCrjSYYaquZq20EUKAzVpjo1FzWRDVrNay6C/HDxT92wXrAVCH3ASqq5VqEtv1WZ13Mdwf8LFyyKECNbgHHAObWhScf4Wnj9CbQpPzWYU3UFoX3qkhlG8AY2BTQt5/EA7qaEPQsgGLWied0A8VKrHAsCC1eJ6EFoUd1v6GoPOaRAtDPViUr/wPzkIFV9AaAZGtYB568VyJfijV+ZBzlVZJ3W7XHB2RESGe4opXIGzRTdjcAupOK09RA6kzr1NTrTj7V1ugM4VgPGWEw+e39CxO6JUw5XhhKihmaDacU2GiR0Ohcc4cZ+Kq3AjlEnEeRSazLs6/9b/kh4eTC+hngE3QQD7Yyclxsrf3cpxsPXn+cFdenF9aqlBXMXaDiEyfyfawBz2RqC/O9WF1ysacOpytlUSoqNrtfbS642+4D4CS9V3xb4u8P/ACI4O810efRu6KsC0QnjHJGaq4IOGUjWTo/YDZDB3xSIxcGyNlWcTucb4T3in/3IaueNrZyX0lGOrWndstOr+w21UlVFokILjJLFhPukbVY8OmwNQ3nZgNJNmKDccusSb4UIe+gtkI+9/bSLJDjqn763f5CQ5TLApmICkqwR0QnUPKZFIUnoozWcQuRbC0Km02knj0tPYx63furGs3x/iPnz83zJDVNtdP3QAAAABJRU5ErkJggg==)](https://usegalaxy.eu/root?tool_id=deeptools_compute_matrix)
 ![test](https://github.com/deeptools/deepTools/actions/workflows/test.yml/badge.svg)
-![planemo](https://github.com/deeptools/deepTools/actions/workflows/planemo.yml/badge.svg)
 
 
 ## User-friendly tools for exploring deep-sequencing data
@@ -34,35 +33,26 @@ Our [Gallery](http://deeptools.readthedocs.org/en/latest/content/example_gallery
 
 deepTools are available for:
 
-* Command line usage (via pip/anaconda/github)
+* Command line usage (via pip / conda / github)
 * Integration into Galaxy servers (via toolshed/API/web-browser)
 
-There are many easy ways to install deepTools. Details can be found [here](https://deeptools.readthedocs.io/en/latest/content/installation.html)
+There are many easy ways to install deepTools. More details can be found [here](https://deeptools.readthedocs.io/en/latest/content/installation.html).
 
-**Install by cloning this repository:**
+In Brief:
 
-You can install any one of the deepTools branches on command line (linux/mac) by cloning this git repository :
+**Install through pypi**
 
-	$ git clone https://github.com/deeptools/deepTools
-	$ cd deepTools
-	$ python setup.py install
-
-By default, the script will install the python library and executable
-codes globally, which means you need to be root or administrator of
-the machine to complete the installation. If you need to
-provide a nonstandard install prefix, or any other nonstandard
-options, you can provide many command line options to the install
-script.
+	$ pip install deeptools
 
-	$ python setup.py --help
+**Install via conda**
 
-For example, to install under a specific location use:
+	$ conda install -c bioconda deeptools
 
-	$ python setup.py install --prefix <target directory>
+**Install by cloning the repository**
 
-To install into your home directory, use:
-
-	$ python setup.py install --user
+	$ git clone https://github.com/deeptools/deepTools
+	$ cd deepTools
+	$ pip install .
 
 <a name="galaxy"/></a>
 ### Galaxy Installation

deeptools/alignmentSieve.py

@@ -7,11 +7,8 @@
 from deeptools import parserCommon
 from deeptools.bamHandler import openBam
 from deeptools.mapReduce import mapReduce
-try:  # keep python 3.7 support.
-    from importlib.metadata import version
-except ModuleNotFoundError:
-    from importlib_metadata import version
 from deeptools.utilities import getTLen, smartLabels, getTempFileName
+from importlib.metadata import version
 
 
 def parseArguments():

deeptools/bamPEFragmentSize.py

@@ -18,10 +18,7 @@
 # own tools
 from deeptools.parserCommon import writableFile
 from deeptools.getFragmentAndReadSize import get_read_and_fragment_length
-try:  # keep python 3.7 support.
-    from importlib.metadata import version
-except ModuleNotFoundError:
-    from importlib_metadata import version
+from importlib.metadata import version
 
 
 def parse_arguments():

deeptools/bigwigAverage.py

@@ -1,23 +1,19 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-import argparse  # to parse command line arguments
+import argparse
 import sys
-import multiprocessing
-import os
 import numpy as np
 from deeptools import parserCommon
 from deeptools import writeBedGraph_bam_and_bw
-import deeptools.deepBlue as db
 
 debug = 0
 
 
 def parse_arguments(args=None):
     parentParser = parserCommon.getParentArgParse()
     outputParser = parserCommon.output()
-    dbParser = parserCommon.deepBlueOptionalArgs()
     parser = argparse.ArgumentParser(
-        parents=[parentParser, outputParser, dbParser],
+        parents=[parentParser, outputParser],
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
         description='This tool average multiple bigWig files based on the number '
         'of mapped reads. To average the bigWig files, the genome is '
@@ -59,7 +55,7 @@ def parse_arguments(args=None):
 
 def getType(fname):
     """
-    Tries to determine if a file is a wiggle file from deepBlue or a bigWig file.
+    Tries to determine if a file is a wiggle file a bigWig file.
     Returns 'wiggle' if the file name ends with .wig, otherwise 'bigwig'
     """
     if fname.endswith(".wig") or fname.endswith(".wiggle"):
@@ -119,29 +115,6 @@ def main(args=None):
     FUNC = average
     function_args = {'scaleFactors': scaleFactors}
 
-    # Preload deepBlue files, which need to then be deleted
-    deepBlueFiles = []
-    for idx, fname in enumerate(args.bigwigs):
-        if db.isDeepBlue(fname):
-            deepBlueFiles.append([fname, idx])
-    if len(deepBlueFiles) > 0:
-        sys.stderr.write("Preloading the following deepBlue files: {}\n".format(",".join([x[0] for x in deepBlueFiles])))
-        foo = db.deepBlue(deepBlueFiles[0][0], url=args.deepBlueURL, userKey=args.userKey)
-        regs = db.makeChromTiles(foo)
-        for x in deepBlueFiles:
-            x.extend([args, regs])
-        if len(deepBlueFiles) > 1 and args.numberOfProcessors > 1:
-            pool = multiprocessing.Pool(args.numberOfProcessors)
-            res = pool.map_async(db.preloadWrapper, deepBlueFiles).get(9999999)
-        else:
-            res = list(map(db.preloadWrapper, deepBlueFiles))
-
-        # substitute the file names with the temp files
-        for (ftuple, r) in zip(deepBlueFiles, res):
-            args.bigwigs[ftuple[1]] = r
-        deepBlueFiles = [[x[0], x[1]] for x in deepBlueFiles]
-        del regs
-
     writeBedGraph_bam_and_bw.writeBedGraph(
         [(b, getType(b)) for b in args.bigwigs],
         args.outFileName, 0, FUNC,
@@ -154,12 +127,3 @@ def main(args=None):
         smoothLength=False,
         missingDataAsZero=not args.skipNonCoveredRegions,
         extendPairedEnds=False)
-
-    # Clean up temporary bigWig files, if applicable
-    if not args.deepBlueKeepTemp:
-        for k, v in deepBlueFiles:
-            os.remove(args.bigwigs[v])
-    else:
-        for k, v in deepBlueFiles:
-            foo = args.bigwigs[v]
-            print("{} is stored in {}".format(k, foo))

deeptools/bigwigCompare.py

@@ -1,23 +1,18 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-import argparse  # to parse command line arguments
-import sys
-import multiprocessing
-import os
+import argparse
 from deeptools import parserCommon
 from deeptools.getRatio import getRatio
 from deeptools import writeBedGraph_bam_and_bw
-import deeptools.deepBlue as db
 
 debug = 0
 
 
 def parse_arguments(args=None):
     parentParser = parserCommon.getParentArgParse()
     outputParser = parserCommon.output()
-    dbParser = parserCommon.deepBlueOptionalArgs()
     parser = argparse.ArgumentParser(
-        parents=[parentParser, outputParser, dbParser],
+        parents=[parentParser, outputParser],
         formatter_class=argparse.ArgumentDefaultsHelpFormatter,
         description='This tool compares two bigWig files based on the number '
         'of mapped reads. To compare the bigWig files, the genome is '
@@ -104,7 +99,7 @@ def parse_arguments(args=None):
 
 def getType(fname):
     """
-    Tries to determine if a file is a wiggle file from deepBlue or a bigWig file.
+    Tries to determine if a file is a wiggle file or a bigWig file.
     Returns 'wiggle' if the file name ends with .wig, otherwise 'bigwig'
     """
     if fname.endswith(".wig") or fname.endswith(".wiggle"):
@@ -136,32 +131,6 @@ def main(args=None):
                      'scaleFactors': scaleFactors,
                      'pseudocount': args.pseudocount}
 
-    # Preload deepBlue files, which need to then be deleted
-    deepBlueFiles = []
-    for idx, fname in enumerate([args.bigwig1, args.bigwig2]):
-        if db.isDeepBlue(fname):
-            deepBlueFiles.append([fname, idx])
-    if len(deepBlueFiles) > 0:
-        sys.stderr.write("Preloading the following deepBlue files: {}\n".format(",".join([x[0] for x in deepBlueFiles])))
-        foo = db.deepBlue(deepBlueFiles[0][0], url=args.deepBlueURL, userKey=args.userKey)
-        regs = db.makeChromTiles(foo)
-        for x in deepBlueFiles:
-            x.extend([args, regs])
-        if len(deepBlueFiles) > 1 and args.numberOfProcessors > 1:
-            pool = multiprocessing.Pool(args.numberOfProcessors)
-            res = pool.map_async(db.preloadWrapper, deepBlueFiles).get(9999999)
-        else:
-            res = list(map(db.preloadWrapper, deepBlueFiles))
-
-        # substitute the file names with the temp files
-        for (ftuple, r) in zip(deepBlueFiles, res):
-            if ftuple[1] == 0:
-                args.bigwig1 = r
-            else:
-                args.bigwig2 = r
-        deepBlueFiles = [[x[0], x[1]] for x in deepBlueFiles]
-        del regs
-
     writeBedGraph_bam_and_bw.writeBedGraph(
         [(args.bigwig1, getType(args.bigwig1)),
          (args.bigwig2, getType(args.bigwig2))],
@@ -176,17 +145,3 @@ def main(args=None):
         missingDataAsZero=not args.skipNonCoveredRegions,
         extendPairedEnds=False,
         fixedStep=args.fixedStep)
-
-    # Clean up temporary bigWig files, if applicable
-    if not args.deepBlueKeepTemp:
-        for k, v in deepBlueFiles:
-            if v == 0:
-                os.remove(args.bigwig1)
-            else:
-                os.remove(args.bigwig2)
-    else:
-        for k, v in deepBlueFiles:
-            foo = args.bigwig1
-            if v == 1:
-                foo = args.bigwig2
-            print("{} is stored in {}".format(k, foo))