bacterial_genome_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Treponema genome analysis workflow - Grillova *et al.* 2019"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>[Jan Oppelt, PhD](https://orcid.org/0000-0002-3076-4840)  \n",
    "Bioinformatics Core Facility, CEITEC Masaryk University, Brno, CZE  \n",
    "jan.oppelt@gmail.com</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**If you use this analysis or its part in your work we would like to kindly ask you to cite this article as well as the tools used in the section(s) you used in your publication.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## The environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Tested software versions and operation system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The workflow has been tested on Linux machine (Ubuntu 16.04 LTS, 4.4.0-148-generic) with Jupyter notebook running on Python 3.7.1 using Conda 4.6.14, Jupyter notebook 5.7.4 and bash_kernel 0.7.1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Linux BioDA-server 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux\n",
      "No LSB modules are available.\n",
      "Distributor ID:\tUbuntu\n",
      "Description:\tUbuntu 16.04.6 LTS\n",
      "Release:\t16.04\n",
      "Codename:\txenial\n",
      "Python 3.7.1\n",
      "conda 4.6.14\n",
      "5.7.4\n"
     ]
    }
   ],
   "source": [
    "uname -a\n",
    "lsb_release -a\n",
    "#export PATH=/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/bin:$PATH # In case we use different version of Conda than the system-wide installation\n",
    "/opt/install/dir/anaconda/bin/python3 --version\n",
    "/opt/install/dir/anaconda/bin/conda --version\n",
    "/opt/install/dir/anaconda/bin/jupyter notebook --version"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## The analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The Conda environment has been created under Miniconda2 4.6.14 using Python 2.7.16."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, we can activate the environment and check. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) \n",
      "     active environment : treponema\n",
      "    active env location : /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema\n",
      "            shell level : 2\n",
      "       user config file : /mnt/nfs/home/325073/000000-My_Documents/VM-home/.condarc\n",
      " populated config files : /mnt/nfs/home/325073/000000-My_Documents/VM-home/.condarc\n",
      "          conda version : 4.6.14\n",
      "    conda-build version : not installed\n",
      "         python version : 2.7.16.final.0\n",
      "       base environment : /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2  (read only)\n",
      "           channel URLs : https://conda.anaconda.org/conda-forge/linux-64\n",
      "                          https://conda.anaconda.org/conda-forge/noarch\n",
      "                          https://repo.anaconda.com/pkgs/main/linux-64\n",
      "                          https://repo.anaconda.com/pkgs/main/noarch\n",
      "                          https://repo.anaconda.com/pkgs/free/linux-64\n",
      "                          https://repo.anaconda.com/pkgs/free/noarch\n",
      "                          https://repo.anaconda.com/pkgs/r/linux-64\n",
      "                          https://repo.anaconda.com/pkgs/r/noarch\n",
      "                          https://conda.anaconda.org/r/linux-64\n",
      "                          https://conda.anaconda.org/r/noarch\n",
      "                          https://conda.anaconda.org/bioconda/linux-64\n",
      "                          https://conda.anaconda.org/bioconda/noarch\n",
      "          package cache : /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/pkgs\n",
      "                          /mnt/nfs/home/325073/000000-My_Documents/VM-home/.conda/pkgs\n",
      "       envs directories : /mnt/nfs/home/325073/000000-My_Documents/VM-home/.conda/envs\n",
      "                          /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs\n",
      "               platform : linux-64\n",
      "             user-agent : conda/4.6.14 requests/2.21.0 CPython/2.7.16 Linux/4.4.0-148-generic ubuntu/16.04.6 glibc/2.23\n",
      "                UID:GID : 100134:100006\n",
      "             netrc file : None\n",
      "           offline mode : False\n",
      "\n",
      "(treponema) # packages in environment at /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema:\n",
      "#\n",
      "# Name                    Version                   Build  Channel\n",
      "_r-mutex                  1.0.0               anacondar_1  \n",
      "aragorn                   1.2.38               h470a237_2    bioconda\n",
      "asn1crypto                0.24.0                py27_1003    conda-forge\n",
      "atk                       2.32.0               haf93ef1_0    conda-forge\n",
      "augustus                  3.2.3               boost1.60_0    bioconda\n",
      "backports                 1.0                        py_2    conda-forge\n",
      "backports.functools_lru_cache 1.5                        py_1    conda-forge\n",
      "backports_abc             0.5                        py_1    conda-forge\n",
      "bamtools                  2.4.1                         1    bioconda\n",
      "barrnap                   0.8                           1    bioconda\n",
      "bbmap                     38.22                h14c3975_1    bioconda\n",
      "bcftools                  1.4.1                         0    bioconda\n",
      "besst                     2.2.8                      py_2    bioconda\n",
      "binutils_impl_linux-64    2.31.1               h6176602_1  \n",
      "binutils_linux-64         2.31.1               h6176602_7  \n",
      "bioconductor-biobase      2.42.0           r351h14c3975_1    bioconda\n",
      "bioconductor-biocgenerics 0.28.0                   r351_1    bioconda\n",
      "bioconductor-biocparallel 1.16.6           r351h1c2f66e_0    bioconda\n",
      "bioconductor-biostrings   2.50.2           r351h14c3975_0    bioconda\n",
      "bioconductor-delayedarray 0.8.0            r351h14c3975_0    bioconda\n",
      "bioconductor-genomeinfodb 1.18.1                   r351_0    bioconda\n",
      "bioconductor-genomeinfodbdata 1.2.1                    r351_0    bioconda\n",
      "bioconductor-genomicalignments 1.18.1           r351h14c3975_0    bioconda\n",
      "bioconductor-genomicranges 1.34.0           r351h14c3975_0    bioconda\n",
      "bioconductor-iranges      2.16.0           r351h14c3975_0    bioconda\n",
      "bioconductor-noiseq       2.26.1                   r351_0    bioconda\n",
      "bioconductor-rsamtools    1.34.0           r351hf484d3e_0    bioconda\n",
      "bioconductor-rtracklayer  1.42.1           r351h9d9f1b6_1    bioconda\n",
      "bioconductor-s4vectors    0.20.1           r351h14c3975_0    bioconda\n",
      "bioconductor-summarizedexperiment 1.12.0                   r351_0    bioconda\n",
      "bioconductor-xvector      0.22.0           r351h14c3975_0    bioconda\n",
      "bioconductor-zlibbioc     1.28.0           r351h14c3975_0    bioconda\n",
      "blas                      1.0                         mkl  \n",
      "blast                     2.9.0           pl526h979a64d_3    bioconda\n",
      "boost                     1.60.0                   py27_0  \n",
      "boost-cpp                 1.67.0               h3a22d5f_0    conda-forge\n",
      "busco                     3.0.2                    py27_8    bioconda\n",
      "bwa                       0.7.15                        1    bioconda\n",
      "bwidget                   1.9.11                        1  \n",
      "bz2file                   0.98                       py_0    conda-forge\n",
      "bzip2                     1.0.6             h14c3975_1002    conda-forge\n",
      "ca-certificates           2019.6.16            hecc5488_0    conda-forge\n",
      "cairo                     1.16.0            ha4e643d_1000    conda-forge\n",
      "certifi                   2019.6.16                py27_0    conda-forge\n",
      "cffi                      1.12.3           py27h8022711_0    conda-forge\n",
      "chardet                   3.0.4                 py27_1003    conda-forge\n",
      "chrpath                   0.16              h14c3975_1001    conda-forge\n",
      "circos                    0.69.6                        5    bioconda\n",
      "click                     7.0                        py_0    conda-forge\n",
      "clustalw                  2.1                  h6bb024c_4    bioconda\n",
      "colormath                 2.1.1                    py27_1    conda-forge\n",
      "cryptography              2.7              py27h72c5cf5_0    conda-forge\n",
      "curl                      7.64.1               hf8cf82a_0    conda-forge\n",
      "cutadapt                  1.18             py27h14c3975_1    bioconda\n",
      "cycler                    0.10.0                     py_1    conda-forge\n",
      "dbus                      1.13.6               he372182_0    conda-forge\n",
      "decorator                 4.4.0                      py_0    conda-forge\n",
      "entrez-direct             11.0                    pl526_0    bioconda\n",
      "enum34                    1.1.6                 py27_1001    conda-forge\n",
      "expat                     2.2.5             hf484d3e_1002    conda-forge\n",
      "fastqc                    0.11.8                        1    bioconda\n",
      "font-ttf-dejavu-sans-mono 2.37                 h6964260_0  \n",
      "fontconfig                2.13.1            he4413a7_1000    conda-forge\n",
      "freebayes                 0.9.21.26                     0    bioconda\n",
      "freetype                  2.10.0               he983fc9_0    conda-forge\n",
      "functools32               3.2.3.2                    py_3    conda-forge\n",
      "future                    0.17.1                py27_1000    conda-forge\n",
      "futures                   3.2.0                 py27_1000    conda-forge\n",
      "gatk                      3.7                      py27_1    bioconda\n",
      "gcc_impl_linux-64         7.3.0                habb00fd_1    conda-forge\n",
      "gcc_linux-64              7.3.0                h553295d_7    conda-forge\n",
      "gdk-pixbuf                2.36.12           h7a26e22_1003    conda-forge\n",
      "gettext                   0.19.8.1          hc5be6a0_1002    conda-forge\n",
      "gfortran_impl_linux-64    7.3.0                hdf63c60_1  \n",
      "gfortran_linux-64         7.3.0                h553295d_7  \n",
      "giflib                    5.1.9                h516909a_0    conda-forge\n",
      "glib                      2.58.3            h6f030ca_1001    conda-forge\n",
      "glimmerhmm                3.0.4                h2d50403_2    bioconda\n",
      "gmp                       6.1.2             hf484d3e_1000    conda-forge\n",
      "gnuplot                   5.2.7                h213187a_0    conda-forge\n",
      "gnutls                    3.5.19               h2a4e5f8_1    conda-forge\n",
      "gobject-introspection     1.56.1          py27h2da5eee_1002    conda-forge\n",
      "graphite2                 1.3.13            hf484d3e_1000    conda-forge\n",
      "graphviz                  2.40.1               h0dab3d1_0    conda-forge\n",
      "gsl                       2.4                  h14c3975_4  \n",
      "gst-plugins-base          1.14.5               h0935bb2_0    conda-forge\n",
      "gstreamer                 1.14.5               h36ae1b5_0    conda-forge\n",
      "gtk2                      2.24.31           hb68c50a_1001    conda-forge\n",
      "gxx_impl_linux-64         7.3.0                hdf63c60_1    conda-forge\n",
      "gxx_linux-64              7.3.0                h553295d_7    conda-forge\n",
      "harfbuzz                  2.4.0                h37c48d4_1    conda-forge\n",
      "hmmer                     3.2.1                hf484d3e_1    bioconda\n",
      "htslib                    1.9                  h4da6232_3    bioconda\n",
      "icu                       58.2              hf484d3e_1000    conda-forge\n",
      "idna                      2.8                   py27_1000    conda-forge\n",
      "infernal                  1.1.2                h14c3975_2    bioconda\n",
      "intel-openmp              2019.4                      243  \n",
      "ipaddress                 1.0.22                     py_1    conda-forge\n",
      "jinja2                    2.10.1                     py_0    conda-forge\n",
      "joblib                    0.13.2                     py_0    conda-forge\n",
      "jpeg                      9c                h14c3975_1001    conda-forge\n",
      "kiwisolver                1.1.0            py27hc9558a2_0    conda-forge\n",
      "kraken2                   2.0.8_beta      pl526h6bb024c_0    bioconda\n",
      "krb5                      1.16.3            h05b26f9_1001    conda-forge\n",
      "libcurl                   7.64.1               hda55be3_0    conda-forge\n",
      "libdb                     6.1.26                        0    bioconda\n",
      "libdeflate                1.2                  h14c3975_0    bioconda\n",
      "libedit                   3.1.20170329      hf8c457e_1001    conda-forge\n",
      "libffi                    3.2.1             he1b5a44_1006    conda-forge\n",
      "libgcc                    7.2.0                h69d50b8_2    conda-forge\n",
      "libgcc-ng                 9.1.0                hdf63c60_0  \n",
      "libgd                     2.2.5             h0d07dcb_1005    conda-forge\n",
      "libgfortran               3.0.0                         1    conda-forge\n",
      "libgfortran-ng            7.3.0                hdf63c60_0  \n",
      "libiconv                  1.15              h516909a_1005    conda-forge\n",
      "libidn11                  1.34                 h1cef754_0    conda-forge\n",
      "libidn2                   2.1.1                h14c3975_0    conda-forge\n",
      "libpng                    1.6.37               hed695b0_0    conda-forge\n",
      "libssh2                   1.8.2                h22169c7_2    conda-forge\n",
      "libstdcxx-ng              9.1.0                hdf63c60_0  \n",
      "libtiff                   4.0.10            h57b8799_1003    conda-forge\n",
      "libtool                   2.4.6             h14c3975_1002    conda-forge\n",
      "libunistring              0.9.10               h14c3975_0    conda-forge\n",
      "libuuid                   2.32.1            h14c3975_1000    conda-forge\n",
      "libwebp                   1.0.2                h576950b_1    conda-forge\n",
      "libxcb                    1.13              h14c3975_1002    conda-forge\n",
      "libxml2                   2.9.9                h13577e0_0    conda-forge\n",
      "libxslt                   1.1.32            hae48121_1003    conda-forge\n",
      "lp_solve                  5.5.2.5           h14c3975_1001    conda-forge\n",
      "lz4-c                     1.8.3             he1b5a44_1001    conda-forge\n",
      "lzstring                  1.0.4                   py_1001    conda-forge\n",
      "make                      4.2.1             h14c3975_2004    conda-forge\n",
      "markdown                  2.6.11                     py_0    conda-forge\n",
      "markupsafe                1.1.1            py27h14c3975_0    conda-forge\n",
      "mathstats                 0.2.6.5                    py_0    bioconda\n",
      "matplotlib                2.2.3            py27hb69df0a_0  \n",
      "matplotlib-base           2.2.3            py27h60b886d_1    conda-forge\n",
      "metis                     5.1.0             hf484d3e_1003    conda-forge\n",
      "minced                    0.4.0                         0    bioconda\n",
      "mkl                       2019.4                      243  \n",
      "mkl_fft                   1.0.13           py27h516909a_1    conda-forge\n",
      "mkl_random                1.0.4            py27hf2d7682_0    conda-forge\n",
      "multiqc                   1.3                      py27_2    bioconda\n",
      "mummer                    3.23                    pl526_8    bioconda\n",
      "ncurses                   6.1               hf484d3e_1002    conda-forge\n",
      "nettle                    3.3                           0    conda-forge\n",
      "networkx                  1.9.1                    py27_0  \n",
      "numpy                     1.16.4           py27h7e9f1db_0  \n",
      "numpy-base                1.16.4           py27hde5b4d6_0  \n",
      "openblas                  0.2.20                        8    conda-forge\n",
      "openjdk                   8.0.192           h14c3975_1003    conda-forge\n",
      "openssl                   1.1.1b               h14c3975_1    conda-forge\n",
      "paml                      4.9                  h14c3975_4    bioconda\n",
      "pango                     1.40.14           he7ab937_1005    conda-forge\n",
      "parallel                  20190522                      0    conda-forge\n",
      "pcre                      8.43                 he6710b0_0  \n",
      "perl                      5.26.2            h516909a_1006    conda-forge\n",
      "perl-aceperl              1.92                    pl526_2    bioconda\n",
      "perl-algorithm-diff       1.1903                  pl526_2    bioconda\n",
      "perl-algorithm-munkres    0.08                    pl526_1    bioconda\n",
      "perl-apache-test          1.40                    pl526_1    bioconda\n",
      "perl-app-cpanminus        1.7044                  pl526_1    bioconda\n",
      "perl-appconfig            1.71                    pl526_1    bioconda\n",
      "perl-archive-tar          2.32                    pl526_0    bioconda\n",
      "perl-array-compare        3.0.1                   pl526_1    bioconda\n",
      "perl-autoloader           5.74                    pl526_2    bioconda\n",
      "perl-base                 2.23                    pl526_1    bioconda\n",
      "perl-bio-asn1-entrezgene  1.73                    pl526_0    bioconda\n",
      "perl-bio-coordinate       1.007001                pl526_0    bioconda\n",
      "perl-bio-featureio        1.6.905                 pl526_1    bioconda\n",
      "perl-bio-phylo            0.58                    pl526_1    bioconda\n",
      "perl-bio-samtools         1.43            pl526h1341992_1    bioconda\n",
      "perl-bio-tools-phylo-paml 1.7.3                   pl526_0    bioconda\n",
      "perl-bio-tools-run-alignment-clustalw 1.7.4                   pl526_0    bioconda\n",
      "perl-bio-tools-run-alignment-tcoffee 1.7.4                   pl526_1    bioconda\n",
      "perl-bioperl              1.7.2                  pl526_10    bioconda\n",
      "perl-bioperl-core         1.007002                pl526_1    bioconda\n",
      "perl-bioperl-run          1.007002                pl526_3    bioconda\n",
      "perl-business-isbn        3.004                   pl526_0    bioconda\n",
      "perl-business-isbn-data   20140910.003            pl526_0    bioconda\n",
      "perl-cache-cache          1.08                    pl526_0    bioconda\n",
      "perl-capture-tiny         0.48                    pl526_0    bioconda\n",
      "perl-carp                 1.38                    pl526_3    bioconda\n",
      "perl-cgi                  4.44            pl526h14c3975_1    bioconda\n",
      "perl-class-data-inheritable 0.08                    pl526_1    bioconda\n",
      "perl-class-inspector      1.34                    pl526_0    bioconda\n",
      "perl-class-load           0.25                    pl526_0    bioconda\n",
      "perl-class-load-xs        0.10            pl526h6bb024c_2    bioconda\n",
      "perl-class-method-modifiers 2.12                    pl526_0    bioconda\n",
      "perl-clone                0.41            pl526h14c3975_1    bioconda\n",
      "perl-common-sense         3.74                    pl526_2    bioconda\n",
      "perl-compress-raw-bzip2   2.086           pl526hf484d3e_0    bioconda\n",
      "perl-compress-raw-zlib    2.086           pl526h6bb024c_1    bioconda\n",
      "perl-config-general       2.63                    pl526_0    bioconda\n",
      "perl-constant             1.33                    pl526_1    bioconda\n",
      "perl-convert-binary-c     0.78            pl526h6bb024c_3    bioconda\n",
      "perl-convert-binhex       1.125                   pl526_1    bioconda\n",
      "perl-cpan-meta            2.150010                pl526_0    bioconda\n",
      "perl-cpan-meta-requirements 2.140                   pl526_0    bioconda\n",
      "perl-cpan-meta-yaml       0.018                   pl526_0    bioconda\n",
      "perl-crypt-rc4            2.02                    pl526_1    bioconda\n",
      "perl-data-dumper          2.173                   pl526_0    bioconda\n",
      "perl-data-optlist         0.110                   pl526_2    bioconda\n",
      "perl-data-stag            0.14                    pl526_1    bioconda\n",
      "perl-date-format          2.30                    pl526_2    bioconda\n",
      "perl-db-file              1.852           pl526h14c3975_0    bioconda\n",
      "perl-dbd-sqlite           1.62            pl526h14c3975_1    bioconda\n",
      "perl-dbi                  1.642                   pl526_0    bioconda\n",
      "perl-devel-globaldestruction 0.14                    pl526_0    bioconda\n",
      "perl-devel-overloadinfo   0.005                   pl526_0    bioconda\n",
      "perl-devel-stacktrace     2.04                    pl526_0    bioconda\n",
      "perl-digest-hmac          1.03                    pl526_3    bioconda\n",
      "perl-digest-md5           2.55                    pl526_0    bioconda\n",
      "perl-digest-perl-md5      1.9                     pl526_1    bioconda\n",
      "perl-digest-sha1          2.13            pl526h6bb024c_1    bioconda\n",
      "perl-dist-checkconflicts  0.11                    pl526_2    bioconda\n",
      "perl-dynaloader           1.25                    pl526_1    bioconda\n",
      "perl-email-date-format    1.005                   pl526_2    bioconda\n",
      "perl-encode               2.88                    pl526_1    bioconda\n",
      "perl-encode-locale        1.05                    pl526_6    bioconda\n",
      "perl-error                0.17027                 pl526_1    bioconda\n",
      "perl-eval-closure         0.14            pl526h6bb024c_4    bioconda\n",
      "perl-exception-class      1.44                    pl526_0    bioconda\n",
      "perl-exporter             5.72                    pl526_1    bioconda\n",
      "perl-exporter-tiny        1.002001                pl526_0    bioconda\n",
      "perl-extutils-cbuilder    0.280230                pl526_1    bioconda\n",
      "perl-extutils-makemaker   7.36                    pl526_1    bioconda\n",
      "perl-extutils-manifest    1.72                    pl526_0    bioconda\n",
      "perl-extutils-parsexs     3.35                    pl526_0    bioconda\n",
      "perl-file-listing         6.04                    pl526_1    bioconda\n",
      "perl-file-path            2.16                    pl526_0    bioconda\n",
      "perl-file-slurp-tiny      0.004                   pl526_1    bioconda\n",
      "perl-file-sort            1.01                    pl526_2    bioconda\n",
      "perl-file-temp            0.2304                  pl526_2    bioconda\n",
      "perl-file-which           1.23                    pl526_0    bioconda\n",
      "perl-font-afm             1.20                    pl526_2    bioconda\n",
      "perl-font-ttf             1.06                    pl526_0    bioconda\n",
      "perl-gd                   2.71            pl526he860b03_0    bioconda\n",
      "perl-getopt-long          2.50                    pl526_1    bioconda\n",
      "perl-graph                0.9704                  pl526_1    bioconda\n",
      "perl-graphviz             2.24            pl526h734ff71_0    bioconda\n",
      "perl-html-element-extended 1.18                    pl526_1    bioconda\n",
      "perl-html-entities-numbered 0.04                    pl526_1    bioconda\n",
      "perl-html-formatter       2.16                    pl526_0    bioconda\n",
      "perl-html-parser          3.72            pl526h6bb024c_5    bioconda\n",
      "perl-html-tableextract    2.13                    pl526_2    bioconda\n",
      "perl-html-tagset          3.20                    pl526_3    bioconda\n",
      "perl-html-tidy            1.60                    pl526_0    bioconda\n",
      "perl-html-tree            5.07                    pl526_1    bioconda\n",
      "perl-html-treebuilder-xpath 0.14                    pl526_1    bioconda\n",
      "perl-http-cookies         6.04                    pl526_0    bioconda\n",
      "perl-http-daemon          6.01                    pl526_1    bioconda\n",
      "perl-http-date            6.02                    pl526_3    bioconda\n",
      "perl-http-message         6.18                    pl526_0    bioconda\n",
      "perl-http-negotiate       6.01                    pl526_3    bioconda\n",
      "perl-image-info           1.38                    pl526_1    bioconda\n",
      "perl-image-size           3.300                   pl526_2    bioconda\n",
      "perl-io-compress          2.086           pl526hf484d3e_0    bioconda\n",
      "perl-io-html              1.001                   pl526_2    bioconda\n",
      "perl-io-sessiondata       1.03                    pl526_1    bioconda\n",
      "perl-io-socket-ssl        2.066                   pl526_0    bioconda\n",
      "perl-io-string            1.08                    pl526_3    bioconda\n",
      "perl-io-stringy           2.111                   pl526_1    bioconda\n",
      "perl-io-tty               1.12                    pl526_1    bioconda\n",
      "perl-io-zlib              1.10                    pl526_2    bioconda\n",
      "perl-ipc-cmd              1.02                    pl526_0    bioconda\n",
      "perl-ipc-run              20180523.0              pl526_0    bioconda\n",
      "perl-ipc-sharelite        0.17            pl526h6bb024c_1    bioconda\n",
      "perl-jcode                2.07                    pl526_2    bioconda\n",
      "perl-json                 4.02                    pl526_0    bioconda\n",
      "perl-json-pp              4.02                    pl526_0    bioconda\n",
      "perl-json-xs              2.34            pl526h6bb024c_3    bioconda\n",
      "perl-lib                  0.63                    pl526_1    bioconda\n",
      "perl-libwww-perl          6.39                    pl526_0    bioconda\n",
      "perl-libxml-perl          0.08                    pl526_2    bioconda\n",
      "perl-list-moreutils       0.428                   pl526_1    bioconda\n",
      "perl-list-moreutils-xs    0.428                   pl526_0    bioconda\n",
      "perl-locale-maketext-simple 0.21                    pl526_2    bioconda\n",
      "perl-lwp-mediatypes       6.04                    pl526_0    bioconda\n",
      "perl-lwp-protocol-https   6.07                    pl526_4    bioconda\n",
      "perl-lwp-simple           6.15            pl526h470a237_4    bioconda\n",
      "perl-mailtools            2.21                    pl526_0    bioconda\n",
      "perl-math-bezier          0.01                    pl526_1    bioconda\n",
      "perl-math-cdf             0.1             pl526h14c3975_5    bioconda\n",
      "perl-math-derivative      1.01                    pl526_0    bioconda\n",
      "perl-math-random          0.72            pl526h14c3975_2    bioconda\n",
      "perl-math-round           0.07                    pl526_1    bioconda\n",
      "perl-math-spline          0.02                    pl526_2    bioconda\n",
      "perl-math-vecstat         0.08                    pl526_1    bioconda\n",
      "perl-mime-base64          3.15                    pl526_1    bioconda\n",
      "perl-mime-lite            3.030                   pl526_1    bioconda\n",
      "perl-mime-tools           5.508                   pl526_1    bioconda\n",
      "perl-mime-types           2.17                    pl526_0    bioconda\n",
      "perl-mldbm                2.05                    pl526_1    bioconda\n",
      "perl-module-build         0.4224                  pl526_3    bioconda\n",
      "perl-module-corelist      5.20190524              pl526_0    bioconda\n",
      "perl-module-implementation 0.09                    pl526_2    bioconda\n",
      "perl-module-load          0.32                    pl526_1    bioconda\n",
      "perl-module-load-conditional 0.68                    pl526_2    bioconda\n",
      "perl-module-metadata      1.000036                pl526_0    bioconda\n",
      "perl-module-runtime       0.016                   pl526_1    bioconda\n",
      "perl-module-runtime-conflicts 0.003                   pl526_0    bioconda\n",
      "perl-moo                  2.003004                pl526_0    bioconda\n",
      "perl-moose                2.2011          pl526hf484d3e_1    bioconda\n",
      "perl-mozilla-ca           20180117                pl526_1    bioconda\n",
      "perl-mro-compat           0.13                    pl526_0    bioconda\n",
      "perl-net-http             6.19                    pl526_0    bioconda\n",
      "perl-net-ssleay           1.88            pl526h90d6eec_0    bioconda\n",
      "perl-ntlm                 1.09                    pl526_4    bioconda\n",
      "perl-number-format        1.75                    pl526_3    bioconda\n",
      "perl-ole-storage_lite     0.19                    pl526_3    bioconda\n",
      "perl-package-deprecationmanager 0.17                    pl526_0    bioconda\n",
      "perl-package-stash        0.38            pl526hf484d3e_1    bioconda\n",
      "perl-package-stash-xs     0.28            pl526hf484d3e_1    bioconda\n",
      "perl-params-check         0.38                    pl526_1    bioconda\n",
      "perl-params-util          1.07            pl526h6bb024c_4    bioconda\n",
      "perl-params-validate      1.29            pl526h14c3975_1    bioconda\n",
      "perl-parent               0.236                   pl526_1    bioconda\n",
      "perl-parse-recdescent     1.967015                pl526_0    bioconda\n",
      "perl-pathtools            3.75            pl526h14c3975_1    bioconda\n",
      "perl-pdf-api2             2.033                   pl526_0    bioconda\n",
      "perl-perl-ostype          1.010                   pl526_1    bioconda\n",
      "perl-pod-escapes          1.07                    pl526_1    bioconda\n",
      "perl-pod-usage            1.69                    pl526_1    bioconda\n",
      "perl-postscript           0.06                    pl526_2    bioconda\n",
      "perl-readonly             2.05                    pl526_0    bioconda\n",
      "perl-regexp-common        2017060201              pl526_0    bioconda\n",
      "perl-role-tiny            2.000006                pl526_0    bioconda\n",
      "perl-scalar-list-utils    1.50            pl526h14c3975_0    bioconda\n",
      "perl-set-intspan          1.19                    pl526_1    bioconda\n",
      "perl-set-scalar           1.29                    pl526_2    bioconda\n",
      "perl-soap-lite            1.19                    pl526_1    bioconda\n",
      "perl-socket               2.027                   pl526_1    bioconda\n",
      "perl-sort-naturally       1.03                    pl526_2    bioconda\n",
      "perl-spreadsheet-parseexcel 0.65                    pl526_2    bioconda\n",
      "perl-spreadsheet-writeexcel 2.40                    pl526_2    bioconda\n",
      "perl-statistics-basic     1.6611                  pl526_2    bioconda\n",
      "perl-statistics-descriptive 3.0702                  pl526_0    bioconda\n",
      "perl-storable             3.15            pl526h14c3975_0    bioconda\n",
      "perl-sub-exporter         0.987                   pl526_2    bioconda\n",
      "perl-sub-exporter-progressive 0.001013                pl526_0    bioconda\n",
      "perl-sub-identify         0.14            pl526h14c3975_0    bioconda\n",
      "perl-sub-install          0.928                   pl526_2    bioconda\n",
      "perl-sub-name             0.21                    pl526_1    bioconda\n",
      "perl-sub-quote            2.006003                pl526_1    bioconda\n",
      "perl-sub-uplevel          0.2800          pl526h14c3975_2    bioconda\n",
      "perl-svg                  2.84                    pl526_0    bioconda\n",
      "perl-svg-graph            0.02                    pl526_3    bioconda\n",
      "perl-task-weaken          1.06                    pl526_0    bioconda\n",
      "perl-template-toolkit     2.26                    pl526_1    bioconda\n",
      "perl-test                 1.26                    pl526_1    bioconda\n",
      "perl-test-deep            1.128                   pl526_1    bioconda\n",
      "perl-test-differences     0.67                    pl526_0    bioconda\n",
      "perl-test-exception       0.43                    pl526_2    bioconda\n",
      "perl-test-harness         3.42                    pl526_0    bioconda\n",
      "perl-test-leaktrace       0.16            pl526h14c3975_2    bioconda\n",
      "perl-test-most            0.35                    pl526_0    bioconda\n",
      "perl-test-requiresinternet 0.05                    pl526_0    bioconda\n",
      "perl-test-warn            0.36                    pl526_1    bioconda\n",
      "perl-text-abbrev          1.02                    pl526_0    bioconda\n",
      "perl-text-diff            1.45                    pl526_0    bioconda\n",
      "perl-text-format          0.59                    pl526_2    bioconda\n",
      "perl-text-parsewords      3.30                    pl526_0    bioconda\n",
      "perl-tie-ixhash           1.23                    pl526_2    bioconda\n",
      "perl-time-hires           1.9760          pl526h14c3975_1    bioconda\n",
      "perl-time-local           1.28                    pl526_1    bioconda\n",
      "perl-timedate             2.30                    pl526_1    bioconda\n",
      "perl-tree-dag_node        1.31                    pl526_0    bioconda\n",
      "perl-try-tiny             0.30                    pl526_1    bioconda\n",
      "perl-type-tiny            1.004004                pl526_0    bioconda\n",
      "perl-types-serialiser     1.0                     pl526_2    bioconda\n",
      "perl-unicode-map          0.112           pl526h6bb024c_3    bioconda\n",
      "perl-uri                  1.76                    pl526_0    bioconda\n",
      "perl-vcftools-vcf         0.1.15                  pl526_2    bioconda\n",
      "perl-version              0.9924                  pl526_0    bioconda\n",
      "perl-www-robotrules       6.02                    pl526_3    bioconda\n",
      "perl-xml-dom              1.46                    pl526_0    bioconda\n",
      "perl-xml-dom-xpath        0.14                    pl526_1    bioconda\n",
      "perl-xml-filter-buffertext 1.01                    pl526_2    bioconda\n",
      "perl-xml-libxml           2.0132          pl526hbc14f71_0    bioconda\n",
      "perl-xml-libxslt          1.94                    pl526_1    bioconda\n",
      "perl-xml-namespacesupport 1.12                    pl526_0    bioconda\n",
      "perl-xml-parser           2.44            pl526h4e0c4b3_7    bioconda\n",
      "perl-xml-regexp           0.04                    pl526_2    bioconda\n",
      "perl-xml-sax              1.02                    pl526_0    bioconda\n",
      "perl-xml-sax-base         1.09                    pl526_0    bioconda\n",
      "perl-xml-sax-expat        0.51                    pl526_3    bioconda\n",
      "perl-xml-sax-writer       0.57                    pl526_0    bioconda\n",
      "perl-xml-simple           2.25                    pl526_1    bioconda\n",
      "perl-xml-twig             3.52                    pl526_2    bioconda\n",
      "perl-xml-writer           0.625                   pl526_2    bioconda\n",
      "perl-xml-xpath            1.44                    pl526_0    bioconda\n",
      "perl-xml-xpathengine      0.14                    pl526_2    bioconda\n",
      "perl-xsloader             0.24                    pl526_0    bioconda\n",
      "perl-yaml                 1.29                    pl526_0    bioconda\n",
      "picard                    2.9.2                         2    bioconda\n",
      "pigz                      2.3.4                         0    conda-forge\n",
      "pip                       19.1.1                   py27_0    conda-forge\n",
      "pixman                    0.34.0            h14c3975_1003    conda-forge\n",
      "popt                      1.16                          1    bioconda\n",
      "prodigal                  2.6.3                         1    bioconda\n",
      "prokka                    1.13.7                  pl526_0    bioconda\n",
      "pthread-stubs             0.4               h14c3975_1001    conda-forge\n",
      "pycparser                 2.19                     py27_1    conda-forge\n",
      "pyopenssl                 19.0.0                   py27_0    conda-forge\n",
      "pyparsing                 2.4.0                      py_0    conda-forge\n",
      "pyqt                      5.9.2            py27hcca6a23_0    conda-forge\n",
      "pysam                     0.11.2.2                 py27_1    bioconda\n",
      "pysocks                   1.7.0                    py27_0    conda-forge\n",
      "python                    2.7.15            h721da81_1008    conda-forge\n",
      "python-dateutil           2.8.0                      py_0    conda-forge\n",
      "pytz                      2019.1                     py_0    conda-forge\n",
      "pyyaml                    5.1.1            py27h516909a_0    conda-forge\n",
      "qt                        5.9.7                h52cfd70_2    conda-forge\n",
      "qualimap                  2.2.2b                        1    bioconda\n",
      "quast                     5.0.2           py27pl526ha92aebf_0    bioconda\n",
      "r                         3.5.1                 r351_1000    conda-forge\n",
      "r-assertthat              0.2.1            r351h6115d3f_0    conda-forge\n",
      "r-backports               1.1.4            r351hcdcec82_0    conda-forge\n",
      "r-base                    3.5.1             h271c98b_1006    conda-forge\n",
      "r-bh                      1.69.0_1         r351h6115d3f_0    conda-forge\n",
      "r-bitops                  1.0_6           r351h96ca727_1002    conda-forge\n",
      "r-boot                    1.3_22                   r351_0    conda-forge\n",
      "r-class                   7.3_15          r351h96ca727_1000    conda-forge\n",
      "r-cli                     1.1.0            r351h6115d3f_0    conda-forge\n",
      "r-cluster                 2.0.9            r351h9bbef5b_0    conda-forge\n",
      "r-codetools               0.2_16          r351h6115d3f_1000    conda-forge\n",
      "r-colorspace              1.4_1            r351hcdcec82_0    conda-forge\n",
      "r-crayon                  1.3.4           r351h6115d3f_1001    conda-forge\n",
      "r-digest                  0.6.19           r351h0357c0b_0    conda-forge\n",
      "r-fansi                   0.4.0           r351h96ca727_1000    conda-forge\n",
      "r-foreign                 0.8_71          r351h96ca727_1002    conda-forge\n",
      "r-formatr                 1.7               r35h6115d3f_0    conda-forge\n",
      "r-futile.logger           1.4.3           r351h6115d3f_1001    conda-forge\n",
      "r-futile.options          1.0.1           r351h6115d3f_1000    conda-forge\n",
      "r-getopt                  1.20.3                   r351_0    conda-forge\n",
      "r-ggplot2                 3.2.0             r35h6115d3f_0    conda-forge\n",
      "r-glue                    1.3.1            r351hcdcec82_0    conda-forge\n",
      "r-gtable                  0.3.0            r351h6115d3f_0    conda-forge\n",
      "r-kernsmooth              2.23_15         r351ha65eedd_1002    conda-forge\n",
      "r-labeling                0.3             r351h6115d3f_1001    conda-forge\n",
      "r-lambda.r                1.2.3           r351h6115d3f_1000    conda-forge\n",
      "r-lattice                 0.20_38         r351h96ca727_1000    conda-forge\n",
      "r-lazyeval                0.2.2            r351hcdcec82_0    conda-forge\n",
      "r-magrittr                1.5             r351h6115d3f_1001    conda-forge\n",
      "r-mass                    7.3_51.4         r351hcdcec82_0    conda-forge\n",
      "r-matrix                  1.2_17           r351hcdcec82_0    conda-forge\n",
      "r-matrixstats             0.54.0          r351h96ca727_1000    conda-forge\n",
      "r-mgcv                    1.8_28           r351hcdcec82_0    conda-forge\n",
      "r-munsell                 0.5.0           r351h6115d3f_1001    conda-forge\n",
      "r-nlme                    3.1_140          r351h9bbef5b_0    conda-forge\n",
      "r-nnet                    7.3_12          r351h96ca727_1002    conda-forge\n",
      "r-optparse                1.6.1            r351h6115d3f_0    conda-forge\n",
      "r-pillar                  1.4.1                h6115d3f_0    conda-forge\n",
      "r-pkgconfig               2.0.2           r351h6115d3f_1001    conda-forge\n",
      "r-plyr                    1.8.4           r351h29659fb_1002    conda-forge\n",
      "r-r6                      2.4.0            r351h6115d3f_0    conda-forge\n",
      "r-rcolorbrewer            1.1_2           r351h6115d3f_1001    conda-forge\n",
      "r-rcpp                    1.0.1            r351h0357c0b_0    conda-forge\n",
      "r-rcurl                   1.95_4.12        r351hcdcec82_0    conda-forge\n",
      "r-recommended             3.5.1                 r351_1001    conda-forge\n",
      "r-reshape2                1.4.3           r351h29659fb_1003    conda-forge\n",
      "r-rlang                   0.3.4            r351hcdcec82_0    conda-forge\n",
      "r-rpart                   4.1_15           r351hcdcec82_0    conda-forge\n",
      "r-scales                  1.0.0           r351h29659fb_1001    conda-forge\n",
      "r-snow                    0.4_3           r351h6115d3f_1000    conda-forge\n",
      "r-spatial                 7.3_11          r351h96ca727_1002    conda-forge\n",
      "r-stringi                 1.4.3            r351h0357c0b_0    conda-forge\n",
      "r-stringr                 1.4.0            r351h6115d3f_0    conda-forge\n",
      "r-survival                2.44_1.1         r351hcdcec82_0    conda-forge\n",
      "r-tibble                  2.1.3             r35hcdcec82_0    conda-forge\n",
      "r-utf8                    1.1.4           r351h96ca727_1000    conda-forge\n",
      "r-vctrs                   0.1.0           r351h96ca727_1000    conda-forge\n",
      "r-viridislite             0.3.0           r351h6115d3f_1001    conda-forge\n",
      "r-withr                   2.1.2           r351h6115d3f_1000    conda-forge\n",
      "r-xml                     3.98_1.20         r35hcdcec82_0    conda-forge\n",
      "r-zeallot                 0.1.0           r351h6115d3f_1000    conda-forge\n",
      "readline                  7.0               hf8c457e_1001    conda-forge\n",
      "reaper                    16.098               ha92aebf_2    bioconda\n",
      "requests                  2.22.0                   py27_0    conda-forge\n",
      "rsync                     3.1.3             h84994c4_1002    conda-forge\n",
      "samtools                  1.4.1                         0    bioconda\n",
      "scipy                     1.2.1            py27h7c811a0_0  \n",
      "seqtk                     1.3                  h84994c4_1    bioconda\n",
      "setuptools                41.0.1                   py27_0    conda-forge\n",
      "simplejson                3.8.1                    py27_0    bioconda\n",
      "singledispatch            3.4.0.3               py27_1000    conda-forge\n",
      "sip                       4.19.8          py27hf484d3e_1000    conda-forge\n",
      "six                       1.12.0                py27_1000    conda-forge\n",
      "snpeff                    4.3.1t                        2    bioconda\n",
      "spades                    3.13.0                        0    bioconda\n",
      "spectra                   0.0.7                    py27_0    conda-forge\n",
      "sqlite                    3.28.0               h8b20d00_0    conda-forge\n",
      "subprocess32              3.5.4            py27h516909a_0    conda-forge\n",
      "suitesparse               5.2.0                h9e4a6bb_0  \n",
      "t_coffee                  11.0.8           py27hea885bf_8    bioconda\n",
      "tbb                       2019.7               hc9558a2_0    conda-forge\n",
      "tbl2asn                   25.6                          3    bioconda\n",
      "tidyp                     1.04                          1    bioconda\n",
      "tk                        8.6.9             hed695b0_1002    conda-forge\n",
      "tktable                   2.10                 h14c3975_0  \n",
      "tornado                   5.1.1           py27h14c3975_1000    conda-forge\n",
      "urllib3                   1.24.3                   py27_0    conda-forge\n",
      "vcflib                    1.0.0_rc2            h56106d0_2    bioconda\n",
      "vcftools                  0.1.15               he941832_2    bioconda\n",
      "virtualenv                16.0.0                   py27_0    anaconda\n",
      "wget                      1.20.1               h90d6eec_0    conda-forge\n",
      "wheel                     0.33.4                   py27_0    conda-forge\n",
      "xopen                     0.7.0                      py_0    bioconda\n",
      "xorg-kbproto              1.0.7             h14c3975_1002    conda-forge\n",
      "xorg-libice               1.0.9             h516909a_1004    conda-forge\n",
      "xorg-libsm                1.2.3             h84519dc_1000    conda-forge\n",
      "xorg-libx11               1.6.7             h14c3975_1000    conda-forge\n",
      "xorg-libxau               1.0.9                h14c3975_0    conda-forge\n",
      "xorg-libxdmcp             1.1.3                h516909a_0    conda-forge\n",
      "xorg-libxext              1.3.4                h516909a_0    conda-forge\n",
      "xorg-libxpm               3.5.12            h14c3975_1002    conda-forge\n",
      "xorg-libxrender           0.9.10            h516909a_1002    conda-forge\n",
      "xorg-libxt                1.1.5             h516909a_1003    conda-forge\n",
      "xorg-renderproto          0.11.1            h14c3975_1002    conda-forge\n",
      "xorg-xextproto            7.3.0             h14c3975_1002    conda-forge\n",
      "xorg-xproto               7.0.31            h14c3975_1007    conda-forge\n",
      "xz                        5.2.4             h14c3975_1001    conda-forge\n",
      "yaml                      0.1.7             h14c3975_1001    conda-forge\n",
      "zlib                      1.2.11            h14c3975_1004    conda-forge\n",
      "zstd                      1.4.0                h3b9ef0a_0    conda-forge\n",
      "(treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "conda activate treponema\n",
    "conda info\n",
    "conda list \n",
    "#conda env export > treponema_mod.yml # In case we modified the environment and we want to export it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Running the workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We strongly recommend running the workflow manually for the first time to see what are the individual steps and variables. After you are confident about the analysis and you set the variables correctly you can run the whole analysis at once, export it as a bash script or however it is the most suitable for you. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Input variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, we have to set a few variables we will be using throughout the analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "INPUT_DIR=\"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/raw\" # Directory with the raw paired input files in fastq.gz; first pair has to be named xxxR1.fastq.gz and the second xxxR2.fastq.gz otherwise you would have to change the input suffixes for the individual steps\n",
    "OUTPUT_DIR=\"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results\" # Directory where we want to save the results\n",
    "\n",
    "REFERENCES=\"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references\" # Directory with the reference genomes and databases\n",
    "\n",
    "ADAPTER_R1=\"CTGTCTCTTATACACATCTC\" # R1 3' adapeter; first 20 bp should be enough\n",
    "ADAPTER_R2=\"CTGTCTCTTATACACATCTG\" # R2 3' adapter; first 20 bp should be enough\n",
    "\n",
    "#REFERENCE=\"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa.gz\" # Bacteria reference genome; in this analysis - ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.fna.gz\n",
    "#HOST_GENOME=\"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCF_000001405.36_GRCh38.p10_genomic.fna.gz\" # Host genome reference sequence; for human - ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.36_GRCh38.p10/GCF_000001405.36_GRCh38.p10_genomic.fna.gz\n",
    "\n",
    "THREADS=12 # Number of threads we will use in the analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we don't have the target reference genome and the host genome, we have to download them first. In this example, we will use *SS14* reference (GenBank CP004011.1) as the reference genome and human (GRCh38.p10) as the host genome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) --2019-07-01 16:49:47--  ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.fna.gz\n",
      "           => ‘GCA_000410555.1_ASM41055v1_genomic.fna.gz’\n",
      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 130.14.250.13, 2607:f220:41e:250::12\n",
      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|130.14.250.13|:21... connected.\n",
      "Logging in as anonymous ... Logged in!\n",
      "==> SYST ... done.    ==> PWD ... done.\n",
      "==> TYPE I ... done.  ==> CWD (1) /genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1 ... done.\n",
      "==> SIZE GCA_000410555.1_ASM41055v1_genomic.fna.gz ... 340895\n",
      "==> PASV ... done.    ==> RETR GCA_000410555.1_ASM41055v1_genomic.fna.gz ... done.\n",
      "Length: 340895 (333K) (unauthoritative)\n",
      "\n",
      "GCA_000410555.1_ASM 100%[===================>] 332.91K   824KB/s    in 0.4s    \n",
      "\n",
      "2019-07-01 16:49:49 (824 KB/s) - ‘GCA_000410555.1_ASM41055v1_genomic.fna.gz’ saved [340895]\n",
      "\n",
      "(treponema) --2019-07-01 16:49:49--  ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.36_GRCh38.p10/GCF_000001405.36_GRCh38.p10_genomic.fna.gz\n",
      "           => ‘GCF_000001405.36_GRCh38.p10_genomic.fna.gz’\n",
      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 130.14.250.13, 2607:f220:41e:250::12\n",
      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|130.14.250.13|:21... connected.\n",
      "Logging in as anonymous ... Logged in!\n",
      "==> SYST ... done.    ==> PWD ... done.\n",
      "==> TYPE I ... done.  ==> CWD (1) /genomes/all/GCF/000/001/405/GCF_000001405.36_GRCh38.p10 ... done.\n",
      "==> SIZE GCF_000001405.36_GRCh38.p10_genomic.fna.gz ... 955587620\n",
      "==> PASV ... done.    ==> RETR GCF_000001405.36_GRCh38.p10_genomic.fna.gz ... done.\n",
      "Length: 955587620 (911M) (unauthoritative)\n",
      "\n",
      "GCF_000001405.36_GR 100%[===================>] 911.32M  29.2MB/s    in 35s     \n",
      "\n",
      "2019-07-01 16:50:25 (26.4 MB/s) - ‘GCF_000001405.36_GRCh38.p10_genomic.fna.gz’ saved [955587620]\n",
      "\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $REFERENCES\n",
    "cd $REFERENCES/\n",
    "\n",
    "wget ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.fna.gz\n",
    "wget ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.36_GRCh38.p10/GCF_000001405.36_GRCh38.p10_genomic.fna.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We like to apply some \"fixes\" to the name of the reference genome to avoid issues with the subsequent steps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "zcat GCA_000410555.1_ASM41055v1_genomic.fna.gz | sed 's/,//g' | sed 's/ /_/g' | sed 's/\\./_/g' | gzip -c > SS14.fa.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we didn't specify the reference sequences before, we can do it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "REFERENCE=$REFERENCES/SS14.fa.gz # Bacteria reference genome; in this analysis - ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.fna.gz\n",
    "HOST_GENOME=$REFERENCES/GCF_000001405.36_GRCh38.p10_genomic.fna.gz # Host genome reference sequence; for human - ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.36_GRCh38.p10/GCF_000001405.36_GRCh38.p10_genomic.fna.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Initial quality check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It is always a good idea to run an initial quality check on your data. Using FastQC and MultiQC we can get the general quality overview of our input data. Outputs are described [here](#Raw-reads-quality-check-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) Doing initial QC on samples CW56_S1_R1_001.fastq.gz\n",
      "CW56_S1_R2_001.fastq.gz\n",
      "(treponema) (treponema) Started analysis of CW56_S1_R1_001.fastq.gz\n",
      "Started analysis of CW56_S1_R2_001.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R2_001.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R1_001.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R2_001.fastq.gz\n",
      "Analysis complete for CW56_S1_R1_001.fastq.gz\n",
      "Analysis complete for CW56_S1_R2_001.fastq.gz\n",
      "(treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  configs = yaml.load(f)\n",
      "/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  sp = yaml.load(f)\n",
      "[WARNING]         multiqc : MultiQC Version v1.7 now available!\n",
      "[INFO   ]         multiqc : This is MultiQC v1.3\n",
      "[INFO   ]         multiqc : Template    : default\n",
      "[INFO   ]         multiqc : Searching '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/fastqc/raw/'\n",
      "[INFO   ]          fastqc : Found 2 reports\n",
      "[INFO   ]         multiqc : Compressing plot data\n",
      "[INFO   ]         multiqc : Report      : ../results/qc/fastqc/raw/multiqc_report.html\n",
      "[INFO   ]         multiqc : Data        : ../results/qc/fastqc/raw/multiqc_data\n",
      "[INFO   ]         multiqc : MultiQC complete\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/qc/fastqc/raw\n",
    "\n",
    "cd $INPUT_DIR/\n",
    "\n",
    "echo \"Doing initial QC on samples $(ls *.gz)\"\n",
    "\n",
    "fastqc --threads $THREADS --outdir $OUTPUT_DIR/qc/fastqc/raw *.gz\n",
    "multiqc --outdir $OUTPUT_DIR/qc/fastqc/raw $OUTPUT_DIR/qc/fastqc/raw/ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Read preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Since we are working with DNA data and aiming for results including polymorphisms, we should do a careful preprocessing to remove the adapter sequences and remove low-quality ends. Outputs are described [here](#Read-preprocessing-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) Now I am processing CW56_S1_R1_001.fastq.gz as first in a pair and CW56_S1_R2_001.fastq.gz as a second in a pair.\n",
      "Done processing CW56_S1_R1_001.fastq.gz and CW56_S1_R2_001.fastq.gz\n",
      "(treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  configs = yaml.load(f)\n",
      "/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  sp = yaml.load(f)\n",
      "[WARNING]         multiqc : MultiQC Version v1.7 now available!\n",
      "[INFO   ]         multiqc : This is MultiQC v1.3\n",
      "[INFO   ]         multiqc : Template    : default\n",
      "[INFO   ]         multiqc : Searching '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/cutadapt/'\n",
      "[INFO   ]        cutadapt : Found 1 reports\n",
      "[INFO   ]         multiqc : Compressing plot data\n",
      "[INFO   ]         multiqc : Report      : ../results/qc/cutadapt/multiqc_report.html\n",
      "[INFO   ]         multiqc : Data        : ../results/qc/cutadapt/multiqc_data\n",
      "[INFO   ]         multiqc : MultiQC complete\n",
      "(treponema) (treponema) Started analysis of CW56_S1_R1.trimmed.fastq.gz\n",
      "Started analysis of CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "Analysis complete for CW56_S1_R1.trimmed.fastq.gz\n",
      "Analysis complete for CW56_S1_R2.trimmed.fastq.gz\n",
      "(treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  configs = yaml.load(f)\n",
      "/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  sp = yaml.load(f)\n",
      "[WARNING]         multiqc : MultiQC Version v1.7 now available!\n",
      "[INFO   ]         multiqc : This is MultiQC v1.3\n",
      "[INFO   ]         multiqc : Template    : default\n",
      "[INFO   ]         multiqc : Searching '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/fastqc/preprocessed/'\n",
      "[INFO   ]          fastqc : Found 2 reports\n",
      "[INFO   ]         multiqc : Compressing plot data\n",
      "[INFO   ]         multiqc : Report      : ../results/qc/fastqc/preprocessed/multiqc_report.html\n",
      "[INFO   ]         multiqc : Data        : ../results/qc/fastqc/preprocessed/multiqc_data\n",
      "[INFO   ]         multiqc : MultiQC complete\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/data/preprocessed\n",
    "mkdir $OUTPUT_DIR/qc/cutadapt\n",
    "mkdir $OUTPUT_DIR/qc/fastqc/preprocessed\n",
    "\n",
    "for sample in *R1*.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "    echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "    echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "\n",
    "    echo \"Now I am processing ${FORWARD} as first in a pair and ${REVERSE} as a second in a pair.\"\n",
    "\n",
    "    # Remove adapters and low-quality ends\n",
    "    cutadapt -a $ADAPTER_R1 -A $ADAPTER_R2 \\\n",
    "    --times 1 --quality-cutoff 15,15 --trim-n \\\n",
    "    --error-rate 0.10 -O 3 --minimum-length 35 --max-n 0 \\\n",
    "    --output $OUTPUT_DIR/data/preprocessed/${FORWARD%$extension}.trimmed.fastq.gz \\\n",
    "    --paired-output $OUTPUT_DIR/data/preprocessed/${REVERSE%$extension}.trimmed.fastq.gz \\\n",
    "    $FORWARD $REVERSE &>$OUTPUT_DIR/qc/cutadapt/${FORWARD%R1${extension}}.cutadapt.out\n",
    "\n",
    "    echo \"Done processing $FORWARD and $REVERSE\"\n",
    "done\n",
    "\n",
    "multiqc --outdir $OUTPUT_DIR/qc/cutadapt $OUTPUT_DIR/qc/cutadapt/\n",
    "\n",
    "fastqc --threads $THREADS --outdir $OUTPUT_DIR/qc/fastqc/preprocessed $OUTPUT_DIR/data/preprocessed/*.gz\n",
    "multiqc --outdir $OUTPUT_DIR/qc/fastqc/preprocessed $OUTPUT_DIR/qc/fastqc/preprocessed/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Host genome contamination removal"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Removal of the host genome DNA before the analysis speeds up the analysis as we will work with much smaller files. Also, we avoid unnecessary cross-mappings. Outputs are described [here](#Host-genome-cleaning-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Before the removal, we have to generate a host genome DNA reference index. This is run just once for one reference and one BBMap version. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) java -Djava.library.path=/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/bbmap-38.22-1/jni/ -ea -Xmx35g -cp /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/bbmap-38.22-1/current/ align2.BBMap build=1 overwrite=true fastareadlen=500 ref=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCF_000001405.36_GRCh38.p10_genomic.fna.gz -Xmx35g path=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/bbmap_index/GCF_000001405.36_GRCh38.p10_genomic.fna.gz\n",
      "Executing align2.BBMap [build=1, overwrite=true, fastareadlen=500, ref=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCF_000001405.36_GRCh38.p10_genomic.fna.gz, -Xmx35g, path=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/bbmap_index/GCF_000001405.36_GRCh38.p10_genomic.fna.gz]\n",
      "Version 38.22\n",
      "\n",
      "No output file.\n",
      "Writing reference.\n",
      "Executing dna.FastaToChromArrays2 [/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCF_000001405.36_GRCh38.p10_genomic.fna.gz, 1, writeinthread=false, genscaffoldinfo=true, retain, waitforwriting=false, gz=true, maxlen=536670912, writechroms=true, minscaf=1, midpad=300, startpad=8000, stoppad=8000, nodisk=false]\n",
      "\n",
      "Set genScaffoldInfo=true\n",
      "Writing chunk 1\n",
      "Writing chunk 2\n",
      "Writing chunk 3\n",
      "Writing chunk 4\n",
      "Writing chunk 5\n",
      "Writing chunk 6\n",
      "Writing chunk 7\n",
      "Set genome to 1\n",
      "\n",
      "Loaded Reference:\t0.049 seconds.\n",
      "Loading index for chunk 1-7, build 1\n",
      "No index available; generating from reference genome: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/bbmap_index/GCF_000001405.36_GRCh38.p10_genomic.fna.gz/ref/index/1/chr1-3_index_k13_c2_b1.block\n",
      "No index available; generating from reference genome: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/bbmap_index/GCF_000001405.36_GRCh38.p10_genomic.fna.gz/ref/index/1/chr4-7_index_k13_c2_b1.block\n",
      "Indexing threads started for block 4-7\n",
      "Indexing threads started for block 0-3\n",
      "Indexing threads finished for block 0-3\n",
      "Indexing threads finished for block 4-7\n",
      "Generated Index:\t594.459 seconds.\n",
      "Finished Writing:\t18.447 seconds.\n",
      "No reads to process; quitting.\n",
      "\n",
      "Total time:     \t663.970 seconds.\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir $(dirname $HOST_GENOME)/bbmap_index\n",
    "\n",
    "# If the host genome index does not exist create it\n",
    "if [ ! -d \"$(dirname $HOST_GENOME)/bbmap_index/$(basename $HOST_GENOME)\" ]\n",
    "then\n",
    "    mkdir $(dirname $HOST_GENOME)/bbmap_index/$(basename $HOST_GENOME)\n",
    "    bbmap.sh ref=$HOST_GENOME -Xmx35g path=$(dirname $HOST_GENOME)/bbmap_index/$(basename $HOST_GENOME)\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once the index is done, we can launch the host-genome removal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) Now I am processing CW56_S1_R1.trimmed.fastq.gz as first in a pair and CW56_S1_R2.trimmed.fastq.gz as a second in a pair with reference /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCF_000001405.36_GRCh38.p10_genomic.fna.gz.\n",
      "java -ea -Xmx12g -cp /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/bbmap-38.22-1/current/ jgi.ReformatReads -Xmx12g verifypaired in=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1.clean.fastq.gz out1=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1_R1.clean.fastq.gz out2=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1_R2.clean.fastq.gz\n",
      "Executing jgi.ReformatReads [-Xmx12g, verifypaired, in=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1.clean.fastq.gz, out1=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1_R1.clean.fastq.gz, out2=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/data/preprocessed/CW56_S1_R2.clean.fastq.gz]\n",
      "\n",
      "Input is being processed as paired\n",
      "Input:                  \t9901750 reads          \t1398779654 bases\n",
      "Output:                 \t9901750 reads (100.00%) \t1398779654 bases (100.00%)\n",
      "\n",
      "Time:                         \t16.382 seconds.\n",
      "Reads Processed:       9901k \t604.44k reads/sec\n",
      "Bases Processed:       1398m \t85.39m bases/sec\n",
      "Names appear to be correctly paired.\n",
      "(treponema) (treponema) (treponema) (treponema) Started analysis of CW56_S1_R1.clean.fastq.gz\n",
      "Started analysis of CW56_S1_R2.clean.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 5% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 10% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 15% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 20% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 25% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 30% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 35% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 40% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 45% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 50% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 55% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 60% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 65% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 70% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 75% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 80% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 85% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 90% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R1.clean.fastq.gz\n",
      "Approx 95% complete for CW56_S1_R2.clean.fastq.gz\n",
      "Analysis complete for CW56_S1_R1.clean.fastq.gz\n",
      "Analysis complete for CW56_S1_R2.clean.fastq.gz\n",
      "(treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  configs = yaml.load(f)\n",
      "/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  sp = yaml.load(f)\n",
      "[WARNING]         multiqc : MultiQC Version v1.7 now available!\n",
      "[INFO   ]         multiqc : This is MultiQC v1.3\n",
      "[INFO   ]         multiqc : Template    : default\n",
      "[INFO   ]         multiqc : Searching '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/fastqc/clean/'\n",
      "[INFO   ]          fastqc : Found 2 reports\n",
      "[INFO   ]         multiqc : Compressing plot data\n",
      "[INFO   ]         multiqc : Report      : ../../qc/fastqc/clean/multiqc_report.html\n",
      "[INFO   ]         multiqc : Data        : ../../qc/fastqc/clean/multiqc_data\n",
      "[INFO   ]         multiqc : MultiQC complete\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir $OUTPUT_DIR/qc/bbmap\n",
    "\n",
    "cd $OUTPUT_DIR/data/preprocessed/\n",
    "\n",
    "for sample in *R1*trimmed.fastq.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "        echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "        echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "\n",
    "    echo \"Now I am processing ${FORWARD} as first in a pair and ${REVERSE} as a second in a pair with reference $HOST_GENOME.\"\n",
    "\n",
    "    # Start mapping\n",
    "    bbmap.sh threads=$THREADS -Xmx25g minid=0.95 maxindel=3 bandwidthratio=0.16 \\\n",
    "    bandwidth=12 quickmatch fast minhits=2 path=$(dirname $HOST_GENOME)/bbmap_index/$(basename $HOST_GENOME) unpigz pigz \\\n",
    "    in=${FORWARD} in2=${REVERSE} outu=$OUTPUT_DIR/data/preprocessed/${FORWARD%_R1${extension}}.clean.fastq.gz \\\n",
    "    outm=$OUTPUT_DIR/data/preprocessed/${FORWARD%_R1${extension}}.dirty.fastq.gz &>$OUTPUT_DIR/qc/bbmap/${FORWARD%_R1${extension}}.bbmap.out # qtrim=rl trimq=10 untrim  # We already have preprocessed data, no need for this\n",
    "\n",
    "    # De-interleave\n",
    "    reformat.sh -Xmx12g verifypaired in=$OUTPUT_DIR/data/preprocessed/${FORWARD%_R1${extension}}.clean.fastq.gz \\\n",
    "    out1=$OUTPUT_DIR/data/preprocessed/${FORWARD%${extension}}.clean.fastq.gz out2=$OUTPUT_DIR/data/preprocessed/${REVERSE%${extension}}.clean.fastq.gz\n",
    "\n",
    "    # Remove the host genome mapped reads (usefull for mapping precision check)\n",
    "    rm $OUTPUT_DIR/data/preprocessed/${FORWARD%_R1${extension}}.clean.fastq.gz\n",
    "    rm $OUTPUT_DIR/data/preprocessed/${FORWARD%_R1${extension}}.dirty.fastq.gz\n",
    "done\n",
    "\n",
    "mkdir $OUTPUT_DIR/qc/fastqc/clean\n",
    "\n",
    "fastqc --threads $THREADS --outdir $OUTPUT_DIR/qc/fastqc/clean $OUTPUT_DIR/data/preprocessed/*.clean.fastq.gz\n",
    "\n",
    "multiqc --outdir $OUTPUT_DIR/qc/fastqc/clean $OUTPUT_DIR/qc/fastqc/clean/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We can check how many reads we had original (raw), how many after the preprocessing and how many after the host-genome removal. It's enough if we do it just for one of the read pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Raw read count for file echo CW56_S1_R1_001.fastq.gz\n",
      "14919358\n",
      "(treponema) (treponema) Preprocessed read count for file echo CW56_S1_R1.trimmed.fastq.gz\n",
      "14882182\n",
      "(treponema) (treponema) Cleaned read count for file echo CW56_S1_R1.clean.fastq.gz\n",
      "4950875\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "# Number of raw reads\n",
    "for sample in $INPUT_DIR/*R1*gz\n",
    "do\n",
    "    echo \"Raw read count for file echo $(basename $sample)\"\n",
    "    echo $(zcat $sample | wc -l)/4 | bc\n",
    "done\n",
    "\n",
    "# Number of the preprocessed reads\n",
    "for sample in $OUTPUT_DIR/data/preprocessed/*R1*trimmed.fastq.gz\n",
    "do\n",
    "    echo \"Preprocessed read count for file echo $(basename $sample)\"\n",
    "    echo $(zcat $sample | wc -l)/4 | bc\n",
    "done\n",
    "\n",
    "# Number of preprocessed and host-cleaned reads\n",
    "for sample in $OUTPUT_DIR/data/preprocessed/*R1*clean.fastq.gz\n",
    "do\n",
    "    echo \"Cleaned read count for file echo $(basename $sample)\"\n",
    "    echo $(zcat $sample | wc -l)/4 | bc\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Bacteria contamination scan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Before we start with the alignment, we can quickly scan for possible bacterial contamination in our dataset. This scan uses default StrainSeeker database which is most likely outdated but StrainSeeker offers a possibility to generate your [own index](http://bioinfo.ut.ee/strainseeker/index.php?r=site/page&view=manual#database) for the scanning with their builder script. One advantage over tools such as [Kraken2](https://ccb.jhu.edu/software/kraken2/) (a great tool) is that it consumes much less RAM. However, the latest releases of MiniKraken2 could be used as well. Outputs are described [here](#Contamination-scan-output). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Before we can use StrainSeeker, we have to download the database to scan. We can use the database provided at the StrainSeeker webpage or create our own. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) --2019-07-01 19:18:08--  http://bioinfo.ut.ee/strainseeker/executables/ss_db_w32_4324.tar.gz\n",
      "Resolving bioinfo.ut.ee (bioinfo.ut.ee)... 193.40.23.25\n",
      "Connecting to bioinfo.ut.ee (bioinfo.ut.ee)|193.40.23.25|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 7777508503 (7.2G) [application/x-gzip]\n",
      "Saving to: ‘ss_db_w32_4324.tar.gz’\n",
      "\n",
      "ss_db_w32_4324.tar. 100%[===================>]   7.24G  68.2MB/s    in 1m 52s  \n",
      "\n",
      "2019-07-01 19:20:00 (66.5 MB/s) - ‘ss_db_w32_4324.tar.gz’ saved [7777508503/7777508503]\n",
      "\n",
      "(treponema) ss_db_w32_4324/\n",
      "ss_db_w32_4324/db_binary\n",
      "ss_db_w32_4324/info.txt\n",
      "(treponema) (treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "STRAINSEEKER_DB=$REFERENCES/strainseekerdb\n",
    "\n",
    "mkdir $STRAINSEEKER_DB\n",
    "cd $STRAINSEEKER_DB/\n",
    "wget http://bioinfo.ut.ee/strainseeker/executables/ss_db_w32_4324.tar.gz\n",
    "tar xvzf ss_db_w32_4324.tar.gz\n",
    "\n",
    "STRAINSEEKER_DB=$STRAINSEEKER_DB/ss_db_w32_4324"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once we have the database we can start the scan. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) Working on sample CW56_S1_R1.clean.fastq.gz\n",
      "Subsampling\n",
      "Scanning\n",
      "Read tree info file...\n",
      "Getting k-mer counts from sample...\n",
      "Searching...\n",
      "------\n",
      "Reading tree info file: 0.0s\n",
      "Get k-mer counts from sample: 34.0s\n",
      "Search process: 7.0s\n",
      "Cleanup: 0.0s\n",
      "Database root-leaf count: 878\n",
      "TOTAL RUN TIME: 41.0s\n",
      "------\n",
      "Input files:\n",
      "CW56_S1_R1.clean.fastq.gz.sub\n",
      "Results saved in: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/strainseeker/CW56_S1_R1.clean.seeker.txt\n",
      "Working on sample CW56_S1_R2.clean.fastq.gz\n",
      "Subsampling\n",
      "Scanning\n",
      "Read tree info file...\n",
      "Getting k-mer counts from sample...\n",
      "Searching...\n",
      "------\n",
      "Reading tree info file: 0.0s\n",
      "Get k-mer counts from sample: 33.0s\n",
      "Search process: 5.0s\n",
      "Cleanup: 0.0s\n",
      "Database root-leaf count: 878\n",
      "TOTAL RUN TIME: 38.0s\n",
      "------\n",
      "Input files:\n",
      "CW56_S1_R2.clean.fastq.gz.sub\n",
      "Results saved in: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/qc/strainseeker/CW56_S1_R2.clean.seeker.txt\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir $OUTPUT_DIR/qc/strainseeker\n",
    "\n",
    "cd $OUTPUT_DIR/data/preprocessed/\n",
    "\n",
    "for sample in *.clean.fastq.gz\n",
    "do \n",
    "    echo \"Working on sample $sample\"\n",
    "\n",
    "    echo \"Subsampling\"\n",
    "    seqtk sample -s100 $sample 1000000 > $sample.sub # Subsample fastq to 1M\n",
    "\n",
    "    echo \"Scanning\"\n",
    "    seeker.pl -i $sample.sub -d $STRAINSEEKER_DB -o $OUTPUT_DIR/qc/strainseeker/${sample%.fastq.gz}.seeker.txt\n",
    "\n",
    "    rm $sample.sub\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Bacteria contamination scan - Kraken2 (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If you have enough resources you can use [Kraken2](https://ccb.jhu.edu/software/kraken2/). It scans the most recent bacterial, viral and fungal databases (or their subselection) and evaluates the possible distributions of individual species. If you decided to use Kraken2, you have to build the Kraken2 database. Please note this might take a while and will consume quite a lot of [RAM](https://ccb.jhu.edu/software/kraken2/index.shtml?t=manual#kraken-2-databases).  Outputs are described [here](#Contamination-scan-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "KRAKEN2_DB=$REFERENCES/kraken2db # Directory with stored StrainSeeker database\n",
    "mkdir $KRAKEN2_DB\n",
    "\n",
    "DATE=$(date +'%m%d%Y') # Save current date -> database version\n",
    "\n",
    "kraken2-build --standard --threads $THREADS --db $KRAKEN2_DB/$DATE\n",
    "\n",
    "KRAKEN2_DB=$KRAKEN2_DB/$DATE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once we have the database we can start the scan. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) Now I am processing CW56_S1_R1.clean.fastq.gz as first in a pair and CW56_S1_R2.clean.fastq.gz as a second in a pair with database /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/kraken2db/06182019.\n",
      "Loading database information... done.\n",
      "4950875 sequences (1398.78 Mbp) processed in 45.126s (6582.8 Kseq/m, 1859.85 Mbp/m).\n",
      "  3342486 sequences classified (67.51%)\n",
      "  1608389 sequences unclassified (32.49%)\n",
      "[1] 42073\n",
      "[2] 42074\n",
      "[3] 42075\n",
      "[4] 42076\n",
      "(treponema) (treponema) [1]   Done                    gzip $output\n",
      "[3]-  Done                    gzip $output\n",
      "[4]+  Done                    gzip $output\n",
      "[2]+  Done                    gzip $output\n",
      "(treponema) (treponema) (treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/qc/kraken2/reads\n",
    "\n",
    "cd $OUTPUT_DIR/data/preprocessed/\n",
    "\n",
    "for sample in *R1*.clean.fastq.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "        echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "        echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "\n",
    "    echo \"Now I am processing ${FORWARD} as first in a pair and ${REVERSE} as a second in a pair with database $KRAKEN2_DB.\"\n",
    "    \n",
    "    # Scan the contamination\n",
    "    kraken2 --use-names --paired --threads $THREADS --gzip-compressed --classified-out $OUTPUT_DIR/qc/kraken2/reads/${FORWARD%R1*}.classified-out#.fastq --unclassified-out $OUTPUT_DIR/qc/kraken2/reads/${FORWARD%R1*}.unclassified-out#.fastq --output $OUTPUT_DIR/qc/kraken2/reads/${FORWARD%R1*}.kraken.txt --report $OUTPUT_DIR/qc/kraken2/reads/${FORWARD%R1*}.kraken.report.txt --db $KRAKEN2_DB $FORWARD $REVERSE\n",
    "\n",
    "    for output in $OUTPUT_DIR/qc/kraken2/reads/*classified*.fastq\n",
    "    do\n",
    "        gzip $output &\n",
    "    done \n",
    "done\n",
    "\n",
    "wait\n",
    "\n",
    "# Get only species lines\n",
    "for report in $OUTPUT_DIR/qc/kraken2/reads/*.kraken.report.txt\n",
    "do \n",
    "    grep -P 'unclassified|root|\\tS\\t' $report | sort -k1,1nr > ${report%.kraken.report.txt*}.S.kraken.report.txt\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Bacteria reference genome mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "With host genome DNA cleaned data, we can proceed to the alignment to the reference. Please note we apply few filterings already at this step, mainly the minimal mapping quality, pairing of the reads and read duplication. Outputs are described [here](#Genome-mapping-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) [bwa_index] Pack FASTA... 0.01 sec\n",
      "[bwa_index] Construct BWT for the packed sequence...\n",
      "[bwa_index] 0.28 seconds elapse.\n",
      "[bwa_index] Update BWT... 0.01 sec\n",
      "[bwa_index] Pack forward-only FASTA... 0.01 sec\n",
      "[bwa_index] Construct SA from BWT and Occ... 0.12 sec\n",
      "[main] Version: 0.7.15-r1140\n",
      "[main] CMD: bwa index /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa\n",
      "[main] Real time: 0.420 sec; CPU: 0.416 sec\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) Now I am processing CW56_S1_R1.clean.fastq.gz as first in a pair and CW56_S1_R2.clean.fastq.gz as a second in a pair with reference /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (83, 132, 235)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 539)\n",
      "[M::mem_pestat] mean and std.dev: (168.32, 124.55)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 691)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 335)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 673)\n",
      "[M::mem_pestat] mean and std.dev: (257.98, 128.34)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 842)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (52, 100, 277)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 727)\n",
      "[M::mem_pestat] mean and std.dev: (113.93, 89.19)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 952)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (68, 184, 272)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 680)\n",
      "[M::mem_pestat] mean and std.dev: (186.00, 113.58)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 884)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (168, 237, 340)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 684)\n",
      "[M::mem_pestat] mean and std.dev: (261.60, 129.83)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 856)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (54, 104, 203)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 501)\n",
      "[M::mem_pestat] mean and std.dev: (114.17, 97.68)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 650)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (75, 146, 224)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 522)\n",
      "[M::mem_pestat] mean and std.dev: (163.36, 93.22)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 671)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (167, 237, 341)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 689)\n",
      "[M::mem_pestat] mean and std.dev: (261.79, 131.01)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 863)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (52, 73, 246)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 634)\n",
      "[M::mem_pestat] mean and std.dev: (125.24, 99.03)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 828)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (88, 152, 235)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 529)\n",
      "[M::mem_pestat] mean and std.dev: (168.62, 111.45)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 676)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (167, 237, 340)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 686)\n",
      "[M::mem_pestat] mean and std.dev: (261.30, 131.10)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 859)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (78, 186, 424)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 1116)\n",
      "[M::mem_pestat] mean and std.dev: (260.95, 255.05)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 1462)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (76, 123, 193)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 427)\n",
      "[M::mem_pestat] mean and std.dev: (121.47, 66.53)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 544)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (167, 236, 338)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 680)\n",
      "[M::mem_pestat] mean and std.dev: (260.69, 130.30)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 851)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (59, 100, 252)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 638)\n",
      "[M::mem_pestat] mean and std.dev: (116.82, 82.52)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 831)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (123, 160, 261)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 537)\n",
      "[M::mem_pestat] mean and std.dev: (170.52, 69.39)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 675)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (169, 237, 339)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 679)\n",
      "[M::mem_pestat] mean and std.dev: (261.66, 128.87)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 849)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (56, 104, 364)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 980)\n",
      "[M::mem_pestat] mean and std.dev: (185.73, 211.36)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 1288)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (68, 140, 287)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 725)\n",
      "[M::mem_pestat] mean and std.dev: (178.38, 134.36)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 944)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 333)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 667)\n",
      "[M::mem_pestat] mean and std.dev: (256.90, 126.67)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 834)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (80, 175, 309)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 767)\n",
      "[M::mem_pestat] mean and std.dev: (155.17, 101.11)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 996)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (49, 119, 184)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 454)\n",
      "[M::mem_pestat] mean and std.dev: (125.05, 104.55)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 589)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (165, 232, 330)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 660)\n",
      "[M::mem_pestat] mean and std.dev: (254.45, 125.30)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 825)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (80, 117, 288)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 704)\n",
      "[M::mem_pestat] mean and std.dev: (123.55, 78.35)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 912)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (59, 118, 161)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 365)\n",
      "[M::mem_pestat] mean and std.dev: (106.88, 60.97)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 467)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (164, 232, 329)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 659)\n",
      "[M::mem_pestat] mean and std.dev: (253.99, 125.20)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 824)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (85, 104, 288)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 694)\n",
      "[M::mem_pestat] mean and std.dev: (145.70, 100.67)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 897)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (90, 163, 266)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 618)\n",
      "[M::mem_pestat] mean and std.dev: (187.00, 122.20)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 794)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (164, 232, 330)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 662)\n",
      "[M::mem_pestat] mean and std.dev: (254.21, 125.92)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 828)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (72, 117, 252)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 612)\n",
      "[M::mem_pestat] mean and std.dev: (134.93, 106.99)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 792)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (72, 183, 291)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 729)\n",
      "[M::mem_pestat] mean and std.dev: (199.52, 164.44)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 948)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (165, 233, 330)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 660)\n",
      "[M::mem_pestat] mean and std.dev: (255.11, 125.29)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 825)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (84, 194, 892)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2508)\n",
      "[M::mem_pestat] mean and std.dev: (307.31, 371.91)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 3316)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FF...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (94, 173, 260)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 592)\n",
      "[M::mem_pestat] mean and std.dev: (187.25, 114.43)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 758)\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (172, 241, 343)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 685)\n",
      "[M::mem_pestat] mean and std.dev: (265.44, 129.93)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 856)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation RR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (78, 100, 995)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 2829)\n",
      "[M::mem_pestat] mean and std.dev: (338.50, 467.50)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 3746)\n",
      "[M::mem_pestat] skip orientation FF\n",
      "[M::mem_pestat] skip orientation RR\n",
      "[main] Version: 0.7.15-r1140\n",
      "[main] CMD: bwa mem -t 12 -T 20 -v 1 -M -R @RG\\tID:1\\tLB:CW56_S1_R1\\tPL:Illumina\\tSM:CW56_S1_R1\\tPU:CW56_S1_R1 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa CW56_S1_R1.clean.fastq.gz CW56_S1_R2.clean.fastq.gz\n",
      "[main] Real time: 65.747 sec; CPU: 694.320 sec\n",
      "Mapping finished\n",
      "Starting post-alignment processing and basic filtering\n",
      "[1] 54588\n",
      "[1]+  Done                    samtools flagstat $i > $OUTPUT_DIR/qc/alignment_stats/${i%.*}.flagstat\n",
      "Filtering finished\n",
      "Starting indel realignment\n",
      "[Mon Jul 01 22:15:40 CEST 2019] picard.sam.CreateSequenceDictionary REFERENCE=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa OUTPUT=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.dict    TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:15:40 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "[Mon Jul 01 22:15:40 CEST 2019] picard.sam.CreateSequenceDictionary done. Elapsed time: 0.00 minutes.\n",
      "Runtime.totalMemory()=514850816\n",
      "INFO  22:15:42,415 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:42,418 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18 \n",
      "INFO  22:15:42,418 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute \n",
      "INFO  22:15:42,418 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk \n",
      "INFO  22:15:42,418 HelpFormatter - [Mon Jul 01 22:15:42 CEST 2019] Executing on Linux 4.4.0-148-generic amd64 \n",
      "INFO  22:15:42,418 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_192-b01 \n",
      "INFO  22:15:42,422 HelpFormatter - Program Args: -T RealignerTargetCreator --num_threads 12 -R /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -I CW56_S1_.SS14.filt.bam -o /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/CW56_S1_.SS14.filt.forIndelRealigner.intervals \n",
      "INFO  22:15:42,430 HelpFormatter - Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01. \n",
      "INFO  22:15:42,430 HelpFormatter - Date/Time: 2019/07/01 22:15:42 \n",
      "INFO  22:15:42,431 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:42,431 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:42,436 GenomeAnalysisEngine - Strictness is SILENT \n",
      "INFO  22:15:42,529 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 \n",
      "INFO  22:15:42,537 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,560 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.02 \n",
      "INFO  22:15:42,585 MicroScheduler - Running the GATK in parallel mode with 12 total threads, 1 CPU thread(s) for each of 12 data thread(s), of 70 processors available on this machine \n",
      "INFO  22:15:42,721 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files \n",
      "INFO  22:15:42,753 GenomeAnalysisEngine - Done preparing for traversal \n",
      "INFO  22:15:42,754 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] \n",
      "INFO  22:15:42,754 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining \n",
      "INFO  22:15:42,754 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime \n",
      "INFO  22:15:42,762 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,767 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,768 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,771 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,771 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,774 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,775 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,777 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,778 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,780 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,781 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,784 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,784 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,787 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,788 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,791 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,792 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,795 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,795 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,798 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:42,798 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:42,801 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.00 \n",
      "INFO  22:15:57,657 ProgressMeter -            done   1139569.0    14.0 s      13.0 s      100.0%    14.0 s       0.0 s \n",
      "INFO  22:15:57,659 ProgressMeter - Total runtime 14.90 secs, 0.25 min, 0.00 hours \n",
      "INFO  22:15:57,659 MicroScheduler - 68802 reads were filtered out during the traversal out of approximately 3985556 total reads (1.73%) \n",
      "INFO  22:15:57,659 MicroScheduler -   -> 0 reads (0.00% of total) failing BadCigarFilter \n",
      "INFO  22:15:57,660 MicroScheduler -   -> 0 reads (0.00% of total) failing BadMateFilter \n",
      "INFO  22:15:57,660 MicroScheduler -   -> 0 reads (0.00% of total) failing DuplicateReadFilter \n",
      "INFO  22:15:57,660 MicroScheduler -   -> 0 reads (0.00% of total) failing FailsVendorQualityCheckFilter \n",
      "INFO  22:15:57,660 MicroScheduler -   -> 0 reads (0.00% of total) failing MalformedReadFilter \n",
      "INFO  22:15:57,661 MicroScheduler -   -> 0 reads (0.00% of total) failing MappingQualityUnavailableFilter \n",
      "INFO  22:15:57,661 MicroScheduler -   -> 68802 reads (1.73% of total) failing MappingQualityZeroFilter \n",
      "INFO  22:15:57,661 MicroScheduler -   -> 0 reads (0.00% of total) failing NotPrimaryAlignmentFilter \n",
      "INFO  22:15:57,661 MicroScheduler -   -> 0 reads (0.00% of total) failing Platform454Filter \n",
      "INFO  22:15:57,662 MicroScheduler -   -> 0 reads (0.00% of total) failing UnmappedReadFilter \n",
      "------------------------------------------------------------------------------------------\n",
      "Done. There were no warn messages.\n",
      "------------------------------------------------------------------------------------------\n",
      "INFO  22:15:59,528 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:59,531 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18 \n",
      "INFO  22:15:59,531 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute \n",
      "INFO  22:15:59,531 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk \n",
      "INFO  22:15:59,531 HelpFormatter - [Mon Jul 01 22:15:59 CEST 2019] Executing on Linux 4.4.0-148-generic amd64 \n",
      "INFO  22:15:59,532 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_192-b01 \n",
      "INFO  22:15:59,535 HelpFormatter - Program Args: -I CW56_S1_.SS14.filt.bam -R /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -T IndelRealigner -LOD 2.5 --consensusDeterminationModel USE_SW -targetIntervals /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/CW56_S1_.SS14.filt.forIndelRealigner.intervals -o /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/CW56_S1_.SS14.filt.indelRealigned.bam \n",
      "INFO  22:15:59,543 HelpFormatter - Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01. \n",
      "INFO  22:15:59,544 HelpFormatter - Date/Time: 2019/07/01 22:15:59 \n",
      "INFO  22:15:59,544 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:59,544 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  22:15:59,612 GenomeAnalysisEngine - Strictness is SILENT \n",
      "INFO  22:15:59,701 GenomeAnalysisEngine - Downsampling Settings: No downsampling \n",
      "INFO  22:15:59,708 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  22:15:59,731 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.02 \n",
      "INFO  22:15:59,884 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files \n",
      "INFO  22:15:59,888 GenomeAnalysisEngine - Done preparing for traversal \n",
      "INFO  22:15:59,889 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] \n",
      "INFO  22:15:59,889 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining \n",
      "INFO  22:15:59,889 ProgressMeter -        Location |     reads | elapsed |     reads | completed | runtime |   runtime \n",
      "INFO  22:15:59,917 ReadShardBalancer$1 - Loading BAM index data \n",
      "INFO  22:15:59,918 ReadShardBalancer$1 - Done loading BAM index data \n",
      "INFO  22:16:29,894 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:46037    100002.0    30.0 s       5.0 m        4.0%    12.4 m      11.9 m \n",
      "INFO  22:16:59,897 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:96654    300036.0    60.0 s       3.3 m        8.5%    11.8 m      10.8 m \n",
      "INFO  22:17:29,921 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:150374    500046.0    90.0 s       3.0 m       13.2%    11.4 m       9.9 m \n",
      "INFO  22:17:59,925 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:206001    700061.0   120.0 s       2.9 m       18.1%    11.1 m       9.1 m \n",
      "INFO  22:18:29,952 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:263236    900105.0     2.5 m       2.8 m       23.1%    10.8 m       8.3 m \n",
      "INFO  22:19:00,137 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:323854   1200115.0     3.0 m       2.5 m       28.4%    10.6 m       7.6 m \n",
      "INFO  22:19:30,139 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:401043   1400127.0     3.5 m       2.5 m       35.2%     9.9 m       6.4 m \n",
      "INFO  22:20:00,141 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:453573   1600148.0     4.0 m       2.5 m       39.8%    10.0 m       6.0 m \n",
      "INFO  22:20:30,144 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:524111   1800157.0     4.5 m       2.5 m       46.0%     9.8 m       5.3 m \n",
      "INFO  22:21:00,146 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:577892   2000169.0     5.0 m       2.5 m       50.7%     9.9 m       4.9 m \n",
      "INFO  22:21:30,148 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:644329   2200174.0     5.5 m       2.5 m       56.5%     9.7 m       4.2 m \n",
      "INFO  22:22:00,150 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:718068   2400187.0     6.0 m       2.5 m       63.0%     9.5 m       3.5 m \n",
      "INFO  22:22:30,153 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:778275   2600201.0     6.5 m       2.5 m       68.3%     9.5 m       3.0 m \n",
      "INFO  22:23:00,155 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:844636   2900217.0     7.0 m       2.4 m       74.1%     9.4 m       2.4 m \n",
      "INFO  22:23:30,158 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:906866   3100229.0     7.5 m       2.4 m       79.6%     9.4 m     115.0 s \n",
      "INFO  22:24:00,161 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:968295   3300241.0     8.0 m       2.4 m       85.0%     9.4 m      84.0 s \n",
      "INFO  22:24:30,187 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:1025605   3500265.0     8.5 m       2.4 m       90.0%     9.4 m      56.0 s \n",
      "INFO  22:25:00,191 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:1083016   3700272.0     9.0 m       2.4 m       95.0%     9.5 m      28.0 s \n",
      "INFO  22:25:27,281 ProgressMeter -            done   3960606.0     9.5 m       2.4 m      100.0%     9.5 m       0.0 s \n",
      "INFO  22:25:27,281 ProgressMeter - Total runtime 567.39 secs, 9.46 min, 0.16 hours \n",
      "INFO  22:25:27,286 MicroScheduler - 0 reads were filtered out during the traversal out of approximately 3960606 total reads (0.00%) \n",
      "INFO  22:25:27,286 MicroScheduler -   -> 0 reads (0.00% of total) failing BadCigarFilter \n",
      "INFO  22:25:27,287 MicroScheduler -   -> 0 reads (0.00% of total) failing MalformedReadFilter \n",
      "------------------------------------------------------------------------------------------\n",
      "Done. There were no warn messages.\n",
      "------------------------------------------------------------------------------------------\n",
      "rm: cannot remove 'rm': No such file or directory\n",
      "rm: cannot remove 'CW56_S1_.SS14.filt.bam': No such file or directory\n",
      "Finished indel realignment\n",
      "Starting read duplicate removal\n",
      "[Mon Jul 01 22:25:29 CEST 2019] picard.sam.markduplicates.MarkDuplicates INPUT=[CW56_S1_.SS14.filt.indelRealigned.bam] OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.bam METRICS_FILE=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/picard_dup/CW56_S1_.SS14.filt.indelRealigned.dedupStats.txt REMOVE_DUPLICATES=true OPTICAL_DUPLICATE_PIXEL_DISTANCE=100    MAX_SEQUENCES_FOR_DISK_READ_ENDS_MAP=50000 MAX_FILE_HANDLES_FOR_READ_ENDS_MAP=8000 SORTING_COLLECTION_SIZE_RATIO=0.25 TAG_DUPLICATE_SET_MEMBERS=false REMOVE_SEQUENCING_DUPLICATES=false TAGGING_POLICY=DontTag ASSUME_SORTED=false DUPLICATE_SCORING_STRATEGY=SUM_OF_BASE_QUALITIES PROGRAM_RECORD_ID=MarkDuplicates PROGRAM_GROUP_NAME=MarkDuplicates READ_NAME_REGEX=<optimized capture of last three ':' separated fields as numeric values> VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:25:29 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "INFO\t2019-07-01 22:25:29\tMarkDuplicates\tStart of doWork freeMemory: 507090464; totalMemory: 514850816; maxMemory: 954728448\n",
      "INFO\t2019-07-01 22:25:29\tMarkDuplicates\tReading input file and constructing read end information.\n",
      "INFO\t2019-07-01 22:25:29\tMarkDuplicates\tWill retain up to 3459161 data points before spilling to disk.\n",
      "INFO\t2019-07-01 22:25:35\tMarkDuplicates\tRead     1,000,000 records.  Elapsed time: 00:00:06s.  Time for last 1,000,000:    6s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:266,234\n",
      "INFO\t2019-07-01 22:25:35\tMarkDuplicates\tTracking 188 as yet unmatched pairs. 188 records in RAM.\n",
      "INFO\t2019-07-01 22:25:41\tMarkDuplicates\tRead     2,000,000 records.  Elapsed time: 00:00:12s.  Time for last 1,000,000:    5s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:570,353\n",
      "INFO\t2019-07-01 22:25:41\tMarkDuplicates\tTracking 230 as yet unmatched pairs. 230 records in RAM.\n",
      "INFO\t2019-07-01 22:25:54\tMarkDuplicates\tRead     3,000,000 records.  Elapsed time: 00:00:24s.  Time for last 1,000,000:   12s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:872,641\n",
      "INFO\t2019-07-01 22:25:54\tMarkDuplicates\tTracking 208 as yet unmatched pairs. 208 records in RAM.\n",
      "INFO\t2019-07-01 22:26:02\tMarkDuplicates\tRead 3960606 records. 0 pairs never matched.\n",
      "INFO\t2019-07-01 22:26:07\tMarkDuplicates\tAfter buildSortedReadEndLists freeMemory: 723137512; totalMemory: 888143872; maxMemory: 954728448\n",
      "INFO\t2019-07-01 22:26:07\tMarkDuplicates\tWill retain up to 29835264 duplicate indices before spilling to disk.\n",
      "INFO\t2019-07-01 22:26:07\tMarkDuplicates\tTraversing read pair information and detecting duplicates.\n",
      "INFO\t2019-07-01 22:26:08\tMarkDuplicates\tTraversing fragment information and detecting duplicates.\n",
      "INFO\t2019-07-01 22:26:09\tMarkDuplicates\tSorting list of duplicate records.\n",
      "INFO\t2019-07-01 22:26:09\tMarkDuplicates\tAfter generateDuplicateIndexes freeMemory: 638643600; totalMemory: 885522432; maxMemory: 954728448\n",
      "INFO\t2019-07-01 22:26:09\tMarkDuplicates\tMarking 3091336 records as duplicates.\n",
      "INFO\t2019-07-01 22:26:09\tMarkDuplicates\tFound 108 optical duplicate clusters.\n",
      "INFO\t2019-07-01 22:26:09\tMarkDuplicates\tReads are assumed to be ordered by: coordinate\n",
      "INFO\t2019-07-01 22:26:31\tMarkDuplicates\tBefore output close freeMemory: 889288992; totalMemory: 898629632; maxMemory: 954728448\n",
      "INFO\t2019-07-01 22:26:31\tMarkDuplicates\tAfter output close freeMemory: 882470240; totalMemory: 891813888; maxMemory: 954728448\n",
      "[Mon Jul 01 22:26:31 CEST 2019] picard.sam.markduplicates.MarkDuplicates done. Elapsed time: 1.03 minutes.\n",
      "Runtime.totalMemory()=891813888\n",
      "Finished read duplicate removal\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "# Prepare references\n",
    "if [[ ${REFERENCE##*.} == \"gz\" ]] # Uncompress the reference if it is in gz archive\n",
    "then\n",
    "    gunzip -c $REFERENCE > ${REFERENCE%.gz*}\n",
    "    REFERENCE=${REFERENCE%.gz*}\n",
    "fi\n",
    "\n",
    "if [[ `echo $REFERENCE | grep \".fna$\"` != \"\" ]] || [[ `echo $REFERENCE | grep \".fasta$\"` != \"\" ]] # Make a link to the original to have \".fa\" suffix\n",
    "then\n",
    "    ln -s $REFERENCE ${REFERENCE%.*}.fa\n",
    "fi\n",
    "\n",
    "# Prepare indexes\n",
    "bwa index $REFERENCE\n",
    "samtools faidx $REFERENCE\n",
    "\n",
    "cd $OUTPUT_DIR/data/preprocessed\n",
    "\n",
    "# Adjust output directory to reflect the used reference genome\n",
    "OUTPUT_DIR=${OUTPUT_DIR}/$(basename $REFERENCE .fa)\n",
    "\n",
    "mkdir -p $OUTPUT_DIR/alignment\n",
    "mkdir -p $OUTPUT_DIR/qc/alignment_stats\n",
    "\n",
    "for sample in *R1*clean.fastq.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "        echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "        echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "\n",
    "    # Start mapping\n",
    "    echo \"Now I am processing ${FORWARD} as first in a pair and ${REVERSE} as a second in a pair with reference $REFERENCE\"\n",
    "\n",
    "    bwa mem -t $THREADS -T 20 -v 1 -M -R \"@RG\\tID:1\\tLB:${sample%%.*}\\tPL:Illumina\\tSM:${sample%%.*}\\tPU:${sample%%.*}\" $REFERENCE ${FORWARD} ${REVERSE} | samtools view -F 4 -@ $THREADS -b - | samtools sort -@ $THREADS - > $OUTPUT_DIR/alignment/${FORWARD%R1*}.$(basename $REFERENCE .fa).bam \n",
    "\n",
    "    echo \"Mapping finished\"\n",
    "\n",
    "    echo \"Starting post-alignment processing and basic filtering\"\n",
    "    \n",
    "    cd $OUTPUT_DIR/alignment\n",
    "\n",
    "    i=${FORWARD%R1*}.$(basename $REFERENCE .fa).bam\n",
    "\n",
    "    # Index BAM files and perform basic filtering\n",
    "    samtools index -@ $THREADS $i\n",
    "    samtools flagstat $i > $OUTPUT_DIR/qc/alignment_stats/${i%.*}.flagstat &\n",
    "    samtools view -@ $THREADS -h -F 12 -f 2 -F 256 -b $i | samtools sort -n -@ $THREADS - | samtools fixmate -O bam - - | samtools sort -@ $THREADS - > ${i%.*}.filt.bam\n",
    "    samtools index -@ $THREADS ${i%.*}.filt.bam\n",
    "    samtools flagstat ${i%.*}.filt.bam > $OUTPUT_DIR/qc/alignment_stats/${i%.*}.filt.flagstat\n",
    "\n",
    "    echo \"Filtering finished\"\n",
    "\n",
    "    # Indel realignment\n",
    "    echo \"Starting indel realignment\"\n",
    "    # Create input seqence dictionary and index reference\n",
    "    rm ${REFERENCE%.*}.dict\n",
    "    picard CreateSequenceDictionary R=$REFERENCE O=${REFERENCE%.*}.dict\n",
    "\n",
    "    # Realign\n",
    "    i=${i%.*}.filt.bam\n",
    "\n",
    "    gatk -T RealignerTargetCreator --num_threads $THREADS -R $REFERENCE -I ${i} -o $OUTPUT_DIR/alignment/${i%.*}.forIndelRealigner.intervals # Prepare intervals\n",
    "    gatk -I ${i} -R $REFERENCE -T IndelRealigner -LOD 2.5 --consensusDeterminationModel USE_SW -targetIntervals $OUTPUT_DIR/alignment/${i%.*}.forIndelRealigner.intervals -o $OUTPUT_DIR/alignment/${i%.*}.indelRealigned.bam # Run re-alignment\n",
    "\n",
    "    samtools index -@ $THREADS ${i%.*}.indelRealigned.bam\n",
    "\n",
    "    rm $i $i*\n",
    "    rm $OUTPUT_DIR/alignment/${i%.*}.forIndelRealigner.intervals\n",
    "\n",
    "    echo \"Finished indel realignment\"\n",
    "\n",
    "    echo \"Starting read duplicate removal\"\n",
    "\n",
    "    # Remove duplicates\n",
    "    i=${i%.*}.indelRealigned.bam\n",
    "\n",
    "    mkdir $OUTPUT_DIR/qc/picard_dup\n",
    "    \n",
    "    picard MarkDuplicates INPUT=$i OUTPUT=${i%.*}.dedup.bam METRICS_FILE=$OUTPUT_DIR/qc/picard_dup/${i%.*}.dedupStats.txt REMOVE_DUPLICATES=true OPTICAL_DUPLICATE_PIXEL_DISTANCE=100 # Taggs ALL duplicates, PCR and optical and remove them\n",
    "\n",
    "    samtools index -@ $THREADS ${i%.*}.dedup.bam\n",
    "    samtools flagstat ${i%.*}.dedup.bam > $OUTPUT_DIR/qc/alignment_stats/${i%.*}.dedup.flagstat\n",
    "\n",
    "    rm $i ${i%.bam}.bai\n",
    "\n",
    "    echo \"Finished read duplicate removal\"\n",
    "done\n",
    "\n",
    "multiqc -o $OUTPUT_DIR/qc/picard_dup $OUTPUT_DIR/qc/picard_dup/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Post-alignment filtering"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The most important part of this part is the **post-alignment filtering**. `BWA MEM` is known to be very sensitive but not very specific. Right now, we have a lot of alignments which in reality do not belong to our reference. This is due to several factors used in `BWA MEM`, such as minimum length of alignment (default: 19, can be adjusted by `-k [INT]`), allowance of extensive soft-clipping, allowed hard-clipping and allowing almost any number of mismatches. The simplest way to see the alignment artifacts and cross-mappings is to check the reference alignment coverage and check highly uneven coverage peaks. Outputs are described [here](#Alignment-filtering-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "We apply the following filters:\n",
    "\n",
    "1. To many mismatches\n",
    "    * max. 0.05% mismatches and\n",
    "    * max. 5 mismatches\n",
    "2. Very short alignments\n",
    "     * min 35 bp mapping (measured on the read, not the reference)\n",
    "3. Too much soft-clipped \n",
    "     * max 0.05% soft-clipped\n",
    "4. Supplementary/chimeric reads \n",
    "    * `samtools -F 2048` flag\n",
    "5. Too much hard-clipped\n",
    "     * max 0.00% hard-clipped (no hard-clipping allowed)\n",
    "6. MAPQ 10 \n",
    "    * very often repetitive alignments\n",
    "7. MAPQ 40 \n",
    "    * get only high-quality alignment\n",
    "8. Singletons \n",
    "    * remove singleton reads (not paired) after all the mapping filtering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) Processing sample CW56_S1_.SS14.filt.indelRealigned.dedup.bam\n",
      "                                                                                \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bDone! (0:25)\n",
      "851264 kept\n",
      "18006 failed\n",
      "                                                                                \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bDone! (0:20)\n",
      "843621 kept\n",
      "7643 failed\n",
      "Number of too much mismatched reads is 23531 CW56_S1_.SS14.filt.indelRealigned.dedup.mm.fail.txt.tmp for sample  CW56_S1_.SS14.filt.indelRealigned.dedup.bam\n",
      "[Mon Jul 01 22:28:48 CEST 2019] picard.sam.FilterSamReads INPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.bam FILTER=includeReadList [OUTPUT SAM/BAM will contain reads that are supplied in the READ_LIST_FILE file] READ_LIST_FILE=CW56_S1_.SS14.filt.indelRealigned.dedup.mm.fail.txt.tmp OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.mm.filtOut.bam    WRITE_READS_FILES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:28:48 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "INFO\t2019-07-01 22:28:52\tFilterSamReads\tFiltering [presorted=true] CW56_S1_.SS14.filt.indelRealigned.dedup.bam -> OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.mm.filtOut.bam [sortorder=coordinate]\n",
      "INFO\t2019-07-01 22:28:55\tFilterSamReads\t47,062 SAMRecords written to CW56_S1_.SS14.filt.indelRealigned.dedup.mm.filtOut.bam\n",
      "[Mon Jul 01 22:28:55 CEST 2019] picard.sam.FilterSamReads done. Elapsed time: 0.12 minutes.\n",
      "Runtime.totalMemory()=707264512\n",
      "[1] 55202\n",
      "[1]+  Done                    samtools view -h -@ $THREADS ${sample%.*}.mm2.bam | perl -slane '$l = 0; $F[5] =~ s/(\\d+)[MX=DN]/$l+=$1/eg; print if $l < $MIN_LENGTH_MAPPED or /^@/' -- -MIN_LENGTH_MAPPED=$MIN_LENGTH_MAPPED | samtools view -@ $THREADS -b - > ${sample%.*}.short.filtOut.bam\n",
      "                                                                                \b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\b\bDone! (0:19)\n",
      "Wrote: 840174 reads\n",
      "Altered: 12314\n",
      "Unmapped: 0\n",
      "Number of too much soft clipped reads is 9472 CW56_S1_.SS14.filt.indelRealigned.dedup.read_names_to_remove_highSoftClip.txt for sample  CW56_S1_.SS14.filt.indelRealigned.dedup.bam\n",
      "[1] 55293\n",
      "[Mon Jul 01 22:29:27 CEST 2019] picard.sam.FilterSamReads INPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.short.bam FILTER=excludeReadList [OUTPUT bam will contain reads that are *not* supplied in the READ_LIST_FILE file] READ_LIST_FILE=CW56_S1_.SS14.filt.indelRealigned.dedup.read_names_to_remove_highSoftClip.txt OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.scf.bam    WRITE_READS_FILES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:29:27 CEST 2019] picard.sam.FilterSamReads INPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.short.bam FILTER=includeReadList [OUTPUT SAM/BAM will contain reads that are supplied in the READ_LIST_FILE file] READ_LIST_FILE=CW56_S1_.SS14.filt.indelRealigned.dedup.read_names_to_remove_highSoftClip.txt OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.scf.filtOut.bam    WRITE_READS_FILES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:29:27 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "[Mon Jul 01 22:29:27 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "INFO\t2019-07-01 22:29:31\tFilterSamReads\tFiltering [presorted=true] CW56_S1_.SS14.filt.indelRealigned.dedup.short.bam -> OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.scf.bam [sortorder=coordinate]\n",
      "INFO\t2019-07-01 22:29:31\tFilterSamReads\tFiltering [presorted=true] CW56_S1_.SS14.filt.indelRealigned.dedup.short.bam -> OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.scf.filtOut.bam [sortorder=coordinate]\n",
      "INFO\t2019-07-01 22:29:33\tFilterSamReads\t17,136 SAMRecords written to CW56_S1_.SS14.filt.indelRealigned.dedup.scf.filtOut.bam\n",
      "[Mon Jul 01 22:29:34 CEST 2019] picard.sam.FilterSamReads done. Elapsed time: 0.10 minutes.\n",
      "Runtime.totalMemory()=703594496\n",
      "INFO\t2019-07-01 22:29:39\tFilterSamReads\t823,038 SAMRecords written to CW56_S1_.SS14.filt.indelRealigned.dedup.scf.bam\n",
      "[Mon Jul 01 22:29:42 CEST 2019] picard.sam.FilterSamReads done. Elapsed time: 0.24 minutes.\n",
      "Runtime.totalMemory()=709885952\n",
      "[1]+  Done                    picard FilterSamReads I=${sample%.*}.short.bam O=${sample%.*}.scf.filtOut.bam READ_LIST_FILE=${sample%.*}.read_names_to_remove_highSoftClip.txt FILTER=includeReadList\n",
      "[1] 55476\n",
      "[1]+  Done                    samtools view -@ $THREADS -h -f 2048 -b ${sample%.*}.scf.bam > ${sample%.*}.sup.filtOut.bam\n",
      "[1] 55508\n",
      "[1]+  Done                    samtools view -@ $THREADS -h ${sample%.*}.sup.bam | awk '$6 ~ /H/{print}' | samtools view -bh - > ${sample%.*}.hcf.filtOut.bam\n",
      "[1] 55541\n",
      "[2] 55573\n",
      "[1]-  Done                    samtools view -@ $THREADS -h ${sample%.*}.sup.bam | awk -v var=\"$MAPQ\" '$5 < var || $1 ~ /^@/' | samtools view -b - > ${sample%.*}.MAPQ${MAPQ}.filtOut.bam\n",
      "[2]+  Done                    samtools view -@ $THREADS -h ${sample%.*}.MAPQ${MAPQ}.bam | awk -v var=\"$MAPQ_FINAL\" '$5 < var || $1 ~ /^@/' | samtools view -b - > ${sample%.*}.MAPQ${MAPQ_FINAL}.filtOut.bam\n",
      "Number of singleton reads after filtering is 20005 CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.singleAfterFilt.txt for sample  CW56_S1_.SS14.filt.indelRealigned.dedup.bam\n",
      "[1] 55620\n",
      "[Mon Jul 01 22:30:11 CEST 2019] picard.sam.FilterSamReads INPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam FILTER=includeReadList [OUTPUT SAM/BAM will contain reads that are supplied in the READ_LIST_FILE file] READ_LIST_FILE=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.singleAfterFilt.txt OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.singletons.filtOut.bam    WRITE_READS_FILES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:30:11 CEST 2019] picard.sam.FilterSamReads INPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam FILTER=excludeReadList [OUTPUT bam will contain reads that are *not* supplied in the READ_LIST_FILE file] READ_LIST_FILE=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.singleAfterFilt.txt OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam.tmp    WRITE_READS_FILES=true VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 22:30:11 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "[Mon Jul 01 22:30:11 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "INFO\t2019-07-01 22:30:14\tFilterSamReads\tFiltering [presorted=true] CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam -> OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam.tmp [sortorder=coordinate]\n",
      "INFO\t2019-07-01 22:30:14\tFilterSamReads\tFiltering [presorted=true] CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam -> OUTPUT=CW56_S1_.SS14.filt.indelRealigned.dedup.singletons.filtOut.bam [sortorder=coordinate]\n",
      "INFO\t2019-07-01 22:30:17\tFilterSamReads\t20,005 SAMRecords written to CW56_S1_.SS14.filt.indelRealigned.dedup.singletons.filtOut.bam\n",
      "[Mon Jul 01 22:30:17 CEST 2019] picard.sam.FilterSamReads done. Elapsed time: 0.10 minutes.\n",
      "Runtime.totalMemory()=708837376\n",
      "INFO\t2019-07-01 22:30:22\tFilterSamReads\t787,234 SAMRecords written to CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam.tmp\n",
      "[Mon Jul 01 22:30:25 CEST 2019] picard.sam.FilterSamReads done. Elapsed time: 0.24 minutes.\n",
      "Runtime.totalMemory()=709361664\n",
      "[1]+  Done                    picard FilterSamReads I=${sample%.*}.MAPQ${MAPQ_FINAL}.bam O=${sample%.*}.singletons.filtOut.bam READ_LIST_FILE=${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt FILTER=includeReadList\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) mv: cannot move '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/filtered' to a subdirectory of itself, '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/filtered/filt/filtered'\n",
      "(treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "MAX_PERC_OF_MM=\"0.05\" # Maximum percentage of mismatches compared to the read length - bad if we have to error-prone reads but helps to remove false-positives\n",
    "MAX_NUMBER_OF_MM=\"5\" # Maximum number of mismatches\n",
    "MIN_LENGTH_MAPPED=\"35\" # Remove mappings that mapped with too few bases (remove excessive soft-clipping)\n",
    "MAX_SOFTCLIP=\"0.05\" # Maximal percentage of the reads allowed to be soft-clipped\n",
    "MAX_HARDCLIP=\"0.00\" # Maximal percentage of the reads allowed to be hard-clipped\n",
    "MAPQ=\"10\" # Minimal MAPQ for (probably) repetitive regions\n",
    "MAPQ_FINAL=\"40\" # Minimal MAPQ for final results\n",
    "\n",
    "cd $OUTPUT_DIR/alignment\n",
    "\n",
    "mkdir $OUTPUT_DIR/tmp\n",
    "\n",
    "# Filter mappings\n",
    "for sample in *.filt.indelRealigned.dedup.bam\n",
    "do \n",
    "    echo \"Processing sample $sample\"\n",
    "\n",
    "    samtools index -@ $THREADS $sample\n",
    "\n",
    "    # 1) Remove too many mismatches\n",
    "    bamutils filter $sample ${sample%.*}.mm1.bam -failed ${sample%.*}.mm1.fail.txt -maximum_mismatch_ratio $MAX_PERC_OF_MM # Filter by perc. of mismatches; It's more filtering on edit distance = how many nucleotides have to be changed to get exactly the reference sequence; indels are counted as many times as they are \"long\"; error in the source https://github.com/ngsutils/ngsutils/pull/18\n",
    "    bamutils filter ${sample%.*}.mm1.bam ${sample%.*}.mm2.bam -failed ${sample%.*}.mm2.fail.txt -mismatch $MAX_NUMBER_OF_MM # Filter by num. of mismatches\n",
    "\n",
    "    cat ${sample%.*}.mm1.fail.txt ${sample%.*}.mm2.fail.txt > ${sample%.*}.mm.fail.txt # Merge mapping filtered by mismatches\n",
    "\n",
    "    # Use extracted filtered read name and filter it out from the original bam but ONLY when the mismatches filtering had some results\n",
    "    if [ -s ${sample%.*}.mm.fail.txt ]\n",
    "    then\n",
    "        cat ${sample%.*}.mm.fail.txt | awk '{print $1}' | sort -T $OUTPUT_DIR/tmp --parallel=$THREADS | uniq > ${sample%.*}.mm.fail.txt.tmp\n",
    "        echo \"Number of too much mismatched reads is\" `wc -l ${sample%.*}.mm.fail.txt.tmp` \"for sample \" $sample\n",
    "        picard FilterSamReads I=$sample O=${sample%.*}.mm.filtOut.bam READ_LIST_FILE=${sample%.*}.mm.fail.txt.tmp FILTER=includeReadList # Include reads\n",
    "        rm ${sample%.*}.mm.fail.txt.tmp\n",
    "    else\n",
    "        echo \"There are none too much mismatched reads with \" $MAX_PERC_OF_MM \" % and \" $MAX_NUMBER_OF_MM \" mismatches for sample \" $sample\". Nothing to report.\"\n",
    "    fi\n",
    "\n",
    "    # 2) Remove too short alignment\n",
    "    # Filter out mappings that mapped with length shorter than MIN_LENGTH_MAPPED https://www.biostars.org/p/12406/ - this filters the mapping by calculating the length of mapping on the read itself; example 106S8M1D21M14S with MIN_LENGTH_MAPPED=30 is filtered out because 8M+21M=29\n",
    "    # Another filtering by mapped read length could be taken from here https://www.biostars.org/p/151510/ - this filters the mapping by calculating the length of mapping on the reference; example 106S8M1D21M14S with MIN_LENGTH_MAPPED=30 is NOT filtered out because 8M+1D+21M=30, it is filtered out when set to MIN_LENGTH_MAPPED=31\n",
    "    #samjs.jar -e '!record.readUnmappedFlag  && record.cigar.referenceLength  >= $MIN_LENGTH_MAPPED '$sample | samtools view -@ $THREADS -b -o ${sample%.*}.short.bam -\n",
    "    samtools view -h -@ $THREADS ${sample%.*}.mm2.bam | perl -slane '$l = 0; $F[5] =~ s/(\\d+)[MX=DN]/$l+=$1/eg; print if $l < $MIN_LENGTH_MAPPED or /^@/' -- -MIN_LENGTH_MAPPED=$MIN_LENGTH_MAPPED | samtools view -@ $THREADS -b - > ${sample%.*}.short.filtOut.bam &\n",
    "    samtools view -h -@ $THREADS ${sample%.*}.mm2.bam | perl -slane '$l = 0; $F[5] =~ s/(\\d+)[MX=DN]/$l+=$1/eg; print if $l >= $MIN_LENGTH_MAPPED or /^@/' -- -MIN_LENGTH_MAPPED=$MIN_LENGTH_MAPPED | samtools view -@ $THREADS -b - > ${sample%.*}.short.bam\n",
    "\n",
    "    # 3) Too much softclipped\n",
    "    # Running it on the whole file and saving it creates error in BAM validation; we have to just take the read names from here and remove them from the alignment\n",
    "    bamutils removeclipping ${sample%.*}.short.bam ${sample%.*}.scf.bam.tmp\n",
    "    samtools view -@ $THREADS ${sample%.*}.scf.bam.tmp | grep 'ZC:f:' | awk '{for (i=1;i<=NF;i++){if ($sample ~/ZC:f:/) {print $1, $sample}}}' | sed 's/ZC:f://' | awk -v MAX_SOFTCLIP=\"$MAX_SOFTCLIP\" ' $2 > MAX_SOFTCLIP {print $1}' > ${sample%.*}.read_names_to_remove_highSoftClip.txt.tmp # Get only softclipped reads above the threshold\n",
    "    rm ${sample%.*}.scf.bam.tmp # Remove temporary BAMs\n",
    "    cat ${sample%.*}.read_names_to_remove_highSoftClip.txt.tmp | cut -f 1 | sort -T $OUTPUT_DIR/tmp --parallel=$THREADS | uniq > ${sample%.*}.read_names_to_remove_highSoftClip.txt # Get unique read names with softclipping!\n",
    "    #cat ${sample%.*}.read_names_to_remove_highSoftClip.txt | cut -f 1 | sort -T $OUTPUT_DIR/tmp --parallel=$THREADS | uniq -d > tmp; mv tmp ${sample%.*}.read_names_to_remove_highSoftClip.txt # Get unique read names with softclipping if whole pair failed the filtering!\n",
    "    rm ${sample%.*}.read_names_to_remove_highSoftClip.txt.tmp\n",
    "\n",
    "    # Use extracted filtered read name and filter it out from the original bam but ONLY when the soft clipping filtering had some results\n",
    "    if [ -s ${sample%.*}.read_names_to_remove_highSoftClip.txt ]\n",
    "    then\n",
    "        echo \"Number of too much soft clipped reads is\" `wc -l ${sample%.*}.read_names_to_remove_highSoftClip.txt` \"for sample \" $sample\n",
    "        picard FilterSamReads I=${sample%.*}.short.bam O=${sample%.*}.scf.filtOut.bam READ_LIST_FILE=${sample%.*}.read_names_to_remove_highSoftClip.txt FILTER=includeReadList & # Include reads\n",
    "        picard FilterSamReads I=${sample%.*}.short.bam O=${sample%.*}.scf.bam READ_LIST_FILE=${sample%.*}.read_names_to_remove_highSoftClip.txt FILTER=excludeReadList # Exclude reads\n",
    "    else\n",
    "        echo \"There are none to much soft clipped reads with \" $MAX_SOFTCLIP \" % soft clipping for sample \" $sample. Continue without filtering.\n",
    "        mv ${sample%.*}.short.bam ${sample%.*}.scf.bam\n",
    "    fi\n",
    "\n",
    "    samtools index -@ $THREADS ${sample%.*}.scf.bam\n",
    "\n",
    "    # 4) Supplementary/chimeric alignment\n",
    "    samtools view -@ $THREADS -h -f 2048 -b ${sample%.*}.scf.bam > ${sample%.*}.sup.filtOut.bam & # get -F 2048 \"chimeric/non-linear alignments\" - might be interesting for translocations; possible overlap with hardclipping \n",
    "    samtools view -@ $THREADS -h -F 2048 -b ${sample%.*}.scf.bam > ${sample%.*}.sup.bam # remove -F 2048 \"chimeric/non-linear alignments\" - might be interesting for translocations; possible overlap with hardclipping\n",
    "\n",
    "    # 5) Hardclipped\n",
    "    samtools view -@ $THREADS -h ${sample%.*}.sup.bam | awk '$6 ~ /H/{print}' | samtools view -bh - > ${sample%.*}.hcf.filtOut.bam & # Get only hardclipped mappings; hardclipped alignments might mean the part of the alignment that is hardclipped might map to different part of the genome - possible chimeric reads? https://www.biostars.org/p/109333/\n",
    "    samtools view -@ $THREADS -h ${sample%.*}.sup.bam | awk '$6 !~ /H/{print}' | samtools view -bh - > ${sample%.*}.hcf.bam # https://www.biostars.org/p/137461/\n",
    "\n",
    "    # 6) MAPQ10 – very often repetitive alignments\n",
    "    samtools view -@ $THREADS -h ${sample%.*}.sup.bam | awk -v var=\"$MAPQ\" '$5 < var || $1 ~ /^@/' | samtools view -b - > ${sample%.*}.MAPQ${MAPQ}.filtOut.bam &\n",
    "    samtools view -@ $THREADS -h -bq $MAPQ ${sample%.*}.sup.bam > ${sample%.*}.MAPQ${MAPQ}.bam # BWA-MEM -T settings should filter all low MAPQ mappings; -F 2048 \"chimeric/non-linear alignments\" - might be interesting for translocations\n",
    "\n",
    "    # 7) MAPQ40 – Only high quality alignments\n",
    "    samtools view -@ $THREADS -h ${sample%.*}.MAPQ${MAPQ}.bam | awk -v var=\"$MAPQ_FINAL\" '$5 < var || $1 ~ /^@/' | samtools view -b - > ${sample%.*}.MAPQ${MAPQ_FINAL}.filtOut.bam &\n",
    "    samtools view -@ $THREADS -h -bq $MAPQ_FINAL ${sample%.*}.MAPQ${MAPQ}.bam > ${sample%.*}.MAPQ${MAPQ_FINAL}.bam # Get only very high quality mappings\n",
    "\n",
    "    # 8) Singletons\n",
    "    # Remove reads that remained as singletons after all filtering steps - works only for paired-end sequencing!\n",
    "    samtools view -@ $THREADS ${sample%.*}.MAPQ${MAPQ_FINAL}.bam | cut -f 1 | sort -T $OUTPUT_DIR/tmp --parallel=$THREADS | uniq -u > ${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt\n",
    "\n",
    "    if [ -s ${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt ]\n",
    "    then\n",
    "        echo \"Number of singleton reads after filtering is\" `wc -l ${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt` \"for sample \" $sample\n",
    "        picard FilterSamReads I=${sample%.*}.MAPQ${MAPQ_FINAL}.bam O=${sample%.*}.singletons.filtOut.bam READ_LIST_FILE=${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt FILTER=includeReadList & # Include reads\n",
    "        picard FilterSamReads I=${sample%.*}.MAPQ${MAPQ_FINAL}.bam O=${sample%.*}.MAPQ${MAPQ_FINAL}.bam.tmp READ_LIST_FILE=${sample%.*}.MAPQ${MAPQ_FINAL}.singleAfterFilt.txt FILTER=excludeReadList # Exclude reads\n",
    "        mv ${sample%.*}.MAPQ${MAPQ_FINAL}.bam.tmp ${sample%.*}.MAPQ${MAPQ_FINAL}.bam\n",
    "    else\n",
    "        echo \"There are none singleton reads after filtering for sample \" $sample. Continue without filtering.\n",
    "    fi\n",
    "\n",
    "    samtools index -@ $THREADS ${sample%.*}.MAPQ${MAPQ_FINAL}.bam\n",
    "\n",
    "    wait\n",
    "done\n",
    "\n",
    "for sample in $OUTPUT_DIR/alignment/*.bam\n",
    "do\n",
    "    samtools index -@ $THREADS $sample\n",
    "done\n",
    "\n",
    "# Do some cleaning\n",
    "rm $OUTPUT_DIR/alignment/*.read_names_to_remove_highSoftClip.txt $OUTPUT_DIR/alignment/*.reads $OUTPUT_DIR/alignment/*.singleAfterFilt.txt $OUTPUT_DIR/alignment/*.fail.txt\n",
    "\n",
    "mkdir -p $OUTPUT_DIR/alignment/filtered/filtOut\n",
    "mkdir -p $OUTPUT_DIR/alignment/filtered/filt\n",
    "\n",
    "mv $OUTPUT_DIR/alignment/*.filtOut.bam* $OUTPUT_DIR/alignment/filtered/filtOut/\n",
    "mv $OUTPUT_DIR/alignment/* $OUTPUT_DIR/alignment/filtered/filt/\n",
    "mv $OUTPUT_DIR/alignment/filtered/filt/*.MAPQ40.bam* $OUTPUT_DIR/alignment/filtered/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Please note that the filtering applied is rather strict. We might lose some true positive mappings but in this case, we rather focus on strongly supported alignments to be sure we are detecting what we want to detect."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Alignment statistics and reference genome coverage "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "One of the quality checks after the filtering and before all the other steps is the genome coverage statistics...and of course general mapping statistics. Outputs are described [here](#Alignment-statistics-and-coverage-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) Java memory size is set to 1200M\n",
      "Launching application...\n",
      "\n",
      "QualiMap v.2.2.2-dev\n",
      "Built on 2017-08-28 08:37\n",
      "\n",
      "Selected tool: bamqc\n",
      "Available memory (Mb): 32\n",
      "Max memory (Mb): 1118\n",
      "Mon Jul 01 23:18:37 CEST 2019\t\tWARNING\tOutput folder already exists, the results will be saved there\n",
      "\n",
      "Starting bam qc....\n",
      "Loading sam header...\n",
      "Loading locator...\n",
      "Loading reference...\n",
      "Number of windows: 400, effective number of windows: 400\n",
      "Chunk of reads size: 1000\n",
      "Number of threads: 12\n",
      "Processed 50 out of 400 windows...\n",
      "Processed 100 out of 400 windows...\n",
      "Processed 150 out of 400 windows...\n",
      "Processed 200 out of 400 windows...\n",
      "Processed 250 out of 400 windows...\n",
      "Processed 300 out of 400 windows...\n",
      "Processed 350 out of 400 windows...\n",
      "Processed 400 out of 400 windows...\n",
      "Total processed windows:400\n",
      "Number of reads: 787234\n",
      "Number of valid reads: 787234\n",
      "Number of correct strand reads:0\n",
      "\n",
      "Inside of regions...\n",
      "Num mapped reads: 787234\n",
      "Num mapped first of pair: 393617\n",
      "Num mapped second of pair: 393617\n",
      "Num singletons: 0\n",
      "Time taken to analyze reads: 7\n",
      "Computing descriptors...\n",
      "numberOfMappedBases: 105669066\n",
      "referenceSize: 1139569\n",
      "numberOfSequencedBases: 105665514\n",
      "numberOfAs: 24824813\n",
      "Computing per chromosome statistics...\n",
      "Computing histograms...\n",
      "Overall analysis time: 7\n",
      "end of bam qc\n",
      "Computing report...\n",
      "Writing PDF report...\n",
      "PDF file created successfully \n",
      "\n",
      "Finished\n",
      "Java memory size is set to 1200M\n",
      "Launching application...\n",
      "\n",
      "QualiMap v.2.2.2-dev\n",
      "Built on 2017-08-28 08:37\n",
      "\n",
      "Selected tool: bamqc\n",
      "Available memory (Mb): 32\n",
      "Max memory (Mb): 1118\n",
      "Mon Jul 01 23:18:47 CEST 2019\t\tWARNING\tOutput folder already exists, the results will be saved there\n",
      "\n",
      "Starting bam qc....\n",
      "Loading sam header...\n",
      "Loading locator...\n",
      "Loading reference...\n",
      "Number of windows: 400, effective number of windows: 400\n",
      "Chunk of reads size: 1000\n",
      "Number of threads: 12\n",
      "Processed 50 out of 400 windows...\n",
      "Processed 100 out of 400 windows...\n",
      "Processed 150 out of 400 windows...\n",
      "Processed 200 out of 400 windows...\n",
      "Processed 250 out of 400 windows...\n",
      "Processed 300 out of 400 windows...\n",
      "Processed 350 out of 400 windows...\n",
      "Processed 400 out of 400 windows...\n",
      "Total processed windows:400\n",
      "Number of reads: 787234\n",
      "Number of valid reads: 787234\n",
      "Number of correct strand reads:0\n",
      "\n",
      "Inside of regions...\n",
      "Num mapped reads: 787234\n",
      "Num mapped first of pair: 393617\n",
      "Num mapped second of pair: 393617\n",
      "Num singletons: 0\n",
      "Time taken to analyze reads: 8\n",
      "Computing descriptors...\n",
      "numberOfMappedBases: 105669066\n",
      "referenceSize: 1139569\n",
      "numberOfSequencedBases: 105665514\n",
      "numberOfAs: 24824813\n",
      "Computing per chromosome statistics...\n",
      "Computing histograms...\n",
      "Overall analysis time: 9\n",
      "end of bam qc\n",
      "Computing report...\n",
      "Writing HTML report...\n",
      "HTML report created successfully\n",
      "\n",
      "Finished\n",
      "(treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  configs = yaml.load(f)\n",
      "/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/lib/python2.7/site-packages/multiqc/utils/config.py:51: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.\n",
      "  sp = yaml.load(f)\n",
      "[WARNING]         multiqc : MultiQC Version v1.7 now available!\n",
      "[INFO   ]         multiqc : This is MultiQC v1.3\n",
      "[INFO   ]         multiqc : Template    : default\n",
      "[INFO   ]         multiqc : Searching '/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/qualimap/'\n",
      "\u001b[?25lSearching 50 files..  [####################################]  100%\u001b[?25h\n",
      "[INFO   ]        qualimap : Found 1 BamQC reports\n",
      "[INFO   ]         multiqc : Compressing plot data\n",
      "[INFO   ]         multiqc : Report      : ../../qc/qualimap/multiqc_report.html\n",
      "[INFO   ]         multiqc : Data        : ../../qc/qualimap/multiqc_data\n",
      "[INFO   ]         multiqc : MultiQC complete\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) [Mon Jul 01 23:19:07 CEST 2019] picard.sam.CreateSequenceDictionary REFERENCE=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa OUTPUT=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.dict    TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 23:19:07 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "[Mon Jul 01 23:19:07 CEST 2019] picard.sam.CreateSequenceDictionary done. Elapsed time: 0.00 minutes.\n",
      "Runtime.totalMemory()=514850816\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) Going to process files CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam\n",
      "(treponema) (treponema) INFO  23:19:10,296 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:19:10,299 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18 \n",
      "INFO  23:19:10,299 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute \n",
      "INFO  23:19:10,299 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk \n",
      "INFO  23:19:10,299 HelpFormatter - [Mon Jul 01 23:19:10 CEST 2019] Executing on Linux 4.4.0-148-generic amd64 \n",
      "INFO  23:19:10,299 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_192-b01 \n",
      "INFO  23:19:10,302 HelpFormatter - Program Args: -T DepthOfCoverage -R /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -I /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/filtered/input_bams.list -o /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/coverage/SS14.coverage -ct 3 -ct 5 -ct 10 \n",
      "INFO  23:19:10,311 HelpFormatter - Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01. \n",
      "INFO  23:19:10,311 HelpFormatter - Date/Time: 2019/07/01 23:19:10 \n",
      "INFO  23:19:10,311 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:19:10,311 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:19:10,321 GenomeAnalysisEngine - Strictness is SILENT \n",
      "INFO  23:19:10,415 GenomeAnalysisEngine - Downsampling Settings: No downsampling \n",
      "INFO  23:19:10,423 SAMDataSource$SAMReaders - Initializing SAMRecords in serial \n",
      "INFO  23:19:10,447 SAMDataSource$SAMReaders - Done initializing BAM readers: total time 0.02 \n",
      "INFO  23:19:10,607 GenomeAnalysisEngine - Preparing for traversal over 1 BAM files \n",
      "INFO  23:19:10,641 GenomeAnalysisEngine - Done preparing for traversal \n",
      "INFO  23:19:10,641 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] \n",
      "INFO  23:19:10,641 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining \n",
      "INFO  23:19:10,642 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime \n",
      "INFO  23:19:40,648 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:352765    344064.0    30.0 s      87.0 s       31.0%    96.0 s      66.0 s \n",
      "INFO  23:20:10,651 ProgressMeter - CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:756865    753664.0    60.0 s      79.0 s       66.4%    90.0 s      30.0 s \n",
      "INFO  23:20:38,635 DepthOfCoverage - Printing summary info \n",
      "INFO  23:20:38,643 DepthOfCoverage - Printing locus summary \n",
      "INFO  23:20:38,668 ProgressMeter -            done   1139569.0    88.0 s      77.0 s      100.0%    88.0 s       0.0 s \n",
      "INFO  23:20:38,668 ProgressMeter - Total runtime 88.03 secs, 1.47 min, 0.02 hours \n",
      "INFO  23:20:38,673 MicroScheduler - 0 reads were filtered out during the traversal out of approximately 785414 total reads (0.00%) \n",
      "INFO  23:20:38,674 MicroScheduler -   -> 0 reads (0.00% of total) failing BadCigarFilter \n",
      "INFO  23:20:38,674 MicroScheduler -   -> 0 reads (0.00% of total) failing DuplicateReadFilter \n",
      "INFO  23:20:38,674 MicroScheduler -   -> 0 reads (0.00% of total) failing FailsVendorQualityCheckFilter \n",
      "INFO  23:20:38,674 MicroScheduler -   -> 0 reads (0.00% of total) failing MalformedReadFilter \n",
      "INFO  23:20:38,675 MicroScheduler -   -> 0 reads (0.00% of total) failing NotPrimaryAlignmentFilter \n",
      "INFO  23:20:38,675 MicroScheduler -   -> 0 reads (0.00% of total) failing UnmappedReadFilter \n",
      "------------------------------------------------------------------------------------------\n",
      "Done. There were no warn messages.\n",
      "------------------------------------------------------------------------------------------\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "COVER_INT=\"-ct 3 -ct 5 -ct 10\" # Intervals of coverage to make the statistics for, can be multiple values; --summaryCoverageThreshold\n",
    "\n",
    "cd $OUTPUT_DIR/alignment/filtered\n",
    "\n",
    "# Get general alignment statistics\n",
    "mkdir $OUTPUT_DIR/qc/qualimap\n",
    "\n",
    "# Run Qualimap\n",
    "for sample in *.MAPQ${MAPQ_FINAL}.bam\n",
    "do\n",
    "    # PDF version is better for browsing\n",
    "    qualimap bamqc -bam $sample -nt $THREADS -c -outformat PDF -outdir $OUTPUT_DIR/qc/qualimap -outfile ${sample%.*}.qualimap.pdf\n",
    "    mv $OUTPUT_DIR/qc/qualimap/genome_results.txt $OUTPUT_DIR/qc/qualimap/${sample%.*}.genome_results.txt\n",
    "    # HTML version is necessary for multiQC\n",
    "    mkdir -p $OUTPUT_DIR/qc/qualimap/html/${sample%.*}\n",
    "    qualimap bamqc -bam $sample -nt $THREADS -c -outformat HTML -outdir $OUTPUT_DIR/qc/qualimap/html/${sample%.*} # Has to be redirected to separate folder for each file\n",
    "done\n",
    "\n",
    "multiqc -o $OUTPUT_DIR/qc/qualimap $OUTPUT_DIR/qc/qualimap/\n",
    "\n",
    "# Calculate the coverage\n",
    "mkdir -p $OUTPUT_DIR/qc/coverage\n",
    "\n",
    "# Prepare indexes\n",
    "samtools faidx $REFERENCE\n",
    "rm ${REFERENCE%.*}.dict\n",
    "picard CreateSequenceDictionary R=$REFERENCE O=${REFERENCE%.*}.dict\n",
    "\n",
    "# Run GATK DepthOfCoverage\n",
    "ls *.MAPQ40.bam | tr ' ' '\\n' > $OUTPUT_DIR/alignment/filtered//input_bams.list\n",
    "\n",
    "echo \"Going to process files\" `cat $OUTPUT_DIR/alignment/filtered//input_bams.list`\n",
    "\n",
    "gatk \\\n",
    "-T DepthOfCoverage \\\n",
    "-R $REFERENCE \\\n",
    "-I $OUTPUT_DIR/alignment/filtered/input_bams.list \\\n",
    "-o $OUTPUT_DIR/qc/coverage/$(basename $REFERENCE .fa)${INPUT_BAM%.bam}.coverage \\\n",
    "$COVER_INT # --outputFormat csv # Default is readable table (rtable)\n",
    "\n",
    "rm $OUTPUT_DIR/alignment/filtered/input_bams.list\n",
    "\n",
    "pigz -p $THREADS $OUTPUT_DIR/qc/coverage/*.coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Alignment downsampling (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In some cases, our resulting SAM/BAM is very deep which causes problems during the subsequent analysis. An option to lower down the computational demands is to downsample to a specific depth. Outputs are described [here](#Alignment-downsampling-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) INFO\t2019-07-01 23:57:05\tSortingCollection\tCreating merging iterator from 16 files\n",
      "[INFO][Biostar154220]Alloc memory for contig CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome N=1139569*sizeof(int)\n",
      "[INFO][Biostar154220]Last M70401:55:000000000-B24PV:1:2104:25239:14742;CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome;606,736 11 seconds\n",
      "[INFO][Biostar154220]. Completed. N=1,180,852. That took:18 seconds\n",
      "[INFO][SortSamRefName]done: N=787234\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "MAX_COVER=250 # Set maximum coverage to put the limit on\n",
    "\n",
    "for sample in *.filt.indelRealigned.dedup.MAPQ40.bam\n",
    "do\n",
    "    java -jar -Xmx4g $CONDA_PREFIX/bin/sortsamrefname.jar --tmpDir $OUTPUT_DIR/tmp $sample |  java -jar -Xmx4g $CONDA_PREFIX/bin/downsamplebam.jar -n $MAX_COVER | samtools sort -@ $THREADS - > ${sample%.bam*}.downsamp${MAX_COVER}.bam\n",
    "    samtools index -@ $THREADS ${sample%.bam*}.downsamp${MAX_COVER}.bam\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "\n",
    "### Alignment consensus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In this phase, we can generate the mapping consensus. Please note this step is closely related to the previous step as it will take only the filtered mappings into consideration. Here, we use a \"simple\" variant call performed by `samtools`&`bcftools` and `GATK`. You might consider alternating this step with a more sophisticated variant call if you think it is necessary. The difference is that the \"samtools\" version gives you lower-case letters where coverage was low and *N* where there was no coverage. Here, we convert all low-coverage positions to *n* (lower-case). The \"GATK\" version copies the reference sequence where is no coverage. \"samtools\" version doesn't like to include indels whereas \"GATK\" version should be OK with it. Outputs are described [here](#Alignment-consensus-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Fix reference name and SAM header (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "`bcftools` and `vcfutils` might have a problem with \".\" or any \"strange\" symbol in the reference name. If you haven't \"fix\" your reference name you can do it here but remember you have to replace it in the SAM/BAM header as well. Or go back to the beginning and redo the whole analysis (joke)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "An example of re-heading of a SAM/BAM file is provided below. Since we \"fixed\" the header at the beginning, we will not run this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Replace the reference name\n",
    "sed -i 's/gi_511533127_gb_CP004011\\.1__Treponema_pallidum_subsp\\._pallidum_SS14,_complete_genome/gi_511533127_gb_CP004011_1_Treponema_pallidum_subsp_pallidum_SS14_complete_genome/g' $REFERENCE \n",
    "\n",
    "# Replace the SAM header\n",
    "for sample in *.filt.indelRealigned.dedup.MAPQ40.bam\n",
    "do\n",
    "    samtools view -@ $THREADS -H $sample > $sample.header.sam  # extract header only\n",
    "\n",
    "    sed -i 's/gi_511533127_gb_CP004011\\.1__Treponema_pallidum_subsp\\._pallidum_SS14,_complete_genome/gi_511533127_gb_CP004011_1_Treponema_pallidum_subsp_pallidum_SS14_complete_genome/g' $sample.header.sam\n",
    "\n",
    "    samtools reheader $sample.header.sam $sample > $sample.tmp\n",
    "    mv $sample.tmp $sample\n",
    "\n",
    "    rm $sample.header.sam\n",
    "done\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### Mapping consensus (~ alignment-guided assembly)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) [Mon Jul 01 23:47:14 CEST 2019] picard.sam.CreateSequenceDictionary REFERENCE=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa OUTPUT=/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.dict    TRUNCATE_NAMES_AT_WHITESPACE=true NUM_SEQUENCES=2147483647 VERBOSITY=INFO QUIET=false VALIDATION_STRINGENCY=STRICT COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Mon Jul 01 23:47:14 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "[Mon Jul 01 23:47:14 CEST 2019] picard.sam.CreateSequenceDictionary done. Elapsed time: 0.00 minutes.\n",
      "Runtime.totalMemory()=514850816\n",
      "(treponema) (treponema) [mpileup] 1 samples in 1 input files\n",
      "INFO  23:50:30,413 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:50:30,416 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18 \n",
      "INFO  23:50:30,416 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute \n",
      "INFO  23:50:30,416 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk \n",
      "INFO  23:50:30,416 HelpFormatter - [Mon Jul 01 23:50:30 CEST 2019] Executing on Linux 4.4.0-148-generic amd64 \n",
      "INFO  23:50:30,417 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_192-b01 \n",
      "INFO  23:50:30,420 HelpFormatter - Program Args: -T FastaAlternateReferenceMaker -R /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -o /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/consensus/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.cns.gatk.fasta --variant /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/consensus/other/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.vcf \n",
      "INFO  23:50:30,432 HelpFormatter - Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01. \n",
      "INFO  23:50:30,433 HelpFormatter - Date/Time: 2019/07/01 23:50:30 \n",
      "INFO  23:50:30,433 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:50:30,433 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:50:30,443 GenomeAnalysisEngine - Strictness is SILENT \n",
      "INFO  23:50:30,537 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 \n",
      "INFO  23:50:37,484 RMDTrackBuilder - Writing Tribble index to disk for file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/consensus/other/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.vcf.idx \n",
      "INFO  23:50:38,101 GenomeAnalysisEngine - Preparing for traversal \n",
      "INFO  23:50:38,103 GenomeAnalysisEngine - Done preparing for traversal \n",
      "INFO  23:50:38,103 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] \n",
      "INFO  23:50:38,103 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining \n",
      "INFO  23:50:38,104 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime \n",
      "INFO  23:50:46,630 ProgressMeter -            done   1139569.0     8.0 s       7.0 s      100.0%     8.0 s       0.0 s \n",
      "INFO  23:50:46,632 ProgressMeter - Total runtime 8.53 secs, 0.14 min, 0.00 hours \n",
      "------------------------------------------------------------------------------------------\n",
      "Done. There were no warn messages.\n",
      "------------------------------------------------------------------------------------------\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "MIN_COVER=3 # All bases bellow this coverage are converted to ns\n",
    "PLOIDY=1 # Ploidy for the bcftools\n",
    "\n",
    "mkdir -p $OUTPUT_DIR/consensus/other\n",
    "\n",
    "# Make sure we have reference index and dictionary (! have to redo if you used the previous \"fix\" step)\n",
    "samtools faidx $REFERENCE\n",
    "rm ${REFERENCE%.*}.dict\n",
    "picard CreateSequenceDictionary R=$REFERENCE O=${REFERENCE%.*}.dict\n",
    "\n",
    "for sample in *.filt.indelRealigned.dedup.MAPQ40.bam\n",
    "do\n",
    "    samtools index -@ $THREADS $sample\n",
    "    samtools mpileup --max-depth 10000 -E --min-BQ 13 --fasta-ref $REFERENCE --min-MQ 40 -g -u --max-idepth 100000 --min-ireads 5 --gap-frac 0.002 --excl-flags UNMAP,SECONDARY,QCFAIL,DUP --output-tags DP,AD,ADF,ADR,SP,INFO/AD,INFO/ADF,INFO/ADR --reference $REFERENCE $sample | bcftools call --ploidy $PLOIDY -c --keep-masked-ref --output-type v --threads $THREADS - > $OUTPUT_DIR/consensus/other/${sample%.*}.vcf\n",
    "\n",
    "    cat $OUTPUT_DIR/consensus/other/${sample%.*}.vcf | vcfutils.pl vcf2fq -d $MIN_COVER | seqtk seq -A - > $OUTPUT_DIR/consensus/${sample%.*}.cns.def.fasta\n",
    "\n",
    "    # Convert all low-coverage (=lower-case) positions to \"n\"\n",
    "    sed -e '/^>/! s/[[:lower:]]/n/g' $OUTPUT_DIR/consensus/${sample%.*}.cns.def.fasta > $OUTPUT_DIR/consensus/${sample%.*}.cns.final.fasta\n",
    "\n",
    "    gatk -T FastaAlternateReferenceMaker -R $REFERENCE -o $OUTPUT_DIR/consensus/${sample%.*}.cns.gatk.fasta --variant $OUTPUT_DIR/consensus/other/${sample%.*}.vcf\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Variant call"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To get the most relevant variants directly from the alignments we could either use the variants used for the consensus generation or we can use probably more suitable tools for bacteria variant call such as [`freebayes`](https://github.com/ekg/freebayes). The reason why we don't use other tools than `samtools`&`bcftools` for the generation of consensus is the compatibility of the approach. Outputs are described [here](#Variant-call-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) Processing CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.bam\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) Processing  CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf\n",
      "INFO  23:59:37,969 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:59:37,972 HelpFormatter - The Genome Analysis Toolkit (GATK) v3.7-0-gcfedb67, Compiled 2016/12/12 11:21:18 \n",
      "INFO  23:59:37,972 HelpFormatter - Copyright (c) 2010-2016 The Broad Institute \n",
      "INFO  23:59:37,972 HelpFormatter - For support and documentation go to https://software.broadinstitute.org/gatk \n",
      "INFO  23:59:37,972 HelpFormatter - [Mon Jul 01 23:59:37 CEST 2019] Executing on Linux 4.4.0-148-generic amd64 \n",
      "INFO  23:59:37,972 HelpFormatter - OpenJDK 64-Bit Server VM 1.8.0_192-b01 \n",
      "INFO  23:59:37,975 HelpFormatter - Program Args: -T VariantFiltration -R /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -o /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/variants/freebayes/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.filt.vcf --variant:VCF CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf --filterExpression DP < 3 --filterName LowCoverage --filterExpression QUAL > 0 && QUAL < 20 --filterName VeryLowQual --filterExpression QUAL > 20 && QUAL < 50 --filterName LowQual --filterExpression QUAL == 0 --filterName TechnicalQual \n",
      "INFO  23:59:37,992 HelpFormatter - Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01. \n",
      "INFO  23:59:37,992 HelpFormatter - Date/Time: 2019/07/01 23:59:37 \n",
      "INFO  23:59:37,992 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:59:37,993 HelpFormatter - -------------------------------------------------------------------------------- \n",
      "INFO  23:59:38,011 GenomeAnalysisEngine - Strictness is SILENT \n",
      "INFO  23:59:38,105 GenomeAnalysisEngine - Downsampling Settings: Method: BY_SAMPLE, Target Coverage: 1000 \n",
      "INFO  23:59:38,603 RMDTrackBuilder - Writing Tribble index to disk for file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/variants/freebayes/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf.idx \n",
      "INFO  23:59:38,718 GenomeAnalysisEngine - Preparing for traversal \n",
      "INFO  23:59:38,719 GenomeAnalysisEngine - Done preparing for traversal \n",
      "INFO  23:59:38,720 ProgressMeter - [INITIALIZATION COMPLETE; STARTING PROCESSING] \n",
      "INFO  23:59:38,720 ProgressMeter -                 | processed |    time |    per 1M |           |   total | remaining \n",
      "INFO  23:59:38,721 ProgressMeter -        Location |     sites | elapsed |     sites | completed | runtime |   runtime \n",
      "INFO  23:59:39,715 ProgressMeter -            done      7687.0     0.0 s       2.2 m       99.9%     0.0 s       0.0 s \n",
      "INFO  23:59:39,716 ProgressMeter - Total runtime 1.00 secs, 0.02 min, 0.00 hours \n",
      "------------------------------------------------------------------------------------------\n",
      "Done. There were no warn messages.\n",
      "------------------------------------------------------------------------------------------\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "MIN_DP=$MIN_COVER # Set minimal coverage depth of the mapping\n",
    "QUAL_SET=50 # Set minimal variant quality for the most certain variants; default it 20\n",
    "\n",
    "mkdir -p $OUTPUT_DIR/variants/freebayes\n",
    "mkdir $OUTPUT_DIR/qc/variants\n",
    "\n",
    "for sample in *filt.indelRealigned.dedup.MAPQ40.bam\n",
    "do\n",
    "    echo \"Processing $sample\"\n",
    "    \n",
    "    samtools index $sample\n",
    "\n",
    "    freebayes --bam $sample --vcf $OUTPUT_DIR/variants/freebayes/${sample%.*}.full.vcf --fasta-reference $REFERENCE --theta 0.001 --ploidy $PLOIDY --min-mapping-quality 10 --min-base-quality 15 --min-alternate-fraction 0.01 --min-alternate-count 2 --genotype-qualities --report-all-haplotype-alleles # You can add \"--report-monomorphic\" to get all positions\n",
    "done\n",
    "\n",
    "# Variant post-processing and filtering\n",
    "cd $OUTPUT_DIR/variants/freebayes/\n",
    "\n",
    "for sample in *full.vcf\n",
    "do\n",
    "    echo \"Processing \" $sample\n",
    "\n",
    "    # GATK filtering\n",
    "    gatk -T VariantFiltration -R $REFERENCE -o $OUTPUT_DIR/variants/freebayes/${sample%.full*}.filt.vcf --variant:VCF $sample --filterExpression \"DP < ${MIN_DP}\" --filterName \"LowCoverage\" --filterExpression \"QUAL > 0 && QUAL < 20\" --filterName \"VeryLowQual\" --filterExpression \"QUAL > 20 && QUAL < ${QUAL_SET}\" --filterName \"LowQual\" --filterExpression \"QUAL == 0\" --filterName \"TechnicalQual\" # This might be too strict for FreeBayes\n",
    "    # gatk -T VariantFiltration -R $REFERENCE -o ${sample%.full*}.filt.vcf --variant:VCF $sample --filterExpression \"DP < ${MIN_DP}\" --filterName \"LowCoverage\" --filterExpression \"QUAL > 0 && QUAL < 20\" --filterName \"VeryLowQual\" --filterExpression \"QUAL == 0\" --filterName \"TechnicalQual\" # This would be the way FreeBayes author recommends to use the filtering\n",
    "\n",
    "    mv ${sample%.full*}.filt.vcf $sample\n",
    "    rm ${sample%.full*}.filt.vcf.idx\n",
    "    \n",
    "    # Get only variants (PASS filtering)\n",
    "    grep \"^#\" $sample > ${sample%.full*}.var.vcf # Get only header\n",
    "    awk '$7 == \"PASS\" {print $0}' $sample >> ${sample%.full*}.var.vcf\n",
    "\n",
    "    # Get only SNPs and/or indels\n",
    "#    vcffilter -f \"TYPE = snp\" ${sample%.full*}.var.vc > ${sample%.full*}.var.snp.vcf # Get only SNP\n",
    "#    vcffilter -f \"( TYPE = ins | TYPE = del )\" ${sample%.full*}.var.vcf > ${sample%.full*}.var.indel.vcf # Get only indels\n",
    "\n",
    "    # Convert vct to tab vcftools\n",
    "    cat ${sample%.full*}.var.vcf | vcf-to-tab > ${sample%.full*}.var.tsv\n",
    "\n",
    "    # Get stats of the variant calling\n",
    "    # Stats from raw and filtered SNPs\n",
    "    vcfstats $sample > $OUTPUT_DIR/qc/variants/${sample%.vcf}.stats.txt\n",
    "    vcfstats ${sample%.full*}.var.vcf > $OUTPUT_DIR/qc/variants/${sample%.full*}.var.stats.txt\n",
    "\n",
    "    # Compare ts/tv ration between high/low quality variants\n",
    "    echo $sample > $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    " \n",
    "    echo \"Low quality variants\" >> $OUTPUT_DIR/qc/variants/${sample%.*}.ratio.stats.txt # Low quality variants\n",
    "    vcffilter -f \"QUAL < 20\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "    echo \"High quality variants\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt # High quality variants\n",
    "    vcffilter -f \"QUAL > 20\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "    echo \"Custom quality variants - $QUAL_SET\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt # Custom quality variants\n",
    "    vcffilter -f \"QUAL > $QUAL_SET\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) Processing  CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd $OUTPUT_DIR/variants/freebayes/\n",
    "\n",
    "for sample in *full.vcf\n",
    "do\n",
    "    echo \"Processing \" $sample\n",
    "\n",
    "    # Compare ts/tv ration between high/low quality variants\n",
    "    echo $sample > $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    " \n",
    "    echo \"Low quality variants\" >> $OUTPUT_DIR/qc/variants/${sample%.*}.ratio.stats.txt # Low quality variants\n",
    "    vcffilter -f \"QUAL < 20\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "    echo \"High quality variants\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt # High quality variants\n",
    "    vcffilter -f \"QUAL > 20\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "    echo \"Custom quality variants - $QUAL_SET\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt # Custom quality variants\n",
    "    vcffilter -f \"QUAL > $QUAL_SET\" $sample | vcfstats | grep \"ts\\|bial\" >> $OUTPUT_DIR/qc/variants/${sample%.vcf}.ratio.stats.txt\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Variant annotation (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Having variants in VCF is nice but often you might be interested in what is their effect or purpose. Unless you immediately know the important positions variant annotation might bring you some additional information. [SnpEff](http://snpeff.sourceforge.net/) allows you to build your own annotation database and apply it to your VCF. Outputs are described [here](#Variant-annotation-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, we build the annotation database. This has to be done only once per genome. Note: there are some pre-loaded annotations in SnpEff but since we might be using different genome version it might be safer to generate our own database. You can generate the database with GFF and FASTA (hint [here](http://lab.loman.net/2012/11/16/how-to-get-snpeff-working-with-bacterial-genomes-from-ncbi/)) or you can do it from **GenBank full** format (suffix **.gbff** at the GenBank FTP). The advantage of going for **GenBank full** is that both gene annotation and genome sequence is included in one file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) --2019-07-02 02:34:49--  ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.gbff.gz\n",
      "           => ‘/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCA_000410555.1_ASM41055v1_genomic.gbff.gz’\n",
      "Resolving ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)... 130.14.250.13, 2607:f220:41e:250::7\n",
      "Connecting to ftp.ncbi.nlm.nih.gov (ftp.ncbi.nlm.nih.gov)|130.14.250.13|:21... connected.\n",
      "Logging in as anonymous ... Logged in!\n",
      "==> SYST ... done.    ==> PWD ... done.\n",
      "==> TYPE I ... done.  ==> CWD (1) /genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1 ... done.\n",
      "==> SIZE GCA_000410555.1_ASM41055v1_genomic.gbff.gz ... 748077\n",
      "==> PASV ... done.    ==> RETR GCA_000410555.1_ASM41055v1_genomic.gbff.gz ... done.\n",
      "Length: 748077 (731K) (unauthoritative)\n",
      "\n",
      "GCA_000410555.1_ASM 100%[===================>] 730.54K  1.41MB/s    in 0.5s    \n",
      "\n",
      "2019-07-02 02:34:51 (1.41 MB/s) - ‘/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/GCA_000410555.1_ASM41055v1_genomic.gbff.gz’ saved [748077]\n",
      "\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/\n",
      "(treponema) (treponema) (treponema) (treponema) 00:00:00\tSnpEff version SnpEff 4.3t (build 2017-11-24 10:18), by Pablo Cingolani\n",
      "00:00:00\tCommand: 'build'\n",
      "00:00:00\tBuilding database for 'SS14_CP004011.1'\n",
      "00:00:00\tReading configuration file 'snpEff.config'. Genome: 'SS14_CP004011.1'\n",
      "00:00:00\tReading config file: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/snpEff.config\n",
      "00:00:00\tdone\n",
      "Chromosome: 'CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome'\tlength: 1139569\n",
      "\n",
      "\tCreate exons from CDS (if needed): ...............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................\n",
      "\tExons created for 975 transcripts.\n",
      "\n",
      "\tDeleting redundant exons (if needed): \n",
      "\t\tTotal transcripts with deleted exons: 0\n",
      "\n",
      "\tCollapsing zero length introns (if needed): \n",
      "\t\tTotal collapsed transcripts: 0\n",
      "\t\tAdding genomic sequences to exons: \tDone (975 sequences added, 0 ignored).\n",
      "\n",
      "\tAdjusting transcripts: \n",
      "\tAdjusting genes: \n",
      "\tAdjusting chromosomes lengths: \n",
      "\tRanking exons: \n",
      "\tCreate UTRs from CDS (if needed): \n",
      "\tRemove empty chromosomes: \n",
      "\n",
      "\tMarking as 'coding' from CDS information: \n",
      "\tDone: 0 transcripts marked\n",
      "00:00:00\tCaracterizing exons by splicing (stage 1) : \n",
      "\t\n",
      "00:00:00\tCaracterizing exons by splicing (stage 2) : \n",
      "\t00:00:00\tdone.\n",
      "00:00:00\t[Optional] Rare amino acid annotations\n",
      "00:00:00\tWarning: Cannot read optional protein sequence file '/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/./data/SS14_CP004011.1/protein.fa', nothing done.\n",
      "00:00:00\tProtein check file: '/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/./data/SS14_CP004011.1/genes.gbk'\n",
      "\n",
      "00:00:00\tChecking database using protein sequences\n",
      "00:00:00\tComparing Proteins...\n",
      "\tLabels:\n",
      "\t\t'+' : OK\n",
      "\t\t'.' : Missing\n",
      "\t\t'*' : Error\n",
      "\t+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++*+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\t++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "\n",
      "\tProtein check:\tSS14_CP004011.1\tOK: 974\tNot found: 0\tErrors: 1\tError percentage: 0.10256410256410256%\n",
      "00:00:01\tSaving database\n",
      "00:00:01\t[Optional] Reading regulation elements: GFF\n",
      "00:00:01\tWarning: Cannot read optional regulation file '/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/./data/SS14_CP004011.1/regulation.gff', nothing done.\n",
      "00:00:01\t[Optional] Reading regulation elements: BED \n",
      "00:00:01\tCannot find optional regulation dir '/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/./data/SS14_CP004011.1/regulation.bed/', nothing done.\n",
      "00:00:01\t[Optional] Reading motifs: GFF\n",
      "00:00:01\tWarning: Cannot open PWMs file /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/snpeff-4.3.1t-2/./data/SS14_CP004011.1/pwms.bin. Nothing done\n",
      "00:00:01\tDone\n",
      "00:00:01\tLogging\n",
      "00:00:02\tChecking for updates...\n",
      "\n",
      "\n",
      "NEW VERSION!\n",
      "\tThere is a new SnpEff version available: \n",
      "\t\tVersion      : 4.4\n",
      "\t\tRelease date : 2019-01-26\n",
      "\t\tDownload URL : http://sourceforge.net/projects/snpeff/files/snpEff_latest_core.zip\n",
      "\n",
      "00:00:03\tDone.\n",
      "(treponema) (treponema) #-----------------------------------------------\n",
      "# Genome name                : 'Treponema pallidum subsp. pallidum SS14'\n",
      "# Genome version             : 'SS14_CP004011.1'\n",
      "# Genome ID                  : 'SS14_CP004011.1[0]'\n",
      "# Has protein coding info    : true\n",
      "# Has Tr. Support Level info : true\n",
      "# Genes                      : 1032\n",
      "# Protein coding genes       : 975\n",
      "#-----------------------------------------------\n",
      "# Transcripts                : 975\n",
      "# Avg. transcripts per gene  : 0.94\n",
      "# TSL transcripts            : 0\n",
      "#-----------------------------------------------\n",
      "# Checked transcripts        : \n",
      "#               AA sequences :    974 ( 99.90% )\n",
      "#              DNA sequences :      0 ( 0.00% )\n",
      "#-----------------------------------------------\n",
      "# Protein coding transcripts : 975\n",
      "#              Length errors :      0 ( 0.00% )\n",
      "#  STOP codons in CDS errors :      0 ( 0.00% )\n",
      "#         START codon errors :      0 ( 0.00% )\n",
      "#        STOP codon warnings :      0 ( 0.00% )\n",
      "#              UTR sequences :      0 ( 0.00% )\n",
      "#               Total Errors :      0 ( 0.00% )\n",
      "# WARNING                    : No protein coding transcript has UTR\n",
      "#-----------------------------------------------\n",
      "# Cds                        : 975\n",
      "# Exons                      : 975\n",
      "# Exons with sequence        : 975\n",
      "# Exons without sequence     : 0\n",
      "# Avg. exons per transcript  : 1.00\n",
      "#-----------------------------------------------\n",
      "# Number of chromosomes      : 1\n",
      "# Chromosomes                : Format 'chromo_name size codon_table'\n",
      "#\t\t'CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome'\t1139569\tBacterial_and_Plant_Plastid\n",
      "#-----------------------------------------------\n",
      "\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:3-1397, strand:1, id:TPASS_20001, name:dnaA\n",
      "Transcipts:\n",
      "\tCP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:3-1397, strand: +, id:TPASS_20001, Protein, AA check\n",
      "\t\tExons:\n",
      "\t\tCP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:3-1397 'EXON_CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome_3_1397', rank: 1, frame: ., sequence: atggacgcagtagggtatgaagtattctggaacgagacactcagccagatacggagtgaatcgaccgaagcagaatttaacatgtggtttgctcatttgttctttatcgcatcttttgaaaacgctatcgaaatagcagtaccttcagactttttccgaatacagtttagccaaaaatatcaagaaaagcttgagcgcaagttcctcgaactttctggacaccccattaaacttttgtttgccgttaaaaaaggcacccctcatggaaatactgctccccccaaacacgtgcatacctacctggagaaaaactctcctgcagaggttccttccaaaaagagctttcaccccgacctgaacagagactataccttcgagaactttgtatccggagaagaaaccaaattcagccatagcgctgctatctccgtatcaaaaaacccaggcacttcctacaatccgttacttatctacggtggagtgggactaggaaaaacccaccttatgcaggctattggacacgagatctacaagacaacagacctgaacgtcatatacgtcactgcggagaattttggaaatgaattcatttccacattactcaataaaaagacccaggattttaaaaaaaaataccgctacaccgcggatgtacttcttatagatgacattcatttttttgaaaacaaagacggattacaagaagagcttttctatacgttcaacgaacttttcgagaaaaaaaaacaaattatctttacctgcgacaggcctgtacaagaattgaaaaatctctcttctcgcttacgctcgaggtgctcccgagggcttagcactgatctgaatatgccatgttttgaaacgcgctgtgctatcttgattaaaaaaatacaaaactataacagcacctatcctcacaaagccatccacatttcagacgatgttgtccgacttgtttctgaaaacatttcttcaaatatcagggatcttgagggggcattaacaaaaattatcgctttcattgaagtgtcgggatccatcacgatagatatcgttccctctctcctaaaagagttcttcctctctgcaaggccaaaacacatcacagtagaaactattcttcatgtagttgcagatcactttaacatttcgtattcagatctaaagggtaagaaacgcaataaaagcgttgtttatcctcggcaaatcgctatgtttctctcaaaggaactgacagagctctccactactgaacttggtatcgaatttggtggcagagatcattcaaccgtcatttacggatgtcaaaaaatagaaggagaaattctcactaatccttcgttacaggcaaatcttgatttgctgaaaagtaaagttcaagattcaatccgctag\n",
      "\t\tCDS     :\tatggacgcagtagggtatgaagtattctggaacgagacactcagccagatacggagtgaatcgaccgaagcagaatttaacatgtggtttgctcatttgttctttatcgcatcttttgaaaacgctatcgaaatagcagtaccttcagactttttccgaatacagtttagccaaaaatatcaagaaaagcttgagcgcaagttcctcgaactttctggacaccccattaaacttttgtttgccgttaaaaaaggcacccctcatggaaatactgctccccccaaacacgtgcatacctacctggagaaaaactctcctgcagaggttccttccaaaaagagctttcaccccgacctgaacagagactataccttcgagaactttgtatccggagaagaaaccaaattcagccatagcgctgctatctccgtatcaaaaaacccaggcacttcctacaatccgttacttatctacggtggagtgggactaggaaaaacccaccttatgcaggctattggacacgagatctacaagacaacagacctgaacgtcatatacgtcactgcggagaattttggaaatgaattcatttccacattactcaataaaaagacccaggattttaaaaaaaaataccgctacaccgcggatgtacttcttatagatgacattcatttttttgaaaacaaagacggattacaagaagagcttttctatacgttcaacgaacttttcgagaaaaaaaaacaaattatctttacctgcgacaggcctgtacaagaattgaaaaatctctcttctcgcttacgctcgaggtgctcccgagggcttagcactgatctgaatatgccatgttttgaaacgcgctgtgctatcttgattaaaaaaatacaaaactataacagcacctatcctcacaaagccatccacatttcagacgatgttgtccgacttgtttctgaaaacatttcttcaaatatcagggatcttgagggggcattaacaaaaattatcgctttcattgaagtgtcgggatccatcacgatagatatcgttccctctctcctaaaagagttcttcctctctgcaaggccaaaacacatcacagtagaaactattcttcatgtagttgcagatcactttaacatttcgtattcagatctaaagggtaagaaacgcaataaaagcgttgtttatcctcggcaaatcgctatgtttctctcaaaggaactgacagagctctccactactgaacttggtatcgaatttggtggcagagatcattcaaccgtcatttacggatgtcaaaaaatagaaggagaaattctcactaatccttcgttacaggcaaatcttgatttgctgaaaagtaaagttcaagattcaatccgctag\n",
      "\t\tProtein :\tMDAVGYEVFWNETLSQIRSESTEAEFNMWFAHLFFIASFENAIEIAVPSDFFRIQFSQKYQEKLERKFLELSGHPIKLLFAVKKGTPHGNTAPPKHVHTYLEKNSPAEVPSKKSFHPDLNRDYTFENFVSGEETKFSHSAAISVSKNPGTSYNPLLIYGGVGLGKTHLMQAIGHEIYKTTDLNVIYVTAENFGNEFISTLLNKKTQDFKKKYRYTADVLLIDDIHFFENKDGLQEELFYTFNELFEKKKQIIFTCDRPVQELKNLSSRLRSRCSRGLSTDLNMPCFETRCAILIKKIQNYNSTYPHKAIHISDDVVRLVSENISSNIRDLEGALTKIIAFIEVSGSITIDIVPSLLKEFFLSARPKHITVETILHVVADHFNISYSDLKGKKRNKSVVYPRQIAMFLSKELTELSTTELGIEFGGRDHSTVIYGCQKIEGEILTNPSLQANLDLLKSKVQDSIR*\n",
      "\n",
      "\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:1640-2755, strand:1, id:TPASS_20002, name:dnaN\n",
      "Transcipts:\n",
      "\tCP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:1640-2755, strand: +, id:TPASS_20002, Protein, AA check\n",
      "\t\tExons:\n",
      "\n",
      "\n",
      "NEW VERSION!\n",
      "\tThere is a new SnpEff version available: \n",
      "\t\tVersion      : 4.4\n",
      "\t\tRelease date : 2019-01-26\n",
      "\t\tDownload URL : http://sourceforge.net/projects/snpeff/files/snpEff_latest_core.zip\n",
      "\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "# Get the GenBank full annotation - reference genome version and the gene annotation must be identical!\n",
    "wget -P $REFERENCES ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/bacteria/Treponema_pallidum/all_assembly_versions/GCA_000410555.1_ASM41055v1/GCA_000410555.1_ASM41055v1_genomic.gbff.gz\n",
    "gunzip $REFERENCES/GCA_000410555.1_ASM41055v1_genomic.gbff.gz\n",
    "\n",
    "cd $REFERENCES/\n",
    "\n",
    "DB_NAME=\"SS14_CP004011.1\" # How you want to call the database?\n",
    "fasta_header=`head -1 $REFERENCE | sed 's/>//g' | cut -d ' ' -f 1` # Get only the fasta header without the initial \">\" and split after first empty space\n",
    "\n",
    "# Create custom snpEff.config\n",
    "echo \"$DB_NAME.genome : Treponema pallidum subsp. pallidum SS14\" > ${REFERENCE%.fa*}.snpEffect.config # Database name and description\n",
    "echo -e \"\\\\t$DB_NAME.chromosomes : $fasta_header\" >> ${REFERENCE%.fa*}.snpEffect.config # Database name and fasta header\n",
    "echo -e \"\\\\t$DB_NAME.$fasta_header.codonTable : Bacterial_and_Plant_Plastid\" >> ${REFERENCE%.fa*}.snpEffect.config # Database name, fasta header and codon table\n",
    "\n",
    "SNPEFF_DIR=$(ls -d $CONDA_PREFIX/share/snpeff-*/)\n",
    "echo $SNPEFF_DIR\n",
    "ANNOT_DIR=$SNPEFF_DIR/data/$DB_NAME # Data directory to store the added genome annotation\n",
    "\n",
    "# Make the annotation database - only once per genome!\n",
    "if [ ! -d \"$ANNOT_DIR\" ]; then  # Check if the genome folder already exists\n",
    "    mkdir -p $ANNOT_DIR\n",
    "    cd $ANNOT_DIR/\n",
    "\n",
    "    cat $REFERENCES/GCA_000410555.1_ASM41055v1_genomic.gbff > $ANNOT_DIR/genes.gbk\n",
    "\n",
    "    seq_name=`head -1 $ANNOT_DIR/genes.gbk | awk '{print $2}'` # Cheat a little and replace chromosome id with out sequence name - will work only with a single chromosome reference!!!\n",
    "    sed -i \"s/$seq_name/$fasta_header/g\" $ANNOT_DIR/genes.gbk\n",
    " \n",
    "    cat ${REFERENCE%.fa*}.snpEffect.config >> $SNPEFF_DIR/snpEff.config\n",
    "\n",
    "    cd $SNPEFF_DIR/\n",
    "    snpEff build -genbank -v $DB_NAME\n",
    "fi\n",
    "\n",
    "snpEff dump $DB_NAME | head -50 # Check if everything is OK "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "When we have the database ready, we can run the variant effect prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/variants/freebayes/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf\n",
      "\n",
      "\n",
      "NEW VERSION!\n",
      "\tThere is a new SnpEff version available: \n",
      "\t\tVersion      : 4.4\n",
      "\t\tRelease date : 2019-01-26\n",
      "\t\tDownload URL : http://sourceforge.net/projects/snpeff/files/snpEff_latest_core.zip\n",
      "\n",
      "/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/variants/freebayes/CW56_S1_.SS14.filt.indelRealigned.dedup.MAPQ40.var.vcf\n",
      "\n",
      "\n",
      "NEW VERSION!\n",
      "\tThere is a new SnpEff version available: \n",
      "\t\tVersion      : 4.4\n",
      "\t\tRelease date : 2019-01-26\n",
      "\t\tDownload URL : http://sourceforge.net/projects/snpeff/files/snpEff_latest_core.zip\n",
      "\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cd $SNPEFF_DIR/\n",
    "\n",
    "for sample in $OUTPUT_DIR/variants/freebayes/*.vcf\n",
    "do\n",
    "    echo $sample\n",
    "    \n",
    "    snpEff $DB_NAME $sample > ${sample%.vcf}.$DB_NAME.annot.vcf\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### *De novo* assembly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "To accompany the alignment-guided assembly and to balance for the potential pitfalls of that approach we strongly recommend to run *de novo* assembly as well. You can run *de novo* assembly on the host-cleaned data but since we assume there could be a mixture of organisms present we recommend to run the assembly only from the reads mapped to the reference. Probably the most suitable set at the mapped reads from the \"rough\" initial mapping with not filtering applied. Outputs are described [here](#De-novo-assembly-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### SAM/BAM to fastq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "In the preparation step, we first extract the mapped reads from the SAM/BAM."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) Processing file CW56_S1_.SS14.bam\n",
      "[Tue Jul 02 02:41:04 CEST 2019] picard.sam.FixMateInformation INPUT=[CW56_S1_.SS14.mapped.bam] OUTPUT=tmp.bam VALIDATION_STRINGENCY=LENIENT    ASSUME_SORTED=false ADD_MATE_CIGAR=true IGNORE_MISSING_MATES=true VERBOSITY=INFO QUIET=false COMPRESSION_LEVEL=5 MAX_RECORDS_IN_RAM=500000 CREATE_INDEX=false CREATE_MD5_FILE=false GA4GH_CLIENT_SECRETS=client_secrets.json\n",
      "[Tue Jul 02 02:41:04 CEST 2019] Executing as 325073@BioDA-server on Linux 4.4.0-148-generic amd64; OpenJDK 64-Bit Server VM 1.8.0_192-b01; Picard version: 2.9.2-SNAPSHOT\n",
      "INFO\t2019-07-02 02:41:04\tFixMateInformation\tSorting input into queryname order.\n",
      "INFO\t2019-07-02 02:41:59\tFixMateInformation\tSorting by queryname complete.\n",
      "INFO\t2019-07-02 02:41:59\tFixMateInformation\tOutput will be sorted by coordinate\n",
      "INFO\t2019-07-02 02:41:59\tFixMateInformation\tTraversing query name sorted records and fixing up mate pair information.\n",
      "INFO\t2019-07-02 02:42:08\tFixMateInformation\tProcessed     1,000,000 records.  Elapsed time: 00:00:08s.  Time for last 1,000,000:    8s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:861,691\n",
      "INFO\t2019-07-02 02:42:21\tFixMateInformation\tProcessed     2,000,000 records.  Elapsed time: 00:00:22s.  Time for last 1,000,000:   13s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:823,603\n",
      "INFO\t2019-07-02 02:42:35\tFixMateInformation\tProcessed     3,000,000 records.  Elapsed time: 00:00:36s.  Time for last 1,000,000:   13s.  Last read position: CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome:515,805\n",
      "INFO\t2019-07-02 02:42:47\tFixMateInformation\tFinished processing reads; re-sorting output file.\n",
      "[Tue Jul 02 02:43:28 CEST 2019] picard.sam.FixMateInformation done. Elapsed time: 2.39 minutes.\n",
      "Runtime.totalMemory()=822607872\n",
      "[M::bam2fq_mainloop_singletontrack] discarded 0 singletons\n",
      "[M::bam2fq_mainloop_singletontrack] processed 3960606 reads\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/alignment/fastq\n",
    "\n",
    "cd $OUTPUT_DIR/alignment/filtered/filt/\n",
    "\n",
    "for sample in *.$(basename $REFERENCE .fa).bam\n",
    "do\n",
    "    echo \"Processing file $sample\"\n",
    "\n",
    "    samtools view -H $sample > ${sample%.*}.sam\n",
    "\n",
    "    samtools view -@ $THREADS -F 4 -f 3 $sample  >> ${sample%.*}.sam\n",
    "    samtools view -@ $THREADS -b ${sample%.*}.sam | samtools sort -@ $THREADS -o ${sample%.*}.mapped.bam -\n",
    "    rm ${sample%.*}.sam\n",
    "\n",
    "    picard FixMateInformation I=${sample%.*}.mapped.bam O=tmp.bam VALIDATION_STRINGENCY=LENIENT\n",
    "    mv tmp.bam ${sample%.*}.mapped.bam\n",
    "    samtools index -@ $THREADS ${sample%.*}.mapped.bam\n",
    "\n",
    "    samtools collate -O -@ $THREADS ${sample%.*}.mapped.bam - | samtools fastq -@ $THREADS -s $OUTPUT_DIR/alignment/fastq/${sample%.*}_unpaired.mapped.fastq -1 $OUTPUT_DIR/alignment/fastq/${sample%.*}_R1.mapped.fastq -2 $OUTPUT_DIR/alignment/fastq/${sample%.*}_R2.mapped.fastq -\n",
    "done\n",
    "\n",
    "cd $OUTPUT_DIR/alignment/fastq/\n",
    "\n",
    "rm *_unpaired.mapped.fastq\n",
    "\n",
    "for sample in *.fastq\n",
    "do\n",
    "    pigz -p $THREADS $sample\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "#### *De novo* assembly"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can do the *de novo* assembly itself. In this section, we are using SPAdes but you can consider using [Unicycler](https://github.com/rrwick/Unicycler) or [metaSPADES](http://cab.spbu.ru/software/meta-spades/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) Command line: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/spades.py\t--careful\t--threads\t12\t--cov-cutoff\t5\t--tmp-dir\t/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp\t-k\t15,21,27,33,55,77,99,127\t-1\t/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz\t-2\t/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\t-o\t/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_\t\n",
      "\n",
      "System information:\n",
      "  SPAdes version: 3.13.0\n",
      "  Python version: 2.7.15\n",
      "  OS: Linux-4.4.0-148-generic-x86_64-with-debian-stretch-sid\n",
      "\n",
      "Output dir: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_\n",
      "Mode: read error correction and assembling\n",
      "Debug mode is turned OFF\n",
      "\n",
      "Dataset parameters:\n",
      "  Multi-cell mode (you should set '--sc' flag if input data was obtained with MDA (single-cell) technology or --meta flag if processing metagenomic dataset)\n",
      "  Reads:\n",
      "    Library number: 1, library type: paired-end\n",
      "      orientation: fr\n",
      "      left reads: ['/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz']\n",
      "      right reads: ['/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz']\n",
      "      interlaced reads: not specified\n",
      "      single reads: not specified\n",
      "      merged reads: not specified\n",
      "Read error correction parameters:\n",
      "  Iterations: 1\n",
      "  PHRED offset will be auto-detected\n",
      "  Corrected reads will be compressed\n",
      "Assembly parameters:\n",
      "  k: [15, 21, 27, 33, 55, 77, 99, 127]\n",
      "  Repeat resolution is enabled\n",
      "  Mismatch careful mode is turned ON\n",
      "  MismatchCorrector will be used\n",
      "  Coverage cutoff is turned ON and threshold is 5.0\n",
      "Other parameters:\n",
      "  Dir for temp files: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp\n",
      "  Threads: 12\n",
      "  Memory limit (in Gb): 246\n",
      "\n",
      "\n",
      "======= SPAdes pipeline started. Log can be found here: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/spades.log\n",
      "\n",
      "\n",
      "===== Read error correction started. \n",
      "\n",
      "\n",
      "== Running read error correction tool: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-hammer /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/corrected/configs/config.info\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  75)   Starting BayesHammer, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  76)   Loading config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/corrected/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  78)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  86)   Trying to determine PHRED offset\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (main.cpp                  :  92)   Determined value is 33\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (hammer_tools.cpp          :  36)   Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (main.cpp                  : 113)   Size of aux. kmer data 24 bytes\n",
      "     === ITERATION 0 begins ===\n",
      "  0:00:00.002     4M / 4M    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 349525\n",
      "  0:00:00.038     6G / 6G    INFO   K-mer Splitting          (kmer_data.cpp             :  97)   Processing /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz\n",
      "  0:00:13.508     6G / 6G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 1980303 reads\n",
      "  0:00:13.509     6G / 6G    INFO   K-mer Splitting          (kmer_data.cpp             :  97)   Processing /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "  0:00:27.394     6G / 6G    INFO   K-mer Splitting          (kmer_data.cpp             : 107)   Processed 3960606 reads\n",
      "  0:00:27.394     6G / 6G    INFO   K-mer Splitting          (kmer_data.cpp             : 112)   Total 3960606 reads processed\n",
      "  0:00:27.950    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:28.586    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 40328484 kmers in total.\n",
      "  0:00:28.587    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:29.085    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:30.307    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:30.810    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 18708344 bytes occupied (3.71119 bits per kmer).\n",
      "  0:00:30.811    48M / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 356)   Arranging kmers in hash map order\n",
      "  0:00:32.742   664M / 6G    INFO    General                 (main.cpp                  : 148)   Clustering Hamming graph.\n",
      "  0:01:32.746   664M / 6G    INFO    General                 (main.cpp                  : 155)   Extracting clusters\n",
      "  0:02:02.034   664M / 6G    INFO    General                 (main.cpp                  : 167)   Clustering done. Total clusters: 11909008\n",
      "  0:02:02.057   356M / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 376)   Collecting K-mer information, this takes a while.\n",
      "  0:02:02.628     1G / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 382)   Processing /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz\n",
      "  0:02:47.775     1G / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 382)   Processing /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "  0:03:33.627     1G / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 389)   Collection done, postprocessing.\n",
      "  0:03:34.028     1G / 6G    INFO   K-mer Counting           (kmer_data.cpp             : 403)   There are 40328484 kmers in total. Among them 33569694 (83.2407%) are singletons.\n",
      "  0:03:34.028     1G / 6G    INFO    General                 (main.cpp                  : 173)   Subclustering Hamming graph\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 649)   Subclustering done. Total 2 non-read kmers were generated.\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 650)   Subclustering statistics:\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 651)     Total singleton hamming clusters: 9600580. Among them 232576 (2.42252%) are good\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 652)     Total singleton subclusters: 11750. Among them 11708 (99.6426%) are good\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 653)     Total non-singleton subcluster centers: 2341279. Among them 2279392 (97.3567%) are good\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 654)     Average size of non-trivial subcluster: 13.1244 kmers\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 655)     Average number of sub-clusters per non-singleton cluster: 1.01932\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 656)     Total solid k-mers: 2523676\n",
      "  0:03:38.548     1G / 6G    INFO   Hamming Subclustering    (kmer_cluster.cpp          : 657)     Substitution probabilities: [4,4]((0.939505,0.0300912,0.0177624,0.0126413),(0.0224521,0.954234,0.0103599,0.0129541),(0.0129237,0.0103271,0.954256,0.0224933),(0.012609,0.0177274,0.0300477,0.939616))\n",
      "  0:03:38.567     1G / 6G    INFO    General                 (main.cpp                  : 178)   Finished clustering.\n",
      "  0:03:38.567     1G / 6G    INFO    General                 (main.cpp                  : 197)   Starting solid k-mers expansion in 12 threads.\n",
      "  0:04:13.420     1G / 6G    INFO    General                 (main.cpp                  : 218)   Solid k-mers iteration 0 produced 60852 new k-mers.\n",
      "  0:04:47.446     1G / 6G    INFO    General                 (main.cpp                  : 218)   Solid k-mers iteration 1 produced 5884 new k-mers.\n",
      "  0:05:22.223     1G / 6G    INFO    General                 (main.cpp                  : 218)   Solid k-mers iteration 2 produced 0 new k-mers.\n",
      "  0:05:22.223     1G / 6G    INFO    General                 (main.cpp                  : 222)   Solid k-mers finalized\n",
      "  0:05:22.223     1G / 6G    INFO    General                 (hammer_tools.cpp          : 220)   Starting read correction in 12 threads.\n",
      "  0:05:22.223     1G / 6G    INFO    General                 (hammer_tools.cpp          : 233)   Correcting pair of reads: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz and /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "  0:05:32.227     2G / 6G    INFO    General                 (hammer_tools.cpp          : 168)   Prepared batch 0 of 1200000 reads.\n",
      "  0:05:56.831     2G / 6G    INFO    General                 (hammer_tools.cpp          : 175)   Processed batch 0\n",
      "  0:06:00.745     2G / 6G    INFO    General                 (hammer_tools.cpp          : 185)   Written batch 0\n",
      "  0:06:07.858     2G / 6G    INFO    General                 (hammer_tools.cpp          : 168)   Prepared batch 1 of 780303 reads.\n",
      "  0:06:23.007     2G / 6G    INFO    General                 (hammer_tools.cpp          : 175)   Processed batch 1\n",
      "  0:06:25.685     2G / 6G    INFO    General                 (hammer_tools.cpp          : 185)   Written batch 1\n",
      "  0:06:28.875     1G / 6G    INFO    General                 (hammer_tools.cpp          : 274)   Correction done. Changed 1398025 bases in 577934 reads.\n",
      "  0:06:28.875     1G / 6G    INFO    General                 (hammer_tools.cpp          : 275)   Failed to correct 0 bases out of 567567605.\n",
      "  0:06:29.048    48M / 6G    INFO    General                 (main.cpp                  : 255)   Saving corrected dataset description to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/corrected/corrected.yaml\n",
      "  0:06:29.049    48M / 6G    INFO    General                 (main.cpp                  : 262)   All done. Exiting.\n",
      "\n",
      "== Compressing corrected reads (with pigz)\n",
      "\n",
      "== Dataset description file was created: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/corrected/corrected.yaml\n",
      "\n",
      "\n",
      "===== Read error correction finished. \n",
      "\n",
      "\n",
      "===== Assembling started.\n",
      "\n",
      "\n",
      "== Running assembler: K15\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K15/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K15/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=15\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  77)   Converting reads to binary format for library #0 (takes a while)\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  78)   Converting paired reads\n",
      "  0:00:00.252   112M / 112M  INFO    General                 (binary_converter.hpp      :  93)   16384 reads processed\n",
      "  0:00:00.446   128M / 128M  INFO    General                 (binary_converter.hpp      :  93)   32768 reads processed\n",
      "  0:00:00.832   156M / 156M  INFO    General                 (binary_converter.hpp      :  93)   65536 reads processed\n",
      "  0:00:01.605   212M / 212M  INFO    General                 (binary_converter.hpp      :  93)   131072 reads processed\n",
      "  0:00:03.158   328M / 328M  INFO    General                 (binary_converter.hpp      :  93)   262144 reads processed\n",
      "  0:00:06.264   556M / 556M  INFO    General                 (binary_converter.hpp      :  93)   524288 reads processed\n",
      "  0:00:16.077   912M / 912M  INFO    General                 (binary_converter.hpp      :  93)   1048576 reads processed\n",
      "  0:00:32.539   196M / 912M  INFO    General                 (binary_converter.hpp      : 117)   1973079 reads written\n",
      "  0:00:32.715     4M / 912M  INFO    General                 (read_converter.hpp        :  87)   Converting single reads\n",
      "  0:00:32.915   152M / 912M  INFO    General                 (binary_converter.hpp      : 117)   7214 reads written\n",
      "  0:00:32.930     4M / 912M  INFO    General                 (read_converter.hpp        :  95)   Converting merged reads\n",
      "  0:00:33.069   148M / 912M  INFO    General                 (binary_converter.hpp      : 117)   0 reads written\n",
      "  0:00:33.081     4M / 912M  INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:33.081     4M / 912M  INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:33.081     4M / 912M  INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:33.081     4M / 912M  INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:33.082     4M / 912M  INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:33.082     4M / 912M  INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:33.082     4M / 912M  INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 466033\n",
      "  0:00:42.596     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:42.596     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:43.099    48M / 6G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:43.099    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:43.185    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2447403 kmers in total.\n",
      "  0:00:43.185    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:43.223    48M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:43.223    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:43.223    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:43.223    48M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:43.223    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:43.223    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 349525\n",
      "  0:00:43.468     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2447403 kmers\n",
      "  0:00:43.468     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2447403 kmers.\n",
      "  0:00:43.554    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:43.632    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2413146 kmers in total.\n",
      "  0:00:43.632    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:43.674    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:43.790    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:43.821    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1127384 bytes occupied (3.73747 bits per kmer).\n",
      "  0:00:43.824    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:43.961    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:43.961    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Early tip clipping\n",
      "  0:00:43.961    52M / 6G    INFO    General                 (construction.cpp          : 253)   Early tip clipper length bound set as (RL - K)\n",
      "  0:00:43.961    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 181)   Early tip clipping\n",
      "  0:00:44.265    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 184)   681219 16-mers were removed by early tip clipper\n",
      "  0:00:44.265    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:44.265    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:44.379    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 96849 sequences extracted\n",
      "  0:00:44.444    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:44.471    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 0 loops collected\n",
      "  0:00:44.506    56M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:44.506    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:44.506    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:44.577    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1141160 bytes occupied (3.73019 bits per kmer).\n",
      "  0:00:44.584    68M / 6G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:54.873    68M / 6G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:54.961    68M / 6G    INFO    General                 (construction.cpp          : 464)   Processed 193556 edges\n",
      "  0:00:54.972    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:54.972    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 14\n",
      "  0:00:54.972    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 411\n",
      "  0:00:54.972    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 428. Coverage mad: 137.882\n",
      "  0:00:54.972    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:55.193    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:55.746    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:57.745    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:01:01.798    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:01:04.234    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 445.013. Fitted coverage std. dev: 135.008\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.998592\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 196\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 196\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1105438\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 445.013\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 196\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 4.47661\n",
      "  0:01:04.245    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:01:04.245    52M / 6G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:01:04.250    52M / 6G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:01:04.250    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:01:04.258    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:01:04.258    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:01:04.258    52M / 6G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:01:04.258    52M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:01:04.258    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:01:04.258    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:04.273    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 433 times\n",
      "  0:01:04.273    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:05.591    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 17091 times\n",
      "  0:01:05.591    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.043    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 14004 times\n",
      "  0:01:06.043    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:01:06.043    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.057    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 540 times\n",
      "  0:01:06.057    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.071    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 97 times\n",
      "  0:01:06.071    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.075    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 156 times\n",
      "  0:01:06.075    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:01:06.075    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.075    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 5 times\n",
      "  0:01:06.075    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.076    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 6 times\n",
      "  0:01:06.076    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.077    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 56 times\n",
      "  0:01:06.077    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:01:06.077    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.077    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:01:06.077    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.078    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 4 times\n",
      "  0:01:06.078    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 27 times\n",
      "  0:01:06.079    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 4 times\n",
      "  0:01:06.079    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.080    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 27 times\n",
      "  0:01:06.080    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:01:06.080    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.080    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.080    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 3 times\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 15 times\n",
      "  0:01:06.080    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.080    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 15 times\n",
      "  0:01:06.081    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 19 times\n",
      "  0:01:06.081    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.081    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 28 times\n",
      "  0:01:06.082    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.082    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.083    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 36 times\n",
      "  0:01:06.083    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:01:06.083    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.084    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 6 times\n",
      "  0:01:06.084    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.098    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:01:06.098    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:01:06.100    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:01:06.100    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:01:06.100    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:01:06.100    52M / 6G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:01:06.101    52M / 6G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:01:06.101    52M / 6G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:01:06.101    52M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:01:06.101    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.101    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.101    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.114    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.114    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:01:06.115    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:01:06.115    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:01:06.128    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:01:06.128    52M / 6G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:01:06.131    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:01:06.132    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 2 times\n",
      "  0:01:06.132    52M / 6G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 4.47661 and less\n",
      "  0:01:06.132    52M / 6G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:01:06.134    52M / 6G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 449.72\n",
      "  0:01:06.134    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:01:06.134    52M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/assembly_graph_with_scaffolds.gfa\n",
      "  0:01:06.163    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/before_rr.fasta\n",
      "  0:01:06.205    52M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/assembly_graph.fastg\n",
      "  0:01:06.320    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/simplified_contigs.fasta\n",
      "  0:01:06.362    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/final_contigs.fasta\n",
      "  0:01:06.403    52M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:01:06.403    52M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/assembly_graph_with_scaffolds.gfa\n",
      "  0:01:06.431    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/before_rr.fasta\n",
      "  0:01:06.473    52M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/assembly_graph.fastg\n",
      "  0:01:06.588    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/simplified_contigs.fasta\n",
      "  0:01:06.654    52M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K15/final_contigs.fasta\n",
      "  0:01:06.709    52M / 6G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:01:06.744    48M / 6G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 1 minutes 6 seconds\n",
      "Max read length detected as 151\n",
      "\n",
      "== Running assembler: K21\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K21/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K21/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=21\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 466033\n",
      "  0:00:09.219     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:09.219     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:09.943    48M / 6G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:09.944    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.039    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2558992 kmers in total.\n",
      "  0:00:10.039    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.084    48M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:10.084    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:10.084    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:10.084    48M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:10.084    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:10.084    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 349525\n",
      "  0:00:10.357     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2558992 kmers\n",
      "  0:00:10.357     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2558992 kmers.\n",
      "  0:00:10.441    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.509    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2544734 kmers in total.\n",
      "  0:00:10.509    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.553    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:10.677    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:10.707    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1188456 bytes occupied (3.73621 bits per kmer).\n",
      "  0:00:10.710    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:10.885    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:10.886    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Early tip clipping\n",
      "  0:00:10.886    52M / 6G    INFO    General                 (construction.cpp          : 253)   Early tip clipper length bound set as (RL - K)\n",
      "  0:00:10.886    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 181)   Early tip clipping\n",
      "  0:00:11.130    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 184)   1005529 22-mers were removed by early tip clipper\n",
      "  0:00:11.130    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:11.131    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:11.237    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 44447 sequences extracted\n",
      "  0:00:11.297    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:11.328    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 2 loops collected\n",
      "  0:00:11.350    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:11.350    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:11.350    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:11.424    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1192832 bytes occupied (3.72907 bits per kmer).\n",
      "  0:00:11.431    64M / 6G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:21.498    64M / 6G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:21.582    64M / 6G    INFO    General                 (construction.cpp          : 464)   Processed 88795 edges\n",
      "  0:00:21.595    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:21.595    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 14\n",
      "  0:00:21.595    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 364\n",
      "  0:00:21.595    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 405. Coverage mad: 131.951\n",
      "  0:00:21.595    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:21.789    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:22.317    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:24.028    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:27.469    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:27.902    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 424.195. Fitted coverage std. dev: 130.121\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.998584\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 190\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 190\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1104445\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 424.195\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 190\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 4.26726\n",
      "  0:00:27.912    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:27.912    48M / 6G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:27.912    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:27.919    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:27.919    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:27.919    48M / 6G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:27.920    48M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:27.920    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:27.920    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:27.936    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 627 times\n",
      "  0:00:27.936    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.458    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 11542 times\n",
      "  0:00:28.458    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.558    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2649 times\n",
      "  0:00:28.558    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:28.558    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.568    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 167 times\n",
      "  0:00:28.568    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.576    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 52 times\n",
      "  0:00:28.576    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.585    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 75 times\n",
      "  0:00:28.585    52M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:28.585    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.585    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 6 times\n",
      "  0:00:28.585    52M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.586    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 2 times\n",
      "  0:00:28.586    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 23 times\n",
      "  0:00:28.587    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 2 times\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 5 times\n",
      "  0:00:28.587    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 13 times\n",
      "  0:00:28.588    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.588    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:28.589    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.589    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 7 times\n",
      "  0:00:28.590    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:00:28.590    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:28.591    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:28.591    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:28.591    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 4 times\n",
      "  0:00:28.591    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:28.591    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.593    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:00:28.593    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.594    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:00:28.594    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:28.595    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:28.595    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:28.595    48M / 6G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:28.595    48M / 6G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:28.595    48M / 6G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:28.595    48M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.595    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.596    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.596    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.596    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.596    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.597    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.597    48M / 6G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:28.598    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:28.601    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 200 times\n",
      "  0:00:28.601    48M / 6G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 4.26726 and less\n",
      "  0:00:28.601    48M / 6G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:28.601    48M / 6G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 421.738\n",
      "  0:00:28.601    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.601    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.615    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/before_rr.fasta\n",
      "  0:00:28.632    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/assembly_graph.fastg\n",
      "  0:00:28.685    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/simplified_contigs.fasta\n",
      "  0:00:28.700    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/final_contigs.fasta\n",
      "  0:00:28.716    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.716    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.728    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/before_rr.fasta\n",
      "  0:00:28.744    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/assembly_graph.fastg\n",
      "  0:00:28.795    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/simplified_contigs.fasta\n",
      "  0:00:28.811    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K21/final_contigs.fasta\n",
      "  0:00:28.826    48M / 6G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:28.831    48M / 6G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 28 seconds\n",
      "\n",
      "== Running assembler: K27\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K27/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K27/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=27\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 466033\n",
      "  0:00:08.621     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:08.621     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:09.305    48M / 6G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:09.305    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:09.382    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2625375 kmers in total.\n",
      "  0:00:09.382    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:09.427    48M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:09.427    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:09.427    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:09.427    48M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:09.427    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:09.427    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 349525\n",
      "  0:00:09.661     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2625375 kmers\n",
      "  0:00:09.662     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2625375 kmers.\n",
      "  0:00:09.745    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:09.822    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2615943 kmers in total.\n",
      "  0:00:09.822    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:09.869    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:10.009    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:10.040    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1221504 bytes occupied (3.73557 bits per kmer).\n",
      "  0:00:10.043    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:10.189    52M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:10.189    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Early tip clipping\n",
      "  0:00:10.189    52M / 6G    INFO    General                 (construction.cpp          : 253)   Early tip clipper length bound set as (RL - K)\n",
      "  0:00:10.189    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 181)   Early tip clipping\n",
      "  0:00:10.430    52M / 6G    INFO   Early tip clipping       (early_simplification.hpp  : 184)   1110467 28-mers were removed by early tip clipper\n",
      "  0:00:10.430    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:10.431    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:10.534    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 31720 sequences extracted\n",
      "  0:00:10.586    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:10.617    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 2 loops collected\n",
      "  0:00:10.633    52M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:10.633    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:10.633    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:10.698    52M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1223632 bytes occupied (3.72863 bits per kmer).\n",
      "  0:00:10.705    64M / 6G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:20.166    64M / 6G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:20.247    64M / 6G    INFO    General                 (construction.cpp          : 464)   Processed 63359 edges\n",
      "  0:00:20.259    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:20.260    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 14\n",
      "  0:00:20.260    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 355\n",
      "  0:00:20.260    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 385. Coverage mad: 127.504\n",
      "  0:00:20.260    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:20.417    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:20.851    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:22.621    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:26.061    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:28.049    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 403.334. Fitted coverage std. dev: 125.13\n",
      "  0:00:28.057    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.998476\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 182\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 182\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1103092\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 403.334\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 182\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 4.0579\n",
      "  0:00:28.058    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:28.058    48M / 6G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:28.058    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:28.060    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:28.060    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:28.060    48M / 6G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:28.060    48M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:28.060    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:28.060    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.072    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 730 times\n",
      "  0:00:28.072    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.408    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 7929 times\n",
      "  0:00:28.408    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.483    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1823 times\n",
      "  0:00:28.483    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:28.483    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.491    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 116 times\n",
      "  0:00:28.491    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.495    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 39 times\n",
      "  0:00:28.495    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.505    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 62 times\n",
      "  0:00:28.505    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:28.505    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.505    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 4 times\n",
      "  0:00:28.505    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 4 times\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 13 times\n",
      "  0:00:28.506    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 3 times\n",
      "  0:00:28.506    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 7 times\n",
      "  0:00:28.507    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 3 times\n",
      "  0:00:28.507    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.507    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 7 times\n",
      "  0:00:28.508    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 3 times\n",
      "  0:00:28.508    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:28.508    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.508    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 3 times\n",
      "  0:00:28.509    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 5 times\n",
      "  0:00:28.509    48M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.509    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.510    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.510    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:28.510    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:28.510    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:28.510    48M / 6G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:28.510    48M / 6G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:28.510    48M / 6G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:28.510    48M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:28.510    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.510    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.511    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.512    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.512    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.512    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.512    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.512    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.512    48M / 6G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:28.513    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:28.517    48M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 478 times\n",
      "  0:00:28.517    48M / 6G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 4.0579 and less\n",
      "  0:00:28.517    48M / 6G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:28.518    48M / 6G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 399.116\n",
      "  0:00:28.518    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.518    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.530    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/before_rr.fasta\n",
      "  0:00:28.545    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/assembly_graph.fastg\n",
      "  0:00:28.596    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/simplified_contigs.fasta\n",
      "  0:00:28.611    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/final_contigs.fasta\n",
      "  0:00:28.626    48M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.626    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.639    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/before_rr.fasta\n",
      "  0:00:28.654    48M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/assembly_graph.fastg\n",
      "  0:00:28.705    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/simplified_contigs.fasta\n",
      "  0:00:28.720    48M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K27/final_contigs.fasta\n",
      "  0:00:28.735    48M / 6G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:28.739    48M / 6G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 28 seconds\n",
      "\n",
      "== Running assembler: K33\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K33/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K33/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=33\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 233016\n",
      "  0:00:10.433     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7111785 reads\n",
      "  0:00:11.349     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:11.350     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:12.320    48M / 7G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:12.320    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:12.450    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2672326 kmers in total.\n",
      "  0:00:12.450    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:12.517    48M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:12.517    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:12.517    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:12.517    48M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:12.517    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:12.517    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 174762\n",
      "  0:00:12.836     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2672326 kmers\n",
      "  0:00:12.836     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2672326 kmers.\n",
      "  0:00:12.931    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:13.032    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2665424 kmers in total.\n",
      "  0:00:13.032    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:13.103    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:13.231    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:13.297    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1244352 bytes occupied (3.7348 bits per kmer).\n",
      "  0:00:13.301    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:13.468    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:13.468    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Early tip clipping\n",
      "  0:00:13.468    52M / 7G    INFO    General                 (construction.cpp          : 253)   Early tip clipper length bound set as (RL - K)\n",
      "  0:00:13.468    52M / 7G    INFO   Early tip clipping       (early_simplification.hpp  : 181)   Early tip clipping\n",
      "  0:00:13.727    52M / 7G    INFO   Early tip clipping       (early_simplification.hpp  : 184)   1161283 34-mers were removed by early tip clipper\n",
      "  0:00:13.727    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:13.728    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:13.834    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 25387 sequences extracted\n",
      "  0:00:13.890    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:13.925    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 1 loops collected\n",
      "  0:00:13.939    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:13.939    52M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:13.939    52M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:14.009    52M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1245360 bytes occupied (3.72817 bits per kmer).\n",
      "  0:00:14.016    64M / 7G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:23.194    64M / 7G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:23.259    64M / 7G    INFO    General                 (construction.cpp          : 464)   Processed 50702 edges\n",
      "  0:00:23.275    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:23.275    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 15\n",
      "  0:00:23.275    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 358\n",
      "  0:00:23.275    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 366. Coverage mad: 121.573\n",
      "  0:00:23.275    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:23.447    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:23.888    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:25.588    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:28.645    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:29.737    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 382.42. Fitted coverage std. dev: 119.999\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.998013\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 173\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 173\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1101587\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 382.42\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 173\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 3.84855\n",
      "  0:00:29.744    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:29.744    48M / 7G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:29.744    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:29.748    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:29.748    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:29.748    48M / 7G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:29.748    48M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:29.748    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:29.748    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:29.763    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 835 times\n",
      "  0:00:29.763    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.017    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 6217 times\n",
      "  0:00:30.017    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.078    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1303 times\n",
      "  0:00:30.078    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:30.078    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.082    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 56 times\n",
      "  0:00:30.082    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.085    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 31 times\n",
      "  0:00:30.085    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.093    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 56 times\n",
      "  0:00:30.093    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:30.093    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.093    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 7 times\n",
      "  0:00:30.093    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:30.094    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 2 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 9 times\n",
      "  0:00:30.094    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:30.094    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.094    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 3 times\n",
      "  0:00:30.095    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 5 times\n",
      "  0:00:30.095    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 2 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:30.095    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:30.095    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:30.095    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:30.095    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.096    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 1 times\n",
      "  0:00:30.096    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:30.097    48M / 7G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:30.097    48M / 7G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:30.097    48M / 7G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:30.097    48M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.097    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.099    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.099    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.099    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.099    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.099    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.099    48M / 7G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:30.100    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:30.105    48M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 714 times\n",
      "  0:00:30.105    48M / 7G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 3.84855 and less\n",
      "  0:00:30.106    48M / 7G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:30.106    48M / 7G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 378.244\n",
      "  0:00:30.106    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:30.106    48M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:30.118    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/before_rr.fasta\n",
      "  0:00:30.133    48M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/assembly_graph.fastg\n",
      "  0:00:30.182    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/simplified_contigs.fasta\n",
      "  0:00:30.197    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/final_contigs.fasta\n",
      "  0:00:30.211    48M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:30.211    48M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:30.223    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/before_rr.fasta\n",
      "  0:00:30.237    48M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/assembly_graph.fastg\n",
      "  0:00:30.286    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/simplified_contigs.fasta\n",
      "  0:00:30.301    48M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K33/final_contigs.fasta\n",
      "  0:00:30.315    48M / 7G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:30.318    48M / 7G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 30 seconds\n",
      "\n",
      "== Running assembler: K55\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K55/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K55/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=55\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  76)   Will need read mapping, kmer mapper will be attached\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 233016\n",
      "  0:00:09.262     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:09.262     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:10.173    48M / 7G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:10.173    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.295    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2762907 kmers in total.\n",
      "  0:00:10.295    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.369    48M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:10.369    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:10.369    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:10.369    48M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:10.369    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:10.369    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 174762\n",
      "  0:00:10.631     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2762907 kmers\n",
      "  0:00:10.631     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2762907 kmers.\n",
      "  0:00:10.732    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.838    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2761177 kmers in total.\n",
      "  0:00:10.838    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.915    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:11.047    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:11.118    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1288896 bytes occupied (3.73434 bits per kmer).\n",
      "  0:00:11.121    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:11.308    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:11.308    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:11.330    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:11.542    60M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 136236 sequences extracted\n",
      "  0:00:11.649    60M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:11.687    60M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 0 loops collected\n",
      "  0:00:11.749    68M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:11.750    68M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:11.750    68M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:11.815   104M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1287352 bytes occupied (3.72753 bits per kmer).\n",
      "  0:00:11.822   116M / 7G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:19.604   108M / 7G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:19.790   108M / 7G    INFO    General                 (construction.cpp          : 464)   Processed 272427 edges\n",
      "  0:00:19.806    88M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:19.806    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 20\n",
      "  0:00:19.806    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 266\n",
      "  0:00:19.806    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 290. Coverage mad: 100.817\n",
      "  0:00:19.806    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:19.932    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:20.276    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:21.504    88M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:23.980    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:25.386    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 306.107. Fitted coverage std. dev: 100.722\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.991455\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 139\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 139\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1095090\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 306.107\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 139\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 3.08092\n",
      "  0:00:25.393    76M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:25.393    76M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:25.394    76M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:25.394    76M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:25.395    76M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:25.395    76M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83127 Gb\n",
      "  0:00:25.395    76M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 174762\n",
      "  0:00:25.587     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 272427 edges\n",
      "  0:00:25.587     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 272427 sequences.\n",
      "  0:00:25.672    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:25.777    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2762907 kmers in total.\n",
      "  0:00:25.777    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:25.852    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:25.965    68M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:26.036    68M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1289632 bytes occupied (3.73413 bits per kmer).\n",
      "  0:00:26.077   132M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:26.181   132M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:26.182   132M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:26.332   140M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:28.447   172M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:28.447   172M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:28.494   156M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:28.778   156M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 34 gaps after checking 3642 candidates\n",
      "  0:00:28.791   152M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:28.800    88M / 7G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:28.801    88M / 7G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:28.801    88M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:28.812    88M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:28.812    88M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:28.812    88M / 7G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:28.812    88M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:28.812    88M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:28.812    88M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:29.724    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 62462 times\n",
      "  0:00:29.724    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.304    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 3168 times\n",
      "  0:00:30.304    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.344    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 519 times\n",
      "  0:00:30.344    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:30.344    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.346    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 20 times\n",
      "  0:00:30.346    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.352    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 17 times\n",
      "  0:00:30.352    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.357    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 27 times\n",
      "  0:00:30.357    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:30.357    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.357    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 4 times\n",
      "  0:00:30.357    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 3 times\n",
      "  0:00:30.358    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 4 times\n",
      "  0:00:30.358    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.358    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:30.359    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 1 times\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.359    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:30.359    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:30.360    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:30.360    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:30.360    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:30.360    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.361    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 2 times\n",
      "  0:00:30.361    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:30.362    72M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:30.362    72M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:30.363    72M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:30.363    72M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:30.363    72M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:30.363    72M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:30.363    72M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:30.363    72M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83138 Gb\n",
      "  0:00:30.363    72M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 174762\n",
      "  0:00:30.487     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 4288 edges\n",
      "  0:00:30.487     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 4288 sequences.\n",
      "  0:00:30.569    72M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:30.634    72M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 1309258 kmers in total.\n",
      "  0:00:30.634    72M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:30.679    72M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:30.742    72M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:30.775    72M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 615584 bytes occupied (3.76142 bits per kmer).\n",
      "  0:00:30.803   104M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:30.860   104M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:30.860   104M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:30.864   104M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:32.279   104M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:32.279   104M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:32.283   104M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:32.291   104M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 1 gaps after checking 58 candidates\n",
      "  0:00:32.294   104M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:32.294   104M / 7G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:32.294   104M / 7G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:32.294   104M / 7G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:32.294   104M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:32.294   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:32.295   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:32.295   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:32.295   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:32.295   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:32.296   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:32.296   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:32.296   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:32.296   104M / 7G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:32.298   104M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:32.429   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 1927 times\n",
      "  0:00:32.429   100M / 7G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 3.08092 and less\n",
      "  0:00:32.431   100M / 7G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:32.432   100M / 7G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 301.97\n",
      "  0:00:32.432   100M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:32.432   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:32.443   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/before_rr.fasta\n",
      "  0:00:32.458   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/assembly_graph.fastg\n",
      "  0:00:32.505   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/simplified_contigs.fasta\n",
      "  0:00:32.519   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/final_contigs.fasta\n",
      "  0:00:32.532   100M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:32.532   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:32.544   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/before_rr.fasta\n",
      "  0:00:32.557   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/assembly_graph.fastg\n",
      "  0:00:32.603   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/simplified_contigs.fasta\n",
      "  0:00:32.616   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K55/final_contigs.fasta\n",
      "  0:00:32.630   100M / 7G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:32.734    52M / 7G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 32 seconds\n",
      "\n",
      "== Running assembler: K77\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K77/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K77/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=77\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  76)   Will need read mapping, kmer mapper will be attached\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 155344\n",
      "  0:00:08.874     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7821556 reads\n",
      "  0:00:09.007     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:09.007     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:10.074    48M / 7G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:10.074    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.248    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2742260 kmers in total.\n",
      "  0:00:10.248    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.357    48M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:10.357    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:10.357    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:10.357    48M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:10.357    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:10.358    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 116508\n",
      "  0:00:10.714     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2742260 kmers\n",
      "  0:00:10.714     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2742260 kmers.\n",
      "  0:00:10.818    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:10.940    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2745925 kmers in total.\n",
      "  0:00:10.941    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:11.049    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:11.166    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:11.272    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1281744 bytes occupied (3.73424 bits per kmer).\n",
      "  0:00:11.276    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:11.426    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:11.426    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:11.440    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:11.614    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 129292 sequences extracted\n",
      "  0:00:11.730    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:11.769    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 0 loops collected\n",
      "  0:00:11.834    60M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:11.834    60M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:11.834    60M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:11.916   104M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1277912 bytes occupied (3.72805 bits per kmer).\n",
      "  0:00:11.923   116M / 7G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:18.290   108M / 7G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:18.448   108M / 7G    INFO    General                 (construction.cpp          : 464)   Processed 258555 edges\n",
      "  0:00:18.470    64M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:18.470    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 20\n",
      "  0:00:18.470    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 209\n",
      "  0:00:18.470    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 218. Coverage mad: 80.0604\n",
      "  0:00:18.470    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:18.565    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:18.831    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:19.756    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:21.485    64M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:22.052    60M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 230.772. Fitted coverage std. dev: 81.1236\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.969638\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 104\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 104\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1085812\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 230.772\n",
      "  0:00:22.054    60M / 7G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 104\n",
      "  0:00:22.055    60M / 7G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 2.31328\n",
      "  0:00:22.055    60M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:22.055    60M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:22.055    60M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:22.055    60M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:22.056    60M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:22.056    60M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83171 Gb\n",
      "  0:00:22.056    60M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 116508\n",
      "  0:00:22.273     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 258555 edges\n",
      "  0:00:22.273     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 258555 sequences.\n",
      "  0:00:22.362    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:22.500    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2742260 kmers in total.\n",
      "  0:00:22.500    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:22.616    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:22.775    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:22.879    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1279968 bytes occupied (3.73405 bits per kmer).\n",
      "  0:00:22.919   120M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:23.030   120M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:23.031   120M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:23.179   128M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:24.882   176M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:24.882   176M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:24.930   144M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:25.203   144M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 54 gaps after checking 3678 candidates\n",
      "  0:00:25.221   132M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:25.230    60M / 7G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:25.230    60M / 7G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:25.230    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:25.241    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:25.241    60M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:25.241    60M / 7G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:25.241    60M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:25.241    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:25.241    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.156    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 61196 times\n",
      "  0:00:26.156    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.452    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 971 times\n",
      "  0:00:26.452    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.475    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 213 times\n",
      "  0:00:26.475    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:26.475    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.477    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 28 times\n",
      "  0:00:26.477    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.481    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 10 times\n",
      "  0:00:26.481    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:26.483    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:26.483    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:26.483    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:26.483    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:26.483    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.483    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:26.484    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:26.484    64M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:26.484    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:26.486    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:26.486    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:26.487    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:26.487    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:26.488    64M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:26.488    64M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:26.488    64M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:26.489    64M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:26.489    64M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:26.489    64M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:26.489    64M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.8316 Gb\n",
      "  0:00:26.489    64M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 116508\n",
      "  0:00:26.635     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 9874 edges\n",
      "  0:00:26.635     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 9874 sequences.\n",
      "  0:00:26.717    64M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:26.816    64M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 1464827 kmers in total.\n",
      "  0:00:26.816    64M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:26.888    64M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:26.979    64M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:27.032    64M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 687568 bytes occupied (3.75508 bits per kmer).\n",
      "  0:00:27.067   100M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:27.125   100M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:27.126   100M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:27.137   104M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:28.443   104M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:28.443   104M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:28.452   100M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:28.465   100M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 2 gaps after checking 92 candidates\n",
      "  0:00:28.469   100M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:28.469   100M / 7G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:28.469   100M / 7G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:28.469   100M / 7G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:28.469   100M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:28.469   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.471   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.471   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.472   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.472   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:28.473   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:28.473   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:28.474   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:28.474   100M / 7G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:28.477   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:28.750   100M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 4776 times\n",
      "  0:00:28.750   100M / 7G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 2.31328 and less\n",
      "  0:00:28.750   100M / 7G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:28.750   100M / 7G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 227.453\n",
      "  0:00:28.750   100M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.750   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.763   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/before_rr.fasta\n",
      "  0:00:28.779   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/assembly_graph.fastg\n",
      "  0:00:28.825   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/simplified_contigs.fasta\n",
      "  0:00:28.839   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/final_contigs.fasta\n",
      "  0:00:28.852   100M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:28.853   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:28.864   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/before_rr.fasta\n",
      "  0:00:28.878   100M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/assembly_graph.fastg\n",
      "  0:00:28.922   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/simplified_contigs.fasta\n",
      "  0:00:28.936   100M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K77/final_contigs.fasta\n",
      "  0:00:28.950   100M / 7G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:29.009    52M / 7G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 29 seconds\n",
      "\n",
      "== Running assembler: K99\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K99/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K99/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=99\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  76)   Will need read mapping, kmer mapper will be attached\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 116508\n",
      "  0:00:07.715     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:07.715     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:08.746    48M / 7G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:08.746    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:08.918    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2575629 kmers in total.\n",
      "  0:00:08.918    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:09.052    48M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:09.052    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:09.052    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:09.052    48M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:09.053    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:09.053    48M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:09.408     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2575629 kmers\n",
      "  0:00:09.408     6G / 7G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2575629 kmers.\n",
      "  0:00:09.516    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:09.673    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2587253 kmers in total.\n",
      "  0:00:09.673    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:09.814    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:09.965    48M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:10.084    48M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1208216 bytes occupied (3.7359 bits per kmer).\n",
      "  0:00:10.087    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:10.223    52M / 7G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:10.224    52M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:10.242    52M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:10.416    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 118879 sequences extracted\n",
      "  0:00:10.525    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:10.557    56M / 7G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 0 loops collected\n",
      "  0:00:10.625    56M / 7G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:10.625    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:10.625    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:10.690   100M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1201256 bytes occupied (3.73115 bits per kmer).\n",
      "  0:00:10.697   112M / 7G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:15.167   108M / 7G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:15.323   108M / 7G    INFO    General                 (construction.cpp          : 464)   Processed 237744 edges\n",
      "  0:00:15.343    80M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:15.343    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 20\n",
      "  0:00:15.343    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 140\n",
      "  0:00:15.343    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 145. Coverage mad: 57.8214\n",
      "  0:00:15.343    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:15.418    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:15.618    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:16.244    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:16.964    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 157.351. Fitted coverage std. dev: 61.5749\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.809699\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 68\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 68\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1073612\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 157.351\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 68\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 1.54565\n",
      "  0:00:16.966    80M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:16.966    80M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:16.967    80M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:16.967    80M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:16.967    80M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:16.967    80M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83116 Gb\n",
      "  0:00:16.967    80M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:17.195     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 237744 edges\n",
      "  0:00:17.195     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 237744 sequences.\n",
      "  0:00:17.288    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:17.405    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2575629 kmers in total.\n",
      "  0:00:17.405    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:17.543    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:17.689    60M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:17.819    60M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 1202944 bytes occupied (3.73639 bits per kmer).\n",
      "  0:00:17.856   120M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:17.982   120M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:17.983   120M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:18.118   132M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:19.484   164M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:19.484   164M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:19.534   128M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:19.820   128M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 147 gaps after checking 3755 candidates\n",
      "  0:00:19.851   120M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:19.857    60M / 7G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:19.857    60M / 7G    INFO    General                 (graph_simplification.hpp  : 662)   Flanking coverage based disconnection disabled\n",
      "  0:00:19.857    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:19.867    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:19.867    60M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:19.867    60M / 7G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:19.867    60M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:19.867    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:19.867    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.693    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 53605 times\n",
      "  0:00:20.693    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.784    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 233 times\n",
      "  0:00:20.784    56M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.793    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 67 times\n",
      "  0:00:20.793    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:20.793    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.794    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 13 times\n",
      "  0:00:20.794    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.796    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 2 times\n",
      "  0:00:20.796    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 6 times\n",
      "  0:00:20.797    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.797    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:20.798    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:20.798    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:20.798    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:20.798    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.798    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:20.799    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:20.799    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:20.799    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.802    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 2 times\n",
      "  0:00:20.802    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.804    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.804    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:20.806    60M / 7G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:20.806    60M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:20.807    60M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:20.807    60M / 7G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:20.807    60M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:20.807    60M / 7G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:20.807    60M / 7G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:20.807    60M / 7G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83171 Gb\n",
      "  0:00:20.807    60M / 7G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:20.998     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 23783 edges\n",
      "  0:00:20.998     6G / 7G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 23783 sequences.\n",
      "  0:00:21.085    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:21.191    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 1718701 kmers in total.\n",
      "  0:00:21.192    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:21.280    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:21.378    56M / 7G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:21.464    56M / 7G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 805280 bytes occupied (3.74832 bits per kmer).\n",
      "  0:00:21.504    96M / 7G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:21.573   100M / 7G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:21.574   100M / 7G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:21.599   100M / 7G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:22.658   100M / 7G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:22.658   100M / 7G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:22.682    96M / 7G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:22.715    96M / 7G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 0 gaps after checking 420 candidates\n",
      "  0:00:22.718    96M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:22.718    96M / 7G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:22.718    96M / 7G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:22.718    96M / 7G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:22.718    96M / 7G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:22.718    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:22.721    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:22.721    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:22.725    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:22.725    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:22.727    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:22.727    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:22.729    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:22.729    96M / 7G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:22.737    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:23.284    96M / 7G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 11756 times\n",
      "  0:00:23.284    96M / 7G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 1.54565 and less\n",
      "  0:00:23.284    96M / 7G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:23.284    96M / 7G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 155.285\n",
      "  0:00:23.284    96M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:23.285    96M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:23.297    96M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/before_rr.fasta\n",
      "  0:00:23.315    92M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/assembly_graph.fastg\n",
      "  0:00:23.363    92M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/simplified_contigs.fasta\n",
      "  0:00:23.376    92M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/final_contigs.fasta\n",
      "  0:00:23.390    92M / 7G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:23.390    92M / 7G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:23.401    92M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/before_rr.fasta\n",
      "  0:00:23.415    92M / 7G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/assembly_graph.fastg\n",
      "  0:00:23.463    92M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/simplified_contigs.fasta\n",
      "  0:00:23.477    92M / 7G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K99/final_contigs.fasta\n",
      "  0:00:23.491    92M / 7G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:23.522    52M / 7G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 23 seconds\n",
      "\n",
      "== Running assembler: K127\n",
      "\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K127/configs/config.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  74)   Loaded config from /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/K127/configs/careful_mode.info\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (memory_limit.cpp          :  49)   Memory limit set to 246 Gb\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  87)   Starting SPAdes, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  88)   Maximum k-mer length: 128\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  89)   Assembling dataset (/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/dataset.info) with K=127\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  90)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  51)   SPAdes started\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  58)   Starting from stage: construction\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  65)   Two-step RR enabled: 0\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (launch.hpp                :  76)   Will need read mapping, kmer mapper will be attached\n",
      "  0:00:00.000     4M / 4M    INFO   StageManager             (stage.cpp                 : 132)   STAGE == de Bruijn graph construction\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (read_converter.hpp        :  59)   Binary reads detected\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 111)   Max read length 151\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (construction.cpp          : 117)   Average read length 143.298\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == k+1-mer counting\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 144 files using 12 threads. This might take a while.\n",
      "  0:00:00.001     4M / 4M    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83322 Gb\n",
      "  0:00:00.002     4M / 4M    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 116508\n",
      "  0:00:02.896     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 289)   Processed 7906744 reads\n",
      "  0:00:02.896     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 295)   Adding contigs from previous K\n",
      "  0:00:03.530    48M / 6G    INFO    General                 (kmer_splitters.hpp        : 308)   Used 7906744 reads\n",
      "  0:00:03.531    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:03.687    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2037679 kmers in total.\n",
      "  0:00:03.687    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:03.794    48M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Extension index construction\n",
      "  0:00:03.794    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:03.794    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:03.794    48M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:03.794    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83203 Gb\n",
      "  0:00:03.794    48M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:04.097     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 380)   Processed 2037679 kmers\n",
      "  0:00:04.097     6G / 6G    INFO    General                 (kmer_splitters.hpp        : 385)   Used 2037679 kmers.\n",
      "  0:00:04.202    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:04.329    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2065269 kmers in total.\n",
      "  0:00:04.329    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:04.445    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:04.565    48M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:04.669    48M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 965864 bytes occupied (3.74136 bits per kmer).\n",
      "  0:00:04.672    48M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build:  99)   Building k-mer extensions from k+1-mers\n",
      "  0:00:04.777    48M / 6G    INFO   DeBruijnExtensionIndexBu (kmer_extension_index_build: 103)   Building k-mer extensions from k+1-mers finished.\n",
      "  0:00:04.777    48M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Condensing graph\n",
      "  0:00:04.798    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 355)   Extracting unbranching paths\n",
      "  0:00:04.930    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 374)   Extracting unbranching paths finished. 99766 sequences extracted\n",
      "  0:00:05.015    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 310)   Collecting perfect loops\n",
      "  0:00:05.041    52M / 6G    INFO   UnbranchingPathExtractor (debruijn_graph_constructor: 343)   Collecting perfect loops finished. 0 loops collected\n",
      "  0:00:05.100    56M / 6G    INFO    General                 (stage.cpp                 : 101)   PROCEDURE == Filling coverage indices (PHM)\n",
      "  0:00:05.100    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:05.100    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:05.164    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 951240 bytes occupied (3.7346 bits per kmer).\n",
      "  0:00:05.170    64M / 6G    INFO    General                 (construction.cpp          : 388)   Collecting k-mer coverage information from reads, this takes a while.\n",
      "  0:00:07.594    64M / 6G    INFO    General                 (construction.cpp          : 508)   Filling coverage and flanking coverage from PHM\n",
      "  0:00:07.724    64M / 6G    INFO    General                 (construction.cpp          : 464)   Processed 199528 edges\n",
      "  0:00:07.740    56M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == EC Threshold Finding\n",
      "  0:00:07.740    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 181)   Kmer coverage valley at: 9\n",
      "  0:00:07.740    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 201)   K-mer histogram maximum: 55\n",
      "  0:00:07.740    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 237)   Estimated median coverage: 57. Coverage mad: 29.652\n",
      "  0:00:07.740    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 259)   Fitting coverage model\n",
      "  0:00:07.781    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 2\n",
      "  0:00:07.885    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 4\n",
      "  0:00:08.214    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 8\n",
      "  0:00:08.795    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 295)   ... iteration 16\n",
      "  0:00:09.574    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 309)   Fitted mean coverage: 68.3634. Fitted coverage std. dev: 36.3081\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 334)   Probability of erroneous kmer at valley: 0.391162\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 358)   Preliminary threshold calculated as: 24\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 362)   Threshold adjusted to: 24\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (kmer_coverage_model.cpp   : 375)   Estimated genome size (ignoring repeats): 1046696\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (genomic_info_filler.cpp   : 112)   Mean coverage was calculated as 68.3634\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (genomic_info_filler.cpp   : 127)   EC coverage threshold value was calculated as 24\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (genomic_info_filler.cpp   : 128)   Trusted kmer low bound: 0.56866\n",
      "  0:00:09.575    56M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:09.575    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:09.575    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:09.576    56M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:09.576    56M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83181 Gb\n",
      "  0:00:09.576    56M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:09.774     6G / 6G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 199528 edges\n",
      "  0:00:09.774     6G / 6G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 199528 sequences.\n",
      "  0:00:09.863    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:09.980    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 2037679 kmers in total.\n",
      "  0:00:09.980    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:10.089    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:10.199    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:10.295    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 953088 bytes occupied (3.74186 bits per kmer).\n",
      "  0:00:10.324   104M / 6G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:10.404   104M / 6G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:10.404   104M / 6G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:10.534   116M / 6G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:11.442   116M / 6G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:11.442   116M / 6G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:11.502   104M / 6G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:11.826   104M / 6G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 343 gaps after checking 3895 candidates\n",
      "  0:00:11.877   104M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Raw Simplification\n",
      "  0:00:11.882    56M / 6G    INFO    General                 (simplification.cpp        : 128)   PROCEDURE == InitialCleaning\n",
      "  0:00:11.882    56M / 6G    INFO    General                 (simplification.cpp        :  62)   Most init cleaning disabled on main iteration\n",
      "  0:00:11.882    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Self conjugate edge remover\n",
      "  0:00:11.892    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Self conjugate edge remover triggered 0 times\n",
      "  0:00:11.892    56M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification\n",
      "  0:00:11.892    56M / 6G    INFO    General                 (simplification.cpp        : 357)   Graph simplification started\n",
      "  0:00:11.892    56M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:11.892    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 1\n",
      "  0:00:11.892    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.474    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 36329 times\n",
      "  0:00:12.474    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.494    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 28 times\n",
      "  0:00:12.494    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.499    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 4 times\n",
      "  0:00:12.499    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 2\n",
      "  0:00:12.499    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.499    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 3 times\n",
      "  0:00:12.499    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 2 times\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:12.501    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 3\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:12.501    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 4\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.501    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 2 times\n",
      "  0:00:12.503    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 5\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:12.503    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 6\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.503    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 7\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:12.504    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 8\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 9\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 1 times\n",
      "  0:00:12.504    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 10\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:12.504    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 11\n",
      "  0:00:12.504    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.511    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 5 times\n",
      "  0:00:12.511    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.516    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.516    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:12.520    56M / 6G    INFO    General                 (simplification.cpp        : 362)   PROCEDURE == Simplification cycle, iteration 12\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Low coverage edge remover\n",
      "  0:00:12.520    56M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Low coverage edge remover triggered 0 times\n",
      "  0:00:12.520    56M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Gap Closer\n",
      "  0:00:12.520    56M / 6G    INFO    General                 (graph_pack.hpp            : 101)   Index refill\n",
      "  0:00:12.521    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 301)   Building kmer index\n",
      "  0:00:12.521    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 117)   Splitting kmer instances into 192 files using 12 threads. This might take a while.\n",
      "  0:00:12.521    56M / 6G    INFO    General                 (file_limit.hpp            :  32)   Open file limit set to 50000\n",
      "  0:00:12.521    56M / 6G    INFO    General                 (kmer_splitters.hpp        :  89)   Memory available for splitting buffers: 6.83181 Gb\n",
      "  0:00:12.521    56M / 6G    INFO    General                 (kmer_splitters.hpp        :  97)   Using cell size of 87381\n",
      "  0:00:12.703     6G / 6G    INFO    General                 (edge_index_builders.hpp   :  77)   Processed 54722 edges\n",
      "  0:00:12.703     6G / 6G    INFO    General                 (edge_index_builders.hpp   :  82)   Used 54722 sequences.\n",
      "  0:00:12.790    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 120)   Starting k-mer counting.\n",
      "  0:00:12.964    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 127)   K-mer counting done. There are 1774468 kmers in total.\n",
      "  0:00:12.964    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 133)   Merging temporary buckets.\n",
      "  0:00:13.061    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 314)   Building perfect hash indices\n",
      "  0:00:13.140    56M / 6G    INFO    General                 (kmer_index_builder.hpp    : 150)   Merging final buckets.\n",
      "  0:00:13.228    56M / 6G    INFO   K-mer Index Building     (kmer_index_builder.hpp    : 336)   Index built. Total 831144 bytes occupied (3.74712 bits per kmer).\n",
      "  0:00:13.264   100M / 6G    INFO    General                 (edge_index_builders.hpp   : 107)   Collecting edge information from graph, this takes a while.\n",
      "  0:00:13.347   100M / 6G    INFO    General                 (edge_index.hpp            :  92)   Index refilled\n",
      "  0:00:13.348   100M / 6G    INFO    General                 (gap_closer.cpp            : 159)   Preparing shift maps\n",
      "  0:00:13.401   108M / 6G    INFO    General                 (gap_closer.cpp            : 119)   Processing paired reads (takes a while)\n",
      "  0:00:14.170   108M / 6G    INFO    General                 (gap_closer.cpp            : 138)   Used 1973079 paired reads\n",
      "  0:00:14.170   108M / 6G    INFO    General                 (gap_closer.cpp            : 140)   Merging paired indices\n",
      "  0:00:14.228   100M / 6G    INFO   GapCloser                (gap_closer.cpp            : 346)   Closing short gaps\n",
      "  0:00:14.339   100M / 6G    INFO   GapCloser                (gap_closer.cpp            : 380)   Closing short gaps complete: filled 2 gaps after checking 1390 candidates\n",
      "  0:00:14.349   100M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Simplification Cleanup\n",
      "  0:00:14.349   100M / 6G    INFO    General                 (simplification.cpp        : 196)   PROCEDURE == Post simplification\n",
      "  0:00:14.349   100M / 6G    INFO    General                 (graph_simplification.hpp  : 453)   Disconnection of relatively low covered edges disabled\n",
      "  0:00:14.349   100M / 6G    INFO    General                 (graph_simplification.hpp  : 489)   Complex tip clipping disabled\n",
      "  0:00:14.349   100M / 6G    INFO    General                 (graph_simplification.hpp  : 634)   Creating parallel br instance\n",
      "  0:00:14.349   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:14.354   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:14.354   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:14.359   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:14.359   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Tip clipper\n",
      "  0:00:14.363   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Tip clipper triggered 0 times\n",
      "  0:00:14.363   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Bulge remover\n",
      "  0:00:14.367   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Bulge remover triggered 0 times\n",
      "  0:00:14.367   100M / 6G    INFO    General                 (simplification.cpp        : 330)   Disrupting self-conjugate edges\n",
      "  0:00:14.383   100M / 6G    INFO   Simplification           (parallel_processing.hpp   : 165)   Running Removing isolated edges\n",
      "  0:00:15.080    96M / 6G    INFO   Simplification           (parallel_processing.hpp   : 167)   Removing isolated edges triggered 27265 times\n",
      "  0:00:15.080    96M / 6G    INFO    General                 (simplification.cpp        : 340)   Removing all the edges having coverage 0.56866 and less\n",
      "  0:00:15.081    96M / 6G    INFO    General                 (simplification.cpp        : 470)   Counting average coverage\n",
      "  0:00:15.081    96M / 6G    INFO    General                 (simplification.cpp        : 476)   Average coverage = 68.3509\n",
      "  0:00:15.081    96M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Mismatch Correction\n",
      "  0:00:15.081    96M / 6G    INFO    General                 (graph_pack.hpp            : 109)   Normalizing k-mer map. Total 7478 kmers to process\n",
      "  0:00:15.087    92M / 6G    INFO    General                 (graph_pack.hpp            : 111)   Normalizing done\n",
      "  0:00:18.074    96M / 6G    INFO    General                 (mismatch_shall_not_pass.hp: 189)   Finished collecting potential mismatches positions\n",
      "  0:00:18.094    92M / 6G    INFO    General                 (mismatch_shall_not_pass.hp: 290)   All edges processed\n",
      "  0:00:18.094    92M / 6G    INFO    General                 (mismatch_correction.cpp   :  27)   Corrected 0 nucleotides\n",
      "  0:00:18.094    92M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:18.094    92M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:18.106    92M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/before_rr.fasta\n",
      "  0:00:18.120    92M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/assembly_graph.fastg\n",
      "  0:00:18.168    92M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/simplified_contigs.fasta\n",
      "  0:00:18.182    92M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/intermediate_contigs.fasta\n",
      "  0:00:18.195    92M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Paired Information Counting\n",
      "  0:00:18.196    92M / 6G    INFO    General                 (graph_pack.hpp            : 109)   Normalizing k-mer map. Total 7478 kmers to process\n",
      "  0:00:18.196    92M / 6G    INFO    General                 (graph_pack.hpp            : 111)   Normalizing done\n",
      "  0:00:18.196    92M / 6G    INFO    General                 (pair_info_count.cpp       : 320)   Min edge length for estimation: 86519\n",
      "  0:00:18.196    92M / 6G    INFO    General                 (pair_info_count.cpp       : 331)   Estimating insert size for library #0\n",
      "  0:00:18.196    92M / 6G    INFO    General                 (pair_info_count.cpp       : 190)   Estimating insert size (takes a while)\n",
      "  0:00:18.365   300M / 6G    INFO    General                 (pair_info_count.cpp       :  39)   Selecting usual mapper\n",
      "  0:00:19.742   300M / 6G    INFO    General                 (sequence_mapper_notifier.h:  98)   Total 1973079 reads processed\n",
      "  0:00:21.003   300M / 6G    INFO    General                 (pair_info_count.cpp       : 209)   Edge pairs: 67108864 (rough upper limit)\n",
      "  0:00:21.003   300M / 6G    INFO    General                 (pair_info_count.cpp       : 213)   953755 paired reads (48.3384% of all) aligned to long edges\n",
      "  0:00:21.019    92M / 6G    INFO    General                 (pair_info_count.cpp       : 354)     Insert size = 289.951, deviation = 131.594, left quantile = 156, right quantile = 479, read length = 151\n",
      "  0:00:21.144   284M / 6G    INFO    General                 (pair_info_count.cpp       : 371)   Filtering data for library #0\n",
      "  0:00:21.144   284M / 6G    INFO    General                 (pair_info_count.cpp       :  39)   Selecting usual mapper\n",
      "  0:00:22.086   284M / 6G    INFO    General                 (sequence_mapper_notifier.h:  98)   Total 1973079 reads processed\n",
      "  0:00:22.086   284M / 6G    INFO    General                 (pair_info_count.cpp       : 383)   Mapping library #0\n",
      "  0:00:22.086   284M / 6G    INFO    General                 (pair_info_count.cpp       : 385)   Mapping paired reads (takes a while)\n",
      "  0:00:22.086   284M / 6G    INFO    General                 (pair_info_count.cpp       : 289)   Left insert size quantile 156, right insert size quantile 479, filtering threshold 2, rounding threshold 0\n",
      "  0:00:22.096   296M / 6G    INFO    General                 (pair_info_count.cpp       :  39)   Selecting usual mapper\n",
      "  0:00:23.529   296M / 6G    INFO    General                 (sequence_mapper_notifier.h:  98)   Total 1973079 reads processed\n",
      "  0:00:23.554    92M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Distance Estimation\n",
      "  0:00:23.555    92M / 6G    INFO    General                 (distance_estimation.cpp   : 173)   Processing library #0\n",
      "  0:00:23.555    92M / 6G    INFO    General                 (distance_estimation.cpp   : 149)   Weight Filter Done\n",
      "  0:00:23.555    92M / 6G    INFO   DistanceEstimator        (distance_estimation.hpp   : 116)   Using SIMPLE distance estimator\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (distance_estimation.cpp   :  34)   Filtering info\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (pair_info_filters.hpp     : 242)   Start filtering; index size: 654\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (pair_info_filters.hpp     : 263)   Done filtering\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (distance_estimation.cpp   : 156)   Refining clustered pair information\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (distance_estimation.cpp   : 158)   The refining of clustered pair information has been finished\n",
      "  0:00:23.557    92M / 6G    INFO    General                 (distance_estimation.cpp   : 160)   Improving paired information\n",
      "  0:00:23.558    92M / 6G    INFO   PairInfoImprover         (pair_info_improver.hpp    : 103)   Paired info stats: missing = 0; contradictional = 58\n",
      "  0:00:23.559    92M / 6G    INFO   PairInfoImprover         (pair_info_improver.hpp    : 103)   Paired info stats: missing = 0; contradictional = 0\n",
      "  0:00:23.559    92M / 6G    INFO    General                 (distance_estimation.cpp   : 103)   Filling scaffolding index\n",
      "  0:00:23.559    92M / 6G    INFO   DistanceEstimator        (distance_estimation.hpp   : 116)   Using SMOOTHING distance estimator\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (distance_estimation.cpp   :  34)   Filtering info\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (pair_info_filters.hpp     : 242)   Start filtering; index size: 224\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (pair_info_filters.hpp     : 263)   Done filtering\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (distance_estimation.cpp   : 182)   Clearing raw paired index\n",
      "  0:00:23.569    92M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Repeat Resolving\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (repeat_resolving.cpp      :  69)   Using Path-Extend repeat resolving\n",
      "  0:00:23.569    92M / 6G    INFO    General                 (launcher.cpp              : 477)   ExSPAnder repeat resolving tool started\n",
      "  0:00:23.570    92M / 6G    INFO    General                 (launcher.cpp              : 392)   Creating main extenders, unique edge length = 2000\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (extenders_logic.cpp       : 275)   Estimated coverage of library #0 is 68.3509\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (extenders_logic.cpp       : 275)   Estimated coverage of library #0 is 68.3509\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (extenders_logic.cpp       : 472)   Using 1 paired-end library\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (extenders_logic.cpp       : 473)   Using 1 paired-end scaffolding library\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (extenders_logic.cpp       : 474)   Using 0 single read libraries\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (launcher.cpp              : 420)   Total number of extenders is 3\n",
      "  0:00:23.571    92M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 0 paths from 91 (0%)\n",
      "  0:00:23.573    92M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 10 paths from 91 (10%)\n",
      "  0:00:23.574    92M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 20 paths from 91 (21%)\n",
      "  0:00:23.578    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 30 paths from 91 (32%)\n",
      "  0:00:23.579    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 40 paths from 91 (43%)\n",
      "  0:00:23.579    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 50 paths from 91 (54%)\n",
      "  0:00:23.580    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 60 paths from 91 (65%)\n",
      "  0:00:23.581    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 70 paths from 91 (76%)\n",
      "  0:00:23.583    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 80 paths from 91 (87%)\n",
      "  0:00:23.584    96M / 6G    INFO    General                 (path_extender.hpp         : 885)   Processed 90 paths from 91 (98%)\n",
      "  0:00:23.584    96M / 6G    INFO    General                 (launcher.cpp              : 234)   Finalizing paths\n",
      "  0:00:23.584    96M / 6G    INFO    General                 (launcher.cpp              : 236)   Deduplicating paths\n",
      "  0:00:23.584    96M / 6G    INFO    General                 (launcher.cpp              : 240)   Paths deduplicated\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 295)   Removing overlaps\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 298)   Sorting paths\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 305)   Marking overlaps\n",
      "  0:00:23.584    96M / 6G    INFO   OverlapRemover           (pe_resolver.hpp           : 130)   Marking start/end overlaps\n",
      "  0:00:23.584    96M / 6G    INFO   OverlapRemover           (pe_resolver.hpp           : 133)   Marking remaining overlaps\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 308)   Splitting paths\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 313)   Deduplicating paths\n",
      "  0:00:23.584    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 315)   Overlaps removed\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 257)   Paths finalized\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 427)   Closing gaps in paths\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 455)   Gap closing completed\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 286)   Traversing tandem repeats\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 296)   Traversed 1 loops\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 234)   Finalizing paths\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 236)   Deduplicating paths\n",
      "  0:00:23.585    96M / 6G    INFO    General                 (launcher.cpp              : 240)   Paths deduplicated\n",
      "  0:00:23.585    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 295)   Removing overlaps\n",
      "  0:00:23.585    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 298)   Sorting paths\n",
      "  0:00:23.585    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 305)   Marking overlaps\n",
      "  0:00:23.585    96M / 6G    INFO   OverlapRemover           (pe_resolver.hpp           : 130)   Marking start/end overlaps\n",
      "  0:00:23.585    96M / 6G    INFO   OverlapRemover           (pe_resolver.hpp           : 133)   Marking remaining overlaps\n",
      "  0:00:23.586    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 308)   Splitting paths\n",
      "  0:00:23.586    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 313)   Deduplicating paths\n",
      "  0:00:23.586    96M / 6G    INFO   PEResolver               (pe_resolver.hpp           : 315)   Overlaps removed\n",
      "  0:00:23.586    96M / 6G    INFO    General                 (launcher.cpp              : 257)   Paths finalized\n",
      "  0:00:23.586    96M / 6G    INFO    General                 (launcher.cpp              : 529)   ExSPAnder repeat resolving tool finished\n",
      "  0:00:23.587    96M / 6G    INFO   StageManager             (stage.cpp                 : 132)   STAGE == Contig Output\n",
      "  0:00:23.587    96M / 6G    INFO    General                 (contig_output_stage.cpp   :  40)   Writing GFA to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/assembly_graph_with_scaffolds.gfa\n",
      "  0:00:23.598    96M / 6G    INFO    General                 (contig_output.hpp         :  22)   Outputting contigs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/before_rr.fasta\n",
      "  0:00:23.612    92M / 6G    INFO    General                 (contig_output_stage.cpp   :  51)   Outputting FastG graph to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/assembly_graph.fastg\n",
      "  0:00:23.660    92M / 6G    INFO    General                 (contig_output_stage.cpp   :  20)   Outputting FastG paths to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/final_contigs.paths\n",
      "  0:00:23.682    96M / 6G    INFO    General                 (contig_output_stage.cpp   :  20)   Outputting FastG paths to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_//K127/scaffolds.paths\n",
      "  0:00:23.704    96M / 6G    INFO    General                 (launch.hpp                : 149)   SPAdes finished\n",
      "  0:00:23.721    48M / 6G    INFO    General                 (main.cpp                  : 109)   Assembling time: 0 hours 0 minutes 23 seconds\n",
      "\n",
      "===== Assembling finished. Used k-mer sizes: 15, 21, 27, 33, 55, 77, 99, 127 \n",
      "\n",
      "\n",
      "===== Mismatch correction started.\n",
      "\n",
      "== Processing of contigs\n",
      "\n",
      "\n",
      "== Running contig polishing tool: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-corrector-core /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/contigs/configs/corrector.info /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta\n",
      "\n",
      "\n",
      "== Dataset description file was created: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/contigs/configs/corrector.info\n",
      "\n",
      "/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/contigs/configs/log.properties  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  58)   Starting MismatchCorrector, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  59)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 195)   Splitting assembly...\n",
      "  0:00:00.000     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 196)   Assembly file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta\n",
      "  0:00:00.028     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 203)   Processing paired sublib of number 0\n",
      "  0:00:00.028     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 206)   /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "  0:00:00.028     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 140)   Running bwa index ...: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa index -a is /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta\n",
      "[bwa_index] Pack FASTA... 0.00 sec\n",
      "[bwa_index] Construct BWT for the packed sequence...\n",
      "[bwa_index] 0.28 seconds elapse.\n",
      "[bwa_index] Update BWT... 0.00 sec\n",
      "[bwa_index] Pack forward-only FASTA... 0.01 sec\n",
      "[bwa_index] Construct SA from BWT and Occ... 0.12 sec\n",
      "[main] Version: 0.7.12-r1039\n",
      "[main] CMD: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa index -a is /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta\n",
      "[main] Real time: 0.436 sec; CPU: 0.416 sec\n",
      "  0:00:00.467     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 149)   Running bwa mem ...:/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa mem  -v 1 -t 12 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz  > /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/corrector_kNDV0F/lib0_WZdPW8/tmp.sam\n",
      "[main] Version: 0.7.12-r1039\n",
      "[main] CMD: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa mem -v 1 -t 12 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_contigs.fasta /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "[main] Real time: 26.317 sec; CPU: 249.340 sec\n",
      "  0:00:26.818     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 209)   Adding samfile /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/corrector_kNDV0F/lib0_WZdPW8/tmp.sam\n",
      "  0:00:30.986    44M / 44M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 1000000reads, flushing\n",
      "  0:00:35.387    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 2000000reads, flushing\n",
      "  0:00:39.652    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 3000000reads, flushing\n",
      "  0:00:44.015    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 4000000reads, flushing\n",
      "  0:00:48.361    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 5000000reads, flushing\n",
      "  0:00:51.346     8M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 235)   Processing contigs\n",
      "  0:00:57.354   168M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_8_length_43722_cov_59.642987 processed with 0 changes in thread 11\n",
      "  0:00:57.383   164M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_10_length_37477_cov_70.532262 processed with 0 changes in thread 10\n",
      "  0:00:57.811   160M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_9_length_43473_cov_64.820237 processed with 0 changes in thread 7\n",
      "  0:00:58.134   156M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_7_length_44530_cov_66.817287 processed with 0 changes in thread 6\n",
      "  0:00:58.161   152M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_11_length_36438_cov_81.403129 processed with 1 changes in thread 9\n",
      "  0:01:02.547   148M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_6_length_63307_cov_77.431466 processed with 0 changes in thread 5\n",
      "  0:01:03.516   140M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_5_length_76950_cov_69.042982 processed with 0 changes in thread 4\n",
      "  0:01:07.588   136M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_4_length_86646_cov_81.668073 processed with 0 changes in thread 3\n",
      "  0:01:09.702   128M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_3_length_126709_cov_62.629307 processed with 0 changes in thread 2\n",
      "  0:01:17.481   112M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_2_length_165590_cov_68.322991 processed with 0 changes in thread 1\n",
      "  0:01:53.634    92M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_1_length_375077_cov_67.153178 processed with 0 changes in thread 0\n",
      "  0:01:53.651    48M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 255)   Gluing processed contigs\n",
      "  0:01:53.655    48M / 172M  INFO    General                 (main.cpp                  :  72)   Correcting time: 0 hours 1 minutes 53 seconds\n",
      "\n",
      "== Processing of scaffolds\n",
      "\n",
      "\n",
      "== Running contig polishing tool: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-corrector-core /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/scaffolds/configs/corrector.info /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta\n",
      "\n",
      "\n",
      "== Dataset description file was created: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/scaffolds/configs/corrector.info\n",
      "\n",
      "/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/mismatch_corrector/scaffolds/configs/log.properties  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  58)   Starting MismatchCorrector, built from refs/heads/spades_3.13.0, git revision 8ea46659e9b2aca35444a808db550ac333006f8b\n",
      "  0:00:00.000     4M / 4M    INFO    General                 (main.cpp                  :  59)   Maximum # of threads to use (adjusted due to OMP capabilities): 12\n",
      "  0:00:00.000     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 195)   Splitting assembly...\n",
      "  0:00:00.000     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 196)   Assembly file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta\n",
      "  0:00:00.028     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 203)   Processing paired sublib of number 0\n",
      "  0:00:00.028     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 206)   /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "  0:00:00.029     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 140)   Running bwa index ...: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa index -a is /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta\n",
      "[bwa_index] Pack FASTA... 0.00 sec\n",
      "[bwa_index] Construct BWT for the packed sequence...\n",
      "[bwa_index] 0.27 seconds elapse.\n",
      "[bwa_index] Update BWT... 0.02 sec\n",
      "[bwa_index] Pack forward-only FASTA... 0.00 sec\n",
      "[bwa_index] Construct SA from BWT and Occ... 0.12 sec\n",
      "[main] Version: 0.7.12-r1039\n",
      "[main] CMD: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa index -a is /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta\n",
      "[main] Real time: 0.431 sec; CPU: 0.416 sec\n",
      "  0:00:00.464     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 149)   Running bwa mem ...:/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa mem  -v 1 -t 12 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz  > /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/corrector_pmBnSV/lib0_UXo2Na/tmp.sam\n",
      "[main] Version: 0.7.12-r1039\n",
      "[main] CMD: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/spades-3.13.0-0/bin/spades-bwa mem -v 1 -t 12 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/misc/assembled_scaffolds.fasta /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R1.mapped.fastq.gz /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/alignment/fastq/CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "[main] Real time: 25.492 sec; CPU: 243.268 sec\n",
      "  0:00:25.989     4M / 4M    INFO   DatasetProcessor         (dataset_processor.cpp     : 209)   Adding samfile /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/corrector_pmBnSV/lib0_UXo2Na/tmp.sam\n",
      "  0:00:30.153    44M / 44M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 1000000reads, flushing\n",
      "  0:00:34.477    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 2000000reads, flushing\n",
      "  0:00:38.758    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 3000000reads, flushing\n",
      "  0:00:43.156    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 4000000reads, flushing\n",
      "  0:00:47.546    44M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 105)   processed 5000000reads, flushing\n",
      "  0:00:50.538     8M / 48M   INFO   DatasetProcessor         (dataset_processor.cpp     : 235)   Processing contigs\n",
      "  0:00:56.612   168M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_8_length_43722_cov_59.642987 processed with 0 changes in thread 6\n",
      "  0:00:56.637   164M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_10_length_37477_cov_70.532262 processed with 0 changes in thread 10\n",
      "  0:00:57.069   160M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_9_length_43473_cov_64.820237 processed with 0 changes in thread 11\n",
      "  0:00:57.342   156M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_11_length_36438_cov_81.403129 processed with 1 changes in thread 9\n",
      "  0:00:57.384   152M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_7_length_44530_cov_66.817287 processed with 0 changes in thread 7\n",
      "  0:01:01.776   148M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_6_length_63307_cov_77.431466 processed with 0 changes in thread 5\n",
      "  0:01:02.868   140M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_5_length_76950_cov_69.042982 processed with 0 changes in thread 4\n",
      "  0:01:06.888   136M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_4_length_86646_cov_81.668073 processed with 0 changes in thread 3\n",
      "  0:01:08.945   128M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_3_length_126709_cov_62.629307 processed with 0 changes in thread 2\n",
      "  0:01:16.897   112M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_2_length_165590_cov_68.322991 processed with 0 changes in thread 1\n",
      "  0:01:50.980    96M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 251)   Contig NODE_1_length_375077_cov_67.153178 processed with 0 changes in thread 0\n",
      "  0:01:51.007    48M / 172M  INFO   DatasetProcessor         (dataset_processor.cpp     : 255)   Gluing processed contigs\n",
      "  0:01:51.011    48M / 172M  INFO    General                 (main.cpp                  :  72)   Correcting time: 0 hours 1 minutes 51 seconds\n",
      "\n",
      "===== Mismatch correction finished.\n",
      "\n",
      " * Corrected reads are in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/corrected/\n",
      " * Assembled contigs are in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/contigs.fasta\n",
      " * Assembled scaffolds are in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/scaffolds.fasta\n",
      " * Assembly graph is in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/assembly_graph.fastg\n",
      " * Assembly graph in GFA format is in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/assembly_graph_with_scaffolds.gfa\n",
      " * Paths in the assembly graph corresponding to the contigs are in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/contigs.paths\n",
      " * Paths in the assembly graph corresponding to the scaffolds are in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/scaffolds.paths\n",
      "\n",
      "======= SPAdes pipeline finished.\n",
      "\n",
      "SPAdes log can be found here: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/CW56_S1_.SS14_/spades.log\n",
      "\n",
      "Thank you for using SPAdes!\n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) Finished\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "KMERS=\"15,21,27,33,55,77,99,127\" # List of kmers for the SPAdes assembly\n",
    "NSEQ=10 # How many longest sequences we will report in short assembly fasta result\n",
    "\n",
    "mkdir -p $OUTPUT_DIR/assembly/spades\n",
    "\n",
    "for sample in *R1.mapped.fastq.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "        echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "        echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "    \n",
    "    NAME=${FORWARD%R1*}\n",
    "\n",
    "    mkdir $OUTPUT_DIR/assembly/spades/$NAME\n",
    "\n",
    "    spades.py --careful --threads $THREADS --cov-cutoff 5 --tmp-dir $OUTPUT_DIR/tmp -k $KMERS -1 $FORWARD -2 $REVERSE -o $OUTPUT_DIR/assembly/spades/$NAME\n",
    "done\n",
    "\n",
    "# Assembly statistics\n",
    "cd $OUTPUT_DIR/assembly/spades\n",
    "\n",
    "dirs_tmp=$(ls -d */)\n",
    "\n",
    "mkdir scaffolds\n",
    "mkdir graphs\n",
    "\n",
    "for sample in $dirs_tmp\n",
    "do\n",
    "    cat $sample/scaffolds.fasta > scaffolds/${sample%/*}.scaffolds.fasta\n",
    "    cat $sample/assembly_graph.fastg > graphs/${sample%/*}.assembly_graph.fastg\n",
    "    cat $sample/assembly_graph_with_scaffolds.gfa > graphs/${sample%/*}.assembly_graph_with_scaffolds.gfa\n",
    "done\n",
    "\n",
    "cd scaffolds/\n",
    "\n",
    "for sample in *.fasta\n",
    "do\n",
    "    awk \"/^>/ {n++} n>$NSEQ {exit} {print}\" $sample > ${sample%.*}.top${NSEQ}.fasta\n",
    "done\n",
    "\n",
    "echo \"Finished\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Assembly quality check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once the assembly is done, we strongly recommend running an assembly quality check. We also recommend looking at the resulting assembly graphs at [Bandage](https://rrwick.github.io/Bandage/). Outputs are described [here](#Assembly-quality-check-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) --2019-07-02 04:15:19--  https://busco.ezlab.org/datasets/spirochaetes_odb9.tar.gz\n",
      "Resolving busco.ezlab.org (busco.ezlab.org)... 129.194.96.125\n",
      "Connecting to busco.ezlab.org (busco.ezlab.org)|129.194.96.125|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 9269492 (8.8M) [application/octet-stream]\n",
      "Saving to: ‘/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/spirochaetes_odb9.tar.gz’\n",
      "\n",
      "/mnt/ssd/ssd_2/biod 100%[===================>]   8.84M  19.1MB/s    in 0.5s    \n",
      "\n",
      "2019-07-02 04:15:20 (19.1 MB/s) - ‘/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/spirochaetes_odb9.tar.gz’ saved [9269492/9269492]\n",
      "\n",
      "(treponema) spirochaetes_odb9/\n",
      "spirochaetes_odb9/hmms/\n",
      "spirochaetes_odb9/hmms/POG091N038E.hmm\n",
      "spirochaetes_odb9/hmms/POG091N014E.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01G6.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02J8.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00LU.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02LM.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03AM.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03CQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00WR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N002X.hmm\n",
      "spirochaetes_odb9/hmms/POG091N016U.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02PI.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02XE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01GP.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01LQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01NE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01QI.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01M7.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01MG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01JG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00HA.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02BT.hmm\n",
      "spirochaetes_odb9/hmms/POG091N036L.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01E5.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01UQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02A7.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01VY.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0073.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01JY.hmm\n",
      "spirochaetes_odb9/hmms/POG091N034Q.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02TE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Y6.hmm\n",
      "spirochaetes_odb9/hmms/POG091N014N.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01ZZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0017.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0180.hmm\n",
      "spirochaetes_odb9/hmms/POG091N000I.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0142.hmm\n",
      "spirochaetes_odb9/hmms/POG091N032R.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00AG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N037H.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00V5.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01GS.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02QS.hmm\n",
      "spirochaetes_odb9/hmms/POG091N030W.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02UD.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0007.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00XL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0369.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01MK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N038C.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02XQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01EQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01M4.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01QV.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00YJ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Q3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N027F.hmm\n",
      "spirochaetes_odb9/hmms/POG091N006P.hmm\n",
      "spirochaetes_odb9/hmms/POG091N020G.hmm\n",
      "spirochaetes_odb9/hmms/POG091N000F.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02NU.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00GB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01SE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01G2.hmm\n",
      "spirochaetes_odb9/hmms/POG091N004E.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0063.hmm\n",
      "spirochaetes_odb9/hmms/POG091N011X.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0294.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00SR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01KQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0158.hmm\n",
      "spirochaetes_odb9/hmms/POG091N018W.hmm\n",
      "spirochaetes_odb9/hmms/POG091N006X.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01OW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03CK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01CJ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00S7.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00SO.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01UW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00YW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00PL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02QZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00V4.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01LO.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00OW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00ZU.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03AJ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N039X.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01IT.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00P8.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0016.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02Q3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0087.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01BV.hmm\n",
      "spirochaetes_odb9/hmms/POG091N011Y.hmm\n",
      "spirochaetes_odb9/hmms/POG091N011W.hmm\n",
      "spirochaetes_odb9/hmms/POG091N020T.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01F3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N003Q.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00E5.hmm\n",
      "spirochaetes_odb9/hmms/POG091N034S.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02PQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03FK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01YD.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01TC.hmm\n",
      "spirochaetes_odb9/hmms/POG091N028X.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01OB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02Y2.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0059.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00JW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N029I.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03F0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01JZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01HD.hmm\n",
      "spirochaetes_odb9/hmms/POG091N013M.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Q5.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01JU.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01LR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00G9.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00Q0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N027V.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02SF.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01N0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N002T.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02J9.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0227.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02ZE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01WI.hmm\n",
      "spirochaetes_odb9/hmms/POG091N000C.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00H3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02H7.hmm\n",
      "spirochaetes_odb9/hmms/POG091N016S.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01D8.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01YG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01LL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01PU.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02AK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01CR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N029P.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01DP.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00EL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01OE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01LH.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02AF.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00QW.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01U9.hmm\n",
      "spirochaetes_odb9/hmms/POG091N017D.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02ZK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00LV.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Z3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00AC.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00PC.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01D3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02FJ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00EM.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02BI.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03E0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N029Z.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01RB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01VR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00IF.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01DC.hmm\n",
      "spirochaetes_odb9/hmms/POG091N017W.hmm\n",
      "spirochaetes_odb9/hmms/POG091N008Z.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02KQ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0089.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01OX.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01P7.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00RL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00RY.hmm\n",
      "spirochaetes_odb9/hmms/POG091N037O.hmm\n",
      "spirochaetes_odb9/hmms/POG091N006Q.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00R3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0186.hmm\n",
      "spirochaetes_odb9/hmms/POG091N037L.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02DN.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01AT.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03CH.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0366.hmm\n",
      "spirochaetes_odb9/hmms/POG091N015Q.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01IG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00XH.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0198.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02KB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03BZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N023H.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01EC.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0331.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0206.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00OG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00NI.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0393.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01DF.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01CT.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01P6.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00IX.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01TB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02AL.hmm\n",
      "spirochaetes_odb9/hmms/POG091N03CE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N025O.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01L4.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02SJ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N027H.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01TE.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01YO.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01J0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00BX.hmm\n",
      "spirochaetes_odb9/hmms/POG091N023F.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01T0.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02OF.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0204.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01EZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02IG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01UB.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0377.hmm\n",
      "spirochaetes_odb9/hmms/POG091N0197.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00TN.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00EN.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Z5.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02CZ.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00PN.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01Z9.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01V2.hmm\n",
      "spirochaetes_odb9/hmms/POG091N008U.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01JK.hmm\n",
      "spirochaetes_odb9/hmms/POG091N02LH.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01T3.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01E2.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01TG.hmm\n",
      "spirochaetes_odb9/hmms/POG091N001J.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00ID.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00UR.hmm\n",
      "spirochaetes_odb9/hmms/POG091N00YX.hmm\n",
      "spirochaetes_odb9/hmms/POG091N006G.hmm\n",
      "spirochaetes_odb9/hmms/POG091N014M.hmm\n",
      "spirochaetes_odb9/hmms/POG091N01HM.hmm\n",
      "spirochaetes_odb9/ancestral\n",
      "spirochaetes_odb9/dataset.cfg\n",
      "spirochaetes_odb9/prfl/\n",
      "spirochaetes_odb9/prfl/POG091N01UQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0393.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0294.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01E2.prfl\n",
      "spirochaetes_odb9/prfl/POG091N006X.prfl\n",
      "spirochaetes_odb9/prfl/POG091N017W.prfl\n",
      "spirochaetes_odb9/prfl/POG091N006Q.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01N0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02IG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01YD.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00IX.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01OE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00ZU.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01OB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N000F.prfl\n",
      "spirochaetes_odb9/prfl/POG091N016U.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01VR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01TE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02AF.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00YW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00JW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01U9.prfl\n",
      "spirochaetes_odb9/prfl/POG091N029I.prfl\n",
      "spirochaetes_odb9/prfl/POG091N011W.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02UD.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02OF.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01M7.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00NI.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00ID.prfl\n",
      "spirochaetes_odb9/prfl/POG091N002X.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01F3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01JU.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02LM.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01JY.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01CJ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Q5.prfl\n",
      "spirochaetes_odb9/prfl/POG091N023F.prfl\n",
      "spirochaetes_odb9/prfl/POG091N037L.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03CK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03E0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00XL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01E5.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01ZZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00BX.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00EL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01YG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01G6.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0186.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00IF.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0063.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02ZK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02NU.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00Q0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01DC.prfl\n",
      "spirochaetes_odb9/prfl/POG091N014N.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00SR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0227.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02FJ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00QW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0158.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01EC.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01EZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01V2.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01SE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01EQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02PI.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01GP.prfl\n",
      "spirochaetes_odb9/prfl/POG091N027F.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00H3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0059.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00PN.prfl\n",
      "spirochaetes_odb9/prfl/POG091N027H.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Z3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02KB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N000C.prfl\n",
      "spirochaetes_odb9/prfl/POG091N025O.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00AC.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01GS.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00WR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0089.prfl\n",
      "spirochaetes_odb9/prfl/POG091N008Z.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01L4.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02BT.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00GB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01JG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01LH.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0366.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01HM.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01JK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00EM.prfl\n",
      "spirochaetes_odb9/prfl/POG091N032R.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02XE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N018W.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03CQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01T3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00V4.prfl\n",
      "spirochaetes_odb9/prfl/POG091N003Q.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00PC.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01HD.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02AL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01TC.prfl\n",
      "spirochaetes_odb9/prfl/POG091N023H.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03F0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01AT.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00RY.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00PL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01G2.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00TN.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01MG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01OW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01DF.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00YX.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Z9.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02BI.prfl\n",
      "spirochaetes_odb9/prfl/POG091N029Z.prfl\n",
      "spirochaetes_odb9/prfl/POG091N000I.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02KQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N020T.prfl\n",
      "spirochaetes_odb9/prfl/POG091N037O.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01J0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01BV.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Y6.prfl\n",
      "spirochaetes_odb9/prfl/POG091N038E.prfl\n",
      "spirochaetes_odb9/prfl/POG091N011Y.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0073.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02J9.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02A7.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0204.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02TE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02J8.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Q3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N037H.prfl\n",
      "spirochaetes_odb9/prfl/POG091N028X.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01OX.prfl\n",
      "spirochaetes_odb9/prfl/POG091N029P.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0180.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02H7.prfl\n",
      "spirochaetes_odb9/prfl/POG091N036L.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02PQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03CE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00V5.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02QZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03AM.prfl\n",
      "spirochaetes_odb9/prfl/POG091N017D.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00S7.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01CR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00XH.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01YO.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02SJ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0206.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01IG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02Q3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01D3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02DN.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01UW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0197.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01IT.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03FK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N006G.prfl\n",
      "spirochaetes_odb9/prfl/POG091N015Q.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01Z5.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03AJ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N030W.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01PU.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01P7.prfl\n",
      "spirochaetes_odb9/prfl/POG091N039X.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02LH.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02CZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01CT.prfl\n",
      "spirochaetes_odb9/prfl/POG091N034Q.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01LL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02ZE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0142.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00E5.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02XQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00R3.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00OG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N016S.prfl\n",
      "spirochaetes_odb9/prfl/POG091N038C.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01LR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N034S.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03CH.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01T0.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01RB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N027V.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02Y2.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01QV.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00SO.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00LU.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01DP.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01TG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N011X.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01LQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01WI.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00YJ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01VY.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00OW.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00UR.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00HA.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01JZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N014M.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0087.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01LO.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0016.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0331.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00AG.prfl\n",
      "spirochaetes_odb9/prfl/POG091N013M.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01P6.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0017.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0198.prfl\n",
      "spirochaetes_odb9/prfl/POG091N001J.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00G9.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01UB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01MK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02SF.prfl\n",
      "spirochaetes_odb9/prfl/POG091N008U.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01NE.prfl\n",
      "spirochaetes_odb9/prfl/POG091N004E.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00LV.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02QS.prfl\n",
      "spirochaetes_odb9/prfl/POG091N002T.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00P8.prfl\n",
      "spirochaetes_odb9/prfl/POG091N020G.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01KQ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00EN.prfl\n",
      "spirochaetes_odb9/prfl/POG091N014E.prfl\n",
      "spirochaetes_odb9/prfl/POG091N00RL.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01TB.prfl\n",
      "spirochaetes_odb9/prfl/POG091N006P.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01D8.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01QI.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0377.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0007.prfl\n",
      "spirochaetes_odb9/prfl/POG091N03BZ.prfl\n",
      "spirochaetes_odb9/prfl/POG091N0369.prfl\n",
      "spirochaetes_odb9/prfl/POG091N02AK.prfl\n",
      "spirochaetes_odb9/prfl/POG091N01M4.prfl\n",
      "spirochaetes_odb9/ancestral_variants\n",
      "spirochaetes_odb9/scores_cutoff\n",
      "spirochaetes_odb9/lengths_cutoff\n",
      "spirochaetes_odb9/info/\n",
      "spirochaetes_odb9/info/ogs.id.info\n",
      "spirochaetes_odb9/info/species.info\n",
      "spirochaetes_odb9/info/spirochaetes_203691_OrthoDB9_orthogroup_info.txt.gz\n",
      "(treponema) (treponema) (treponema) (treponema) WARNING\tAn augustus species is mentioned in the config file, dataset default species (E_coli_K12) will be ignored\n",
      "INFO\t****************** Start a BUSCO 3.0.2 analysis, current time: 07/02/2019 04:15:21 ******************\n",
      "INFO\tConfiguration loaded from /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/../config/config.ini\n",
      "INFO\tInit tools...\n",
      "INFO\tCheck dependencies...\n",
      "INFO\tCheck input file...\n",
      "INFO\tTo reproduce this run: python /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/run_BUSCO.py -i CW56_S1_.SS14_.scaffolds.fasta -o busco_CW56_S1_.SS14_.scaffolds -l /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/spirochaetes_odb9/ -m genome -c 12 -t /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/ -sp E_coli_K12\n",
      "INFO\tMode is: genome\n",
      "INFO\tThe lineage dataset is: spirochaetes_odb9 (prokaryota)\n",
      "INFO\tTemp directory is /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/\n",
      "INFO\t****** Phase 1 of 2, initial predictions ******\n",
      "INFO\t****** Step 1/3, current time: 07/02/2019 04:15:21 ******\n",
      "INFO\tCreate blast database...\n",
      "INFO\t[makeblastdb]\tBuilding a new DB, current time: 07/02/2019 04:15:22\n",
      "INFO\t[makeblastdb]\tNew DB name:   /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/tmp/busco_CW56_S1_.SS14_.scaffolds_3864771\n",
      "INFO\t[makeblastdb]\tNew DB title:  CW56_S1_.SS14_.scaffolds.fasta\n",
      "INFO\t[makeblastdb]\tSequence type: Nucleotide\n",
      "INFO\t[makeblastdb]\tKeep MBits: T\n",
      "INFO\t[makeblastdb]\tMaximum file size: 1000000000B\n",
      "INFO\t[makeblastdb]\tAdding sequences from FASTA; added 66 sequences in 0.0121789 seconds.\n",
      "INFO\t[makeblastdb]\t1 of 1 task(s) completed at 07/02/2019 04:15:22\n",
      "INFO\tRunning tblastn, writing output to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/run_busco_CW56_S1_.SS14_.scaffolds/blast_output/tblastn_busco_CW56_S1_.SS14_.scaffolds.tsv...\n",
      "INFO\t[tblastn]\t1 of 1 task(s) completed at 07/02/2019 04:15:23\n",
      "INFO\t****** Step 2/3, current time: 07/02/2019 04:15:23 ******\n",
      "INFO\tMaximum number of candidate contig per BUSCO limited to: 3\n",
      "INFO\tGetting coordinates for candidate regions...\n",
      "INFO\tPre-Augustus scaffold extraction...\n",
      "INFO\tRunning Augustus prediction using E_coli_K12 as species:\n",
      "INFO\t[augustus]\tPlease find all logs related to Augustus errors here: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/run_busco_CW56_S1_.SS14_.scaffolds/augustus_output/augustus.log\n",
      "INFO\t[augustus]\t23 of 228 task(s) completed at 07/02/2019 04:15:25\n",
      "INFO\t[augustus]\t46 of 228 task(s) completed at 07/02/2019 04:15:26\n",
      "INFO\t[augustus]\t69 of 228 task(s) completed at 07/02/2019 04:15:28\n",
      "INFO\t[augustus]\t92 of 228 task(s) completed at 07/02/2019 04:15:30\n",
      "INFO\t[augustus]\t115 of 228 task(s) completed at 07/02/2019 04:15:31\n",
      "INFO\t[augustus]\t137 of 228 task(s) completed at 07/02/2019 04:15:33\n",
      "INFO\t[augustus]\t160 of 228 task(s) completed at 07/02/2019 04:15:35\n",
      "INFO\t[augustus]\t206 of 228 task(s) completed at 07/02/2019 04:15:38\n",
      "INFO\t[augustus]\t228 of 228 task(s) completed at 07/02/2019 04:15:41\n",
      "INFO\tExtracting predicted proteins...\n",
      "INFO\t****** Step 3/3, current time: 07/02/2019 04:15:42 ******\n",
      "INFO\tRunning HMMER to confirm orthology of predicted proteins:\n",
      "INFO\t[hmmsearch]\t46 of 228 task(s) completed at 07/02/2019 04:15:43\n",
      "INFO\t[hmmsearch]\t115 of 228 task(s) completed at 07/02/2019 04:15:43\n",
      "INFO\t[hmmsearch]\t137 of 228 task(s) completed at 07/02/2019 04:15:43\n",
      "INFO\t[hmmsearch]\t183 of 228 task(s) completed at 07/02/2019 04:15:43\n",
      "INFO\t[hmmsearch]\t228 of 228 task(s) completed at 07/02/2019 04:15:43\n",
      "INFO\tResults:\n",
      "INFO\tC:84.8%[S:84.8%,D:0.0%],F:2.5%,M:12.7%,n:237\n",
      "INFO\t201 Complete BUSCOs (C)\n",
      "INFO\t201 Complete and single-copy BUSCOs (S)\n",
      "INFO\t0 Complete and duplicated BUSCOs (D)\n",
      "INFO\t6 Fragmented BUSCOs (F)\n",
      "INFO\t30 Missing BUSCOs (M)\n",
      "INFO\t237 Total BUSCO groups searched\n",
      "INFO\t****** Phase 2 of 2, predictions using species specific training ******\n",
      "INFO\t****** Step 1/3, current time: 07/02/2019 04:15:43 ******\n",
      "INFO\tExtracting missing and fragmented buscos from the ancestral_variants file...\n",
      "INFO\tRunning tblastn, writing output to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/run_busco_CW56_S1_.SS14_.scaffolds/blast_output/tblastn_busco_CW56_S1_.SS14_.scaffolds_missing_and_frag_rerun.tsv...\n",
      "INFO\t[tblastn]\t1 of 1 task(s) completed at 07/02/2019 04:15:44\n",
      "INFO\tMaximum number of candidate contig per BUSCO limited to: 3\n",
      "INFO\tGetting coordinates for candidate regions...\n",
      "INFO\t****** Step 2/3, current time: 07/02/2019 04:15:44 ******\n",
      "INFO\tTraining Augustus using Single-Copy Complete BUSCOs:\n",
      "INFO\tConverting predicted genes to short genbank files at 07/02/2019 04:15:44...\n",
      "INFO\tAll files converted to short genbank files, now running the training scripts at 07/02/2019 04:15:45...\n",
      "INFO\tPre-Augustus scaffold extraction...\n",
      "INFO\tRe-running Augustus with the new metaparameters, number of target BUSCOs: 36\n",
      "INFO\t[augustus]\t3 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t5 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t8 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t10 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t13 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t15 of 25 task(s) completed at 07/02/2019 04:15:46\n",
      "INFO\t[augustus]\t18 of 25 task(s) completed at 07/02/2019 04:15:47\n",
      "INFO\t[augustus]\t20 of 25 task(s) completed at 07/02/2019 04:15:47\n",
      "INFO\t[augustus]\t23 of 25 task(s) completed at 07/02/2019 04:15:47\n",
      "INFO\t[augustus]\t25 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\tExtracting predicted proteins...\n",
      "INFO\t****** Step 3/3, current time: 07/02/2019 04:15:49 ******\n",
      "INFO\tRunning HMMER to confirm orthology of predicted proteins:\n",
      "INFO\t[hmmsearch]\t8 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\t[hmmsearch]\t10 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\t[hmmsearch]\t18 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\t[hmmsearch]\t20 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\t[hmmsearch]\t23 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\t[hmmsearch]\t25 of 25 task(s) completed at 07/02/2019 04:15:49\n",
      "INFO\tResults:\n",
      "INFO\tC:91.1%[S:91.1%,D:0.0%],F:2.1%,M:6.8%,n:237\n",
      "INFO\t216 Complete BUSCOs (C)\n",
      "INFO\t216 Complete and single-copy BUSCOs (S)\n",
      "INFO\t0 Complete and duplicated BUSCOs (D)\n",
      "INFO\t5 Fragmented BUSCOs (F)\n",
      "INFO\t16 Missing BUSCOs (M)\n",
      "INFO\t237 Total BUSCO groups searched\n",
      "INFO\tBUSCO analysis done with WARNING(s). Total running time: 28.5721929073 seconds\n",
      "INFO\tResults written in /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/run_busco_CW56_S1_.SS14_.scaffolds/\n",
      "\n",
      "(treponema) (treponema) (treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/quast -r /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -t 12 --output-dir /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast CW56_S1_.SS14_.scaffolds.fasta\n",
      "\n",
      "Version: 5.0.2\n",
      "\n",
      "System information:\n",
      "  OS: Linux-4.4.0-148-generic-x86_64-with-debian-stretch-sid (linux_64)\n",
      "  Python version: 2.7.15\n",
      "  CPUs number: 70\n",
      "\n",
      "Started: 2019-07-02 04:15:50\n",
      "\n",
      "Logging to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/quast.log\n",
      "\n",
      "CWD: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds\n",
      "Main parameters: \n",
      "  MODE: default, threads: 12, minimum contig length: 500, minimum alignment length: 65, \\\n",
      "  ambiguity: one, threshold for extensive misassembly size: 1000\n",
      "\n",
      "Reference:\n",
      "  /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa ==> SS14\n",
      "\n",
      "Contigs:\n",
      "  Pre-processing...\n",
      "  CW56_S1_.SS14_.scaffolds.fasta ==> CW56_S1_.SS14_.scaffolds\n",
      "\n",
      "2019-07-02 04:15:52\n",
      "Running Basic statistics processor...\n",
      "  Reference genome:\n",
      "    SS14.fa, length = 1139569, num fragments = 1, GC % = 52.79\n",
      "  Contig files: \n",
      "    CW56_S1_.SS14_.scaffolds\n",
      "  Calculating N50 and L50...\n",
      "    CW56_S1_.SS14_.scaffolds, N50 = 126709, L50 = 3, Total length = 1143162, GC % = 52.73, # N's per 100 kbp =  10.41\n",
      "  Drawing Nx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/Nx_plot.pdf\n",
      "  Drawing NGx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/NGx_plot.pdf\n",
      "  Drawing cumulative plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/cumulative_plot.pdf\n",
      "  Drawing GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/GC_content_plot.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_.SS14_.scaffolds_GC_content_plot.pdf\n",
      "  Drawing Coverage histogram (bin size: 1x)...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/coverage_histogram.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds coverage histogram (bin size: 1x)...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_-SS14_-scaffolds_coverage_histogram.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:15:54\n",
      "Running Contig analyzer...\n",
      "  CW56_S1_.SS14_.scaffolds\n",
      "  Logging to files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/contigs_report_CW56_S1_-SS14_-scaffolds.stdout and contigs_report_CW56_S1_-SS14_-scaffolds.stderr...\n",
      "  Aligning contigs to the reference\n",
      "  Analysis is finished.\n",
      "  Creating total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_report.txt, misassemblies_report.tsv, and misassemblies_report.tex\n",
      "  Transposed version of total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/transposed_report_misassemblies.txt, transposed_report_misassemblies.tsv, and transposed_report_misassemblies.tex\n",
      "  Creating total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/unaligned_report.txt, unaligned_report.tsv, and unaligned_report.tex\n",
      "  Drawing misassemblies by types plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_plot.pdf\n",
      "  Drawing misassemblies FRCurve plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_frcurve_plot.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:15:55\n",
      "Running NA-NGA calculation...\n",
      "  CW56_S1_.SS14_.scaffolds, Largest alignment = 375077, NA50 = 126709, NGA50 = 126709, LA50 = 3, LGA50 = 3\n",
      "  Drawing cumulative plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/cumulative_plot.pdf\n",
      "  Drawing NAx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/NAx_plot.pdf\n",
      "  Drawing NGAx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/NGAx_plot.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:15:56\n",
      "Running Genome analyzer...\n",
      "  NOTICE: No file with genomic features were provided. Use the --features option if you want to specify it.\n",
      "\n",
      "  NOTICE: No file with operons were provided. Use the -O option if you want to specify it.\n",
      "  CW56_S1_.SS14_.scaffolds\n",
      "  Analysis is finished.\n",
      "  Skipping drawing Genome fraction, % histogram... (less than 2 columns histogram makes no sense)\n",
      "Done.\n",
      "\n",
      "NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it.\n",
      "\n",
      "2019-07-02 04:15:56\n",
      "Creating large visual summaries...\n",
      "This may take a while: press Ctrl-C to skip this step..\n",
      "  1 of 2: Creating Icarus viewers...\n",
      "  2 of 2: Creating PDF with all tables and plots...\n",
      "Done\n",
      "\n",
      "2019-07-02 04:15:58\n",
      "RESULTS:\n",
      "  Text versions of total report are saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.txt, report.tsv, and report.tex\n",
      "  Text versions of transposed total report are saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex\n",
      "  HTML version (interactive tables and plots) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.html\n",
      "  PDF version (tables and plots) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.pdf\n",
      "  Icarus (contig browser) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/icarus.html\n",
      "  Log is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/quast.log\n",
      "\n",
      "Finished: 2019-07-02 04:15:58\n",
      "Elapsed time: 0:00:08.174783\n",
      "NOTICEs: 3; WARNINGs: 0; non-fatal ERRORs: 0\n",
      "\n",
      "Thank you for using QUAST!\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/qc/assembly/BUSCO\n",
    "mkdir $OUTPUT_DIR/qc/assembly/quast\n",
    "\n",
    "# Get BUSCO core genes annotation database\n",
    "wget -O $REFERENCES/spirochaetes_odb9.tar.gz https://busco.ezlab.org/datasets/spirochaetes_odb9.tar.gz\n",
    "tar xvzf $REFERENCES/spirochaetes_odb9.tar.gz -C $REFERENCES\n",
    "\n",
    "# -sp XXX is one of the supported organisms by Augustus and should be as close as possible to our organism https://github.com/Gaius-Augustus/Augustus/blob/master/docs/RUNNING-AUGUSTUS.md\n",
    "\n",
    "for sample in *.scaffolds.fasta\n",
    "do\n",
    "    echo \"Running BUSCO for $sample\"\n",
    "\n",
    "    run_BUSCO.py -i $sample -o busco_${sample%.fa*} -l $REFERENCES/spirochaetes_odb9 -m geno \\\n",
    "    -c $THREADS -sp E_coli_K12 -t $OUTPUT_DIR/tmp\n",
    "    \n",
    "    multiqc -o run_busco_${sample%.fa*} run_busco_${sample%.fa*}/\n",
    "done\n",
    "\n",
    "mv run_busco* $OUTPUT_DIR/qc/assembly/BUSCO/\n",
    "\n",
    "quast -r $REFERENCE -t $THREADS --output-dir $OUTPUT_DIR/qc/assembly/quast *.scaffolds.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Additional scaffolding (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "`SPAdes` is quite efficient in the scaffolding and we never had the need to do additional scaffolding but you might find yourself in a situation to do so. We are using [BESST](https://github.com/ksahlin/BESST) but there are other options such as [SSPACE-basic](https://github.com/nsoranzo/sspace_basic), [SSPACE-standard](https://www.baseclear.com/services/bioinformatics/basetools/sspace-standard/) and others. Outputs are described [here](#Additional-scaffolding-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "It is **extremely** important to compare the original assembly scaffolds and the \"re-scaffolded\" ones and determine which ones we want to use. Longer doesn't always mean better. Carefully check the Quast report and compare the original vs. re-scaffolded and make the decision. I would always check \"# misassemblies\" or the whole-genome alignments and consider if it has improved or not or if there is some biology behind the change. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "For the best functionality, you have to know the insert size. You can get this information from the Qualimap report which we created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) (treponema) [bwa_index] Pack FASTA... 0.01 sec\n",
      "[bwa_index] Construct BWT for the packed sequence...\n",
      "[bwa_index] 0.28 seconds elapse.\n",
      "[bwa_index] Update BWT... 0.01 sec\n",
      "[bwa_index] Pack forward-only FASTA... 0.01 sec\n",
      "[bwa_index] Construct SA from BWT and Occ... 0.12 sec\n",
      "[main] Version: 0.7.15-r1140\n",
      "[main] CMD: bwa index /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/CW56_S1_.SS14_.scaffolds.fasta\n",
      "[main] Real time: 0.423 sec; CPU: 0.416 sec\n",
      "[M::bwa_idx_load_from_disk] read 0 ALT contigs\n",
      "[M::process] read 837344 sequences (120000200 bp)...\n",
      "[M::process] read 837422 sequences (120000149 bp)...\n",
      "[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (1, 414269, 1, 0)\n",
      "[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 235, 334)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 670)\n",
      "[M::mem_pestat] mean and std.dev: (258.05, 127.44)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 838)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
      "[M::mem_process_seqs] Processed 837344 reads in 55.740 CPU sec, 5.994 real sec\n",
      "[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (3, 414182, 2, 1)\n",
      "[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 333)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 667)\n",
      "[M::mem_pestat] mean and std.dev: (257.56, 127.15)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 834)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
      "[M::mem_process_seqs] Processed 837422 reads in 47.484 CPU sec, 4.504 real sec\n",
      "[M::process] read 837424 sequences (120000122 bp)...\n",
      "[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (6, 414148, 0, 1)\n",
      "[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 334)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 670)\n",
      "[M::mem_pestat] mean and std.dev: (257.78, 127.57)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 838)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
      "[M::mem_process_seqs] Processed 837424 reads in 51.608 CPU sec, 4.490 real sec\n",
      "[M::process] read 837348 sequences (120000224 bp)...\n",
      "[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (2, 414203, 0, 1)\n",
      "[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 334)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 670)\n",
      "[M::mem_pestat] mean and std.dev: (257.80, 127.42)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 838)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
      "[M::mem_process_seqs] Processed 837348 reads in 47.716 CPU sec, 4.216 real sec\n",
      "[M::process] read 611068 sequences (87566910 bp)...\n",
      "[M::mem_pestat] # candidate unique pairs for (FF, FR, RF, RR): (5, 302298, 1, 0)\n",
      "[M::mem_pestat] skip orientation FF as there are not enough pairs\n",
      "[M::mem_pestat] analyzing insert size distribution for orientation FR...\n",
      "[M::mem_pestat] (25, 50, 75) percentile: (166, 234, 333)\n",
      "[M::mem_pestat] low and high boundaries for computing mean and std.dev: (1, 667)\n",
      "[M::mem_pestat] mean and std.dev: (257.64, 127.48)\n",
      "[M::mem_pestat] low and high boundaries for proper pairs: (1, 834)\n",
      "[M::mem_pestat] skip orientation RF as there are not enough pairs\n",
      "[M::mem_pestat] skip orientation RR as there are not enough pairs\n",
      "[M::mem_process_seqs] Processed 611068 reads in 33.668 CPU sec, 2.954 real sec\n",
      "[main] Version: 0.7.15-r1140\n",
      "[main] CMD: bwa mem -t 12 -w 0 -O 99 /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/CW56_S1_.SS14_.scaffolds.fasta CW56_S1_.SS14_R1.mapped.fastq.gz CW56_S1_.SS14_R2.mapped.fastq.gz\n",
      "[main] Real time: 82.522 sec; CPU: 252.928 sec\n",
      "Number of initial contigs: 66\n",
      "Choosing mode: 202\n",
      "mu_adjusted:267.674563696, sigma_adjusted:141.096532763, skewness_adjusted:1.12554803241\n",
      "mode adj: 202\n",
      "median adj 238\n",
      "Creating contig graph with library:  /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds_besst/CW56_S1_.SS14_.mapped.bam\n",
      "Constructed contig graph. Start BESST algorithm for creating scaffolds. \n",
      "Entering \"find_all_paths_for_start_node\" \n",
      "iterating until maximum of 14 extensions.\n",
      "Number of nodes:132, Number of edges: 89\n",
      "enter Between scaf node:0, scaffold progression 0.0%. \n",
      "Total nr of paths found: 2 with score larger than: 1.5\n",
      "Elapsed time single core pathfinder:  0.000823020935059\n",
      "Writing out scaffolding results for step 1  ...\n",
      "Finished\n",
      "\n",
      " \n",
      "(treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/quast -r /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa -t 12 --output-dir /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast CW56_S1_.SS14_.scaffolds_besst.fasta CW56_S1_.SS14_.scaffolds.fasta CW56_S1_.SS14_.scaffolds.top10.fasta\n",
      "\n",
      "Version: 5.0.2\n",
      "\n",
      "System information:\n",
      "  OS: Linux-4.4.0-148-generic-x86_64-with-debian-stretch-sid (linux_64)\n",
      "  Python version: 2.7.15\n",
      "  CPUs number: 70\n",
      "\n",
      "Started: 2019-07-02 04:22:27\n",
      "\n",
      "Logging to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/quast.log\n",
      "NOTICE: Output directory already exists and looks like a QUAST output dir. Existing results can be reused (e.g. previously generated alignments)!\n",
      "\n",
      "CWD: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds\n",
      "Main parameters: \n",
      "  MODE: default, threads: 12, minimum contig length: 500, minimum alignment length: 65, \\\n",
      "  ambiguity: one, threshold for extensive misassembly size: 1000\n",
      "\n",
      "Reference:\n",
      "  /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/references/SS14.fa ==> SS14\n",
      "\n",
      "Contigs:\n",
      "  Pre-processing...\n",
      "  1  CW56_S1_.SS14_.scaffolds_besst.fasta ==> CW56_S1_.SS14_.scaffolds_besst\n",
      "  2  CW56_S1_.SS14_.scaffolds.fasta ==> CW56_S1_.SS14_.scaffolds\n",
      "  3  CW56_S1_.SS14_.scaffolds.top10.fasta ==> CW56_S1_.SS14_.scaffolds.top10\n",
      "\n",
      "2019-07-02 04:22:28\n",
      "Running Basic statistics processor...\n",
      "  Reference genome:\n",
      "    SS14.fa, length = 1139569, num fragments = 1, GC % = 52.79\n",
      "  Contig files: \n",
      "    1  CW56_S1_.SS14_.scaffolds_besst\n",
      "    2  CW56_S1_.SS14_.scaffolds\n",
      "    3  CW56_S1_.SS14_.scaffolds.top10\n",
      "  Calculating N50 and L50...\n",
      "    1  CW56_S1_.SS14_.scaffolds_besst, N50 = 126709, L50 = 3, Total length = 1143367, GC % = 52.73, # N's per 100 kbp =  10.58\n",
      "    2  CW56_S1_.SS14_.scaffolds, N50 = 126709, L50 = 3, Total length = 1143162, GC % = 52.73, # N's per 100 kbp =  10.41\n",
      "    3  CW56_S1_.SS14_.scaffolds.top10, N50 = 165590, L50 = 2, Total length = 1063481, GC % = 52.63, # N's per 100 kbp =  0.00\n",
      "  Drawing Nx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/Nx_plot.pdf\n",
      "  Drawing NGx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/NGx_plot.pdf\n",
      "  Drawing cumulative plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/cumulative_plot.pdf\n",
      "  Drawing GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/GC_content_plot.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds_besst GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_.SS14_.scaffolds_besst_GC_content_plot.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_.SS14_.scaffolds_GC_content_plot.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds.top10 GC content plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_.SS14_.scaffolds.top10_GC_content_plot.pdf\n",
      "  Drawing Coverage histogram (bin size: 1x)...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/coverage_histogram.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds coverage histogram (bin size: 1x)...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_-SS14_-scaffolds_coverage_histogram.pdf\n",
      "  Drawing CW56_S1_.SS14_.scaffolds.top10 coverage histogram (bin size: 1x)...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/basic_stats/CW56_S1_-SS14_-scaffolds-top10_coverage_histogram.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:22:31\n",
      "Running Contig analyzer...\n",
      "  1  CW56_S1_.SS14_.scaffolds_besst\n",
      "  1  Logging to files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/contigs_report_CW56_S1_-SS14_-scaffolds_besst.stdout and contigs_report_CW56_S1_-SS14_-scaffolds_besst.stderr...\n",
      "  2  CW56_S1_.SS14_.scaffolds\n",
      "  1  Aligning contigs to the reference\n",
      "  2  Logging to files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/contigs_report_CW56_S1_-SS14_-scaffolds.stdout and contigs_report_CW56_S1_-SS14_-scaffolds.stderr...\n",
      "  3  CW56_S1_.SS14_.scaffolds.top10\n",
      "  3  Logging to files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/contigs_report_CW56_S1_-SS14_-scaffolds-top10.stdout and contigs_report_CW56_S1_-SS14_-scaffolds-top10.stderr...\n",
      "  3  Aligning contigs to the reference\n",
      "  2  Using existing alignments... \n",
      "  2  Analysis is finished.\n",
      "  3  Analysis is finished.\n",
      "  1  Analysis is finished.\n",
      "  Creating total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_report.txt, misassemblies_report.tsv, and misassemblies_report.tex\n",
      "  Transposed version of total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/transposed_report_misassemblies.txt, transposed_report_misassemblies.tsv, and transposed_report_misassemblies.tex\n",
      "  Creating total report...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/unaligned_report.txt, unaligned_report.tsv, and unaligned_report.tex\n",
      "  Drawing misassemblies by types plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_plot.pdf\n",
      "  Drawing misassemblies FRCurve plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/contigs_reports/misassemblies_frcurve_plot.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:22:32\n",
      "Running NA-NGA calculation...\n",
      "  1  CW56_S1_.SS14_.scaffolds_besst, Largest alignment = 375077, NA50 = 126709, NGA50 = 126709, LA50 = 3, LGA50 = 3\n",
      "  2  CW56_S1_.SS14_.scaffolds, Largest alignment = 375077, NA50 = 126709, NGA50 = 126709, LA50 = 3, LGA50 = 3\n",
      "  3  CW56_S1_.SS14_.scaffolds.top10, Largest alignment = 375077, NA50 = 165590, NGA50 = 126709, LA50 = 2, LGA50 = 3\n",
      "  Drawing cumulative plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/cumulative_plot.pdf\n",
      "  Drawing NAx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/NAx_plot.pdf\n",
      "  Drawing NGAx plot...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/aligned_stats/NGAx_plot.pdf\n",
      "Done.\n",
      "\n",
      "2019-07-02 04:22:33\n",
      "Running Genome analyzer...\n",
      "  NOTICE: No file with genomic features were provided. Use the --features option if you want to specify it.\n",
      "\n",
      "  NOTICE: No file with operons were provided. Use the -O option if you want to specify it.\n",
      "  1  CW56_S1_.SS14_.scaffolds_besst\n",
      "  2  CW56_S1_.SS14_.scaffolds\n",
      "  3  CW56_S1_.SS14_.scaffolds.top10\n",
      "  1  Analysis is finished.\n",
      "  3  Analysis is finished.\n",
      "  2  Analysis is finished.\n",
      "  Drawing Genome fraction, % histogram...\n",
      "    saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/genome_stats/genome_fraction_histogram.pdf\n",
      "Done.\n",
      "\n",
      "NOTICE: Genes are not predicted by default. Use --gene-finding or --glimmer option to enable it.\n",
      "\n",
      "2019-07-02 04:22:34\n",
      "Creating large visual summaries...\n",
      "This may take a while: press Ctrl-C to skip this step..\n",
      "  1 of 2: Creating Icarus viewers...\n",
      "  2 of 2: Creating PDF with all tables and plots...\n",
      "Done\n",
      "\n",
      "2019-07-02 04:22:37\n",
      "RESULTS:\n",
      "  Text versions of total report are saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.txt, report.tsv, and report.tex\n",
      "  Text versions of transposed total report are saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/transposed_report.txt, transposed_report.tsv, and transposed_report.tex\n",
      "  HTML version (interactive tables and plots) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.html\n",
      "  PDF version (tables and plots) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/report.pdf\n",
      "  Icarus (contig browser) is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/icarus.html\n",
      "  Log is saved to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/qc/assembly/quast/quast.log\n",
      "\n",
      "Finished: 2019-07-02 04:22:37\n",
      "Elapsed time: 0:00:09.989051\n",
      "NOTICEs: 4; WARNINGs: 0; non-fatal ERRORs: 0\n",
      "\n",
      "Thank you for using QUAST!\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir $OUTPUT_DIR/assembly/spades/scaffolds_besst\n",
    "\n",
    "cd $OUTPUT_DIR/alignment/fastq\n",
    "\n",
    "for sample in *R1.mapped.fastq.gz\n",
    "do\n",
    "    FORWARD=$sample\n",
    "    extension=\"${FORWARD##*R1}\"\n",
    "    REVERSE=${FORWARD%R1*}R2${extension}\n",
    "\n",
    "    # Input file check\n",
    "    if [ ! -f ${FORWARD} ]; then\n",
    "        echo \"Input file not found! First in pair.\"; echo ${FORWARD}\n",
    "    fi\n",
    "    if [ ! -f ${REVERSE} ]; then\n",
    "        echo \"Input file not found! Second in pair.\"; echo ${REVERSE}\n",
    "    fi\n",
    "    \n",
    "    contig=$OUTPUT_DIR/assembly/spades/scaffolds/${FORWARD%R1*}.scaffolds.fasta # SPAdes assembly contigs/scaffolds\n",
    "\n",
    "    bwa index $contig\n",
    "\n",
    "    bwa mem -t $THREADS -w 0 -O 99 $contig $FORWARD $REVERSE | samtools view -F 4 -@ $THREADS -b - | samtools fixmate -O bam - - | samtools sort -@ $THREADS - > $OUTPUT_DIR/assembly/spades/scaffolds_besst/${FORWARD%R1*}${extension%.fastq.gz*}.bam # Make alignment for BESST; default is BWA-MEM and recommended settings are used; they have script to do it (reads_to_ctg_map.py) but it has some error - setting taken from there\n",
    "\n",
    "    samtools index -@ $THREADS $OUTPUT_DIR/assembly/spades/scaffolds_besst/${FORWARD%R1*}${extension%.fastq.gz*}.bam\n",
    "\n",
    "    mkdir $OUTPUT_DIR/assembly/spades/scaffolds_besst/${FORWARD%R1*}\n",
    "\n",
    "#  -m MEAN [MEAN ...]    Mean insert size of libraries.\n",
    "#  -s STDDEV [STDDEV ...]\n",
    "#                        Estimated standard deviation of libraries.\n",
    "    runBESST -c $contig -f $OUTPUT_DIR/assembly/spades/scaffolds_besst/${FORWARD%R1*}${extension%.fastq.gz*}.bam -filter_contigs 100 -orientation fr -o $OUTPUT_DIR/assembly/spades/scaffolds_besst/${FORWARD%R1*}\n",
    "done\n",
    "\n",
    "# Clean BWA indexes and alignments\n",
    "cd $OUTPUT_DIR/assembly/spades/scaffolds/\n",
    "rm *.amb *.ann *.bwt *.pac *.sa\n",
    "\n",
    "rm $OUTPUT_DIR/assembly/spades/scaffolds_besst/*.bam*\n",
    "\n",
    "cd $OUTPUT_DIR/assembly/spades/scaffolds_besst\n",
    "\n",
    "for sample in $(ls -d */)\n",
    "do\n",
    "    cat $sample/BESST_output/pass1/Scaffolds_pass1.fa > $OUTPUT_DIR/assembly/spades/scaffolds_besst/${sample%/*}.scaffolds.fasta\n",
    "done\n",
    "\n",
    "cd $OUTPUT_DIR/assembly/spades/scaffolds_besst/\n",
    "\n",
    "for sample in *.scaffolds.fasta\n",
    "do\n",
    "    cat $sample | awk '/^>/ {printf(\"%s%s\\t\",(N>0?\"\\n\":\"\"),$0);N++;next;} {printf(\"%s\",$0);} END {printf(\"\\n\");}' | awk -F '\\t' '{printf(\"%d\\t%s\\n\",length($2),$0);}' | sort -k1,1nr | cut -f 2- | tr \"\\t\" \"\\n\" > $sample.tmp # Longest first; shortest first change sort 'sort -k1,1nr' to 'sort -k1,1n'\n",
    "    mv $sample.tmp $sample\n",
    "    cat $sample > $OUTPUT_DIR/assembly/spades/scaffolds/${sample%.scaffolds.fasta}.scaffolds_besst.fasta \n",
    "done\n",
    "\n",
    "cd $OUTPUT_DIR/assembly/spades/scaffolds\n",
    "\n",
    "quast -r $REFERENCE -t $THREADS --output-dir $OUTPUT_DIR/qc/assembly/quast *.scaffolds*.fasta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Scaffold contamination scan (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Once we have the assembly checked, ready and potentially extended using the additional scaffolding we can run the contamination scan once more. This time not on the sequencing reads themselves but on the resulting scaffolds. We will be using Kraken2 but be aware that Kraken2 evaluates the percentages (the first column) based on the number of reads assigned to the individual phylogenetic groups. Since we now have scaffolds of different lengths the statistics might be misleading. It's better to keep only longer scaffolds to avoid counting unscaffolded short sequences which might be heavily over represented. Outputs are described [here](#Scaffold-contamination-scan-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) (treponema) java -ea -Xmx200m -cp /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/bbmap-38.22-1/current/ jgi.ReformatReads in=CW56_S1_.SS14_.scaffolds.fasta out=CW56_S1_.SS14_.scaffolds.fasta.tmp.fasta minlength=500\n",
      "Executing jgi.ReformatReads [in=CW56_S1_.SS14_.scaffolds.fasta, out=CW56_S1_.SS14_.scaffolds.fasta.tmp.fasta, minlength=500]\n",
      "\n",
      "Input is being processed as unpaired\n",
      "Input:                  \t66 reads          \t1153398 bases\n",
      "Short Read Discards:    \t33 reads (50.00%) \t10236 bases (0.89%)\n",
      "Output:                 \t33 reads (50.00%) \t1143162 bases (99.11%)\n",
      "\n",
      "Time:                         \t0.369 seconds.\n",
      "Reads Processed:          66 \t0.18k reads/sec\n",
      "Bases Processed:       1153k \t3.13m bases/sec\n",
      "Loading database information... done.\n",
      "33 sequences (1.14 Mbp) processed in 0.380s (5.2 Kseq/m, 180.48 Mbp/m).\n",
      "  33 sequences classified (100.00%)\n",
      "  0 sequences unclassified (0.00%)\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir -p $OUTPUT_DIR/qc/kraken2/scaffolds\n",
    "\n",
    "min_len=500 # Minimal length of scaffold for the analysis\n",
    "\n",
    "for sample in *.scaffolds.fasta\n",
    "do\n",
    "    reformat.sh in=$sample out=$sample.tmp.fasta minlength=$min_len # Get only scaffolds >=$min_len bp\n",
    "\n",
    "    kraken2 --threads $THREADS --use-names --classified-out $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.classified-out.fasta --unclassified-out $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.unclassified-out.fasta --output $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.kraken.txt --report $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.kraken.report.txt --db $KRAKEN2_DB $sample.tmp.fasta\n",
    "\n",
    "    # Extract only the annotated reads and their taxa\n",
    "    cat $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.kraken.txt | cut -f2,3 > $OUTPUT_DIR/qc/kraken2/scaffolds/${sample%.scaffolds.fasta}.kraken.taxid.txt\n",
    "    \n",
    "    rm $sample.tmp.fasta\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "The contamination scan can be used to remove scaffolds which most likely do not belong to our organism. Be careful before you remove a scaffold - recombinations happen all the time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Gene prediction (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "If we are sure out assembly is correct, clean and we want to proceed to further analysis we can try gene prediction on the scaffolded assembly. We use [PROKKA](https://github.com/tseemann/prokka) but there are other options as well such as [RAST](http://rast.nmpdr.org/), [Augustus](http://bioinf.uni-greifswald.de/webaugustus/index.gsp), [funannotate](https://github.com/nextgenusfs/funannotate). Outputs are described [here](#Gene-prediction-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) (treponema) [04:36:46] This is prokka 1.13.7\n",
      "[04:36:46] Written by Torsten Seemann <torsten.seemann@gmail.com>\n",
      "[04:36:46] Homepage is https://github.com/tseemann/prokka\n",
      "[04:36:46] Local time is Tue Jul  2 04:36:46 2019\n",
      "[04:36:46] You are 325073\n",
      "[04:36:46] Operating system is linux\n",
      "[04:36:46] You have BioPerl 1.007002\n",
      "[04:36:46] System has 70 cores.\n",
      "[04:36:46] Will use maximum of 12 cores.\n",
      "[04:36:46] Annotating as >>> Bacteria <<<\n",
      "[04:36:46] Generating locus_tag from 'CW56_S1_.SS14_.scaffolds.fasta' contents.\n",
      "[04:36:46] Setting --locustag COOMILJC from MD5 c886253cd55dea63be8090f981768321\n",
      "[04:36:46] Creating new output folder: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction\n",
      "[04:36:46] Running: mkdir -p \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\n",
      "[04:36:46] Using filename prefix: CW56_S1_.SS14_.scaffolds.XXX\n",
      "[04:36:46] Setting HMMER_NCPU=1\n",
      "[04:36:46] Writing log to: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/CW56_S1_.SS14_.scaffolds.log\n",
      "[04:36:46] Command: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/prokka --cpus 12 --kingdom Bacteria --outdir /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction --prefix CW56_S1_.SS14_.scaffolds CW56_S1_.SS14_.scaffolds.fasta\n",
      "[04:36:46] Appending to PATH: /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin\n",
      "[04:36:46] Looking for 'aragorn' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/aragorn\n",
      "[04:36:46] Determined aragorn version is 1.2\n",
      "[04:36:46] Looking for 'barrnap' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/barrnap\n",
      "[04:36:46] Determined barrnap version is 0.8\n",
      "[04:36:46] Looking for 'blastp' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/blastp\n",
      "[04:36:46] Determined blastp version is 2.9\n",
      "[04:36:46] Looking for 'cmpress' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/cmpress\n",
      "[04:36:46] Determined cmpress version is 1.1\n",
      "[04:36:46] Looking for 'cmscan' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/cmscan\n",
      "[04:36:46] Determined cmscan version is 1.1\n",
      "[04:36:46] Looking for 'egrep' - found /bin/egrep\n",
      "[04:36:46] Looking for 'find' - found /usr/bin/find\n",
      "[04:36:47] Looking for 'grep' - found /bin/grep\n",
      "[04:36:47] Looking for 'hmmpress' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/hmmpress\n",
      "[04:36:47] Determined hmmpress version is 3.2\n",
      "[04:36:47] Looking for 'hmmscan' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/hmmscan\n",
      "[04:36:47] Determined hmmscan version is 3.2\n",
      "[04:36:47] Looking for 'java' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/java\n",
      "[04:36:47] Looking for 'less' - found /usr/bin/less\n",
      "[04:36:47] Looking for 'makeblastdb' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/makeblastdb\n",
      "[04:36:47] Determined makeblastdb version is 2.9\n",
      "[04:36:47] Looking for 'minced' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced\n",
      "[04:36:47] Determined minced version is 3.3\n",
      "[04:36:47] Looking for 'parallel' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/parallel\n",
      "[04:36:47] Determined parallel version is 20190522\n",
      "[04:36:47] Looking for 'prodigal' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/prodigal\n",
      "[04:36:47] Determined prodigal version is 2.6\n",
      "[04:36:47] Looking for 'prokka-genbank_to_fasta_db' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/prokka-genbank_to_fasta_db\n",
      "[04:36:47] Looking for 'sed' - found /bin/sed\n",
      "[04:36:47] Looking for 'tbl2asn' - found /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/tbl2asn\n",
      "[tbl2asn] This copy of tbl2asn is more than a year old.  Please download the current version.\n",
      "[04:36:47] Determined tbl2asn version is 25.6\n",
      "[04:36:47] Using genetic code table 11.\n",
      "[04:36:47] Loading and checking input file: CW56_S1_.SS14_.scaffolds.fasta\n",
      "[04:36:47] Wrote 66 contigs totalling 1153398 bp.\n",
      "[04:36:47] Predicting tRNAs and tmRNAs\n",
      "[04:36:47] Running: aragorn -l -gc11  -w \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.fna\n",
      "[04:36:48] 1 tRNA-Arg c[22148,22222] 35 (tcg)\n",
      "[04:36:48] 2 tRNA-His c[22236,22308] 33 (gtg)\n",
      "[04:36:48] 3 tRNA-Phe [22445,22517] 34 (gaa)\n",
      "[04:36:48] 4 tRNA-Pro [54611,54683] 34 (tgg)\n",
      "[04:36:48] 5 tRNA-Arg [59000,59073] 35 (tct)\n",
      "[04:36:48] 6 tRNA-Val [67228,67299] 33 (cac)\n",
      "[04:36:48] 7 tRNA-Met c[84782,84855] 35 (cat)\n",
      "[04:36:48] 8 tRNA-Pro [92010,92083] 35 (ggg)\n",
      "[04:36:48] 9 tRNA-Lys c[166569,166643] 35 (ttt)\n",
      "[04:36:48] 10 tRNA-Leu [166786,166870] 35 (cag)\n",
      "[04:36:48] 11 tRNA-Ala [327621,327694] 34 (cgc)\n",
      "[04:36:48] 12 tRNA-Thr c[330231,330303] 34 (cgt)\n",
      "[04:36:48] 13 tRNA-Val [342320,342393] 35 (gac)\n",
      "[04:36:48] 1 tRNA-Cys [28306,28377] 33 (gca)\n",
      "[04:36:48] 2 tRNA-Asn c[45660,45734] 35 (gtt)\n",
      "[04:36:48] 3 tRNA-Gly c[86117,86188] 33 (gcc)\n",
      "[04:36:48] 4 tRNA-Leu c[86198,86280] 36 (tag)\n",
      "[04:36:48] 5 tRNA-Lys c[86339,86411] 34 (ctt)\n",
      "[04:36:48] 6 tRNA-Tyr c[103478,103560] 35 (gta)\n",
      "[04:36:48] 7 tRNA-Thr c[103570,103644] 36 (tgt)\n",
      "[04:36:48] 1 tRNA-Gln c[65653,65727] 34 (ttg)\n",
      "[04:36:48] 2 tRNA-Gly [107469,107542] 35 (tcc)\n",
      "[04:36:48] 3 tRNA-Leu [109494,109576] 35 (caa)\n",
      "[04:36:48] 1 tRNA-Ser c[38294,38381] 35 (gga)\n",
      "[04:36:48] 2 tRNA-Ser c[38438,38526] 35 (cga)\n",
      "[04:36:48] 3 tRNA-Arg c[38583,38658] 36 (gcg)\n",
      "[04:36:48] 4 tRNA-Ser c[38688,38776] 36 (gct)\n",
      "[04:36:48] 5 tRNA-Ser c[38815,38900] 35 (tga)\n",
      "[04:36:48] 1 tRNA-Val [3625,3697] 34 (tac)\n",
      "[04:36:48] 2 tRNA-Leu c[14871,14954] 36 (gag)\n",
      "[04:36:48] 3 tRNA-Met [36816,36888] 34 (cat)\n",
      "[04:36:48] 1 tRNA-Glu c[10822,10895] 35 (ctc)\n",
      "[04:36:48] 2 tRNA-Glu c[10930,11003] 35 (ttc)\n",
      "[04:36:48] 3 tRNA-Gln c[11012,11082] 33 (ctg)\n",
      "[04:36:48] 1 tRNA-Arg c[12491,12564] 35 (ccg)\n",
      "[04:36:48] 2 tRNA-Met [28279,28353] 35 (cat)\n",
      "[04:36:48] 3 tRNA-Asp [28421,28497] 36 (gtc)\n",
      "[04:36:48] 4 tRNA-Pro [28547,28621] 33 (cgg)\n",
      "[04:36:48] 5 tRNA-Arg c[28632,28704] 34 (cct)\n",
      "[04:36:48] 1 tRNA-Thr [6907,6979] 34 (ggt)\n",
      "[04:36:48] 2 tRNA-Trp [7232,7306] 35 (cca)\n",
      "[04:36:48] 3 tmRNA [41551,41902] 85,123 ANSDSFDYALAA*\n",
      "[04:36:48] 1 tRNA-Leu c[28222,28307] 35 (taa)\n",
      "[04:36:48] 2 tRNA-Ala c[32353,32429] 36 (ggc)\n",
      "[04:36:48] 1 tRNA-Leu [3,92] 35 (taa)\n",
      "[04:36:48] 2 tRNA-Met [120,196] 35 (cat)\n",
      "[04:36:48] 3 tRNA-Trp [206,281] 34 (cca)\n",
      "[04:36:48] 4 tRNA-Ile [298,374] 35 (gat)\n",
      "[04:36:48] 5 tRNA-Phe [380,455] 34 (gaa)\n",
      "[04:36:48] 6 tRNA-Met [463,537] 35 (cat)\n",
      "[04:36:48] 7 tRNA-Met [545,621] 35 (cat)\n",
      "[04:36:48] 1 tRNA-Gln c[-4,68] 33 (ctg)\n",
      "[04:36:48] 2 tRNA-Pyl c[76,152] 36 (cta)\n",
      "[04:36:48] 3 tRNA-Gly c[172,246] 33 (gcc)\n",
      "[04:36:48] 4 tRNA-Leu c[269,352] 35 (tag)\n",
      "[04:36:48] 5 tRNA-Tyr c[371,455] 35 (gta)\n",
      "[04:36:48] 6 tRNA-Thr c[457,531] 33 (tgt)\n",
      "[04:36:48] 1 tRNA-Cys c[103,177] 33 (gca)\n",
      "[04:36:48] 2 tRNA-Gly c[185,259] 33 (gcc)\n",
      "[04:36:48] 3 tRNA-Phe c[267,342] 34 (gaa)\n",
      "[04:36:48] 4 tRNA-Asp c[347,422] 34 (gtc)\n",
      "[04:36:48] 1 tRNA-Met c[79,154] 34 (cat)\n",
      "[04:36:48] 2 tRNA-Tyr c[164,248] 35 (gta)\n",
      "[04:36:48] 3 tRNA-Thr c[252,327] 34 (tgt)\n",
      "[04:36:48] 1 tRNA-Ala c[132,205] 35 (tgc)\n",
      "[04:36:48] 1 tRNA-Ile c[120,194] 35 (gat)\n",
      "[04:36:48] 1 tRNA-Glu c[-3,69] 34 (ttc)\n",
      "[04:36:48] 2 tRNA-Met c[93,168] 34 (cat)\n",
      "[04:36:48] Found 68 tRNAs\n",
      "[04:36:48] Predicting Ribosomal RNAs\n",
      "[04:36:48] Running Barrnap with 12 threads\n",
      "[04:36:50] 1 NODE_13_length_4751_cov_100.731834 209 16S ribosomal RNA (partial)\n",
      "[04:36:50] 2 NODE_13_length_4751_cov_100.731834 1587 23S ribosomal RNA\n",
      "[04:36:50] 3 NODE_13_length_4751_cov_100.731834 4590 5S ribosomal RNA\n",
      "[04:36:50] 4 NODE_15_length_1767_cov_52.664634 131 16S ribosomal RNA\n",
      "[04:36:50] 5 NODE_16_length_1757_cov_17.273006 117 16S ribosomal RNA\n",
      "[04:36:50] 6 NODE_17_length_1636_cov_9.491716 103 16S ribosomal RNA\n",
      "[04:36:50] Found 6 rRNAs\n",
      "[04:36:50] Skipping ncRNA search, enable with --rfam if desired.\n",
      "[04:36:50] Total of 73 tRNA + rRNA features\n",
      "[04:36:50] Searching for CRISPR repeats\n",
      "[04:36:50] Found 0 CRISPRs\n",
      "[04:36:50] Predicting coding sequences\n",
      "[04:36:50] Contigs total 1153398 bp, so using single mode\n",
      "[04:36:50] Running: prodigal -i \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.fna -c -m -g 11 -p single -f sco -q\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_1_length_375077_cov_67.153178:22088..22294 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_1_length_375077_cov_67.153178:59035..60222 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_1_length_375077_cov_67.153178:165361..166593 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_2_length_165590_cov_68.322991:86049..86486 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_3_length_126709_cov_62.629307:65023..65760 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_3_length_126709_cov_62.629307:107411..107536 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_3_length_126709_cov_62.629307:109558..111099 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_8_length_43722_cov_59.642987:28250..28405 on + strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_9_length_43473_cov_64.820237:6838..6996 on + strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_11_length_36439_cov_81.403129:30715..32391 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (rRNA) at NODE_16_length_1757_cov_17.273006:232..471 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (rRNA) at NODE_17_length_1636_cov_9.491716:137..457 on - strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_32_length_563_cov_6.174312:24..260 on + strand\n",
      "[04:36:52] Excluding CDS which overlaps existing RNA (tRNA) at NODE_37_length_435_cov_6.889610:69..248 on + strand\n",
      "[04:36:52] Found 978 CDS\n",
      "[04:36:52] Connecting features back to sequences\n",
      "[04:36:52] Not using genus-specific database. Try --usegenus to enable it.\n",
      "[04:36:52] Annotating CDS, please be patient.\n",
      "[04:36:52] Will use 12 CPUs for similarity searching.\n",
      "[04:36:53] There are still 978 unannotated CDS left (started with 978)\n",
      "[04:36:53] Will use blast to search against /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/IS with 12 CPUs\n",
      "[04:36:53] Running: cat \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/IS\\.faa | parallel --gnu --plain -j 12 --block 14643 --recstart '>' --pipe blastp -query - -db /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/IS -evalue 1e-30 -qcov_hsp_perc 90 -num_threads 1 -num_descriptions 1 -num_alignments 1 -seg no > \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/IS\\.blast 2> /dev/null\n",
      "[04:36:56] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/IS.faa\n",
      "[04:36:56] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/IS.blast\n",
      "[04:36:57] There are still 978 unannotated CDS left (started with 978)\n",
      "[04:36:57] Will use blast to search against /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/AMR with 12 CPUs\n",
      "[04:36:57] Running: cat \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/AMR\\.faa | parallel --gnu --plain -j 12 --block 14643 --recstart '>' --pipe blastp -query - -db /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/AMR -evalue 9.99999999999999e-301 -qcov_hsp_perc 90 -num_threads 1 -num_descriptions 1 -num_alignments 1 -seg no > \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/AMR\\.blast 2> /dev/null\n",
      "[04:36:59] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/AMR.faa\n",
      "[04:36:59] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/AMR.blast\n",
      "[04:36:59] There are still 978 unannotated CDS left (started with 978)\n",
      "[04:36:59] Will use blast to search against /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/sprot with 12 CPUs\n",
      "[04:36:59] Running: cat \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/sprot\\.faa | parallel --gnu --plain -j 12 --block 14643 --recstart '>' --pipe blastp -query - -db /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/kingdom/Bacteria/sprot -evalue 1e-09 -qcov_hsp_perc 80 -num_threads 1 -num_descriptions 1 -num_alignments 1 -seg no > \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/sprot\\.blast 2> /dev/null\n",
      "[04:37:08] Modify product: UPF0102 protein RPA0323 => hypothetical protein\n",
      "[04:37:08] Modify product: Probable NADH oxidase => putative NADH oxidase\n",
      "[04:37:08] Modify product: Probable peptidyl-prolyl cis-trans isomerase => putative peptidyl-prolyl cis-trans isomerase\n",
      "[04:37:08] Modify product: Uncharacterized protein TP_0956 => putative protein\n",
      "[04:37:08] Modify product: Bifunctional protein FolD => Bifunctional protein FolD protein\n",
      "[04:37:08] Modify product: Dihydroorotate dehydrogenase B (NAD(+)), electron transfer subunit homolog => Dihydroorotate dehydrogenase B (NAD(+)), electron transfer subunit \n",
      "[04:37:08] Modify product: Uncharacterized ABC transporter ATP-binding protein YknY => putative ABC transporter ATP-binding protein YknY\n",
      "[04:37:08] Modify product: Putative DD-carboxypeptidase TP_0574 => Putative DD-carboxypeptidase\n",
      "[04:37:08] Modify product: Probable tRNA sulfurtransferase => putative tRNA sulfurtransferase\n",
      "[04:37:08] Modify product: Uncharacterized protein TP_0751 => putative protein\n",
      "[04:37:08] Modify product: Probable transcriptional regulatory protein aq_1575 => putative transcriptional regulatory protein\n",
      "[04:37:08] Modify product: Outer membrane protein TP0453 => Outer membrane protein\n",
      "[04:37:08] Modify product: Uncharacterized ABC transporter ATP-binding protein YbiT => putative ABC transporter ATP-binding protein YbiT\n",
      "[04:37:08] Modify product: UPF0053 protein HI_0107 => hypothetical protein\n",
      "[04:37:08] Modify product: UPF0053 protein HI_0107 => hypothetical protein\n",
      "[04:37:08] Modify product: Probable ATP synthase YscN => putative ATP synthase YscN\n",
      "[04:37:08] Modify product: Probable peptidoglycan glycosyltransferase FtsW => putative peptidoglycan glycosyltransferase FtsW\n",
      "[04:37:08] Modify product: Cyclic di-GMP binding protein TDE_0214 => Cyclic di-GMP binding protein\n",
      "[04:37:08] Modify product: UPF0051 protein SA0778 => hypothetical protein\n",
      "[04:37:08] Modify product: Probable ABC transporter ATP-binding protein M6_Spy0273 => putative ABC transporter ATP-binding protein\n",
      "[04:37:09] Modify product: Putative zinc metalloprotease aq_1964 => Putative zinc metalloprotease\n",
      "[04:37:09] Modify product: 1,4-alpha-glucan branching enzyme TTHA1902 => 1,4-alpha-glucan branching enzyme\n",
      "[04:37:09] Modify product: Probable dual-specificity RNA methyltransferase RlmN => putative dual-specificity RNA methyltransferase RlmN\n",
      "[04:37:09] Modify product: Putative outer membrane protein assembly factor TP_0326 => Putative outer membrane protein assembly factor\n",
      "[04:37:09] Modify product: UPF0073 inner membrane protein YqfA => hypothetical protein\n",
      "[04:37:09] Modify product: Uncharacterized protein TP_0004 => putative protein\n",
      "[04:37:09] Modify product: Uncharacterized protein TP_0183 => putative protein\n",
      "[04:37:09] Modify product: Oxygen-independent coproporphyrinogen-III oxidase-like protein LL1139 => Oxygen-independent coproporphyrinogen-III oxidase-like protein\n",
      "[04:37:09] Modify product: GTP cyclohydrolase 1 type 2 homolog => GTP cyclohydrolase 1 type 2 \n",
      "[04:37:09] Modify product: Uncharacterized metal-dependent hydrolase YcfH => putative metal-dependent hydrolase YcfH\n",
      "[04:37:09] Modify product: Probable tRNA-dihydrouridine synthase => putative tRNA-dihydrouridine synthase\n",
      "[04:37:09] Modify product: UPF0056 inner membrane protein YhgN => hypothetical protein\n",
      "[04:37:09] Cleaned 32 /product names\n",
      "[04:37:09] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/sprot.faa\n",
      "[04:37:09] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/sprot.blast\n",
      "[04:37:09] There are still 553 unannotated CDS left (started with 978)\n",
      "[04:37:09] Will use hmmer3 to search against /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/hmm/HAMAP.hmm with 12 CPUs\n",
      "[04:37:09] Running: cat \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/HAMAP\\.hmm\\.faa | parallel --gnu --plain -j 12 --block 8078 --recstart '>' --pipe hmmscan --noali --notextw --acc -E 1e-09 --cpu 1 /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/db/hmm/HAMAP.hmm /dev/stdin > \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/HAMAP\\.hmm\\.hmmer3 2> /dev/null\n",
      "[04:37:16] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/HAMAP.hmm.faa\n",
      "[04:37:16] Deleting unwanted file: /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/HAMAP.hmm.hmmer3\n",
      "[04:37:16] Labelling remaining 449 proteins as 'hypothetical protein'\n",
      "[04:37:16] Possible /pseudo 'Flagellin FlaB1' at NODE_1_length_375077_cov_67.153178 position 272233\n",
      "[04:37:16] Possible /pseudo 'Protein YhgF' at NODE_1_length_375077_cov_67.153178 position 329422\n",
      "[04:37:16] Possible /pseudo 'Flagellar basal-body rod protein FlgG' at NODE_1_length_375077_cov_67.153178 position 367173\n",
      "[04:37:16] Possible /pseudo 'Macrolide export ATP-binding/permease protein MacB' at NODE_1_length_375077_cov_67.153178 position 369417\n",
      "[04:37:16] Possible /pseudo 'L-cystine-binding protein FliY' at NODE_7_length_44530_cov_66.817287 position 2420\n",
      "[04:37:16] Found 452 unique /gene codes.\n",
      "[04:37:16] Fixed 2 duplicate /gene - rbn_1 rbn_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - gph_1 gph_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ftsY_1 ftsY_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ntpB_1 ntpB_2\n",
      "[04:37:16] Fixed 3 duplicate /gene - dnaJ_1 dnaJ_2 dnaJ_3\n",
      "[04:37:16] Fixed 2 duplicate /gene - arnB_1 arnB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - flgG_1 flgG_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - fliY_1 fliY_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - lnt_1 lnt_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - fliG_1 fliG_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - clpP_1 clpP_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - deaD_1 deaD_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - pcrA_1 pcrA_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - macB_1 macB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ftsH_1 ftsH_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - flaB1_1 flaB1_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - btuD_1 btuD_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - yhgF_1 yhgF_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - rlmB_1 rlmB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - atpE_1 atpE_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - flhB_1 flhB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ktrA_1 ktrA_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - dbpA_1 dbpA_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ylxH_1 ylxH_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - atpD_1 atpD_2\n",
      "[04:37:16] Fixed 3 duplicate /gene - smc_1 smc_2 smc_3\n",
      "[04:37:16] Fixed 3 duplicate /gene - lapB_1 lapB_2 lapB_3\n",
      "[04:37:16] Fixed 2 duplicate /gene - pgk/tpi_1 pgk/tpi_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - potA_1 potA_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - alsT_1 alsT_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - nth_1 nth_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ruvB_1 ruvB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - ftsW_1 ftsW_2\n",
      "[04:37:16] Fixed 5 duplicate /gene - bepA_1 bepA_2 bepA_3 bepA_4 bepA_5\n",
      "[04:37:16] Fixed 2 duplicate /gene - fusA_1 fusA_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - mglB_1 mglB_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - cmk_1 cmk_2\n",
      "[04:37:16] Fixed 2 duplicate /gene - lepA_1 lepA_2\n",
      "[04:37:16] Fixed 38 colliding /gene names.\n",
      "[04:37:16] Adding /locus_tag identifiers\n",
      "[04:37:16] Assigned 1052 locus_tags to CDS and RNA features.\n",
      "[04:37:16] Writing outputs to /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/gene_prediction/\n",
      "[04:37:17] Generating annotation statistics file\n",
      "[04:37:17] Generating Genbank and Sequin files\n",
      "[04:37:17] Running: tbl2asn -V b -a r10k -l paired-ends -M n -N 1 -y 'Annotated using prokka 1.13.7 from https://github.com/tseemann/prokka' -Z \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.err -i \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.fsa 2> /dev/null\n",
      "[04:37:20] Could not run command: tbl2asn -V b -a r10k -l paired-ends -M n -N 1 -y 'Annotated using prokka 1.13.7 from https://github.com/tseemann/prokka' -Z \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.err -i \\/mnt\\/ssd\\/ssd_2\\/bioda_temp\\/honza\\/scratch_default\\/linda\\/results\\/SS14\\/assembly\\/gene_prediction\\/CW56_S1_\\.SS14_\\.scaffolds\\.fsa 2> /dev/null\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "# If we have a set of HIGHLY trusted proteins we can add the in genebank (gbk) format and assign GBK variable pointing to the file\n",
    "#GBK=/path/to/species_trusted_proteins.gbk\n",
    "\n",
    "for sample in *.scaffolds.fasta\n",
    "do\n",
    "    prokka --cpus $THREADS --kingdom Bacteria --outdir $OUTPUT_DIR/assembly/gene_prediction --prefix ${sample%.fa*} $sample # --proteins $GBK\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "At this point, there is a bug in PROKKA-minced conda installation. If you get an error with `Error: A JNI error has occurred, please check your installation and try again\n",
    "Exception in thread \"main\" java.lang.UnsupportedClassVersionError: minced has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0...` you have to downgrade minced to version 0.3.3 (or rebuild from source). Most likely right now it's at version 0.4.0 and Java at version 1.8.0. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) Error: A JNI error has occurred, please check your installation and try again\n",
      "Exception in thread \"main\" java.lang.UnsupportedClassVersionError: minced has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0\n",
      "\tat java.lang.ClassLoader.defineClass1(Native Method)\n",
      "\tat java.lang.ClassLoader.defineClass(ClassLoader.java:763)\n",
      "\tat java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)\n",
      "\tat java.net.URLClassLoader.defineClass(URLClassLoader.java:468)\n",
      "\tat java.net.URLClassLoader.access$100(URLClassLoader.java:74)\n",
      "\tat java.net.URLClassLoader$1.run(URLClassLoader.java:369)\n",
      "\tat java.net.URLClassLoader$1.run(URLClassLoader.java:363)\n",
      "\tat java.security.AccessController.doPrivileged(Native Method)\n",
      "\tat java.net.URLClassLoader.findClass(URLClassLoader.java:362)\n",
      "\tat java.lang.ClassLoader.loadClass(ClassLoader.java:424)\n",
      "\tat sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)\n",
      "\tat java.lang.ClassLoader.loadClass(ClassLoader.java:357)\n",
      "\tat sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:495)\n",
      "(treponema) (treponema) lrwxrwxrwx 1 323639 domain users 30 Jun 26 22:46 /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/bin/minced -> ../share/minced-0.4.0-0/minced\n",
      "(treponema) (treponema) openjdk version \"1.8.0_192\"\n",
      "OpenJDK Runtime Environment (Zulu 8.33.0.1-linux64) (build 1.8.0_192-b01)\n",
      "OpenJDK 64-Bit Server VM (Zulu 8.33.0.1-linux64) (build 25.192-b01, mixed mode)\n",
      "(treponema) (treponema) mkdir: cannot create directory ‘/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3’: File exists\n",
      "(treponema) --2019-07-02 04:34:40--  https://github.com/ctSkennerton/minced/releases/download/0.3.3/minced.jar\n",
      "Resolving github.com (github.com)... 140.82.118.4\n",
      "Connecting to github.com (github.com)|140.82.118.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/14484942/fd459180-3d00-11e9-82d8-f120121f7368?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T023440Z&X-Amz-Expires=300&X-Amz-Signature=baa642df61db536d95f49b3c2399968ecba688f6e0a730a8e1cbd7c619d29aed&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminced.jar&response-content-type=application%2Foctet-stream [following]\n",
      "--2019-07-02 04:34:40--  https://github-production-release-asset-2e65be.s3.amazonaws.com/14484942/fd459180-3d00-11e9-82d8-f120121f7368?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T023440Z&X-Amz-Expires=300&X-Amz-Signature=baa642df61db536d95f49b3c2399968ecba688f6e0a730a8e1cbd7c619d29aed&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminced.jar&response-content-type=application%2Foctet-stream\n",
      "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.1.140\n",
      "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.1.140|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 25046 (24K) [application/octet-stream]\n",
      "Saving to: ‘/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced.jar’\n",
      "\n",
      "/mnt/nfs/home/32363 100%[===================>]  24.46K  --.-KB/s    in 0.1s    \n",
      "\n",
      "2019-07-02 04:34:41 (241 KB/s) - ‘/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced.jar’ saved [25046/25046]\n",
      "\n",
      "(treponema) --2019-07-02 04:34:41--  https://github.com/ctSkennerton/minced/releases/download/0.3.3/minced\n",
      "Resolving github.com (github.com)... 140.82.118.4\n",
      "Connecting to github.com (github.com)|140.82.118.4|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/14484942/fd459180-3d00-11e9-92d3-9f9d29616d87?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T023441Z&X-Amz-Expires=300&X-Amz-Signature=5689057251aa6f733ae61eefbd689749913df3b58f88425f6d1a6de092829f2d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminced&response-content-type=application%2Foctet-stream [following]\n",
      "--2019-07-02 04:34:41--  https://github-production-release-asset-2e65be.s3.amazonaws.com/14484942/fd459180-3d00-11e9-92d3-9f9d29616d87?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20190702%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190702T023441Z&X-Amz-Expires=300&X-Amz-Signature=5689057251aa6f733ae61eefbd689749913df3b58f88425f6d1a6de092829f2d&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dminced&response-content-type=application%2Foctet-stream\n",
      "Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.1.140\n",
      "Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.1.140|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 791 [application/octet-stream]\n",
      "Saving to: ‘/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced’\n",
      "\n",
      "/mnt/nfs/home/32363 100%[===================>]     791  --.-KB/s    in 0s      \n",
      "\n",
      "2019-07-02 04:34:42 (42.1 MB/s) - ‘/mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced’ saved [791/791]\n",
      "\n",
      "(treponema) (treponema) (treponema) (treponema) /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/share/minced-0.3.3/minced\n",
      "(treponema) minced 0.3.3\n",
      "MinCED - Mining CRISPRs in Environmental Datasets (version 0.3.3)\n",
      "Copyright 2011      Florent Angly     <florent.angly@gmail.com>\n",
      "          2013-2018 Connor Skennerton <c.skennerton@gmail.com>\n",
      "\n",
      "Minced is a modified version of CRT (www.room220.com/crt)\n",
      "Charles Bland et al., CRISPR Recognition Tool (CRT): a tool for automatic\n",
      "detection of clustered regularly interspaced palindromic repeats\n",
      "BMC Bioinformatics 8, no. 1 (2007): 209.\n",
      "Distributed under the GNU General Public License version 3\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "# Results in error\n",
    "minced --version\n",
    "# Check minced version\n",
    "ls -l $CONDA_PREFIX/bin/minced\n",
    "# Check java version\n",
    "java -version\n",
    "\n",
    "# Download minced-0.3.0\n",
    "mkdir $CONDA_PREFIX/share/minced-0.3.3\n",
    "wget https://github.com/ctSkennerton/minced/releases/download/0.3.3/minced.jar -O $CONDA_PREFIX/share/minced-0.3.3/minced.jar\n",
    "wget https://github.com/ctSkennerton/minced/releases/download/0.3.3/minced -O $CONDA_PREFIX/share/minced-0.3.3/minced\n",
    "chmod u+x $CONDA_PREFIX/share/minced-0.3.3/minced\n",
    "export PATH=$CONDA_PREFIX/share/minced-0.3.3:$PATH\n",
    "\n",
    "# Should work fine\n",
    "which minced\n",
    "minced --version\n",
    "\n",
    "rm -r $OUTPUT_DIR/assembly/gene_prediction # PROKKA would complain about already existing output folder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "And now you can rerun the gene prediction (PROKKA)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Whole-genome alignment (optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Another thing we can do with the assembly is to look at the whole-genome similarities and try to fix the orientation of the resulting scaffolds and/or remove potentially wrong assemblies. This can be done using the whole-genome alignment of the scaffolds and the related reference. Outputs are described [here](#Whole-genome-alignment-output)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(treponema) (treponema) 1: PREPARING DATA\n",
      "2,3: RUNNING mummer AND CREATING CLUSTERS\n",
      "# reading input file \"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ntref\" of length 1139570\n",
      "# construct suffix tree for sequence of length 1139570\n",
      "# (maximum reference length is 536870908)\n",
      "# (maximum query length is 4294967295)\n",
      "# process 11395 characters per dot\n",
      "#....................................................................................................\n",
      "# CONSTRUCTIONTIME /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/mummer-3.23/mummer /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ntref 0.33\n",
      "# reading input file \"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/CW56_S1_.SS14_.scaffolds.fasta\" of length 1153463\n",
      "# matching query-file \"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/spades/scaffolds/CW56_S1_.SS14_.scaffolds.fasta\"\n",
      "# against subject-file \"/mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ntref\"\n",
      "# COMPLETETIME /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/mummer-3.23/mummer /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ntref 0.93\n",
      "# SPACE /mnt/nfs/home/323639/000000-My_Documents/VM-home/tools/miniconda2/envs/treponema/opt/mummer-3.23/mummer /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ntref 2.23\n",
      "4: FINISHING DATA\n",
      "Filtering alignments\n",
      "Extracting alignment coordinates\n",
      "Analyzing SNPs\n",
      "Extracting alignment breakpoints\n",
      "Generating report file\n",
      "gnuplot 5.2 patchlevel 7\n",
      "Writing filtered delta file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.filter\n",
      "Reading delta file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.filter\n",
      "Writing plot files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.fplot, /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.rplot, /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.hplot\n",
      "Determining SNPs from sequence and alignment data\n",
      "Writing gnuplot script /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.gp\n",
      "Rendering plot /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.ps\n",
      "gnuplot 5.2 patchlevel 7\n",
      "Writing filtered delta file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.filter\n",
      "Reading delta file /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.filter\n",
      "Writing plot files /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.fplot, /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.rplot, /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.hplot\n",
      "Determining SNPs from sequence and alignment data\n",
      "Writing gnuplot script /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.gp\n",
      "Rendering plot /mnt/ssd/ssd_2/bioda_temp/honza/scratch_default/linda/results/SS14/assembly/genome_alignment/CW56_S1_.SS14_.scaffolds/nucmer.png\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_1_length_375077_cov_67.153178\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_2_length_165590_cov_68.322991\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_3_length_126709_cov_62.629307\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_4_length_86646_cov_81.668073\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_5_length_76950_cov_69.042982\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_6_length_63307_cov_77.431466\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_7_length_44530_cov_66.817287\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_8_length_43722_cov_59.642987\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_9_length_43473_cov_64.820237\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_10_length_37477_cov_70.532262\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_11_length_36439_cov_81.403129\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_12_length_18555_cov_63.726340\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_13_length_4751_cov_100.731834\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_14_length_2653_cov_82.613618\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_19_length_1165_cov_58.982659\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_21_length_821_cov_49.275216\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_45_length_336_cov_51.770335\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_46_length_328_cov_48.885572\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_47_length_323_cov_94.698980\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_50_length_318_cov_52.675393\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_62_length_248_cov_34.669421\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_63_length_187_cov_25.966667\n",
      "CP004011_1_Treponema_pallidum_subsp__pallidum_SS14_complete_genome NODE_65_length_132_cov_100.400000\n",
      "(treponema) "
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "mkdir $OUTPUT_DIR/assembly/genome_alignment\n",
    "\n",
    "for sample in *.scaffolds.fasta\n",
    "do\n",
    "    sample_name=$(basename $sample .fasta)\n",
    "\n",
    "    mkdir $OUTPUT_DIR/assembly/genome_alignment/$sample_name\n",
    "\n",
    "    nucmer -p $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer $REFERENCE $sample\n",
    "\n",
    "    dnadiff -p $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer -d $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.delta\n",
    "\n",
    "    mummerplot -p $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer -s large --SNP --postscript --filter $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.delta\n",
    "    mummerplot -p $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer -s large --SNP --png --filter $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.delta\n",
    "\n",
    "    mkdir $OUTPUT_DIR/assembly/genome_alignment/$sample_name/individual_alignments\n",
    "\n",
    "    grep \">\" $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.delta | cut -d ' ' -f1,2 | sed 's/>//g' | while read list\n",
    "    do\n",
    "        echo $list\n",
    "        outname=`echo $list | sed 's/ /_/g'`\n",
    "        \n",
    "        show-aligns $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.delta $list > $OUTPUT_DIR/assembly/genome_alignment/$sample_name/individual_alignments/$outname.aln\n",
    "    done\n",
    "\n",
    "    # Extract unaligned scaffolds to the reference\n",
    "    cat $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.unqry | cut -f1 > $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.unqry.names\n",
    "    seqtk subseq $sample $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.unqry.names > $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.unqry.fasta\n",
    "    rm $OUTPUT_DIR/assembly/genome_alignment/$sample_name/nucmer.unqry.names\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Scaffolds with suspicious coverage might be either repetitive regions or contamination. Double check the `Kraken2` scan and check the assembly graphs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "**An voila, you have successfully completed the analysis!**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Outputs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "The structure of the results is quite straighforward and follows the analysis. All the directories are relative to the initial `$OUTPUT_DIR`. We will describe only those we think are the most relevant but feel free to explore the others as well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Raw reads quality check output\n",
    "* __*qc/fastqc/raw/multiqc_report.html*__ - summarized quality check of the input reads"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read preprocessing output\n",
    "* _**data/preprocessed/*.trimmed.fastq.gz**_ - adapter and quality trimmed reads\n",
    "* __*qc/fastqc/preprocessed/multiqc_report.html*__ - summarized quality check of the preprocessed reads\n",
    "* __*qc/cutadapt/multiqc_report.html*__ - summarized adapter and quality trimming"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Host-genome cleaning output\n",
    "* _**data/preprocessed/*.clean.fastq.gz**_ - adapter and quality trimmed reads  \n",
    "* *__qc/fastqc/clean/multiqc_report.html__* - summarized quality check of the preprocessed reads  \n",
    "* _**qc/bbmap/*.bbmap.out**_ - report of the host-genome removal mapping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Contamination scan output\n",
    "* _**qc/strainseeker/*.clean.seeker.txt**_ - StrainSeeker bacteria scan\n",
    "* *__qc/kraken2/reads/&ast;.classified-out_1.fastq.gz__* - reads classified to a phylogenetic taxa\n",
    "* __*qc/kraken2/reads/&ast;.unclassified-out_1.fastq.gz*__ - unclassified reads\n",
    "* _**qc/kraken2/reads/*.kraken.report.txt**_ - full `Kraken2` report\n",
    "* _**qc/kraken2/reads/*.S.kraken.report.txt**_ - subset of `Kraken2` report to include only **species**\n",
    "* _**qc/kraken2/reads/*.kraken.txt**_ - `Kraken2` results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this section, all results are specific to the used reference genome and will be placed under **`$OUTPUT_DIR/nameOfReferenceWithoutFa`** (for example **`/results/SS14`** if `SS14.fa` was used as the reference). We will use **SS14** as an example reference name. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "### Genome mapping output\n",
    "* **_SS14/alignment/filtered/filt/*.SS14.bam_** - raw mapping\n",
    "* **_SS14/alignment/filtered/filt/*.SS14.filt.indelRealigned.dedup.bam _** - mapping with basic filtering (both reads of a pair mapped and in a proper pair, only primary alignments), indel realigned and PCR duplicates cleaned\n",
    "* *__SS14/qc/alignment_stats/&ast;.SS14.flagstat__* - basic mapping statistics of the raw mapping\n",
    "* *__SS14/qc/alignment_stats/&ast;.SS14.filt.flagstat__* - basic mapping statistics of the basic filtered mapping\n",
    "* *__SS14/qc/alignment_stats/&ast;.SS14.filt.indelRealigned.dedup.flagstat__* - basic mapping statistics of the filtered, realigned and PCR duplicates cleaned mapping\n",
    "* *__SS14/qc/picard_dup/multiqc_report.html__* - summary of the PCR deduplication"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alignment filtering output\n",
    "Expcept the first bullet point, the mappings are sequenential - they go in the specified order and use the previous mapping as an input\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.mm1.bam__* - mappings remaining after filtering for number of mismatches\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.mm2.bam__* - mappings remaining after filtering for percentage of mismatches\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.mm.filtOut.bam__* - mappings filtered out due to too many mismatches (both number or percentage)\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.short.bam__* - mappings remaining after filtering for too short alignments\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.short.filtOut.bam__* - mappings filtered out due to too short alignment\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.scf.bam__* - mappings remaining after filtering for too extensive soft-clipping\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.scf.filtOut.bam__* - mappings filtered out due to too high soft-clipping\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.sup.bam__* - mappings remaining after filtering for suplementary/chimeric alignments\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.sup.filtOut.bam__*- mappings filtered out due supplementary/chimeric alignments\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.hcf.bam__* - mappings remaining after filtering for too extensive hard-clipping (here no hard-clipping allowed)\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.hcf.filtOut.bam__* - mappings filtered out due to too high hard-clipping\n",
    "* *__SS14/alignment/filtered/filt/&ast;.SS14.filt.indelRealigned.dedup.MAPQ10.bam__* - mappings remaining after filtering for low-quality mappings (here MAPQ10)\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.MAPQ10.filtOut.bam__* - mappings filtered out due low quality (here MAPQ 10)\n",
    "* *__SS14/alignment/filtered/&ast;.SS14.filt.indelRealigned.dedup.MAPQ40.bam__*  - mappings remaining after filtering for low-quality mappings (here MAPQ10); **these are the final mapping used for the subsequent analyses**\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.MAPQ40.filtOut.bam__* - mappings filtered out due lower quality (here MAPQ 40)\n",
    "* *__SS14/alignment/filtered/filtOut/&ast;.SS14.filt.indelRealigned.dedup.singletons.filtOut.bam__* - mappings filtered out due to unpaired mapping (singletons) after the filterings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alignment statistics and coverage output\n",
    "* *__SS14/qc/coverage/SS14.coverage.sample_summary__* - coverage summary for the selected coverages\n",
    "* *__SS14/qc/qualimap/multiqc_report.html__* - alignment statistics and coverage summary\n",
    "* *__SS14/qc/qualimap/&ast;.SS14.filt.indelRealigned.dedup.MAPQ40.qualimap.pdf__* - individual alignment statistics and coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alignment downsampling output\n",
    "* **_SS14/alignment/filtered/*.SS14.filt.indelRealigned.dedup.MAPQ40.downsampXXX.bam_** - downsampled BAM file capped at the *XXX* maximum coverage (here 250)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Alignment consensus output\n",
    "* **_SS14/consensus/*.SS14.filt.indelRealigned.dedup.MAPQ40.cns.final.fasta_** - the final *samtools*-version consensus sequence based on the filtered mappings; all low-coverage positions are converted to lower-case *n*s\n",
    "* **_SS14/consensus/*.SS14.filt.indelRealigned.dedup.MAPQ40.cns.def.fasta_** - the \"default\" *samtools*-version consensus sequence based on the filtered mappings; all low-coverage positions are in lower-case\n",
    "* **_SS14/consensus/*.SS14.filt.indelRealigned.dedup.MAPQ40.cns.gatk.fasta_** - the *gatk*-version consensus sequence based on the filtered mappings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variant call output\n",
    "* *__SS14/variants/freebayes/&ast;.SS14.filt.indelRealigned.dedup.MAPQ40.var.SS14_CP004011.1.annot.vcf__* - annotated high-quality variant call filtered variants\n",
    "* *__SS14/variants/freebayes/&ast;.SS14.filt.indelRealigned.dedup.MAPQ40.full.SS14_CP004011.1.annot.vcf__* - annotated full variant call without any filtering\n",
    "* **_SS14/variants/freebayes/*.SS14.filt.indelRealigned.dedup.MAPQ40.var.vcf_** - high-quality variant call filtered variants\n",
    "* **_SS14/variants/freebayes/*.SS14.filt.indelRealigned.dedup.MAPQ40.full.vcf_** - full variant call without any filtering\n",
    "* **_SS14/variants/freebayes/*.SS14.filt.indelRealigned.dedup.MAPQ40.var.tsv_** - high-quality variant call filtered variants in TSV format\n",
    "* **_SS14/qc/variants/*.SS14.filt.indelRealigned.dedup.MAPQ40.var.stats.txt_** - statistics of full variant call without any filtering\n",
    "* **_SS14/qc/variants/*.SS14.filt.indelRealigned.dedup.MAPQ40.full.stats.txt_** - statistics of high-quality variant call filtered variants\n",
    "* **_SS14/qc/variants/*.SS14.filt.indelRealigned.dedup.MAPQ40.full.ratio.stats.txt_** - ts/tv ratio stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variant annotation output\n",
    "* Please see the Variant call [section](#Variant-call-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### *De novo* assembly output\n",
    "* *__SS14/alignment/fastq/&ast;.SS14_RX.mapped.fastq.gz__* - extracted mapped reads (*X* represents R1 or R2 reads)\n",
    "* **_S14/assembly/spades/scaffolds/*.SS14_.scaffolds.fasta_** - all assembled scaffolds\n",
    "* **_S14/assembly/spades/scaffolds/*.SS14_.scaffolds.topXX.fasta_** - top *XX* (here 10) longest scaffolds\n",
    "* **_S14/assembly/spades/scaffolds/*.SS14_.scaffolds_besst.fasta_** - scaffolds resulting from the additional BESST scaffolding\n",
    "* **_S14/assembly/spades/graphs/*.SS14_.assembly_graph_with_scaffolds.gfa_** - assembly graphs in GFA format\n",
    "* **_S14/assembly/spades/graphs/*.SS14_.assembly_graph.fastg_** - assembly graphs in FASTG format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assembly quality check output\n",
    "* *__SS14/qc/assembly/quast/report.pdf__* - `QUAST` assembly report in PDF format\n",
    "* *__SS14/qc/assembly/quast/report.html__* - `QUAST` assembly report in html format\n",
    "* *__SS14/qc/assembly/BUSCO/run_busco_&ast;.SS14_.scaffolds/multiqc_report.html__*  - `BUSCO` assembly check summary\n",
    "* *__SS14/qc/assembly/BUSCO/run_busco_&ast;.SS14_.scaffolds/short_summary\\_busco_&ast;.SS14_.scaffolds.txt__*  - `BUSCO` assembly check short report"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional scaffolding output\n",
    "* Please see the *De novo* assembly [section](#De-novo-assembly-output)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaffold contamination scan output\n",
    "* *__SS14/qc/kraken2/scaffolds/&ast;.SS14_.kraken.taxid.txt__* - assignment of a taxonomic taxa to the scaffolds\n",
    "* *__SS14/qc/kraken2/scaffolds/&ast;.SS14_.kraken.report.txt__* - `Kraken2` summary report\n",
    "* *__SS14/qc/kraken2/scaffolds/&ast;.SS14_.classified-out.fasta__* - classified scaffolds\n",
    "* *__SS14/qc/kraken2/scaffolds/&ast;.SS14_.unclassified-out.fasta__* - unclassified scaffolds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gene prediction output\n",
    "For the full list of the output files please see the PROKKA [manual](https://github.com/tseemann/prokka/#Output-Files).\n",
    "* *__SS14/assembly/gene_prediction/&ast;.SS14_.scaffolds.gff__* - This is the master annotation in GFF3 format, containing both sequences and annotations. It can be viewed directly in Artemis or IGV.\n",
    "* *__SS14/assembly/gene_prediction/&ast;.SS14_.scaffolds.gbf__* - This is a standard Genbank file derived from the master .gff. If the input to prokka was a multi-FASTA, then this will be a multi-Genbank, with one record for each sequence.\n",
    "* *__SS14/assembly/gene_prediction/&ast;.SS14_.scaffolds.ffn__* - Nucleotide FASTA file of all the prediction transcripts (CDS, rRNA, tRNA, tmRNA, misc_RNA)\n",
    "* *__SS14/assembly/gene_prediction/&ast;.SS14_.scaffolds.faa__* - Protein FASTA file of the translated CDS sequences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Whole-genome alignment output\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/nucmer.png__* - whole-genome alignment synteny plot\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/individual_alignments__* - directory with individual alignments of all scaffolds with the reference sequence\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/nucmer.snps__* - whole-genome-based SNPs\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/nucmer.report__* - final report of the alignment\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/nucmer.unqry__* - unaligned scaffold names\n",
    "* *__SS14/assembly/genome_alignment/&ast;.SS14_.scaffolds/nucmer.unqry.fasta__* - unaligned scaffolds in FASTA format"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}