wilhelm-lab · JSchlensok · Sep 17, 2024 · Jun 20, 2024 · Jul 8, 2024 · Jul 8, 2024
diff --git a/.flake8 b/.flake8
@@ -8,5 +8,6 @@ per-file-ignores =
 	tests/*:S101
 	**/__init__.py:F401,F403
     	docs/conf.py:S404,S607,S603
-	oktoberfest/runner.py:S301,S403
+	oktoberfest/runner.py:C901,S301,S403
+    oktoberfest/predict/dlomix.py:E402
 docstring_style = sphinx
diff --git a/.gitignore b/.gitignore
@@ -151,5 +151,8 @@ tutorials/
 # example data
 data/
 
+# Machine learning artifacts
+wandb/
+
 # doctest IO files
 tests/doctests/
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -40,7 +40,7 @@ repos:
             entry: pyupgrade
             language: system
             types: [python]
-            args: [--py38-plus]
+            args: [--py39-plus, --keep-runtime-typing]
           - id: trailing-whitespace
             name: Trim Trailing Whitespace
             entry: trailing-whitespace-fixer

diff --git a/docs/API.rst b/docs/API.rst
@@ -20,6 +20,7 @@ Preprocessing: :code:`pp`
 .. currentmodule:: oktoberfest
 
 Generating libraries
+~~~~~~~~~~~~~~~~~~~~
 
 .. autosummary::
    :toctree: api/pp
@@ -31,6 +32,7 @@ Generating libraries
    pp.annotate_spectral_library
 
 Spectra preprocessing
+~~~~~~~~~~~~~~~~~~~~~
 
 .. autosummary::
    :toctree: api/pp
@@ -42,6 +44,7 @@ Spectra preprocessing
 
 
 Peptide preprocessing
+~~~~~~~~~~~~~~~~~~~~~
 
 .. autosummary::
    :toctree: api/pp
@@ -57,33 +60,43 @@ Peptide preprocessing
 
 Predicting: :code:`pr`
 ----------------------
+.. TODO
+    add full class documentation through autosummary
 
 .. module:: oktoberfest.pr
 
 .. currentmodule:: oktoberfest
 
-Access to functions that communicate with a Koina server to retrieve predictions from various prediction models.
+Access to functions that interface either a Koina server to retrieve predictions from various prediction models, or DLomix to serve & refinement-learn pre-trained models locally.
 
-High level features
-~~~~~~~~~~~~~~~~~~~
+High-level prediction runner
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. autosummary::
-   :toctree: api/pr
+    :recursive:
+    :toctree: api/pr
 
-   pr.predict_intensities
-   pr.predict_rt
-   pr.ce_calibration
+    pr.Predictor
 
 Koina interface
 ~~~~~~~~~~~~~~~
 
 .. autosummary::
-   :toctree: api/pr
+    :recursive:
+    :toctree: api/pr
 
-   pr.predict
-   pr.predict_at_once
-   pr.predict_in_chunks
+    pr.Koina
 
+DLomix interface
+~~~~~~~~~~~~~~~~
+
+.. autosummary::
+    :recursive:
+    :toctree: api/pr
+
+    pr.DLomix
+    pr.create_dlomix_dataset
+    pr.refine_intensity_predictor
 
 Rescoring: :code:`re`
 ---------------------

diff --git a/docs/_static/custom_cookietemple.css b/docs/_static/custom_cookietemple.css
@@ -75,6 +75,25 @@ table.align-default {
     padding-left: 50px;
 }
 
+.lib-refinement-learning-config-table
+    tbody
+    tr:nth-child(n + 2):nth-child(-n + 5)
+    td:nth-child(1),
+.lib-refinement-learning-config-table tbody tr:nth-child(8) td:nth-child(1) {
+    padding-left: 50px;
+}
+
+.lib-refinement-learning-config-table
+    tbody
+    tr:nth-child(n + 6):nth-child(-n + 7)
+    td:nth-child(1),
+.lib-refinement-learning-config-table
+    tbody
+    tr:nth-child(n + 9):nth-child(-n + 10)
+    td:nth-child(1) {
+    padding-left: 100px;
+}
+
 .rescore-config-table tbody tr:last-child td:first-child {
     padding-left: 50px;
 }

diff --git a/docs/conf.py b/docs/conf.py
@@ -33,6 +33,7 @@
     "sphinx_autodoc_typehints",
     "sphinx.ext.intersphinx",
     "sphinx_click",
+    "sphinx.ext.autosectionlabel",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -250,3 +251,6 @@ def modurl(qualname):
 # and there’s no way to insert filters into those templates
 # so we have to modify the default filters
 DEFAULT_FILTERS["modurl"] = modurl
+
+# -- Options for autosectionlabel mappings -----------------------------
+autosectionlabel_prefix_document = True
diff --git a/docs/config.rst b/docs/config.rst
@@ -1,7 +1,7 @@
 Configuration
 =============
 
-The following provides an overview of all available flags in the configuration file to use the high level API and run jobs. Parameters may be applicable to more than one job type and are collected within indivdual tables.
+The following provides an overview of all available flags in the configuration file to use the high-level API and run jobs. Parameters may be applicable to more than one job type and are collected within indivdual tables.
 
 Always applicable
 -----------------
@@ -18,7 +18,7 @@ Always applicable
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | models                     | Contains information about the used models for peptide property prediction (see following 2 nested parameters)                                                                                                                                                                             |
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
-   |     intensity              | Name of the model used for fragment intensity prediction                                                                                                                                                                                                                                   |
+   |     intensity              | Name or path of the model used for fragment intensity prediction                                                                                                                                                                                                                           |
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    |     irt                    | Name of the model used for indexed retention time prediction                                                                                                                                                                                                                               |
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
@@ -145,4 +145,46 @@ Applicable to in-silico digestion
    |     specialAas             | Special amino acids for decoy generation; default = "KR"                                                                                                           |
    +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    |     db                     | Defines whether the digestion should contain only targets, only decoys or both (concatenated); can be "target", "decoy" or "concat"; default = "concat"            |
-   +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   +----------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+
+Applicable to local intensity prediction
+----------------------------------------
+
+.. table::
+   :class: fixed-table
+
+    +--------------------------+---------------------------------------------------+
+    | Parameter                | Description                                       |
+    +==========================+===================================================+
+    | dlomixInferenceBatchSize | Batch size to use for local inference with DLomix |
+    +--------------------------+---------------------------------------------------+
+
+Applicable to transfer/refinement learning
+------------------------------------------
+
+.. table::
+   :class: fixed-table lib-refinement-learning-config-table
+
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   | Parameter                          |                             Description                                                                                                                            |
+   +====================================+====================================================================================================================================================================+
+   | refinementLearningOptions          | Contains specific settings for local refinement learning of intensity predictor on provided spectra. If not present, no refinement learning will be performed.     |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |     batchSize                      | Defines batch size to use for training; default = 1024                                                                                                             |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |     includeOriginalSequences       | Defines whether unmodified peptide sequences should be kept in processed DLomix dataset for downstream analysis; default = False                                   |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |     improveFurther                 | Defines whether to perform an additional third training phase during refinement learning to further improve the predictor; default = False.                        |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |     wandbOptions                   | Contains specific settings for using WandB when doing refinement learning. If not present, WandB will not be used.                                                 |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |         project                    | Project to save WandB run to; default = "DLomix_auto_RL_TL"                                                                                                        |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |         targets                    | Tags to use for WandB run; default = None                                                                                                                          |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |     datasetFilteringOptions        | Contains specific settings for filtering the refinement/transfer learning dataset. If not provided, will only remove decoys.                                       |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |         searchEngineScoreThreshold | Threshold for included peptides, everything below will be discarded.                                                                                               |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
+   |         numDuplicates              | Number of (peptide, charge, collision energy) duplicates to include.                                                                                               |
+   +------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------+
diff --git a/docs/installation.rst b/docs/installation.rst
@@ -17,7 +17,7 @@ The installer script automatically installs dependencies and creates a new conda
    wget https://raw.githubusercontent.com/wilhelm-lab/oktoberfest/main/installer.sh -O install_oktoberfest.sh
    bash install_oktoberfest.sh
 
-The installer searches for existing anaconda / miniconda installation. If none was found, it will download and install miniconda.
+The installer searches for an existing anaconda / miniconda installation. If none is found, it will download and install miniconda.
 
 Docker Image
 ------------
@@ -44,7 +44,7 @@ This is a step-by-step guide for the manual installation of all mandatory and op
 Install Python
 ~~~~~~~~~~~~~~
 
-Oktoberfest requires python >=3.8 and <=3.11. Best practise is to use a clean conda environment (`Miniconda <https://docs.conda.io/en/latest/miniconda.html>`_).
+Oktoberfest requires python >=3.9 and <=3.11. Best practice is to use a clean conda environment (`Miniconda <https://docs.conda.io/en/latest/miniconda.html>`_).
 Follow the installation guide for your operating system, then create a new environment using
 
 .. code-block:: bash
@@ -55,25 +55,47 @@ Follow the installation guide for your operating system, then create a new envir
 Optional dependencies
 ~~~~~~~~~~~~~~~~~~~~~
 
-There are three dependencies that are required for specific tasks:
+There are multiple optional dependencies depending on job types. Detailed notes and installation instructions can be found below.
 
+.. table::
 
+    +------------------------------------------------------------------+---------------------+---------------------------------------------------------+
+    | Job type                                                         | Dependency          | Notes                                                   |
+    +==================================================================+=====================+=========================================================+
+    | Pre-processing                                                   | ThermoRawFileParser | not required if only mzML files are provided            |
+    +                                                                  +---------------------+---------------------------------------------------------+
+    |                                                                  | `mono`              | required for ThermoRawFileParser to work on Linux/macOS |
+    +------------------------------------------------------------------+---------------------+---------------------------------------------------------+
+    | :ref:`Rescoring <jobs:a) without refinement>`                    | Percolator          |                                                         |
+    +------------------------------------------------------------------+---------------------+---------------------------------------------------------+
+    | :ref:`Rescoring + refinement learning <jobs:b) with refinement>` | DLomix              |                                                         |
+    +------------------------------------------------------------------+---------------------+---------------------------------------------------------+
+
+**ThermoRawFileParser**
 `ThermoRawFileParser v1.4.3 <https://github.com/compomics/ThermoRawFileParser/releases/tag/v1.4.3>`_:
-For conversion of RAW to mzML format. Download and unpack the zip or tar.gz file. The default locations Oktoberfest expects the executable to be at "/opt/compomics/" (Linux/MacOS) or the folder from which you want to execute Oktoberfest (Windows).
-You do not need this package if you only ever provide mzML file. However, it is recommended let Oktoberfest convert RAW files for you, to ensure the mzML files are formatted in the way Oktoberfest expects it.
-For ThermoRawFileParser to work on Linux, you also need to ensure mono is installed using
+For conversion of RAW to mzML format. Download and unpack the zip or tar.gz file. The default locations Oktoberfest expects the executable to be at `/opt/compomics/` (Linux/MacOS) or the folder from which you want to execute Oktoberfest (Windows).
+You do not need this package if you only ever provide mzML files. However, it is recommended to let Oktoberfest convert RAW files for you, to ensure the mzML files are formatted in the way Oktoberfest expects it.
+
+**`mono`**
+For ThermoRawFileParser to work on Linux, you also need to ensure `mono` is installed using
 
 .. code-block:: bash
 
    sudo apt -y update && sudo apt -y install mono-devel  # Debian / Ubuntu
 
 For MacOS, follow the instructions provided by `Mono <https://www.mono-project.com/docs/getting-started/install/mac/>`_.
 
-
+**Percolator**
 `Percolator v3.06.1 <https://github.com/percolator/percolator/releases/tag/rel-3-06-01>`_:
 This is the tool Mokapot is based on. As it has more options and is generally more stable wrt. to FDR cutoffs and deduplication, it is recommended to use this tool instead of Mokapot.
 Installable packages are provided for Linux/MacOS/Windows.
 
+**DLomix**
+`DLomix <https://github.com/wilhelm-lab/dlomix>`_ is a Python framework for deep learning in proteomics. Oktoberfest uses DLomix to refinement-learn intensity predictors on input spectra. It is listed as an optional dependency and can be installed using
+
+.. code-block:: bash
+
+    poetry install -E dlomix
 
 Installing Oktoberfest
 ~~~~~~~~~~~~~~~~~~~~~~
@@ -85,5 +107,9 @@ Oktoberfest is listed on the Python Package Index (PyPI) and can be installed wi
    conda activate oktoberfest
    pip install oktoberfest jupyterlab
 
+For local prediction & refinement learning, you have to install Oktoberfest with the `dlomix` extra:
 
+.. code-block:: bash
 
+   conda activate oktoberfest
+   pip install oktoberfest[dlomix]