Documentation (#149)

* Do not exclude DQM Bokeh app anymore, it is now mock-imported, and we may want to document it (e.g. by converting its README into a proper section in the doc). * Remove commented suppress_warnings, as we now use mock imports instead * Do not explicitlt write `nectarchain` in mock import settings * Changed README as reStructuredText. * Work on documentation * Lint * Fix references to API * Fix target ref, work on doc * Re-introduce default role for Sphinx, to fix reference to API * Some tips about DIRAC * Some doc formatting --------- Co-authored-by: Jean-Philippe Lenain <[email protected]>
cta-observatory · Aug 23, 2024 · 47c56c5 · 47c56c5
1 parent bf6b1f8
commit 47c56c5
Show file tree

Hide file tree

Showing 11 changed files with 180 additions and 91 deletions.
diff --git a/README.md b/README.md
diff --git a/README.rst b/README.rst
@@ -0,0 +1,68 @@
+================================================
+nectarchain |pypi| |conda| |ci| |doc| |codecov|
+================================================
+
+.. |ci| image:: https://github.com/cta-observatory/nectarchain/actions/workflows/ci.yml/badge.svg?branch=main
+    :target: https://github.com/cta-observatory/nectarchain/actions/workflows/ci.yml?query=workflow%3ACI+branch%3Amain
+    :alt: Test Status
+.. |pypi| image:: https://badge.fury.io/py/nectarchain.svg
+    :target: https://pypi.org/project/nectarchain
+.. |conda| image:: https://anaconda.org/conda-forge/nectarchain/badges/version.svg
+    :target: https://anaconda.org/conda-forge/nectarchain
+.. |codecov| image:: https://codecov.io/github/cta-observatory/nectarchain/graph/badge.svg?token=TDhZlJtbMv
+    :target: https://codecov.io/github/cta-observatory/nectarchain
+.. |doc| image:: https://readthedocs.org/projects/nectarchain/badge/?version=latest
+    :target: https://nectarchain.readthedocs.io/en/latest/?badge=latest
+    :alt: Documentation Status
+
+Repository for the high level analysis of the NectarCAM data, a camera to equip the
+medium-sized telescopes of `CTAO <https://www.ctao.org/>`_ in its Northern site.
+The analysis is heavily based on `ctapipe <https://github.com/cta-observatory/ctapipe>`_,
+adding custom code for NectarCAM calibration.
+
+``nectarchain`` is currently pinned to ``ctapipe`` version 0.19.
+
+This code is under rapid development. It is not recommended for production use unless you are an expert or developer!
+
+* Code: https://github.com/cta-observatory/nectarchain
+* Docs: https://nectarchain.readthedocs.io/
+
+Installation for Users
+======================
+
+``nectarchain`` and its dependencies may be installed using the *Anaconda* or
+*Miniconda* package system. We recommend creating a conda virtual environment
+first, to isolate the installed version and dependencies from your main
+environment (this is optional).
+
+The latest version of ``nectarchain`` can be installed via::
+
+  mamba install -c conda-forge nectarchain
+
+or via::
+
+  pip install nectarchain
+
+**Note**: to install a specific version of ``nectarchain``, take look at the documentation `here <https://nectarchain.readthedocs.io/en/latest/user-guide/index.html>`_.
+
+**Note**: ``mamba`` is a C++ reimplementation of conda and can be found `here <https://github.com/mamba-org/mamba>`_.
+
+Note this is *pre-alpha* software and is not yet stable enough for end-users (expect large API changes until the first stable 1.0 release).
+
+Developers should follow the development install instructions found in the
+`documentation <https://nectarchain.readthedocs.io/en/latest/developer-guide/index.html>`_.
+
+
+Contributing
+============
+
+All contribution are welcome.
+
+Guidelines are the same as `ctapipe's ones <https://ctapipe.readthedocs.io/en/latest/developer-guide/getting-started.html>`_.
+See `here <https://ctapipe.readthedocs.io/en/latest/developer-guide/pullrequests.html#pullrequests>`_ how to make a pull request to contribute.
+
+
+Report issue / Ask a question
+=============================
+
+Please use `GitHub Issues <https://github.com/cta-observatory/nectarchain/issues>`_ to report issues or `GitHub Discussions <https://github.com/cta-observatory/nectarchain/discussions>`_ for questions and discussions.
diff --git a/docs/conf.py b/docs/conf.py
@@ -75,19 +75,17 @@
 # source_suffix = ['.rst', '.md']
 source_suffix = ".rst"
 
+# have all links automatically associated with the right domain.
+default_role = "py:obj"
+
 templates_path = ["_templates"]
 
-exclude_patterns = [
-    "_build",
-    "Thumbs.db",
-    ".DS_Store",
-    "**/*.dqm.bokeh_app**",
-]
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 
 autodoc_mock_imports = [
-    "nectarchain.makers.extractor.charge_extractor",
-    "nectarchain.makers.calibration.core",
-    "nectarchain.dqm.bokeh_app",
+    f"{project}.makers.calibration.core",
+    f"{project}.makers.extractor.charge_extractor",
+    f"{project}.dqm.bokeh_app",
 ]
 
 # intersphinx allows referencing other packages sphinx docs
@@ -101,11 +99,6 @@
     "ctapipe": ("https://ctapipe.readthedocs.io/en/v0.19.3/", None),
 }
 
-# suppress_warnings = [
-#     "autosummary",
-#     "autosummary.import_cycle",
-# ]
-
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = True
 

diff --git a/docs/developer-guide/getting-started.rst b/docs/developer-guide/getting-started.rst
@@ -39,7 +39,7 @@ Please follow the `same conventions as ctapipe <https://ctapipe.readthedocs.io/e
 Optional DIRAC support
 ----------------------
 
-*Note*: this is **not** needed if you are using ``nectarchain`` as a container :ref:`as-a-container`, as DIRAC is already fully installed and configured within.
+*Note*: this is **not** needed if you are using ``nectarchain`` as a container (see :ref:`as-a-container`), as DIRAC is already fully installed and configured within.
 
 To enable support for DIRAC within the same environment, do the following after the installation of ``nectarchain`` described above:
 
@@ -64,13 +64,13 @@ Building the documentation
 
 To locally build the documentation, optional dependencies should be installed with:
 
-.. code-block::
+.. code-block:: console
 
     $ pip install -e ".[docs]"
 
 The documentation can then be compiled with:
 
-.. code-block::
+.. code-block:: console
 
     $ make -C docs html
 

diff --git a/docs/user-guide/dqm.rst b/docs/user-guide/dqm.rst
@@ -0,0 +1,53 @@
+.. _dqm:
+
+Quick recipe for the Data Quality Monitoring script
+===================================================
+
+Run locally
+-----------
+
+To launch the Data Quality Monitoring (DQM), first activate the ``nectarchain`` ``conda`` environment:
+
+.. code-block:: console
+
+    source activate nectarchain
+
+Usage:
+
+.. code-block:: console
+
+    $ python start_dqm.py -h
+
+To automatically find and retrieve run files from DIRAC, use the ``-r`` option:
+
+.. code-block:: console
+
+    $ python start_dqm.py -r 2720 $NECTARCAMDATA $NECTARCAMDATA
+
+See :ref:`env-vars` for the usage of the ``$NECTARCAMDATA`` environment variable.
+
+To manually use local run files, use the ``-i`` option **after** indicating the positional arguments for input and output directories:
+
+.. code-block:: console
+
+    $ python start_dqm.py $NECTARCAMDATA $NECTARCAMDATA -i NectarCAM.Run2720.0000.fits.fz NectarCAM.Run2720.0001.fits.fz
+
+As a DIRAC job
+--------------
+
+The user script `nectarchain/user_scripts/jlenain/dqm_job_submitter/submit_dqm_processor.py` can be used to run the DQM as a DIRAC job:
+
+.. code-block:: console
+
+    $ python submit_dqm_processor.py -h
+
+Under the hood, it calls the ``dqm_processor.sh`` wrapper script, which itself launches an Apptainer instance of the ``nectarchain`` container on the DIRAC worker. This Apptainer image is automatically built and published in CI on releases.
+
+The DQM runs one job per NectarCAM run. It is possible, for instance, to bulk-submit DIRAC jobs for all runs acquired during a given period, e.g.:
+
+.. code-block:: console
+
+    $ d=2023-01-01
+    $ while [ "$d" != 2023-03-01 ]; do python submit_dqm_processor.py -d $d; d=$(date -I -d "$d + 1 day"); done
+
+The script will first assess whether DQM jobs have already been run for a given NectarCAM run, based on the output directory on DIRAC where this script stores its output. Look for the ``$DIRAC_OUTDIR`` environment variable in ``dqm_processor.sh``.
diff --git a/docs/user-guide/env-vars.rst b/docs/user-guide/env-vars.rst
@@ -0,0 +1,18 @@
+.. _env-vars:
+
+Environment variables
+=====================
+
+We strive for relying as little as possible on environment variables across ``nectarchain``.
+However, some environment variables are needed for ``nectarchain`` to work properly, especially to automatically fetch run files from the grid via DIRAC, or to store output results and plots.
+
+Mandatory
+---------
+
+:``NECTARCAMDATA``: path to local NectarCAM data. It can contain ``fits.fz`` run files, `~nectarchain.data.container.waveformsContainer.WaveformsContainer` or `~nectarchain.data.container.chargesContainer.ChargesContainer` HDF5 files. This is also where the `~nectarchain.data.management.DataManagement.findrun` method will automatically store NectarCAM run files when fetched from DIRAC.
+
+Optional
+--------
+
+:``NECTARCHAIN_LOG``: path for output logs for ``nectarchain``, generally defaulting to ``/tmp``.
+:``NECTARCHAIN_FIGURES``: path to store output figures for ``nectarchain``, generally defaulting to ``/tmp``, or configurable (see e.g. `nectarchain.makers.component.spe.spe_algorithm.SPEalgorithm`).
diff --git a/docs/user-guide/howto-dirac.rst b/docs/user-guide/howto-dirac.rst
@@ -75,13 +75,39 @@ NectarCAM data can then be explored using ``dls``:
    NectarCAM.Run5568.0003.fits.fz
    NectarCAM.Run5568.0004.fits.fz
 
-The `~nectarchain.data.management.DataManagement.findrun` method will
+Sometimes, it is useful to quickly find the DIRAC location of a given NectarCAM run. The command ``dfind`` comes in handy. Suppose you want to know where run 5326 is, or how much slice files it contains:
+
+.. code-block:: console
+
+    $ dfind /vo.cta.in2p3.fr/nectarcam | grep 5326
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0000.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0001.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0002.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0003.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0004.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0005.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0006.fits.fz
+    /vo.cta.in2p3.fr/nectarcam/2024/20240520/NectarCAM.Run5326.0007.fits.fz
+    $ dfind /vo.cta.in2p3.fr/nectarcam | grep 5326 | grep -e "fits.fz" | wc -l
+    8
+
+
+The `nectarchain.data.management.DataManagement.findrun` method will
 automatically localize NectarCAM data on DIRAC, given a run number, and fetch
 the run files for you.
 
 Tips
 ====
 
+Web portal
+^^^^^^^^^^
+
+CTA-DIRAC has a web portal available at https://ctadirac-01.cscs.cta-observatory.org/DIRAC/ to interact with it. In particular, the *Job Monitor* application is very useful to interact with your jobs on DIRAC.
+
+.. tip::
+
+    If you submitted your jobs using the ``cta_nectarcam`` role, you would need to re-authenticate in the portal with this role: Click on *Settings* in the bottom left, and select *cta_nectarcam* under *Group*.
+
 Proxy error
 ^^^^^^^^^^^
 

diff --git a/docs/user-guide/index.rst b/docs/user-guide/index.rst
@@ -7,5 +7,7 @@ User Guide
    :maxdepth: 2
 
    getting-started
+   env-vars
    howto-dirac
-   troubleshooting
+   dqm
+   troubleshooting
diff --git a/pyproject.toml b/pyproject.toml
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "nectarchain"
 description = "Analysis chain for the CTA MSTN NectarCAM prototype"
-readme = "README.md"
+readme = "README.rst"
 authors = [{name = "CTAO NectarCAM collaboration"}]
 license = {text = "BSD-3-Clause"}
 classifiers = [

diff --git a/src/nectarchain/README.md b/src/nectarchain/README.md
diff --git a/src/nectarchain/dqm/README.md b/src/nectarchain/dqm/README.md