Add contributor docs for maintaining dependencies

docs/contribute/code/dependencies.rst

@@ -4,26 +4,146 @@
 Maintain Dependencies
 *********************
 
-**Last Updated:** December 2024
+**Last Updated:** January 2025
 
-Tethys Maintained Dependencies
-==============================
+Dependencies are the libraries, packages, and other software that Tethys Platform relies on to function. Most of these dependencies are maintained by others in the open source community, but some are maintained by the developers of Tethys Platform. This document outlines the process for maintaining dependencies for Tethys Platform.
 
-* Tethys Dataset Services
-* Tethys Dask Scheduler
-* CondorPy
+.. _contrib_deps_python:
 
-Third-party Dependencies
-========================
+Python Dependencies
+===================
 
-* Conda Forge Packages
-* Bug Fixes
-* Update Tethys to work with new versions of dependencies
+The Python dependencies are tracked in several Conda :file:`environment.yml` files:
 
-Maintain Dependency Packages
-============================
+* :file:`environment.yml`: Lists all Python dependencies of Tethys Platform, including those needed for optional features. This file is used to build the current |version| release of Tethys Platform.
+* :file:`micro_environment.yml`: Lists only the minimal dependencies needed to run Tethys Platform, excluding the dependencies needed by optional features. This file is used to build the Micro-Tethys release of Tethys Platform (see: :ref:`contrib_deps_optional_dependencies`). 
+* :file:`docs/docs_environment.yml`: Lists the Python dependencies needed to build the Tethys Platform documentation.
 
-* Conda Forge Packages (adding, updating, get added as a maintainer)
-* Tethys Dataset Services
-* Tethys Dask Scheduler
-* JavaScript Dependencies
+
+.. _contrib_deps_optional_dependencies:
+
+Optional Dependencies and Micro-Tethys
+--------------------------------------
+
+As the the Tethys Project has grown and added more features, the number of dependencies have also grown. The large number of dependencies has made it more difficult to install Tethys Platform. This is primarily caused by the complex, interconnected web of dependency version requirements that the Conda solver must sort through. Occasionally, the Conda solver is not able to solve the environment, which prevents users from installing Tethys Platform. Using a faster solver, like `Conda libmamba solver <https://www.anaconda.com/blog/conda-is-fast-now>`_ usually side steps this issue, but not always.
+
+To address this issue, the optional dependencies feature was introduced early in the 4.X series. Optional dependencies are dependencies that are not required to run Tethys Platform, but are required to enable certain features (e.g. captchas, persistent stores, and analytics). By making these dependencies optional, users can install Tethys Platform faster and more easily. Optional dependencies are then installed "on demand" when they are needed for a certain feature.
+
+To support backward compatibility, the primary release of Tethys Platform (4.X) still ships with all of the optional dependencies being installed. However, there is a secondary release, called Micro-Tethys, that ships with only the minimal dependencies needed to run Tethys Platform (see Tip box of the :ref:`Install the tethys-platform Conda Package step <getting_started_install_tethys>`). Users can install the optional dependencies as needed to enable the optional features. In version 5.0, the Micro-Tethys release will become the default release.
+
+Adding Dependencies
+-------------------
+
+When adding new dependencies to Tethys Platform, developers should consider whether the dependency is required for the core functionality of Tethys Platform or if it is supporting an optional feature. If the dependency is required for core functionality, it should be added to both the :file:`environment.yml` and :file:`micro_environment.yml` files. If the dependency is needed for an optional feature, it should be added to the :file:`environment.yml` file only in the Optional Dependency section. The documentation for the features should include instructions for installing the optional dependency to enable the feature. For an example of optional dependency installation instructions in the documentation, see :ref:`persistent_stores_api`.
+
+Add dependencies to the :file:`environment.yml` files as **unpinned** dependencies (without a version number). This enables the dependencies to automatically update to their latest versions during automated testing runs, allowing potential issues introduced by these new versions to be detected. Dependencies are automatically pinned to the versions used in the last successful test run during the release process to ensure greater stability in the releases.
+
+Conda Forge Packages
+--------------------
+
+The Tethys Platform package is published on the `Conda Forge <https://conda-forge.org/>`_ channel so that it can be easily installed using `Conda <https://conda.org/>`_. The Conda Forge channel is a community-maintained collection of Conda packages. One requirement for packages being published on the Conda Forge channel is that all of their dependencies must also be published on the Conda Forge channel. As a result, *all* of the Tethys Platform dependencies must also be published on the Conda Forge channel.
+
+However, not all Python package maintainers release their packages on Conda Forge, but anyone can create a Conda Forge package for any Python library by creating a Conda Forge feedstock for the package. A feedstock is a repository that contains the recipe for building the package and publishing it on the Conda Forge channel. You **do not** have to be a maintainer of a Python library to package it on Conda Forge.
+
+The Tethys Platform developers maintain `Conda Forge feedstocks <https://conda-forge.org/docs/maintainer/adding_pkgs.html>`_ for many of the project's Python dependencies. The following spreadsheet lists the Conda Forge feedstocks that Tethys Platform developers maintain: `Tethys Platform Conda Forge Feedstocks <https://docs.google.com/spreadsheets/d/1H7V-oKfgA3vE7aSko04ELPWQRVEzRziVrZpJsrSV7Ig/edit>`_. To assist with maintaining a feedstock, create a personal fork of the repository, add your GitHub username to the :file:`recipe/meta.yaml`, and submit a pull request.
+
+.. tip:
+
+    For more information about creating and maintaining Conda Forge feedstocks, see `Maintainer Documentation | Conda Forge <https://conda-forge.org/docs/maintainer/>`_.
+
+Project Maintained Dependencies
+-------------------------------
+
+The Tethys Platform developers maintain several of the project's dependencies. These are separate projects that contain functionality that is general purpose enough to be useful outside of Tethys Platform. The following sections briefly describe each of these dependencies and where they can be found.
+
+Tethys Dataset Services
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Tethys Dataset Services (TDS) provides high-level Python APIs for GeoServer and CKAN. These are used to support the Tethys Services functionality in Tethys Platform (see: :ref:`tds_geoserver_reference` and :ref:`tds_ckan_reference`).
+
+* **Source Code**: https://github.com/tethysplatform/tethys_dataset_services
+* **Documentation**: :ref:`tds_geoserver_reference` and :ref:`tds_ckan_reference`
+* **PyPI Package**: https://pypi.org/project/tethys-dataset-services/
+* **Conda Forge Feedstock**: https://github.com/conda-forge/tethys_dataset_services-feedstock
+
+Tethys Dask Scheduler
+~~~~~~~~~~~~~~~~~~~~~
+
+The Tethys Dask Scheduler is an extended version of the Dask Scheduler that is able to communicate with a Tethys Portal to update associated Tethys Job statuses (see: :ref:`tethys_job_dask` and :ref:`tutorials_dask`).
+
+* **Source Code**: https://github.com/tethysplatform/tethys_dask_scheduler
+* **Documentation**: :ref:`tethys_job_dask` and :ref:`tutorials_dask`
+* **PyPI Package**: https://pypi.org/project/tethys-dask-scheduler/
+* **Conda Forge Feedstock**: https://github.com/conda-forge/tethys_dask_scheduler-feedstock
+
+CondorPy
+~~~~~~~~
+
+CondorPy is a Python package that provides a high-level interface for submitting and managing jobs on a HTCondor cluster. It is used by the Tethys Job Manager to submit and manage Tethys Jobs on a HTCondor cluster (see: :ref:`tethys_jobs_condor` and :ref:`tethys_jobs_condor_workflow`).
+
+* **Source Code**: https://github.com/tethysplatform/condorpy
+* **Documentation**: https://www.tethysplatform.org/condorpy/
+* **PyPI Package**: https://pypi.org/project/condorpy/
+* **Conda Forge Feedstock**: https://github.com/conda-forge/condorpy-feedstock
+
+Bokeh Django
+~~~~~~~~~~~~
+
+The bokeh-django package provides support for running Bokeh apps in Django. Tethys Platform uses this package to run Bokeh widgets within Tethys Apps via the handler functionality (see :ref:`handler-decorator`).
+
+* **Source Code**: https://github.com/bokeh/bokeh-django
+* **Documentation**: https://github.com/bokeh/bokeh-django/blob/main/README.md
+* **PyPI Package**: https://pypi.org/project/bokeh-django/
+* **Conda Forge Feedstock**: https://github.com/conda-forge/bokeh-django-feedstock
+
+.. _contrib_deps_javascript:
+
+JavaScript Dependencies
+=======================
+
+Tethys Platform has many JavaScript dependencies that are required to support the dynamic, interactive functionality. These include libraries for plotting, mapping, interactive tables, and interactive controls like date pickers and searchable dropdowns.
+
+CDN and ``npm`` Support
+-----------------------
+
+Tethys Platform supports two modes for hosting JavaScript dependencies: CDN hosted (default) or self hosted.
+
+**CDN Hosted**
+
+The default mode for Tethys Portal to retrieve JavaScript dependencies is to retrieve the packages from the `jsDelivr <https://www.jsdelivr.com/>`_ CDN (Content Delivery Network) as needed. The advantage of using a CDN is that it allows for faster and more reliable delivery of content and offloads the load to third-party servers. CDN servers are able to handle high traffic volumes resulting in overall performance improvements.
+
+The reason the ``jsDelivr`` CDN is used is twofold:
+
+1. ``jsDelivr`` automatically packages all packages published on `npm <https://www.npmjs.com/>`_ (Node Package Manager), the worlds largest software registry. It is likely that any JavaScript dependency that Tethys needs will be on ``npm``, and thus ``jsDelivr``.
+2. To support the ability to download dependencies for the self hosted mode, which is discussed in more detail below.
+
+**Self Hosted**
+
+In some cases, it is not feasible to rely on the CDN mode for JavaScript dependencies, usually due to organization policies or running in an offline environment. For these cases, Tethys Platform provides a self-hosted mode. In this mode, the dependencies are downloaded from ``npm`` using the ``npm`` commandline tool and Tethys Portal hosts the dependencies itself (see: :ref:`self_hosted_deps_config`).
+
+The :file:`dependencies.py` File
+--------------------------------
+
+A list of JavaScript dependencies is maintained in the :file:`dependencies.py` file located in the :file:`tethys_portal` package. In this file is a variable named ``vendor_static_dependencies`` that contains the list (in dictionary form).
+
+Adding Dependencies
+-------------------
+
+To add a new JavaScript dependency to Tethys Platform, do the following:
+
+1. Locate the dependency on `jsDelivr <https://www.jsdelivr.com/>`_.
+2. Open :file:`tethys_portal/dependencies.py`.
+3. Locate the ``vendor_static_dependencies`` dictionary near the end of the file.
+4. Add a new key to the dictionary as a unique lookup key for the dependency (please maintain alphabetical order).
+5. Create a new instance of ``JsDelivrStaticDependency`` for the value, filling in the necessary information:
+
+   * **npm_name**: Name of the NPM package.
+   * **version**: Version of the package for Tethys Platform to use.
+   * **js_path**: Relative path to the JavaScript file, usually a :file:`*.min.js` file.
+   * **js_integrity**: The integrity string for the specific version of the JavaScript file (if available).
+   * **css_path**: Relative path to the CSS file, usually a :file:`*.min.css` file (if applicable).
+   * **css_integrity**: The integrity string for the specific version of the CSS file (if available).
+
+.. tip::
+
+    Compare an existing entry in ``vendor_static_dictionary`` to what you find on ``jsDelivr`` for that package.

docs/contribute/code/getting_started.rst

@@ -155,7 +155,7 @@ In this tutorial you will be making the output of the ``tethys version`` command
 
     If ``pyfiglet`` wasn't on Conda Forge, you would need to add it before the dependency could be added to Tethys Platform. Lucky us.
 
-4. Finally, add ``pyfiglet`` to the :file:`environment.yml` and the :file:`micro_environment.yml` files. The "Gen CLI commands" section seems as good as any:
+4. Finally, add ``pyfiglet`` to the :file:`environment.yml` and the :file:`micro_environment.yml` files (for more details on maintaining dependencies in Tethys Platform, see: :ref:`maintain_dependencies`). The "Gen CLI commands" section seems as good as any:
 
 .. code-block:: yaml
     :emphasize-lines: 4

docs/contribute/documentation/guidelines.rst

@@ -144,12 +144,12 @@ Download files should be stored in a ``resources`` subdirectory near where the d
 
     For more details on the ``download`` role, see `Referencing downloadable files <https://www.sphinx-doc.org/en/master/usage/referencing.html#referencing-downloadable-files>`_.
 
-.. _contribute_docs_headers:
+.. _contribute_docs_headings:
 
-Headers
-=======
+Headings
+========
 
-Every reStructuredText (RST) page should have a title header marked with lines of "*" characters on top and bottom that are the same length as the title:
+Every reStructuredText (RST) page should have a title heading marked with lines of "*" characters on top and bottom that are the same length as the title:
 
 .. code-block:: rst
 
@@ -171,10 +171,10 @@ The **Last Updated** date should follow immediately after the page title, follow
 
     You may encounter some pages with older style dates that include the day; please remove the day and update it to the new format.
 
-Sub Headers
------------
+Sub Headings
+------------
 
-RST syntax allows for multiple levels of headers by using different characters to underline the header text. For consistency, please use "=" for Header 1, "-" for Header 2, and "+" for Header 3. Avoid using more than three levels of headers. 
+RST syntax allows for multiple levels of headings by using different characters to underline the heading text. For consistency, please use "=" for Heading 1, "-" for Heading 2, and "~" for Heading 3, and "+" for Heading 4". Avoid using more than four levels of headings. 
 
 .. code-block:: rst
 
@@ -185,12 +185,15 @@ RST syntax allows for multiple levels of headers by using different characters t
     ---------
 
     Heading 3
+    ~~~~~~~~~
+
+    Heading 4
     +++++++++
 
-Full Page Header Example
-------------------------
+Full Page Heading Example
+-------------------------
 
-Here is an example of the page title, **Last Updated** date, and different levels of headers for a full page:
+Here is an example of the page title, **Last Updated** date, and different levels of headings for a full page:
 
 .. code-block:: rst
 
@@ -211,7 +214,7 @@ Here is an example of the page title, **Last Updated** date, and different level
 
 .. tip::
 
-    For more details on header syntax in RST, see `reStructured Text Primer: Sections <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections>`_.
+    For more details on heading syntax in RST, see `reStructured Text Primer: Sections <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#sections>`_.
 
 .. _contribute_docs_hyperlinks:
 

docs/installation/production/manual/configuration.rst

@@ -37,6 +37,7 @@ These guides describe additional configuration that you can perform to add more
     configuration/advanced/customize
     configuration/advanced/email_config
     configuration/advanced/lockout
+    configuration/advanced/self_hosted_js_deps
     configuration/advanced/social_auth
     configuration/advanced/multi_factor_auth
     configuration/advanced/webanalytics

docs/tethys_portal/configuration.rst

@@ -89,7 +89,7 @@ STATIC_ROOT                                        the Django `STATIC_ROOT <http
 MEDIA_URL                                          the Django `MEDIA_URL <https://docs.djangoproject.com/en/4.2/ref/settings/#media-url>`_ setting. Defaults to ``'/media/'``.
 MEDIA_ROOT                                         the Django `MEDIA_ROOT <https://docs.djangoproject.com/en/4.2/ref/settings/#media-root>`_ setting. Defaults to :file:`~/.tethys/media/`.
 STATICFILES_USE_NPM                                serves JavaScript dependencies through Tethys rather than using a content delivery network (CDN) when ``True``. Defaults to ``False``. When set to ``True`` then you must run ``tethys gen package_json`` to npm install the JS dependencies locally so they can be served by Tethys.
-ADDITIONAL_TEMPLATE_DIRS                           a list of dot-paths to template directories. These will be prepended to Tethys's list of template directories so specific templates can be overriden.
+ADDITIONAL_TEMPLATE_DIRS                           a list of dot-paths to template directories. These will be prepended to Tethys's list of template directories so specific templates can be overridden.
 ADDITIONAL_URLPATTERNS                             a list of dot-paths to list or tuples that define additional URL patterns to register in the portal. Additional URL patterns will precede default URL patterns so URLs will first match against user specified URL patterns.
 ADDITIONAL_SETTINGS_FILES                          a list of dot-paths or file paths to Python files that will be imported into the ``settings.py`` file. Additional settings files are imported at the end of the file and thus will override any previous settings with name conflicts.
 MULTIPLE_APP_MODE                                  boolean indicating if the portal should host multiple apps or be configured for a single standalone app.

docs/tethys_sdk/jobs/condor_job_type.rst

@@ -1,3 +1,5 @@
+.. _tethys_jobs_condor:
+
 ***************
 Condor Job Type
 ***************

docs/tethys_sdk/jobs/condor_workflow_type.rst

@@ -1,3 +1,5 @@
+.. _tethys_jobs_condor_workflow:
+
 ************************
 Condor Workflow Job Type
 ************************

docs/tethys_sdk/jobs/dask_job_type.rst

@@ -1,3 +1,5 @@
+.. _tethys_job_dask:
+
 *************
 Dask Job Type
 *************

docs/tethys_sdk/tethys_services/dataset_service/ckan_reference.rst

@@ -1,3 +1,5 @@
+.. _tds_ckan_reference:
+
 *****************************
 CKAN Dataset Engine Reference
 *****************************

docs/tethys_sdk/tethys_services/persistent_store.rst

@@ -1,3 +1,5 @@
+.. _persistent_stores_api:
+
 *********************
 Persistent Stores API
 *********************

docs/tethys_sdk/tethys_services/spatial_dataset_service/geoserver_reference.rst

@@ -1,3 +1,5 @@
+.. _tds_geoserver_reference:
+
 ****************************************
 GeoServer SpatialDatasetEngine Reference
 ****************************************

docs/tutorials/dask.rst

@@ -1,3 +1,5 @@
+.. _tutorials_dask:
+
 *************
 Dask Tutorial
 *************