From f618b8ceb2abb046fec66cac567e6622abff2b55 Mon Sep 17 00:00:00 2001 From: sdc50 Date: Mon, 23 Sep 2024 17:18:10 -0600 Subject: [PATCH] Update docs for docker production (#1090) Co-authored-by: Nathan Swain --- docs/_static/css/tethys.css | 4 ++ docs/installation/production.rst | 2 +- .../production/docker/docker_compose.rst | 15 ++---- .../production/docker/dockerfile.rst | 4 +- docs/tutorials/bokeh.rst | 40 ++++++---------- docs/tutorials/thredds/plot_at_location.rst | 3 +- docs/tutorials/thredds/visualize_leaflet.rst | 47 ++++++++++++++----- 7 files changed, 63 insertions(+), 52 deletions(-) diff --git a/docs/_static/css/tethys.css b/docs/_static/css/tethys.css index 0c0be9910..b17d61d60 100644 --- a/docs/_static/css/tethys.css +++ b/docs/_static/css/tethys.css @@ -58,3 +58,7 @@ section>section { background-color: #cfef90; } +pre { + padding-left: 1rem; + padding-right: 1rem; +} diff --git a/docs/installation/production.rst b/docs/installation/production.rst index a7790df8d..5b022ce96 100644 --- a/docs/installation/production.rst +++ b/docs/installation/production.rst @@ -58,7 +58,7 @@ We currently have images for the following commercial cloud providers: Docker Deployment ----------------- -This method involves using Docker to package and automate the deployment of a Tethys Portal with that has your apps pre-installed. The advantage of this approach is that the process of installing Tethys Platform and your apps is automated in the Docker image. The other major advantage is portability, the Docker image can be deployed to any Linux server with Docker installed. The disadvantage is that there is a learning curve to get started using Docker for the first time. However, the investment of learning Docker is very much worth your time as most modern web applications are deployed using Docker or a similar container technology. +This method involves using Docker to package and automate the deployment of a Tethys Portal that has your apps pre-installed. The advantage of this approach is that the process of installing Tethys Platform and your apps is automated in the Docker image. The other major advantage is portability, the Docker image can be deployed to any Linux server with Docker installed. The disadvantage is that there is a learning curve to get started using Docker for the first time. However, the investment of learning Docker is very much worth your time as most modern web applications are deployed using Docker or a similar container technology. .. toctree:: :maxdepth: 1 diff --git a/docs/installation/production/docker/docker_compose.rst b/docs/installation/production/docker/docker_compose.rst index e038af829..27b55b408 100644 --- a/docs/installation/production/docker/docker_compose.rst +++ b/docs/installation/production/docker/docker_compose.rst @@ -71,7 +71,6 @@ Use the following instructions to create a Docker Compose file for the custom Te Docker allows directories from the host machine to be mounted into the containers. This is most often used to provide easy access to container data, configuration files, and logs. a. Create the following directories in :file:`tethys_portal_docker` directory: - .. code-block:: bash mkdir -p data/db @@ -84,7 +83,6 @@ a. Create the following directories in :file:`tethys_portal_docker` directory: mkdir -p logs/thredds/tomcat b. Download the default :file:`tomcat-users.xml` file from the `Unidata/thredds-docker GitHub repository `_: - * :download:`tomcat-users.xml ` .. note:: @@ -169,7 +167,6 @@ Add the following definition for the ``db`` service in the :file:`docker-compose * `ports`_: Ports to expose on the container (``:``). * `env_file`_: A file containing the environment variables to create for the container. Environment variables often contain sensitive information that should not be committed with the :file:`docker-compose.yml`. The :file:`db.env` file will be created in Step 7. * `volumes`_: Mount directories from the host into the container or create Docker-managed named volumes. Volumes allow you to preserve data that would otherwise be lost when the container is removed. The syntax shown here is: ``:``. - * ``./data/db:/var/lib/postgresql/data``: The primary data directory for PostgreSQL database. This directory contains the data and configuration files for the database. 4. Define THREDDS Service @@ -184,7 +181,7 @@ Add the following definition for the ``thredds`` service in the :file:`docker-co .. code-block:: yaml thredds: - image: unidata/thredds-docker:latest + image: unidata/thredds-docker:5.5 restart: always networks: - "internal" @@ -207,7 +204,6 @@ Add the following definition for the ``thredds`` service in the :file:`docker-co * `ports`_: Ports to expose on the container (``:``). * `env_file`_: A file containing the environment variables to create for the container. Environment variables often contain sensitive information that should not be committed with the :file:`docker-compose.yml`. The :file:`thredds.env` file will be created in Step 7. * `volumes`_: Mount directories from the host into the container or create Docker-managed named volumes. Volumes allow you to preserve data that would otherwise be lost when the container is removed. The syntax shown here is: ``:``. - * ``./data/thredds/:/usr/local/tomcat/content/thredds``: Main content directory for THREDDS. This directory will contain the data and XML configuration files for THREDDS. * ``./logs/thredds/tomcat/:/usr/local/tomcat/logs/``: Logs for Tomcat, the server running THREDDS. * ``./logs/thredds/thredds/:/usr/local/tomcat/content/thredds/logs/``: Logs for THREDDS. @@ -278,7 +274,6 @@ Add the following definition for the ``web`` service in the :file:`docker-compos * `ports`_: Ports to expose on the container (``:``). * `env_file`_: A file containing the environment variables to create for the container. Environment variables often contain sensitive information that should not be committed with the :file:`docker-compose.yml`. The :file:`web.env` file will be created in Step 7. * `volumes`_: Mount directories from the host into the container or create Docker-managed named volumes. Volumes allow you to preserve data that would otherwise be lost when the container is removed. The syntax shown here is: ``:``. - * ``./data/tethys:/var/lib/tethys_persist``: Main content directory for Tethys Platform. This directory contains the app workspaces, static files, and configuration files including the :file:`portal_config.yml`. * ``./log/tethys:/var/log/tethys``: Logs for Tethys. @@ -290,19 +285,16 @@ Each of the Docker containers can be configured through the environment variable With that said, certain environment variables need to be defined for the custom Tethys Portal Compose recipe to work. This is often the case, so another pattern that is used is to provide default ``.env`` files that users can copy and modify. The default ``.env`` files are committed to the repository and the copies with sensitive information are not. In this step you will create the default ``.env`` files referenced in the `env_file`_ sections of the :file:`docker-compose.yml`. a. Create a new :file:`env` directory in the :file:`tethys_portal_docker` directory for storing the ``.env`` files: - .. code-block:: bash mkdir env b. Create three new empty files in the :file:`env` directory with the same names as those referenced in the `env_file`_ sections of the :file:`docker-compose.yml`: - .. code-block:: bash touch env/db.env env/thredds.env env/web.env c. Add the following contents to each ``.env`` file: - **db.env** .. code-block:: docker @@ -357,6 +349,8 @@ c. Add the following contents to each ``.env`` file: .. code-block:: docker + TERM=xterm + # Domain name of server should be first in the list if multiple entries added ALLOWED_HOSTS="\"[localhost]\"" @@ -463,14 +457,12 @@ Update the contents of the README with instructions for using the repository and The contents of the :file:`data`, :file:`logs`, and :file:`keys` directories should not be committed into the Git repository because they contain large amounts of instance-specific data and sensitive information. a. Create a :file:`.gitignore` file: - .. code-block:: bash touch .gitignore b. Add the following contents to the :file:`.gitignore` file to omit the contents of these directories from being tracked: - .. code-block:: text data/ @@ -478,7 +470,6 @@ b. Add the following contents to the :file:`.gitignore` file to omit the content logs/ c. Stage changes and commit the changes as follows: - .. code-block:: bash git add . diff --git a/docs/installation/production/docker/dockerfile.rst b/docs/installation/production/docker/dockerfile.rst index cd49e6207..dcbfc4e5b 100644 --- a/docs/installation/production/docker/dockerfile.rst +++ b/docs/installation/production/docker/dockerfile.rst @@ -20,7 +20,7 @@ Please ensure you have the following prerequisites before continuing: Project Setup ============= -Before you can start creating the :file:`Dockerfile` there is some setup that needs to be completed. This includes creating a folder to house all of the artifacts that you will use for the Docker build and acquiring the source code for the apps that will be installed in the Tethys Portal. It will also include setting up a Git repository. An important part of creating Docker projects is knowing how to properly version it version control software, so this tutorial will instruct you which files to commit. Follow these instructions to set up the Docker project. +Before you can start creating the :file:`Dockerfile` there is some setup that needs to be completed. This includes creating a folder to house all of the artifacts that you will use for the Docker build and acquiring the source code for the apps that will be installed in the Tethys Portal. It will also include setting up a Git repository. An important part of creating Docker projects is knowing how to properly version it with version control software, so this tutorial will instruct you which files to commit. Follow these instructions to set up the Docker project. 1. Create New Directory ----------------------- @@ -224,7 +224,7 @@ d. Add the following lines to the Dockefile to add the images to the container i The `RUN `_ instruction can be used to run any command during the build. For long commands, the ``\`` (backslash) character can be used to continue a ``RUN`` instruction on the next line for easier readability. -For this image we need to run the ``tethys install`` command for each of our apps. The trickiest part about doing this in a Docker build is activating the ``tethys`` environment, which must be done for each ``RUN`` call. Add the following lines to the :file:`Dockerfile`: +For this image we need to run the ``tethys install`` command for each of our apps. The trickiest part about doing this in a Docker build is activating the ``tethys`` environment, which must be done before installing the apps. Add the following lines to the :file:`Dockerfile`: .. code-block:: dockerfile diff --git a/docs/tutorials/bokeh.rst b/docs/tutorials/bokeh.rst index 25be698b3..aaf8596b3 100644 --- a/docs/tutorials/bokeh.rst +++ b/docs/tutorials/bokeh.rst @@ -55,7 +55,7 @@ To leverage the Bokeh integration with Tethys you will need the ``bokeh`` and `` - conda-forge packages: - bokeh - - bokeh_django + - bokeh-django - bokeh_sampledata pip: @@ -79,9 +79,11 @@ Let's use Bokeh's sea temperature sample data to create a time series plot and l from tethys_sdk.routing import handler + from .app import App + @handler( - template="bokeh_tutorial/home.html", + template=f"{App.package}/home.html", ) def home(document): df = sea_surface_temperature.copy() @@ -101,7 +103,7 @@ This simple handler contains the logic for a time series plot of the sea surface .. code-block:: html+django - {% extends "bokeh_tutorial/base.html" %} + {% extends tethys_app.package|add:"/base.html" %} {% block app_content %}

Bokeh Integration Example

@@ -126,6 +128,9 @@ This is a simple Bokeh plot. We will now add the rest of the logic to make it an ... + @handler( + template=f"{App.package}/home.html", + ) def home(document): df = sea_surface_temperature.copy() source = ColumnDataSource(data=df) @@ -172,27 +177,10 @@ In this example we will build on top of the ``bokeh_tutorial`` app to demonstrat .. code-block:: yaml - # This file should be committed to your app code. - version: 1.0 - # This should match the app - package name in your setup.py - name: bokeh_tutorial - - requirements: - # Putting in a skip true param will skip the entire section. Ignoring the option will assume it be set to False - skip: false - conda: - channels: - - conda-forge - packages: - - bokeh - - bokeh_django - - bokeh_sampledata - - panel - - param - - pip: - - post: + packages: + ... + - panel + - param 3. Add the following objects to a new file called ``param_model.py``. @@ -287,12 +275,14 @@ The added classes depend on ``Bokeh``. The `Circle` and `NGon` classes depend o .. code-block:: python + ... + from .param_model import ShapeViewer ... @handler( - app_package='bokeh_tutorial', + app_package=App.package, ) def shapes(document): viewer = ShapeViewer().panel() diff --git a/docs/tutorials/thredds/plot_at_location.rst b/docs/tutorials/thredds/plot_at_location.rst index 62a28e547..8029f2665 100644 --- a/docs/tutorials/thredds/plot_at_location.rst +++ b/docs/tutorials/thredds/plot_at_location.rst @@ -124,7 +124,7 @@ In this step you'll learn to use another Leaflet plugin: `Leaflet.Draw `_ in a web browser and login if necessary. After the home page loads, inspect the log messages in the terminal where Tethys is running. The ``pprint`` calls in our controller should print the object being returned from the ``parse_dataset`` function in the terminal. It should also populate the options for the **Dataset** control. -4. Create Endpoint for Getting Available WMS Layers + +4. Install Chardet +================== + +In the next step you will create a function to retrieve metadata from the THREDDS server. This will require using the `chardet` library to determine the encoding of the response. + +1. Install `chardet` as follows running the following command in the terminal: + +.. code-block:: + + # with conda + conda install chardet + + # with pip + pip install chardet + +2. The app now depends on `chardet`, so add it to the `install.yml` file: + +.. code-block:: yaml + + dependencies: + ... + - chardet + + +5. Create Endpoint for Getting Available WMS Layers =================================================== Each time a new dataset is selected, the options in the variable and style controls need to be updated to match the variables and styles of the new dataset. This information can be found by querying the WMS endpoint of the dataset provided by THREDDS. Querying the WMS endpoint is most easily accomplished by using the `OWSLib `_ Python library. In this step you will implement a new controller that will use OWSLib to retrieve the information and call it using ``fetch`` anytime a new dataset is selected. @@ -548,7 +573,7 @@ Each time a new dataset is selected, the options in the variable and style contr return JsonResponse(json_response) -5. Stub Out the Variable and Style Control JavaScript Methods +6. Stub Out the Variable and Style Control JavaScript Methods ============================================================= In this step you will use the new ``get-wms-layers`` endpoint to get a list of layers and their attributes (e.g. styles) to update the variable and style controls. @@ -609,7 +634,7 @@ In this step you will use the new ``get-wms-layers`` endpoint to get a list of l init_controls(); }); -6. Implement Variable and Style Control Methods +7. Implement Variable and Style Control Methods =============================================== In this step you will implement the dataset controll JavaScript methods in :file:`public/js/leaflet_map.js`. @@ -711,7 +736,7 @@ Here is a brief explanation of each method that will be implemented in this step 4. Verify that the **Variable** and **Style** controls are updated properly when the dataset changes. Browse to ``_ in a web browser and login if necessary. Use the **Dataset** control to select a new dataset and verify that the **Variable** and **Style** options update accordingly. Inspect the terminal where Tethys is running to see the output from the print statement we added for debugging in Step 4. -7. Add Time-Dimension Plugin to Leaflet Map +8. Add Time-Dimension Plugin to Leaflet Map =========================================== Many of the datasets hosted on THREDDS servers have time as a dimension. In this step you will add the Time-Dimension plugin to the Leaflet map so that it can visualize data with the time dimension. The plugin adds a time slider control to the map and provides a way to load and visualize WMS layers with a time dimension. @@ -760,7 +785,7 @@ Many of the datasets hosted on THREDDS servers have time as a dimension. In this 3. Verify that the Time-Dimension control is enabled. Browse to ``_ in a web browser and login if necessary. There should now be a time slider control at the bottom of the map. -8. Add Selected Dataset Layer to Map +9. Add Selected Dataset Layer to Map ==================================== In this step, you'll create the ``update_layer`` method that will add the THREDDS dataset WMS layer to the Leaflet map. @@ -873,8 +898,8 @@ This controller will act as a proxy to the WMS service. It will take a URL and a 7. Verify that the layers show up on the map. Browse to ``_ in a web browser and login if necessary. Select the "Best GFS Half Degree Forecast Time Series" dataset using the **Dataset** control to test a time-varying layer. Press the **Play** button on the Time-Dimension control to animate the layer. -9. Implement Legend for Layers -============================== +10. Implement Legend for Layers +=============================== The THREDDS implementation of the WMS standard includes support for the ``GetLayerGraphic`` request. In this step you'll use this request to generate a legend image for the layer and style selected. @@ -954,7 +979,7 @@ The THREDDS implementation of the WMS standard includes support for the ``GetLay 6. Verify that the legend has been added to the app. Browse to ``_ in a web browser and login if necessary. The legend should appear under the Query controls in the navigation window on the left. Change the style and verify that the legend updates accordingly. -10. Implement a Map Loading Indicator +11. Implement a Map Loading Indicator ===================================== Depending on the speed of the THREDDS server and the user's internet connection, loading the layers on the map may take some time. In this step you'll add a loading indicator so that the user knows when the app is working on loading layers. @@ -1123,7 +1148,7 @@ Depending on the speed of the THREDDS server and the user's internet connection, }); }; -11. Clean Up +12. Clean Up ============ During development it is common to use print statements. Rather than delete these when you are done, turn them into log statements so that you can use them for debugging in the future. @@ -1230,7 +1255,7 @@ During development it is common to use print statements. Rather than delete thes Logging excessively can impact the performance of your app. Use ``info``, ``error``, and ``warning`` to log minimal, summary information that is useful for monitoring normal operation of the app. Use ``debug`` to log more detailed information to help you assess bugs or other issues with your app without needing to modify the code. In production, the Tethys Portal can be configured to log at different levels of detail using these classifications. See: `Python Logging HOWTO `_ and :ref:`tethys_configuration`. -12. Test and Verify +13. Test and Verify =================== Browse to ``_ in a web browser and login if necessary. Verify the following: @@ -1241,7 +1266,7 @@ Browse to ``_ in a web browser and 4. The map should feature an animation slider. If the dataset selected has time varying data, the slider should display a time step. Otherwise it will say "Time not available". 5. Select the "Best GFS Half Degree Forecast Time Series" dataset using the **Dataset** control to test a time-varying layer. Press the **Play** button on the Time-Dimension control to animate the layer. -13. Solution +14. Solution ============ This concludes the New App Project portion of the THREDDS Tutorial. You can view the solution on GitHub at ``_ or clone it as follows: