Improves installation and docs (#855)

  - Adds docker compose for secure hosting of application
  - Fixes problems with pdf rendering of docs. The problem
    is to the images not being able to fit in one page of
    pdf document.
  - Improves docs for CLI and https installation

README.md

@@ -57,7 +57,7 @@ The [web client](client),
 [library](servers/lib) and
 [runner](servers/execution/runner)
 microservices are functional at present.
-These are packages are available on
+These packages are available on
 [github](https://github.com/orgs/INTO-CPS-Association/packages?repo_name=DTaaS).
 
 Please see the

cli/README.md

@@ -8,7 +8,7 @@ INTO-CPS-Association Digital Twins as a Service.
 Please visit and download the
 [DTaaS Software](https://github.com/INTO-CPS-Association/DTaaS).
 
-The DTaaS service with base users and essential
+The DTaaS application with base users and essential
 containers should be up and running before using the CLI.
 
 ## Installation
@@ -58,6 +58,25 @@ server-dns = "localhost"
 path = "/home/Desktop/DTaaS"
 ```
 
+### Select Template
+
+The _cli_ uses YAML templates provided in this directory to create
+new user workspaces. The templates are:
+
+1. _user.local.yml_: localhost installation
+1. _User.server.yml_: multi-user web application application over HTTP
+1. _user.server.secure.yml_: multi-user web application application over HTTPS
+
+Please note that the _cli_ is not capable of detecting the difference between
+HTTP and HTTPS modes of web application. Thus if you are serving
+web application over HTTPS, please do one extra step.
+
+```bash
+cp user.server.secure.yml user.server.yml
+```
+
+This will change the user template from insecure to secure.
+
 ### Add users
 
 To add new users using the CLI, fill in the
@@ -71,13 +90,21 @@ usernames of the users to be added
 add = ["username1","username2", "username3"]
 ```
 
+Make sure you are in the _cli_ directory.
+
 Then simply:
 
 ```bash
 dtaas admin user add
 ```
 
-#### Caveat
+The command checks for the existence of `files/<username>` directory.
+If it does not exist, a new directory with correct file structure is created.
+The directory, if it exists, must be owned by the user executing
+**dtaas** command on the host operating system. If the files do not
+have the expected ownership rights, the command fails.
+
+#### Caveats
 
 This brings up the containers, without the AuthMS authentication.
 
@@ -91,7 +118,7 @@ This brings up the containers, without the AuthMS authentication.
 - Go to the _docker_ directory
 
 ```bash
-cd <DTaaS>/docker
+cd <DTaaS>/deploy/docker
 ```
 
 - Add three lines to the `conf.server` file
@@ -102,7 +129,7 @@ rule.onlyu3.rule=PathPrefix(`/user3`)
 rule.onlyu3.whitelist = [email protected]
 ```
 
-Run the command for these changes to take effect:
+- Run the command for these changes to take effect:
 
 ```bash
 docker compose -f compose.server.yml --env-file .env up -d --force-recreate traefik-forward-auth
@@ -113,18 +140,27 @@ instance, with authorization enabled.
 
 ### Delete users
 
-TO delete existing users, fill in the _users.delete_ list in
-_dtaas_.toml_ with the Gitlab instance
-usernames of the users to be deleted.
+- To delete existing users, fill in the _users.delete_ list in
+  _dtaas.toml_ with the Gitlab instance
+  usernames of the users to be deleted.
 
-Make sure you are in the _cli_ directory.
+```toml
+[users]
+# matching user info must present in this config file
+delete = ["username1","username2", "username3"]
+```
+
+- Make sure you are in the _cli_ directory.
 
 Then simply:
 
 ```bash
 dtaas admin user delete
 ```
 
+- Remember to remove the rules for deleted users
+  in _conf.server_.
+
 ### Additional Points to Remember
 
 - The _user add_ CLI will add and start a
@@ -138,3 +174,7 @@ dtaas admin user delete
 - _user add_ and _user delete_ CLIs return an
   error if the _add_ and _delete_ lists in
   _dtaas.toml_ are empty, respectively.
+
+- '.' is a special character. Currently, usernames which have
+  '.'s in them cannot be added properly through the CLI.
+  This is an active issue that will be resolved in future releases.

cli/users.local.yml

@@ -1,7 +1,8 @@
 image: mltooling/ml-workspace-minimal:0.13.2
+restart: unless-stopped
 volumes:
-  - ${DTAAS_DIR}/files/common:/workspace/common
-  - ${DTAAS_DIR}/files/${username}:/workspace
+  - "${DTAAS_DIR}/files/common:/workspace/common"
+  - "${DTAAS_DIR}/files/${username}:/workspace"
 environment:
   - AUTHENTICATE_VIA_JUPYTER=
   - WORKSPACE_BASE_URL=${username}
@@ -12,4 +13,4 @@ labels:
   - "traefik.http.routers.${username}.rule=PathPrefix(`/${username}`)"
   - "traefik.http.routers.${username}.middlewares=traefik-forward-auth"
 networks:
-  - users
+  - users

cli/users.server.secure.yml

@@ -0,0 +1,16 @@
+image: mltooling/ml-workspace-minimal:0.13.2
+restart: unless-stopped
+volumes:
+  - "${DTAAS_DIR}/files/common:/workspace/common"
+  - "${DTAAS_DIR}/files/${username}:/workspace"
+environment:
+  - AUTHENTICATE_VIA_JUPYTER=
+  - WORKSPACE_BASE_URL=${username}
+shm_size: 512m
+labels:
+  - "traefik.enable=true"
+  - "traefik.http.routers.${username}.rule=Host(`${SERVER_DNS}`)&&PathPrefix(`/${username}`)"
+  - "traefik.http.routers.${username}.tls=true"
+  - "traefik.http.routers.${username}.middlewares=traefik-forward-auth"
+networks:
+  - users

cli/users.server.yml

@@ -1,7 +1,8 @@
 image: mltooling/ml-workspace-minimal:0.13.2
+restart: unless-stopped
 volumes:
-  - ${DTAAS_DIR}/files/common:/workspace/common
-  - ${DTAAS_DIR}/files/${username}:/workspace
+  - "${DTAAS_DIR}/files/common:/workspace/common"
+  - "${DTAAS_DIR}/files/${username}:/workspace"
 environment:
   - AUTHENTICATE_VIA_JUPYTER=
   - WORKSPACE_BASE_URL=${username}
@@ -12,4 +13,4 @@ labels:
   - "traefik.http.routers.${username}.rule=Host(`${SERVER_DNS}`)&&PathPrefix(`/${username}`)"
   - "traefik.http.routers.${username}.middlewares=traefik-forward-auth"
 networks:
-  - users
+  - users

deploy/docker/README.md

@@ -6,6 +6,6 @@ on both localhost and server.
 There are two installation scenarios each with a dedicated set of instructions:
 
 1. Single user [localhost](./LOCALHOST.md)
-1. Multi-user [web application](./SERVER.md)
+1. Multi-user [web application](./SERVER.md) serving application over **HTTP or HTTPS**
 
 Please choose an appropriate installation scenario.

deploy/docker/SERVER.md

@@ -17,20 +17,32 @@ In the new application configuration, there are two OAuth2 applications.
 
 The installation requirements to run this docker version of the DTaaS are:
 
-- DNS name for the server
-- docker with compose plugin
-- User accounts on a gitlab instance
-- OAuth2 application registrations
+### Domain name
 
-### Create User Accounts
+The DTaaS software is a web application and is preferably hosted
+on a server with a domain name like <http:>_foo.com_</http:>.
+It is also possible to use an IP address in place of domain name.
 
-Create user accounts in a gitlab instance for all the users.
+### TLS / HTTPS Certificate
+
+It is possible to add HTTPS option to the DTaaS software installation.
+Creation of the required TLS certificates is possible through
+[certbot](https://certbot.eff.org/).
+
+### OAuth Provider
+
+**[Gitlab Instance](https://about.gitlab.com/install/)** -
+The DTaaS uses Gitlab OAuth2.0 authorization for user authorization.
+You can either have an on-premise instance of gitlab, or
+use [gitlab.com](https://gitlab.com) itself.
+
+### User Accounts
+
+Create user accounts in a linked gitlab instance for all the users.
 
 The default docker compose file contains two - _user1_ and _user2_.
 These names need to be changed to suitable usernames.
 
-It is possible to use <https://gitlab.com> as well for OAuth2 authorization purposes.
-
 ### OAuth2 Application Registration
 
 The multi-user installation setup requires dedicated authorization
@@ -46,6 +58,8 @@ using OAuth2 protocol.
   The details of this authorization setup are in
   [server docs](../../docs/admin/servers/auth.md).
 
+It is possible to use <https://gitlab.com> or a local installation
+of Gitlab can be used for this purpose.
 Based on your selection of gitlab instance, it is necessary
 to register these two OAuth2 applications and link them
 to your intended DTaaS installation.
@@ -87,15 +101,15 @@ Edit all the fields according to your specific case.
 
   | URL Path | Example Value | Explanation |
   |:------------|:---------------|:---------------|
-  | DTAAS_DIR | '/home/dtaas/Desktop/DTaaS' | Full path to the DTaaS directory. This is an absolute path with no trailing slash. |
+  | DTAAS_DIR | '/Users/username/DTaaS' | Full path to the DTaaS directory. This is an absolute path with no trailing slash. |
   | SERVER_DNS | <http>_foo.com_</http> | The server DNS, if you are deploying with a dedicated server. Remember not use  <http:>http(s)</http:> at the beginning of the DNS string |
   | OAUTH_URL | <http>_gitlab.foo.com_<http/> | The URL of your Gitlab instance. It can be <http>_gitlab.com_<http/> if you are planning to use it for authorization. |
   | CLIENT_ID | 'xx' | The ID of your server OAuth application |
   | CLIENT_SECRET | 'xx' | The Secret of your server OAuth application |
   | OAUTH_SECRET | 'random-secret-string' | Any private random string. This is a password you choose for local installation. |
   | username1 | 'user1' | The gitlab instance username of a user of DTaaS |
   | username2 | 'user2' | The gitlab instance username of a user of DTaaS |
-  | CLIENT_CONFIG | '/home/dtaas/Desktop/DTaaS/deploy/config/client/env.js' | Full path to env.js file for client |
+  | CLIENT_CONFIG | '/Users/username/DTaaS/deploy/config/client/env.js' | Full path to env.js file for client |
 
 :clipboard: Important points to note:
 
@@ -182,8 +196,68 @@ If there are extra routes in `deploy/docker/conf.server` file but these are not
 in `deploy/docker/.env.server` file,
 such routes are not served by traefik; it will give **404 server response**.
 
+## Access Rights Over Files
+
+<!-- markdownlint-disable MD046 -->
+<!-- prettier-ignore -->
+!!! warning
+    The default setting in docker compose file exposes
+    all user files at <http://foo.com/lib/files>.
+    All files of all the users are readable-writable by
+    all logged in users.
+    The `compose.server.yml` / `compose.server.secure.yml` file needs to be
+    updated to expose another directory like common assets directory.
+<!-- markdownlint-enable MD046 -->
+
+If you wish to reduce this scope to only **common assets**,
+please change,
+
+```yaml
+  libms:
+    image: intocps/libms:latest
+    restart: unless-stopped
+    volumes:
+      - ${DTAAS_DIR}/deploy/config/lib.docker:/dtaas/libms/.env
+      - ${DTAAS_DIR}/files/common:/dtaas/libms/files
+```
+
+The change in the last line. The `${DTAAS_DIR}/files`
+got replaced by `${DTAAS_DIR}/files/common`. With this change, only
+common files are readable-writable by all logged in users.
+
+### Add TLS Certificates
+
+The application can be served on HTTPS connection for which TLS certificates
+are needed. The certificates need to be issued for `foo.com` or `*.foo.com`.
+The names of the certificates must be `fullchain.pem` and `privkey.pem`. Copy
+these two certificate files into:
+
+- `certs/foo.com/fullchain.pem`
+- `certs/foo.com/privkey.pem`
+
+Traefik will run with self-issued certificates if the above two certificates
+are either not found or found invalid.
+
+Remember to update `dynamic/tls.yml` with correct path matching your DNS name.
+For example, if your DNS name is `www.foo.com`, then copy the
+TLS certificates of `www.foo.com` to `certs/` directory and update
+`dynamic/tls.yml` as follows.
+
+```yml
+tls:
+  certificates:
+    - certFile: /etc/traefik-certs/www.foo.com/fullchain.pem
+      keyFile: /etc/traefik-certs/www.foo.com/privkey.pem
+      stores:
+        - default
+```
+
 ## Run
 
+### Over HTTP
+
+This docker compose file serves application over HTTP.
+
 The commands to start and stop the appliation are:
 
 ```bash
@@ -197,10 +271,27 @@ To restart only a specific container, for example `client``
 docker compose -f compose.server.yml --env-file .env.server up -d --force-recreate client
 ```
 
+### Over HTTPS
+
+This docker compose file serves application over HTTP.
+
+The commands to start and stop the appliation are:
+
+```bash
+docker compose -f compose.server.secure.yml --env-file .env.server up -d
+docker compose -f compose.server.secure.yml --env-file .env.server down
+```
+
+To restart only a specific container, for example `client``
+
+```bash
+docker compose -f compose.server.secure.yml --env-file .env.server up -d --force-recreate client
+```
+
 ## Use
 
 The application will be accessible at:
-<http://foo.com> from web browser.
+<http(s)://foo.com> from web browser.
 Sign in using your account linked to
 either _gitlab.com_ or your local gitlab instance.
 
@@ -210,7 +301,7 @@ through the single page client now.
 You may have to click Sign in to Gitlab on the Client page
 and authorize access to the shown application.
 
-## Adding a new user
+### Adding a new user
 
 To add a new user to your DTaaS instance, follow these steps: