Swap ml-workspace container and add admin cookbook

deploy/install.sh

@@ -19,7 +19,7 @@ bash script/base.sh || exit
 printf "\n \n Download the required docker images...\n "
 printf ".........\n "
 docker pull traefik:v2.10
-docker pull mltooling/ml-workspace:0.13.2
+docker pull mltooling/ml-workspace-minimal:0.13.2
 printf "\n\n docker images successfully downloaded...\n \n \n "
 
 
@@ -59,7 +59,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="user1" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2 || true
+  mltooling/ml-workspace-minimal:0.13.2 || true
 
 docker run -d \
  -p 8091:8080 \
@@ -70,7 +70,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="user2" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2 || true
+  mltooling/ml-workspace-minimal:0.13.2 || true
 
 #-------------
 printf "\n \n Start the traefik gateway server\n "

deploy/services/services.js

@@ -136,3 +136,26 @@ await $$`sudo chmod 664 /etc/mosquitto/conf.d/default.conf`;
 await $$`sudo chown root:mosquitto /etc/mosquitto/conf.d/default.conf`;
 await $$`sudo systemctl restart mosquitto`;
 await $$`sudo systemctl status mosquitto`;
+
+//---------------
+log(chalk.blue("Start MongoDB server"));
+const mongodbConfig = config.services.mongodb;
+
+try {
+  log(
+    chalk.green(
+      "Attempt to delete any existing MongoDB server docker container"
+    )
+  );
+  await $$`docker stop mongodb`;
+  await $$`docker rm mongodb`;
+} catch (e) {}
+
+log(chalk.green("Start new Mongodb server docker container"));
+await $$`docker run -d -p ${mongodbConfig.port}:27017 \
+--name mongodb \
+-v ${mongodbConfig.datapath}:/data/db \
+-e MONGO_INITDB_ROOT_USERNAME=${mongodbConfig.username} \
+-e MONGO_INITDB_ROOT_PASSWORD=${mongodbConfig.password} \
+mongo:7.0.3`;
+log(chalk.green("MongoDB server docker container started successfully"));

deploy/services/services.yml

@@ -20,3 +20,8 @@ services:
     mqtt:  # usernames and passwords are in deploy/config/services/mqtt/config/password
         username: "dtaas"
         password: "dtaas"
+    mongodb:
+        username: "dtaas"
+        password: "dtaas"
+        port: 27017
+        datapath: "/home/prasad/git/prasadtalasila/DTaaS/deploy/vagrant/two-machine/mongodb" #no spaces in the path

deploy/single-script-install.sh

@@ -97,7 +97,7 @@ printf "\n\n End of installing dependencies...\n\n\n "
 printf "Download the required docker images...\n "
 printf ".........\n\n\n "
 docker pull traefik:v2.10
-docker pull mltooling/ml-workspace:0.13.2
+docker pull mltooling/ml-workspace-minimal:0.13.2
 printf "\n\n docker images successfully downloaded...\n \n \n "
 #----
 
@@ -165,7 +165,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="user1" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2 || true
+  mltooling/ml-workspace-minimal:0.13.2 || true
 
 docker run -d \
  -p 8091:8080 \
@@ -176,7 +176,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="user2" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2 || true
+  mltooling/ml-workspace-minimal:0.13.2 || true
 
 #-------------
 printf "\n\n Start the traefik gateway server\n "

deploy/vagrant/make_boxes/dtaas/README.md

@@ -38,12 +38,13 @@ This vagrant box installed for users will have the following items:
 * yarn v1.22
 * npm v10.2
 * containers
-  * ml-workspace v0.13
+  * ml-workspace-minimal v0.13
   * traefik v2.10
   * gitlab-ce v16.4
   * influxdb v2.7
   * grafana v10.1
   * rabbitmq v3-management
+  * mongodb v7.0
   * eclipse-mosquitto (mqtt) v2
 
 This vagrant box installed for developers will have

deploy/vagrant/make_boxes/dtaas/user.sh

@@ -72,9 +72,10 @@ cat /vagrant/vagrant.pub >> /root/.ssh/authorized_keys
 
 # get the required docker images
 docker pull traefik:v2.10
-docker pull mltooling/ml-workspace:0.13.2
+docker pull mltooling/ml-workspace-minimal:0.13.2
 docker pull grafana/grafana:10.1.4
 docker pull influxdb:2.7
 docker pull rabbitmq:3-management
 docker pull eclipse-mosquitto:2
+docker pull mongo:7.0.3
 docker pull gitlab/gitlab-ce:16.4.1-ce.0

docs/admin/client/CLIENT.md

@@ -125,7 +125,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="alice" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2
+  mltooling/ml-workspace-minimal:0.13.2
 
 docker run -d \
  -p 8091:8080 \
@@ -136,7 +136,7 @@ docker run -d \
   --env WORKSPACE_BASE_URL="bob" \
   --shm-size 512m \
   --restart always \
-  mltooling/ml-workspace:0.13.2
+  mltooling/ml-workspace-minimal:0.13.2
 ```
 
 Given that multiple services are running at different routes,

docs/admin/client/auth.md

@@ -51,6 +51,13 @@ Here are the steps to get started:
 |Callback URL|REACT_APP_REDIRECT_URI|[https://foo.com/Library](https://foo.com/Library)|
 |Scopes|REACT_APP_GITLAB_SCOPES|openid, profile, read_user, read_repository, api|
 
+**7. Create User Accounts:**
+
+Create user accounts in gitlab for all the usernames chosen during
+installation. The _trial_ installation script comes with two default
+usernames - _user1_ and _user2_. For all other installation scenarios,
+accounts with specific usernames need to be created on gitlab.
+
 ## Development Environment
 
 There needs to be a valid callback and logout URLs for development and testing
@@ -102,4 +109,4 @@ DTaaS application URL: https://foo.com/bar
 Gitlab instance URL: https://foo.gitlab.com
 Callback URL: https://foo.com/bar/Library
 Logout URL: https://foo.com/bar
-```
+```

docs/admin/guides/add_service.md

@@ -0,0 +1,112 @@
+# Add other services
+
+<!-- prettier-ignore -->
+!!! Pre-requisite
+    You should read the documentation about
+    the already available [services](../services.md)
+
+This guide will show you how to add more services.
+In the following example we will be adding **MongoDB** as a service,
+but these steps could be modified to install other services as well.
+
+:fontawesome-solid-circle-info:
+**Adding other services requires more RAM and CPU power.**
+**Please make sure the host machine meets the hardware requirements**
+**for running all the services.**
+
+**1. Add the configuration:**
+
+Select configuration parameters for the MongoDB service.
+
+| Configuration Variable Name | Description                                                       |
+| :-------------------------- | :---------------------------------------------------------------- |
+| username                    | the username of the root user in the MongoDB                      |
+| password                    | the password of the root user in the MongoDB                      |
+| port                        | the mapped port on the host machine (default is 27017)            |
+| datapath                    | path on host machine to mount the data from the MongoDB container |
+
+Open the file `/deploy/services/services.yml` and add the configuration for MongoDB:
+
+```yml
+services:
+    rabbitmq:
+        username: "dtaas"
+        password: "dtaas"
+        vhost: "/"
+        ports:
+            main: 5672
+            management: 15672
+    ...
+    mongodb:
+        username: <username>
+        password: <password>
+        port: <port>
+        datapath: <datapath>
+    ...
+```
+
+**2. Add the script:**
+
+The next step is to add the script that sets up the MongoDB container with the configuraiton.
+
+Create new file named `/deploy/services/mongodb.js` and add the following code:
+
+```js
+#!/usr/bin/node
+/* Install the optional platform services for DTaaS */
+import { $ } from "execa";
+import chalk from "chalk";
+import fs from "fs";
+import yaml from "js-yaml";
+
+const $$ = $({ stdio: "inherit" });
+const log = console.log;
+let config;
+
+try {
+  log(chalk.blue("Load services configuration"));
+  config = await yaml.load(fs.readFileSync("services.yml", "utf8"));
+  log(
+    chalk.green(
+      "configuration loading is successful and config is a valid yaml file"
+    )
+  );
+} catch (e) {
+  log(chalk.red("configuration is invalid. Please rectify services.yml file"));
+  process.exit(1);
+}
+
+log(chalk.blue("Start MongoDB server"));
+const mongodbConfig = config.services.mongodb;
+
+try {
+  log(
+    chalk.green(
+      "Attempt to delete any existing MongoDB server docker container"
+    )
+  );
+  await $$`docker stop mongodb`;
+  await $$`docker rm mongodb`;
+} catch (e) {}
+
+log(chalk.green("Start new Mongodb server docker container"));
+await $$`docker run -d -p ${mongodbConfig.port}:27017 \
+--name mongodb \
+-v ${mongodbConfig.datapath}:/data/db \
+-e MONGO_INITDB_ROOT_USERNAME=${mongodbConfig.username} \
+-e MONGO_INITDB_ROOT_PASSWORD=${mongodbConfig.password} \
+mongo:7.0.3`;
+log(chalk.green("MongoDB server docker container started successfully"));
+```
+
+**3. Run the script:**
+
+Go to the directory `/deploy/services/`
+and run services script with the following commands:
+
+```bash
+yarn install
+node mongodb.js
+```
+
+The MongoDB should now be available on **services.foo.com:<port\>**.

docs/admin/guides/add_user.md

@@ -0,0 +1,81 @@
+# Add a new user
+
+This page will guide you on, how to add more users to the DTaas. Please do the following:
+
+<!-- prettier-ignore -->
+!!! important
+    Make sure to replace **<username\>** and **<port\>**
+    Select a port that is not already being used by the system.
+
+__1. Add user:__
+
+Add the new user on the Gitlab instance.
+
+__2. Setup a new workspace:__
+
+The above code creates a new workspace for the new user based on _user2_.
+
+```bash
+cd DTaaS/files
+cp -R user2 <username>
+cd ..
+docker run -d \
+ -p <port>:8080 \
+  --name "ml-workspace-<username>" \
+  -v "${TOP_DIR}/files/<username>:/workspace" \
+  -v "${TOP_DIR}/files/<username>:/workspace/common" \
+  --env AUTHENTICATE_VIA_JUPYTER="" \
+  --env WORKSPACE_BASE_URL="<username>" \
+  --shm-size 512m \
+  --restart always \
+  mltooling/ml-workspace-minimal:0.13.2
+```
+
+__3. Add username and password:__
+
+The following code adds basic authentication for the new user.
+
+```bash
+cd DTaaS/servers/config/gateway
+htpasswd auth <username>
+```
+
+__4. Add 'route' for new user:__
+
+We need to add a new route to the servers ingress.
+
+Open the following file with your preffered editor (e.g. VIM/nano).
+
+```bash
+vi DTaaS/servers/config/gateway/dynamic/fileConfig.yml
+```
+
+Now add the new route and service for the user.
+
+<!-- prettier-ignore -->
+!!! important
+  foo.com should be replaced with your own domain.
+
+```yml
+http:
+  routers:
+    ....
+    <username>:
+      entryPoints:
+        - http
+      rule: 'Host(`foo.com`) && PathPrefix(`/<username>`)'
+      middlewares:
+        - basic-auth
+      service: <username>
+
+  services:
+    ...
+    <username>:
+      loadBalancer:
+        servers:
+          - url: 'http://localhost:<port>'
+```
+
+__5. Access the new user:__
+
+Log into the DTaaS application as new user.

docs/admin/guides/common_workspace_readonly.md

@@ -0,0 +1,40 @@
+# Make common asset area read only
+
+## Why
+
+In some cases you might want to restrict the access rights of some users
+to the common assets.
+In order to make the common area read only,
+you have to change the install script section performing the creation
+of user workspaces.
+
+## How
+
+To make the common assets read-only for user2,
+the following changes need to be made to the install script,
+which is located one of the following places.
+
+- trial installation: `single-script-install.sh`
+
+- production installation: `DTaas/deploy/install.sh`
+
+The line `-v "${TOP_DIR}/files/common:/workspace/common:ro"`
+was added to make the common workspace read-only for user2.
+
+Here's the updated code:
+
+```sh
+docker run -d \
+  -p 8091:8080 \
+  --name "ml-workspace-user2" \
+  -v "${TOP_DIR}/files/user2:/workspace" \
+  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  --env AUTHENTICATE_VIA_JUPYTER="" \
+  --env WORKSPACE_BASE_URL="user2" \
+  --shm-size 512m \
+  --restart always \
+  mltooling/ml-workspace-minimal:0.13.2 || true
+```
+
+This ensures that the common area is read-only for user2,
+while the user's own (private) assets are still writable.

docs/admin/guides/hosting_site_without_https.md

@@ -0,0 +1,6 @@
+# Hosting site without https
+
+In the default trial or production installation setup,
+the https connection is provided by the reverse proxy.
+The DTaaS application by default runs in http mode.
+So removing the reverse proxy removes the https mode.