Updates install scripts and docs (#150)

  - Updates vagrant install scripts. Separates developer
     and user installation scripts
  - Updates the developer environment creation
     script in script/env.sh
  - Corrects docs formatting issues and 
    OAuth documentation

.gitignore

@@ -148,4 +148,7 @@ site/
 !.vscode/settings.json
 !.vscode/tasks.json
 !.vscode/launch.json
-!.vscode/extensions.json
+!.vscode/extensions.json
+
+# yarn cache
+.yarn

README.md

@@ -27,9 +27,10 @@ if there is a suggestion to improve the software.
 
 This is a mono repo containing code for
 both the web client and the microservices code base.
-Only the [web client](client) and
-[library microservice](servers/lib)
-components are functional at present.
+The [web client](client),
+[library](servers/lib) and
+[runner](servers/execution/runner)
+microservices are functional at present.
 Everything else is a work-in-progress.
 
 Please see the
@@ -46,7 +47,8 @@ The DTaaS software platform uses
 [Træfik](https://github.com/traefik/traefik),
 [ML Workspace](https://github.com/ml-tooling/ml-workspace),
 [Grafana](https://github.com/grafana/grafana),
-[InfluxDB](https://github.com/influxdata/influxdb) and
+[InfluxDB](https://github.com/influxdata/influxdb),
+[MQTT](https://github.com/eclipse/mosquitto) and
 [RabbitMQ](https://github.com/rabbitmq/rabbitmq-server)
 open-source components.
 These software components have their own licenses.

client/script/clean.bash

@@ -1,2 +1,2 @@
 #!/bin/bash
-rm -rf build/ node_modules/ coverage/ playwright-report/  *.svg
+rm -rf build/ node_modules/ coverage/ playwright-report/  "*.svg"

deploy/README.md

@@ -1,7 +1,7 @@
 # DTaaS on Linux Operating System
 
 This directory contains code for running DTaaS application
-on a Ubuntu Server 20.04 Operating System.
+on a Ubuntu Server 22.04 Operating System.
 The setup requires a machine which can spare 16GB
 RAM, 8 vCPUs and 50GB Hard Disk space.
 
@@ -31,7 +31,8 @@ The installation guide assumes that you can run the application in HTTPS mode.
 The Traefik gateway configuration is at [fileConfig](../config/gateway/fileConfig.yml).
 Change `localhost` to `foo.com` and user1/user2 to the usernames chosen by you.
 
-**NOTE**: Do not use `http://` or `https://` in [fileConfig](../config/gateway/fileConfig.yml).
+**NOTE**: Do not use `http://` or `https://`
+in [fileConfig](../config/gateway/fileConfig.yml).
 
 #### Authentication
 
@@ -73,13 +74,38 @@ APOLLO_PATH='/lib'
 GRAPHQL_PLAYGROUND='true'
 ```
 
-## Configure react website
+## Configure React Client Website
+
+### Gitlab OAuth application
+
+The DTaaS react website requires Gitlab OAuth provider.
+If you need more help with this step, please see
+the [Authentication page](../docs/admin/client/auth.md).
+
+You need the following information from the OAuth application
+registered on Gitlab:
+
+| Gitlab Variable Name | Variable name in Client env.js | Default Value |
+|:---|:---|:---|
+| OAuth Provider | REACT_APP_AUTH_AUTHORITY | https://gitlab.foo.com/ |
+| Application ID | REACT_APP_CLIENT_ID |
+| Callback URL | REACT_APP_REDIRECT_URI | https://foo.com/Library |
+| Scopes | REACT_APP_GITLAB_SCOPES | openid, profile, read_user, read_repository, api |
+
+You can also see
+[Gitlab help page](https://docs.gitlab.com/ee/integration/oauth_provider.html)
+for getting the Gitlab OAuth application details.
+
+If you need more help with this step, please see
+the [Authentication page](client/auth.md).
+
+### Update Client Config
 
 Change the React website configuration in _deploy/config/client/env.js_.
 
 ```js
 window.env = {
-  REACT_APP_ENVIRONMENT: 'prod',
+  REACT_APP_ENVIRONMENT: 'dev',
   REACT_APP_URL: 'https://foo.com/',
   REACT_APP_URL_BASENAME: 'dtaas',
   REACT_APP_URL_DTLINK: '/lab',

deploy/config/client/env.js

@@ -1,7 +1,7 @@
 window.env = {
   REACT_APP_ENVIRONMENT: 'dev',
   REACT_APP_URL: 'https://foo.com/',
-  REACT_APP_URL_BASENAME: 'dtaas',
+  REACT_APP_URL_BASENAME: '',
   REACT_APP_URL_DTLINK: '/lab',
   REACT_APP_URL_LIBLINK: '',
   REACT_APP_WORKBENCHLINK_TERMINAL: '/terminals/main',

deploy/install.sh

@@ -14,11 +14,13 @@ sleep 60
 
 printf "\n \n Install the system dependencies...\n "
 printf "....\n "
-bash script/env.sh || exit
+bash deploy/make_boxes/dtaas/user.sh || exit
 
 printf "\n \n Download the required docker images...\n "
 printf ".........\n "
-source script/docker.sh || exit
+docker pull traefik:v2.10
+docker pull mltooling/ml-workspace:0.13.2
+printf "\n\n docker images successfully downloaded...\n \n \n "
 
 
 printf "\n \n Continue with the DTaaS installation...\n "
@@ -52,7 +54,7 @@ docker run -d \
  -p 8090:8080 \
   --name "ml-workspace-user1" \
   -v "${TOP_DIR}/files/user1:/workspace" \
-  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  -v "${TOP_DIR}/files/common:/workspace/common" \
   --env AUTHENTICATE_VIA_JUPYTER="" \
   --env WORKSPACE_BASE_URL="user1" \
   --shm-size 512m \
@@ -63,7 +65,7 @@ docker run -d \
  -p 8091:8080 \
   --name "ml-workspace-user2" \
   -v "${TOP_DIR}/files/user2:/workspace" \
-  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  -v "${TOP_DIR}/files/common:/workspace/common" \
   --env AUTHENTICATE_VIA_JUPYTER="" \
   --env WORKSPACE_BASE_URL="user2" \
   --shm-size 512m \
@@ -83,7 +85,7 @@ sudo docker run -d \
  -v "$PWD/auth:/etc/traefik/auth" \
  -v "$PWD/dynamic:/etc/traefik/dynamic" \
  -v /var/run/docker.sock:/var/run/docker.sock \
- traefik:v2.5 || true
+ traefik:v2.10 || true
 
 #----------
 printf "\n \n Create crontabs to run the application in daemon mode.\n "

deploy/single-script-install.sh

@@ -26,9 +26,11 @@ sudo mkdir -p /etc/apt/keyrings
 if [ ! -f /etc/apt/keyrings/docker.gpg ]
 then
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
-  sudo printf \
-  "deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
-  %s stable" "$(dpkg --print-architecture)" "$(lsb_release -cs)"  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+  printf \
+    "deb [arch=%s signed-by=/etc/apt/keyrings/docker.gpg] \
+    https://download.docker.com/linux/ubuntu %s stable" \
+    "$(dpkg --print-architecture)" "$(lsb_release -cs)"  | \
+    sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
 fi
 
 sudo apt-get update -y
@@ -58,14 +60,30 @@ sudo systemctl enable docker.service
 sudo systemctl enable containerd.service
 
 
-# Install nodejs environment
-curl -fsSL https://deb.nodesource.com/setup_18.x | sudo bash -
+sudo apt-get update
+sudo apt-get install -y ca-certificates curl gnupg
+sudo mkdir -p /etc/apt/keyrings
+if [ ! -f /etc/apt/keyrings/nodesource.gpg ]
+then
+  curl -fsSL "https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key" | \
+    sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
+fi
+NODE_MAJOR=18
+printf "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] \
+  https://deb.nodesource.com/node_%s.x nodistro main" "$NODE_MAJOR" | \
+  sudo tee /etc/apt/sources.list.d/nodesource.list
+sudo apt-get update
 sudo apt-get install -y nodejs
+sudo npm install -g [email protected]
+
 
 if [ ! -f /usr/share/keyrings/yarnkey.gpg ]
 then
-  curl -sL https://dl.yarnpkg.com/debian/pubkey.gpg | gpg --dearmor | sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null
-  printf "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] https://dl.yarnpkg.com/debian stable main" | sudo tee /etc/apt/sources.list.d/yarn.list
+  curl -sL "https://dl.yarnpkg.com/debian/pubkey.gpg" | gpg --dearmor | \
+    sudo tee /usr/share/keyrings/yarnkey.gpg >/dev/null
+  printf "deb [signed-by=/usr/share/keyrings/yarnkey.gpg] \
+    https://dl.yarnpkg.com/debian stable main \n" | \
+    sudo tee /etc/apt/sources.list.d/yarn.list
 fi
 
 sudo apt-get update -y
@@ -78,12 +96,8 @@ printf "\n\n End of installing dependencies...\n\n\n "
 # get the required docker images
 printf "Download the required docker images...\n "
 printf ".........\n\n\n "
-docker pull traefik:v2.5
-docker pull influxdb:2.4
+docker pull traefik:v2.10
 docker pull mltooling/ml-workspace:0.13.2
-docker pull grafana/grafana
-docker pull telegraf
-docker pull gitlab/gitlab-ce:15.10.0-ce.0
 printf "\n\n docker images successfully downloaded...\n \n \n "
 #----
 
@@ -128,7 +142,7 @@ yarn build
 {
   printf "PORT='4001'\n "
   printf "MODE='local'\n "
-  printf "LOCAL_PATH ='%s'\n " "$TOP_DIR"
+  printf "LOCAL_PATH ='%s/files'\n " "$TOP_DIR"
   printf "GITLAB_GROUP ='dtaas'\n "
   printf "GITLAB_URL='https://gitlab.com/api/graphql'\n "
   printf "TOKEN='123-sample-token'\n "
@@ -146,7 +160,7 @@ docker run -d \
  -p 8090:8080 \
   --name "ml-workspace-user1" \
   -v "${TOP_DIR}/files/user1:/workspace" \
-  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  -v "${TOP_DIR}/files/common:/workspace/common" \
   --env AUTHENTICATE_VIA_JUPYTER="" \
   --env WORKSPACE_BASE_URL="user1" \
   --shm-size 512m \
@@ -157,7 +171,7 @@ docker run -d \
  -p 8091:8080 \
   --name "ml-workspace-user2" \
   -v "${TOP_DIR}/files/user2:/workspace" \
-  -v "${TOP_DIR}/files/common:/workspace/common:ro" \
+  -v "${TOP_DIR}/files/common:/workspace/common" \
   --env AUTHENTICATE_VIA_JUPYTER="" \
   --env WORKSPACE_BASE_URL="user2" \
   --shm-size 512m \
@@ -177,7 +191,7 @@ docker run -d \
  -v "$PWD/auth:/etc/traefik/auth" \
  -v "$PWD/dynamic:/etc/traefik/dynamic" \
  -v /var/run/docker.sock:/var/run/docker.sock \
- traefik:v2.5 || true
+ traefik:v2.10 || true
 
 
 #----------
@@ -191,6 +205,7 @@ printf "\n\n The installation is complete.\n\n\n "
 
 printf "Continue with the application configuration.\n "
 printf ".........\n\n\n "
-printf "Remember to change foo.com to your local hostname in the following files.\n "
+printf "Remember to change foo.com and Gitlab OAuth details to your \
+local settings in the following files.\n "
 printf "1. %s/client/build/env.js\n " "$TOP_DIR"
 printf "2. %s/servers/config/gateway/dynamic/fileConfig.yml\n " "$TOP_DIR"

deploy/vagrant/make_boxes/dtaas/README.md

@@ -1,21 +1,63 @@
-# DTaaS Boxes
+# DTaaS Vagrant Box
 
-This is a box that has the following items:
+This README provides instructions on creating a custom Operating System
+virtual disk for running the DTaaS software. The virtual disk is managed
+by **vagrant**. The purpose is two fold:
 
-* docker
-* nodejs and yarn
-* jupyter
-* microk8s
+* Provide cross-platform installation of the DTaaS application.
+  Any operating system supporting use of vagrant software utility can
+  support installation of the DTaaS software.
+* Create a ready to use development environment for code contributors.
+
+There are two scripts in this directory:
+
+| Script name | Purpose | Default |
+|:---|:---|:---|
+| `user.sh` | user installation | :white_check_mark: |
+| `developer.sh` | developer installation | :x: |
+
+If you are installing the DTaaS for developers, the default installation
+caters to your needs. You can skip the next step and continue with the
+creation of vagrant box.
+
+If you are a developer and would like additional software installed, you need
+to modify `Vagrantfile`. The existing `Vagrantfile` has two lines:
+
+```sh
+    config.vm.provision "shell", path: "user.sh"
+    #config.vm.provision "shell", path: "developer.sh"
+```
+
+Uncomment the second line to have more software components installed. If you
+are not a developer, no changes are required to the `Vagrantfile`.
+
+This vagrant box installed for users will have the following items:
+
+* docker v24.0
+* nodejs v18.8
+* yarn v1.22
+* npm v10.2
 * containers
-  * mltooling/ml-workspace:0.13.2
-  * traefik2.5
-  * influxdb2.4
-  * grafana
-  * telegraf
-  * gitlab
+  * ml-workspace v0.13
+  * traefik v2.10
+  * gitlab-ce v16.4
+  * influxdb v2.7
+  * grafana v10.1
+  * rabbitmq v3-management
+  * eclipse-mosquitto (mqtt) v2
+
+This vagrant box installed for developers will have
+the following items additional items:
+
+* docker-compose v2.20
+* microk8s v1.27
+* jupyterlab
+* mkdocs
+* containers
+  * telegraf v1.28
 
-Publish a base virtualbox package to be used by
-vagrant to publish all other virtualbox packages
+The upcoming instructions will help with the creation of
+base vagrant box.
 
 ```bash
 #create a key pair
@@ -26,7 +68,6 @@ mv key.pub vagrant.pub
 vagrant up
 
 # let the provisioning be complete
-# replace the vagrant ssh key-pair with personal one
 vagrant ssh
 
 # install the oh-my-zsh
@@ -37,6 +78,7 @@ git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-m
 # inside ~/.zshrc, modify the following line
 plugins=(git zsh-autosuggestions history cp tmux)
 
+# to replace the vagrant ssh key-pair with personal one
 # remove the vagrant default public key - first line of
 # /home/vagrant/.ssh/authorized_keys
 
@@ -47,6 +89,7 @@ cp vagrant .vagrant/machines/default/virtualbox/private_key
 # check
 vagrant ssh #should work
 
+# exit vagrant guest machine and then
 vagrant halt
 
 vagrant package --base dtaas \

deploy/vagrant/make_boxes/dtaas/Vagrantfile

@@ -12,6 +12,7 @@ Vagrant.configure("2") do |config|
       vb.cpus = 8
     end
 
-    config.vm.provision "shell", path: "provision.sh"
+    config.vm.provision "shell", path: "user.sh"
+    #config.vm.provision "shell", path: "developer.sh"
 end