Long term Themes

Target Features by 1-March-2025

1. Integrate gitlab runner into installation setup
2. Integrate git modes for library microservice. See discussion #951.
3. Kasm Ubuntu-based docker container with five services: terminal, file server (cloudcmd), VSCode server, Jupyter Lab, and VNC. These are to be used as user workspaces.
4. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.
5. Create examples of job and service DTs using DevOps features
   1. Potentially using runner + nanomq + piston + Apache airflow

Target Features by 15-June-2025

1. Config generation and validation from dtaas pip package.
2. Support git and seaweed modes in libms. Integrate with gitlab for closed loop DevOps execution.
3. Runner server package
   1. Integrate async execution, reporting and clean up of orphan processes 1 Integrate MQTT protocol
   2. Make runner package cross-platform see issue #637
   3. JWT-based authentication
   4. Integrate nanomq
   5. Clean up all the child processes
4. Fully functional runner client package (use MQTT).
5. Feature complete CLI for admin tasks. Tasks to be taken care of are: install, uninstall, backup and update. Python frameworks to be used for the CLI.
6. Complete code coverage for tests. Feature tests at the top-level of the application.
7. Feature complete DT Lifecycle manager and DT Configuration Validator microservices.
8. Gitlab compliant GraphQL API for library microservice
9. Kubernetes based execution manager. The execution manager should be able to manage DTs running inside docker containers.
10. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.

ML Workspace

• Rebuild the docker image using the latest forks so that
  • the library are updated
  • use the slim version of ML workspace images to see if the problems are less
  • to only include Terminal, VS Code, Jupyter Lab, Jupyter Notebook (with required libraries, kernels reinstalled)
• Problems seen so far (only with ml-workspace but not with ml-workspace-minimal):
  • the docker container loses network connectivity after three days (this is not the case on vagrant machines)
  • the terminal tool doesn't seem to have the ability to refresh itself. If there is an issue, the only solution is to close and reopen the terminal from "open tools" drop down of notebook
  • terminal app does not show at all after some time: terminal always comes if it is open from drop-down menu of Jupyter Notebook, but not as a direct link.
• ml-workspace-minimal only slows down after ten days or so.
  • The completed processes within docker containers become zombies (defunct state). This is because of the absence of init service within docker containers which usually cleans up the zombie process.
• Tools that must run in the new container are: Prosys OPC-UA server, GNU Octave 8.4
• Add Jupyter debuggers and most useful VSCode server extensions in the docker image
• Build docker stacks for different workspace container images needed by the users

Gitlab

• The logout on gitlab does not effect the logged in status on the react app. Gitlab does not have the capability to make a webhook call to oauth end points for logging them out.
• Two concurrent sessions at different instances of DTaaS do not work properly with OAuth. The OAuth for the instance launched next takes the OAuth configuration from the first application.

Security

• Secure folders for all user folders. Generate security keys and use them to encrypt user folders on the server. Only users with their keys can see the files but no one else can.
• Provide SSH access to user workspaces. Users can complete the development on their desktops and then push assets to user workspace (use TCP proxy servers - HA Proxy or Traefik)
• Use port access feature of ml-workspace to provide access to different DT Runners running in user workspace.
• Create skeleton OAuth server (using authentik) for setting up authentication server to be used for 1) tests and 2) simpler local installation scenarios

Documentation

• The markdown to html is good, but not markdown to html to pdf. The mathematical expressions are missing from the pdf documentation. The notices/warnings/tips etc. are not rendered correctly as well.

Installation

• Complete application released as Kubernetes Helm Chart
• All installation managed through the python package

Other priorities

• Integrate Thingsboard and thingsboard-gateway into the DTaaS. Prefer these two over Eclipse Ditto / Hono.