Skip to content

Long term Themes

Jump to bottom
Prasad Talasila edited this page Sep 30, 2024 · 36 revisions

Target Features by 1-Nov-2024

  1. Gitlab DevOps mode - execute, manage and create modes
  2. Integrate gitlab runner into installation setup
  3. Runner
    1. Only permitted commands
    2. Make runner package cross-platform (see issue #637)[https://github.com/INTO-CPS-Association/DTaaS/issues/637]
    3. JWT-based authentication
    4. Integrate nanomq
    5. Async execution and reporting
    6. Clean up all the child processes
  4. Integrate Gitlab and Github modes for library microservice. See discussion #951.
  5. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.
  6. OAuth protection for all components in complete DTaaS installation. (done)
  7. docker compose for both developers and users. (done)
  8. Ten examples of different application domains. (done)

Target Features by 15-Jan-2025

  1. Config generation and validation from dtaas pip package.
  2. Fully functional runner client package (use MQTT).
  3. Feature complete CLI for admin tasks. Tasks to be taken care of are: install, uninstall, backup and update. Python frameworks to be used for the CLI.
  4. Kasm Ubuntu-based docker container with five services: terminal, file server (cloudcmd), VSCode server, Jupyter Lab, and VNC. These are to be used as user workspaces.
  5. Complete code coverage for tests. Feature tests at the top-level of the application.
  6. Feature complete DT Lifecycle manager and DT Configuration Validator microservices.
  7. Gitlab compliant GraphQL API for library microservice
  8. Kubernetes based execution manager. The execution manager should be able to manage DTs running inside docker containers.
  9. Keep support for relative paths in DTs. This means that all of user assets need to be cloned when a DT is executed in DevOps pipeline.

ML Workspace

  • Rebuild the docker image using the latest forks so that
    • the library are updated
    • use the slim version of ML workspace images to see if the problems are less
    • to only include Terminal, VS Code, Jupyter Lab, Jupyter Notebook (with required libraries, kernels reinstalled)
  • Problems seen so far (only with ml-workspace but not with ml-workspace-minimal):
    • the docker container loses network connectivity after three days (this is not the case on vagrant machines)
    • the terminal tool doesn't seem to have the ability to refresh itself. If there is an issue, the only solution is to close and reopen the terminal from "open tools" drop down of notebook
    • terminal app does not show at all after some time: terminal always comes if it is open from drop-down menu of Jupyter Notebook, but not as a direct link.
  • ml-workspace-minimal only slows down after ten days or so.
    • The completed processes within docker containers become zombies (defunct state). This is because of the absence of init service within docker containers which usually cleans up the zombie process.
  • Tools that must run in the new container are: Prosys OPC-UA server, GNU Octave 8.4
  • Add Jupyter debuggers and most useful VSCode server extensions in the docker image
  • Build docker stacks for different workspace container images needed by the users

Gitlab

  • The logout on gitlab does not effect the logged in status on the react app. Gitlab does not have the capability to make a webhook call to oauth end points for logging them out.
  • Two concurrent sessions at different instances of DTaaS do not work properly with OAuth. The OAuth for the instance launched next takes the OAuth configuration from the first application.

Security

  • Secure folders for all user folders. Generate security keys and use them to encrypt user folders on the server. Only users with their keys can see the files but no one else can.
  • Provide SSH access to user workspaces. Users can complete the development on their desktops and then push assets to user workspace (use TCP proxy servers - HA Proxy or Traefik)
  • Use port access feature of ml-workspace to provide access to different DT Runners running in user workspace.
  • Create skeleton OAuth server (using authentik) for setting up authentication server to be used for 1) tests and 2) simpler local installation scenarios

Documentation

  • The markdown to html is good, but not markdown to html to pdf. The mathematical expressions are missing from the pdf documentation. The notices/warnings/tips etc. are not rendered correctly as well.

Installation

  • Complete application released as
    • docker compose
    • Kubernetes Helm Chart
  • All installation managed through the python package

Other priorities

  • Integrate Thingsboard and thingsboard-gateway into the DTaaS. Prefer these two over Eclipse Ditto / Hono.
Clone this wiki locally