Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PeriodicDM as decorator for any DM with data stored in main Nu DB #7364

Open
wants to merge 13 commits into
base: staging
Choose a base branch
from

Conversation

mgoworko
Copy link
Contributor

@mgoworko mgoworko commented Dec 19, 2024

Describe your changes

This PR is a quite huge refactor of the periodic DM.

What is done:

  • The PeriodicDM is now extracted to a new module, completely separate from Flink. The PeriodicDM can be used as a decorator for any DM.
  • DB tables - datasource of PeriodicDM - are now moved to the main Nussknacker DB (previously were completely separated, with different schema and separate migration numbering, could be a completely different db). The old datasource definition is marked as legacy. Can still be used (so it is not necessary to migrate DB immediately), but should be switched to new mechanism.
  • In the deployment-manager-api there is a new interface PeriodicProcessesManager, which encapsulates all operations, that the Nussknacker provides to the DM. It is an abstraction. There are 2 implementations: the new, default one using Nussknacker DB, and the legacy, deprecated implementation using old tables in separate schema.

What was NOT done on purpose:

  • The general structure of PeriodicDM is changed, but there are little changes in the internals of the DM. It is intentional. The structural changes introduced in this PR, like moving repo to core and making PeriodicDM a quasi-decorator, will allow future changes. But it would be way too much in this PR.
  • The DB structure is not extensively refactored. There are some optimizations, like not storing full scenario JSON and allowing underlying DMs other than Flink, but overall tables are almost the same. They are now in main Nu db, so migrations can be introduced in future PRs

Checklist before merge

  • Related issue ID is placed at the beginning of PR title in [brackets] (can be GH issue or Nu Jira issue)
  • Code is cleaned from temporary changes and commented out lines
  • Parts of the code that are not easy to understand are documented in the code
  • Changes are covered by automated tests
  • Showcase in dev-application.conf added to demonstrate the feature
  • Documentation added or updated
  • Added entry in Changelog.md describing the change from the perspective of a public distribution user
  • Added MigrationGuide.md entry in the appropriate subcategory if introducing a breaking change
  • Verify that PR will be squashed during merge

@github-actions github-actions bot added the docs label Dec 30, 2024
@mgoworko mgoworko force-pushed the periodic-dm-as-decorator-for-any-dm branch from 1fb1348 to bbb6345 Compare December 31, 2024 14:35
@mgoworko mgoworko changed the title Draft: Periodic dm as decorator for any dm PeriodicDM as decorator for any DM with data stored in main Nu DB Dec 31, 2024
@mgoworko mgoworko marked this pull request as ready for review December 31, 2024 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant