Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update scheduler folder structure #3883

Merged
merged 13 commits into from
Nov 27, 2024
Merged

Conversation

jpbruinsslot
Copy link
Contributor

@jpbruinsslot jpbruinsslot commented Nov 25, 2024

Changes

Refactoring of folders, cherry-picked from #3839 to make that PR a bit more easier to review in the future.

  • Move alembic/ migrations into storage/
  • Moved config/ into context/ resulted in build-docs problems, left it in config/
  • Renamed connectors/ to clients/
  • Renamed listeners/ to amqp/
  • Renamed services/ to http/
  • Moved queues/ into schedulers/
  • Moved rankers/ into schedulers/

From:

./
├── alembic/
├── config/
├── connectors/
│   ├── listeners/
│   └── services/
├── context/
├── models/
├── queues/
├── rankers/
├── schedulers/
├── server/
├── storage/
└── utils/

To:

./
├── clients/
│   ├── amqp/
│   └── http/
├── context/
├── models/
├── schedulers/
│   ├── queue/
│   ├── rankers/
│   └── schedulers/
├── server/
├── storage/
│   ├── migrations/
│   └── stores/
└── utils/

QA notes

Please add some information for QA on how to test the newly created code.


Code Checklist

  • All the commits in this PR are properly PGP-signed and verified.
  • This PR only contains functionality relevant to the issue.
  • I have written unit tests for the changes or fixes I made.
  • I have checked the documentation and made changes where necessary.
  • I have performed a self-review of my code and refactored it to the best of my abilities.
  • Tickets have been created for newly discovered issues.
  • For any non-trivial functionality, I have added integration and/or end-to-end tests.
  • I have informed others of any required .env changes files if required and changed the .env-dist accordingly.
  • I have included comments in the code to elaborate on what is not self-evident from the code itself, including references to issues and discussions online, or implicit behavior of an interface.

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.


Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

@jpbruinsslot jpbruinsslot added the mula Issues related to the scheduler label Nov 25, 2024
@jpbruinsslot jpbruinsslot self-assigned this Nov 25, 2024
@jpbruinsslot jpbruinsslot requested a review from a team as a code owner November 25, 2024 14:58
ammar92
ammar92 previously approved these changes Nov 26, 2024
Copy link
Contributor

@ammar92 ammar92 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Only the schedulers.schedulers package is a bit confusing, but not blocking for now

mula/scheduler/storage/connection.py Outdated Show resolved Hide resolved
mula/scheduler/clients/http/client.py Outdated Show resolved Hide resolved
@jpbruinsslot
Copy link
Contributor Author

Looks good to me. Only the schedulers.schedulers package is a bit confusing, but not blocking for now

Thanks for checking. Yeah I agree, I'm open to suggestions.

@stephanie0x00
Copy link
Contributor

Checklist for QA:

  • I have checked out this branch, and successfully ran a fresh make reset.
  • I confirmed that there are no unintended functional regressions in this branch:
    • I have managed to pass the onboarding flow
    • Objects and Findings are created properly
    • Tasks are created and completed properly
  • I confirmed that the PR's advertised feature or hotfix works as intended.
  • I checked the logs for errors and/or warnings and made issues where necessary

What works:

Onboarding works, added an extra hostname and enabled some additional boefjes. Scheduled a report which runs on the every minute interval as expected. Subreports can be opened and the main report as well. Aggregate report can be generated. Stopped the scheduler to see what would break. Reports can still be opened and on the Tasks page the expected Error messages are shown.

What doesn't work:

n/a

Bug or feature?:

n/a

* main:
  Increase max number of PostgreSQL connections (#3889)
  Fix for task id as valid UUID (#3744)
  Add `auto_calculate_deadline` attribute to Scheduler (#3869)
  Ignore specific url parameters when following location headers (#3856)
  Let mailserver inherit l1 (#3704)
  Change plugins enabling in report flow to checkboxes (#3747)
  Fix rocky katalogus tests and delete unused fixtures (#3884)
  Enable/disable scheduled reports (#3871)
Copy link

sonarcloud bot commented Nov 27, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
35.9% Coverage on New Code (required ≥ 80%)
5.0% Duplication on New Code (required ≤ 3%)

See analysis details on SonarQube Cloud

@underdarknl underdarknl merged commit 19a3a69 into main Nov 27, 2024
20 of 22 checks passed
@underdarknl underdarknl deleted the feature/mula/restructure-folders branch November 27, 2024 14:42
jpbruinsslot added a commit that referenced this pull request Dec 2, 2024
* main: (64 commits)
  Bug fix: KAT-alogus parameter is now organization member instead of organization code (#3895)
  Remove sigrid workflows (#3920)
  Updated packages (#3898)
  Fix mula migrations Debian package (#3919)
  Adds loggers to report flow (#3872)
  Add additional check if task already run for report scheduler (#3900)
  Create separate finding for Microsoft RDP port (#3882)
  fix: 🐛 allow boefje completion with 404 (#3893)
  Feature/improve rename bulk modal (#3885)
  Update scheduler folder structure (#3883)
  Translations update from Hosted Weblate (#3870)
  Increase max number of PostgreSQL connections (#3889)
  Fix for task id as valid UUID (#3744)
  Add `auto_calculate_deadline` attribute to Scheduler (#3869)
  Ignore specific url parameters when following location headers (#3856)
  Let mailserver inherit l1 (#3704)
  Change plugins enabling in report flow to checkboxes (#3747)
  Fix rocky katalogus tests and delete unused fixtures (#3884)
  Enable/disable scheduled reports (#3871)
  optimize locking in katalogus.py, reuse available data (#3752)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
mula Issues related to the scheduler
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

4 participants