Add configuration setting for number of octopoes workers

[dekkers]

Changes

This adds a setting to octopoes to configure the number of workers. While it is possible to do this directly in celery with CELERY_WORKER_CONCURRENCY, this value does not have a proper default and it not in our documentation. With this change we add a default and it will also automatically be included in the documentation.

This also removes CELERY_WORKER_CONCURRENCY from .env-dist because the default is 4 now and so we don't need this anymore.

QA notes

Check if setting OCTOPOES_WORKERS changes the number of octopoes celery processes that are started.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[underdarknl]

This needs docs changes, and possibly a backward compatible solution for those who already set the celery env-var

[ammar92]

Although this works, I think documenting or referencing the CELERY_WORKER_CONCURRENCY variable would've been easier. Because now we have an extra setting, possibly backwards compatibility issues and settings inconsistency (e.g. you can set other Celery settings, but for CELERY_WORKER_CONCURRENCY you'd now have to use OCTOPOES_WORKERS, which may be confusing for users who have Celery experience).

The last two can probably be solved using field aliases, something like:

workers: int = Field(4, description="Number of Octopoes Celery workers", alias=["OCTOPOES_WORKERS", "CELERY_WORKER_CONCURRENCY"])

(not tested, and note the extra OCTOPOES_ prefix, because env_prefix does not apply to fields with aliases)

I don't have a strong preference, but just giving another perspective

[ammar92]

Extra QA notes: you can verify the setting by checking the octopoes_api_worker logs:

[stephanie0x00]

Checklist for QA:

• I have checked out this branch, and successfully ran a fresh make reset.
• I confirmed that there are no unintended functional regressions in this branch:
  • I have managed to pass the onboarding flow
  • Objects and Findings are created properly
  • Tasks are created and completed properly
• I confirmed that the PR's advertised feature or hotfix works as intended.
• I checked the logs for errors and/or warnings and made issues where necessary

What works:

Works, number of celery workers went from 4 to 6. Onboarding flow still works as expected.

What doesn't work:

n/a

Bug or feature?:

n/a

[underdarknl]

This still needs documentation.

[dekkers]

This still needs documentation.

The documentation of settings is auto generated, so should be online as soon we merge this. We could mention in the release notes when we write those.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud