chore(docker): Set container memory limits in local stack (DEV-2718)

[SamuelBoerlin]

Pull Request Checklist

Task Description/Number

Issue Number: DEV-

Basic Requirements

Please check if your PR fulfills the following requirements:

• Tests for the changes have been added (for bug fixes / features)
• Docs have been added / updated (for bug fixes / features)

PR Type

What kind of change does this PR introduce?

• fix: represents bug fixes
• refactor: represents production code refactoring
• feat: represents a new feature
• docs: documentation changes (no production code change)
• chore: maintenance tasks (no production code change)
• test: all about tests: adding, refactoring tests (no production code change)
• other... Please describe:

Does this PR introduce a breaking change?

• Yes
• No
• Maybe (not 100% sure => check with FE)

Does this PR change client-test-data?

• Yes (don't forget to update the JS-LIB team about the change)
• No

[linear]

DEV-2718 triplestore killed when creating project or uploading data with DSP-TOOLS

Since some days, several RDU members have this problem, on several machines, always on localhost:

When creating a big project from a JSON file, or when uploading a big XML file, the triplestore suddenly becomes unresponsive.

Sometimes it happens already during project creation, sometimes during the xmlupload.

See the attached logs of the respective docker containers: The triplestore says it was killed:

2023-10-04 13:16:54 /docker-entrypoint.sh: line 78:     9 Killed                  exec "$"

(but the docker "db" container is still running)

Steps to reproduce:

• make a fresh clone of the DSP-API repository
• check out main branch on 14f19112 chore: Update Fuseki DEV-2743 (#2869)
• make init-db-test-minimal & make stack-up
• unzip the attached "BIZGregor_test.zip"
• cd BIZGregor_test
• create a new virtual environment: python3 -m venv .venv
• activate the venv:source .venv/bin/activate
• install the latest version of DSP-TOOLS (5.0.1): pip3 install dsp-tools
• dsp-tools create 230905_DM_BIZ.json
• dsp-tools xmlupload biz_complete_noDupl1_mock.xml

BIZGregor_test.zip

diagnosis_info_Johannes_MacBook_2023-10-04 13-16-54.zip

diagnosis_info_MacPro_2023-10-04 14-57-10.zip

[codecov]

Codecov Report

All modified lines are covered by tests ✅

Comparison is base (12402f3) 18.00% compared to head (b4f63ea) 88.12%.
Report is 90 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2874       +/-   ##
===========================================
+ Coverage   18.00%   88.12%   +70.12%     
===========================================
  Files         281      254       -27     
  Lines       28899    23281     -5618     
===========================================
+ Hits         5202    20517    +15315     
+ Misses      23697     2764    -20933     

see 241 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[SamuelBoerlin]

We could probably lower the limits, but for now I've just picked values such that the entire stack should use at most around 20GB memory.

[seakayone]

@SamuelBoerlin thanks for this approach.

I think it really makes sense to limit the resources of each container so that we get a better feedback if one of the services is running out of memory. The chosen defaults make sense in the beginning. We can optimise them in the long run.

Maybe it makes sense to mention the total memory needed in our documentation about the local dev env setup docs/05-internals/development/building-and-running.md, wdyt?

[SamuelBoerlin]

@SamuelBoerlin thanks for this approach.

I think it really makes sense to limit the resources of each container so that we get a better feedback if one of the services is running out of memory. The chosen defaults make sense in the beginning. We can optimise them in the long run.

Maybe it makes sense to mention the total memory needed in our documentation about the local dev env setup docs/05-internals/development/building-and-running.md, wdyt?

Sure, makes sense. I'll add it.

[SamuelBoerlin]

Added documentation