Fix issue with growing number of allocated direct buffers #8
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Example showing the problematic scenario is here in master branch: https://github.com/[kochkozharov/hadroNIO-memory-leak](https://github.com/kochkozharov/hadroNIO-memory-leak). It's pretty much the same as in my previous PR.
docker compose -f docker-compose.nofix.yml up --build
You can see unlimited growing number of off-heap buffers, which can cause OOM error or hang.
There are some charts from visualvm (shows memory usage and amount of direct buffers in taskmanager):
Same situation appears in jobmanager.
In this fix I suggest to implicitly free objects like
new UnsafeBuffer(ByteBuffer.allocateDirect(Long.BYTES))
and replace direct buffer in callback with on-heap one.Running the cluster with the changes:
docker compose up --build
Visualvm results in taskmanager:
The number of direct buffers has been significantly reduced.