High memory usage from monitoring_thread_loop #153
Comments
ckreibich
added a commit
to zeek/zeek-testing-cluster
that referenced
this issue
Jul 2, 2024
This avoids problems with libkqueue allocating many gigabytes of data based on the maximum number of open files, which can default to > 1B in containerd. mheily/libkqueue#153 containerd/containerd#7566
|
This issue caused Zeek, the above-mentioned user of libkqueue, to grow from <300M of memory used to 4.2GB of resident ram used (16GB virtual) when run on a system that has this large number of file descriptors (which appears to be RHEL 9/centos stream 9/alma 9/ and rocky 9. Is it possible to adjust the size of these arrays to avoid having zeek allocate this much ram, triggering the oom-killer? |
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 20, 2024
This triggers a reporter warning when the limit gets reduced. This will happen by default in Docker environments until libkqueue gets fixed, or containerd changes its default-uncapped behavior. mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 20, 2024
This triggers a reporter warning when the limit gets reduced. This will happen by default in Docker environments until libkqueue gets fixed, or containerd changes its default-uncapped behavior. mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 21, 2024
This triggers a reporter warning when Zeek adjusts the limit, in the absence of any ZEEK_NOFILE_MAX configuration. (With ZEEK_NOFILE_MAX, it's clear the user is aware of the tuning mechanism and the warning seems unnecessary.) This adjustment will happen by default in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior, and prevents libkqueue from needlessly allocating gigabytes of memory. mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 21, 2024
This triggers a reporter warning when Zeek adjusts the limit, in the absence of any ZEEK_NOFILE_MAX configuration. (With ZEEK_NOFILE_MAX, it's clear the user is aware of the tuning mechanism and the warning seems unnecessary.) This adjustment will happen by default in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior, and prevents libkqueue from needlessly allocating gigabytes of memory. mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 21, 2024
This triggers a reporter warning when Zeek adjusts the limit, in the absence of any ZEEK_NOFILE_MAX configuration. (With ZEEK_NOFILE_MAX, it's clear the user is aware of the tuning mechanism and the warning seems unnecessary.) This adjustment will happen by default in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior, and prevents libkqueue from needlessly allocating gigabytes of memory. mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 22, 2024
…ptors The main purpose of this function is to disarm excessive memory use on systems with very large limits for open file descriptors, where libkqueue currently wastes gigabytes of memory by allocating state for every possible file descriptor. This particularly triggers in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior. The function caps the maximum to a default of 1M open fds. The user can override or disable the behavior via the ZEEK_NOFILE_MAX environment variable. References: mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 23, 2024
…ptors The main purpose of this function is to disarm excessive memory use on systems with very large limits for open file descriptors, where libkqueue currently wastes gigabytes of memory by allocating state for every possible file descriptor. This particularly triggers in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior. The function caps the maximum to a default of 1M open fds. The user can override or disable the behavior via the ZEEK_NOFILE_MAX environment variable. References: mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 23, 2024
…ptors The main purpose of this function is to disarm excessive memory use on systems with very large limits for open file descriptors, where libkqueue currently wastes gigabytes of memory by allocating state for every possible file descriptor. This particularly triggers in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior. The function caps the maximum to a default of 1M open fds. The user can override or disable the behavior via the ZEEK_NOFILE_MAX environment variable. References: mheily/libkqueue#153 moby/moby#38814
ckreibich
added a commit
to ckreibich/zeek
that referenced
this issue
Nov 23, 2024
…ptors The main purpose of this function is to disarm excessive memory use on systems with very large limits for open file descriptors, where libkqueue currently wastes gigabytes of memory by allocating state for every possible file descriptor. This particularly triggers in Docker environments until libkqueue gets fixed or containerd changes its default-uncapped fd limit behavior. The function caps the maximum to a default of 1M open fds. The user can override or disable the behavior via the ZEEK_NOFILE_MAX environment variable. References: mheily/libkqueue#153 moby/moby#38814
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We discovered a problem recently where the monitoring thread spawned by libkqueue was using a very large amount of memory. After a bit of digging, we realized that it's allocating two arrays of
ints that are the size of the maximum number of file descriptors on the system. The system we discovered this on is a Rocky 9 system, with a docker container running in containerd. In this environment, containerd sets the maximum number of descriptors to 1073741816. libkqueue then allocates about 8GB of memory.While it is possible to tune the container down to use fewer descriptors, this seems like an upstream problem in libkqueue itself.
The text was updated successfully, but these errors were encountered: