Free memory is wrong on Linux with gcroups v2

[gi0baro]

system: Linux container based on debian:bookworm-slim
sysinfo version: 0.30.6

The actual cgroup v2 readings deduct all the values in /sys/fs/cgroup/memory.stat from available memory in these lines https://github.com/GuillaumeGomez/sysinfo/blob/master/src/unix/linux/system.rs#L560-L562

This is causing wrong reporting on the available memory, as memory.stat includes values already counted in memory.current, plus values which are not inherent to available memory, eg:

root@testpod:/work# cat /sys/fs/cgroup/memory.max
7864320000
root@testpod:/work# cat /sys/fs/cgroup/memory.current
2481475584
root@testpod:/work# cat /sys/fs/cgroup/memory.stat
anon 2472849408
file 57344
kernel_stack 458752
pagetables 7061504
percpu 0
sock 4096
shmem 0
file_mapped 0
file_dirty 0
file_writeback 0
swapcached 0
anon_thp 0
file_thp 0
shmem_thp 0
inactive_anon 2472730624
active_anon 12288
inactive_file 4096
active_file 53248
unevictable 0
slab_reclaimable 371120
slab_unreclaimable 435552
slab 806672
workingset_refault_anon 0
workingset_refault_file 0
workingset_activate_anon 0
workingset_activate_file 0
workingset_restore_anon 0
workingset_restore_file 0
workingset_nodereclaim 0
pgfault 2503991
pgmajfault 0
pgrefill 0
pgscan 0
pgsteal 0
pgactivate 3
pgdeactivate 0
pglazyfree 0
pglazyfreed 0
thp_fault_alloc 0
thp_collapse_alloc 0

The free memory reported by sysinfo in this case is 425495718 which is clearly wrong.
I'm not sure why all memory.stat contents gets subtracted from the available memory.

[GuillaumeGomez]

Is it the free memory of the system in which your container is running by any chance? Also, please show the code you're using. If you want to get cgroup information, you need to use cgroup_limits.

[Dav1dde]

@GuillaumeGomez the code we're using is here: https://github.com/getsentry/relay/blob/be87f135374b2c3ea941a7ec1bd236071d13db13/relay-server/src/services/health_check.rs#L273-L292

        // Use the cgroup if available in case Relay is running in a container.
        if let Some(cgroup) = self.system.cgroup_limits() {
            Memory {
                used: cgroup.total_memory.saturating_sub(cgroup.free_memory),
                total: cgroup.total_memory,
            }
        } else {
            Memory {
                used: self.system.used_memory(),
                total: self.system.total_memory(),
            }
        }

[GuillaumeGomez]

Ok so you were using the correct code already and even already wrote a patch. Awesome! :)

[Dav1dde]

Ok so you were using the correct code already and even already wrote a patch. Awesome! :)

Do you know why that code was there in the first place (reading from memory.stat)? From what @gi0baro found it seems like that is wrong, but I assume there was a reason why it was added (tried to git blame but couldn't find anything)?

[GuillaumeGomez]

It was originally added in #1024. Modified a little bit in #1058 and the wrong update comes from #1119. Do you think we should still subtract slab_reclaimable, file and shmem?

[Dav1dde]

I really don't know. I tried digging through the cgroup man pages and I have no clue what is correct. I wanna say no, since these are all system values and not necessarily relevant for the cgroup?

Maybe you have an idea @gi0baro ?

What we can do is test the fork and see if it reports similar/same values we get from Kubernetes.

[GuillaumeGomez]

It'd be much appreciated. 👍

[gi0baro]

@GuillaumeGomez to my knowledge, and based on the information available at https://docs.kernel.org/admin-guide/cgroup-v2.html, memory.current should already include page cache, in-kernel data structures such as inodes, and network buffers.

I'm not a kernel dev nor expert, but IMHO the only value in memory.stat you may still want to subtract might be file. slab_reclaimable should be already counted in memory.current and shmem should rely on swapped contents, so I won't count that.

[GuillaumeGomez]

Thanks for the information! Let's wait for @Dav1dde's tests results and then they'll update (or not depending on the results) their PR, then I'll merge it and make a new release.

[Dav1dde]

@GuillaumeGomez the results are in, a picture speaks more than a thousand words?

This is the max memory usage of all our pods, top is reported from Kubernetes, the bottom is reported via sysinfo crate (with the patch from #1220 applied). It pretty much matches perfectly (purely empirical evidence).

Also big thanks to @gi0baro he actually did the hard work, I just setup the metrics.