You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently we only allow activedefrag mechanism to run when jemalloc is defined as the memory allocator.
JEmalloc is not compatible with memory instrumentation tools like Valgrind and AddressSanitizer to be used together with activedefrag. Aside from the fact that this limitation prevents the use of activedefrag feature when one wants to use a different allocator, When enabled, activedefrag constantly replaces memory allocations for different allocations like data objects, dictionaries etc... the problem is that in many situations the data object might have several references which might not be be updated and considered when the code was introduced. This might lead to different write-after-free scenarios which are extremly hard to debug and our test coverage will not cover them (as activedefrag will not be enabled when memory instrumentation is used).
Although by default valkey is used together with JEmalloc, having the ability to run activedefrag with other allocators can help identify defrag related issues during tests.
Description of the feature
This proposal is to enable the ability to explicitly define the HAVE_DEFRAG flag during Valkey compilation. In case it was not explicitly enabled the system will behave the same as today.
The active defrag code is very independent and the main dependencies are:
je_mallctl - used in order to get current statistics about the memory allocation state. there is really no good parallel functionality with other allocators like tcmalloc and glibc malloc. it is true that mallinfo2 (supported from glibc 2.33) can provide hints about potential external fragmentation, but AFAIK there is no real way to deduct the fragmentation ratio using it.
I think we can provide a NEW DEBUG subcommand or internal config that will force the return value of getAllocatorFragmentation when compiled in that mode (no jemalloc + HAVE_DEFRAG).
je_get_defrag_hint - this function was added to jemalloc in order to support decision to defrag memory buffer allocation. It compares the utilization ratio of the slab hiolding the allocation in question to the total average among non-full slabs.
I think that since this mode is meant for testing purpose, we can just provide a naive implementation of it which always return true so that we can maximize the propability of locating issues related to defrag.
Alternatives you've considered
JEmalloc also has a debug mode and some other configurations which helps locate memory corruption issues. However these can mainly track down double free and non clean memory allocations which will require to add many type of checks on buffers before reading/writing to them. For example it will not catch case of write-after-free. I think that testing with instrumentation tools (eg AddressSanitizer) would yield much better coverage.
The text was updated successfully, but these errors were encountered:
The problem/use-case that the feature addresses
Currently we only allow activedefrag mechanism to run when jemalloc is defined as the memory allocator.
JEmalloc is not compatible with memory instrumentation tools like Valgrind and AddressSanitizer to be used together with activedefrag. Aside from the fact that this limitation prevents the use of activedefrag feature when one wants to use a different allocator, When enabled, activedefrag constantly replaces memory allocations for different allocations like data objects, dictionaries etc... the problem is that in many situations the data object might have several references which might not be be updated and considered when the code was introduced. This might lead to different write-after-free scenarios which are extremly hard to debug and our test coverage will not cover them (as activedefrag will not be enabled when memory instrumentation is used).
Although by default valkey is used together with JEmalloc, having the ability to run activedefrag with other allocators can help identify defrag related issues during tests.
Description of the feature
This proposal is to enable the ability to explicitly define the HAVE_DEFRAG flag during Valkey compilation. In case it was not explicitly enabled the system will behave the same as today.
The active defrag code is very independent and the main dependencies are:
je_mallctl - used in order to get current statistics about the memory allocation state. there is really no good parallel functionality with other allocators like tcmalloc and glibc malloc. it is true that mallinfo2 (supported from glibc 2.33) can provide hints about potential external fragmentation, but AFAIK there is no real way to deduct the fragmentation ratio using it.
I think we can provide a NEW DEBUG subcommand or internal config that will force the return value of getAllocatorFragmentation when compiled in that mode (no jemalloc + HAVE_DEFRAG).
je_get_defrag_hint - this function was added to jemalloc in order to support decision to defrag memory buffer allocation. It compares the utilization ratio of the slab hiolding the allocation in question to the total average among non-full slabs.
I think that since this mode is meant for testing purpose, we can just provide a naive implementation of it which always return true so that we can maximize the propability of locating issues related to defrag.
Alternatives you've considered
JEmalloc also has a debug mode and some other configurations which helps locate memory corruption issues. However these can mainly track down double free and non clean memory allocations which will require to add many type of checks on buffers before reading/writing to them. For example it will not catch case of write-after-free. I think that testing with instrumentation tools (eg AddressSanitizer) would yield much better coverage.
The text was updated successfully, but these errors were encountered: