Expected use of Memory

[stpp2]

Hey all,

Thank you for this great library!
I'm running a pretty basic kvrocks node with about 20GB of data on disk. Current memory usage is about 2GB.
I believe memory usage is correlated with the total size of the data stored on disk (due to indexes, perhaps?)

My question is - is there a way to limit the memory to, say, 4GB or 8GB and make sure it never uses more than that? (while hopefully keeping decent performance levels)

Thanks!

[git-hulk]

Hi @stpp2

The memory usage is generally affected by the configuration: rocksdb.block_cache_size which the default value is 4096MiB. You can reduce it to a smaller value, but keeping this value larger will benefit the read performance.

[stpp2]

Hi @git-hulk,

Thank you so much for the insight.
The thing I didn't understand, however, is if this 4096MB value is the overall set limit, or if memory can be be higher than that.

For example, if I kept that at 4096 but also had 500GB of stored data - would the overall memory use still be around 4GB?
I fully understand how it can affect performance, but I'm also trying to make sure the server doesn't crash due to insufficient memory. (unless this case is already handled by the lib)

Thanks again!

[git-hulk]

For example, if I kept that at 4096 but also had 500GB of stored data - would the overall memory use still be around 4GB?

Yes, it will be around 4GB but it might be a bit larger than that. Because Kvrocks will use some memory for keeping connections and other states, but it won't eat too much memory.

[stpp2]

Got it, thanks!

So there's really shouldn't be a concern that memory would go way beyond those 4GB right? (assuming I'm running on 5-6GB of memory)
Just trying to confirm it won't result in crashing the server or anything like that.

[PragmaTwice]

Yeah, the major part of memory consumption of Kvrocks is cache mechanisms in RocksDB.

[stpp2]

Thank you both @PragmaTwice and @git-hulk !!
Really appreciate your work on this repo, it's been great using it so far.