Is DiskANN also Memory Intensive?

[xiaobingxia-at]

Hello, I'm exploring DiskANN using Milvus. I was assuming that the DiskANN index takes no memory or very little memory.
But after ingesting about 10million 1536-dim vectors, the disk usage comes to ~80GB. And the memory usage comes to ~40GB.
The memory usage is far more than I expected. Is this normal?

[xiaobingxia-at]

PQCodeBudgetGBRatio
SearchCacheBudgetGBRatio
These two parameters should impact the memory usage. But which node memory? Query node or indexing node?

[liliu-z]

DiskANN has three four parts: PQ, Graph, Raw Data and Cache. Graph and Raw Data are placed in disk, and the other two are in memory.
These two params affect memory usage of QueryNodes:

1. PQCodeBudgetGBRatio bigger PQ provides better recall but needs more memory
2. SearchCacheBudgetGBRatio larger cache provides better QPS but still, more memory.

[xiaofan-luan]

DiskANN has three four parts: PQ, Graph, Raw Data and Cache. Graph and Raw Data are placed in disk, and the other two are in memory. These two params affect memory usage of QueryNodes:

1. PQCodeBudgetGBRatio bigger PQ provides better recall but needs more memory
2. SearchCacheBudgetGBRatio larger cache provides better QPS but still, more memory.

What could be a recommened valuae?

[liliu-z]

DiskANN has three four parts: PQ, Graph, Raw Data and Cache. Graph and Raw Data are placed in disk, and the other two are in memory. These two params affect memory usage of QueryNodes:

1. PQCodeBudgetGBRatio bigger PQ provides better recall but needs more memory
2. SearchCacheBudgetGBRatio larger cache provides better QPS but still, more memory.

What could be a recommened valuae?

For 1536dim data, setting PQCodeBudgetGBRatio to 0.075-0.125 is usually safe to get 90-98% recall
And about SearchCacheBudgetGBRatio, as you said, 10%-20% can provide good performance improvement. The performance improvement of higher number will be marginal. But it really depends on your disk resource. Plz ensure to use NVMe SSD.

[xiaobingxia-at]

Thank you so much for your help! The memory we are talking about in this thread, for both parameters, is query node's memory correct?

[xiaobingxia-at]

If SearchCacheBudgetGBRatio=0.1, it means 10% of raw data will be cached in query node's memory, is my understanding correct?

[xiaobingxia-at]

If I decrease PQCodeBudgetGBRatio causing a lower recall. Can I increase the following parameters to increase recall without causing a memory usage increase?
MaxDegree: 56 SearchListSize: 100
Any suggestion about these two parameters?

[liliu-z]

SearchListSize yes, this is a best way to do perf/recall trade
MaxDegree not always, we need to suitable one, 56 is relatively high. 48 can make it better

[xiaobingxia-at]

I have tuned the diskANN parameters, in the end, the diskANN takes memory which is about 16.9% of raw data, which is good.
But once I started searching, the memory usage surged to 31.4% of raw data. Is this expected?

[xiaofan-luan]

I have tuned the diskANN parameters, in the end, the diskANN takes memory which is about 16.9% of raw data, which is good. But once I started searching, the memory usage surged to 31.4% of raw data. Is this expected?

You have to tune the cache param of diskANN param as well.
When search, diskann tend to cache 10-20% of the data in memory. This part can also be change to smaller amount. We do recommend to keep 5% of the cache and it give you 50%-100% performance improvement

[xiaobingxia-at]

SearchCacheBudgetGBRatio=0.01
I had this parameter being effective when I do this benchmark. Does it mean, it only caches 1% of data as cache?

[liliu-z]

@xiaobingxia-at Just double check, you meant after loading index it only use 16.9% of your mem and 31.4% while searching?
If so, some more clarification Qs

1. Are you working on a pure vector search or with some scalar data involved?
2. How big is your data and query? Did you return raw vectors?

SearchCacheBudgetGBRatio=0.01 I had this parameter being effective when I do this benchmark. Does it mean, it only caches 1% of data as cache?