Is there any relationship between compaction and search performance?

[hjlee9182]

I have 2 collections.
A collection is loaded and searched. B collection is not loaded and upserted.
Each collection have almost 2 Million entity and 512 emb.

While I was upserting to B collection, the search performance of A collection fell. When I upserted in a lot at once, the performance went up to 200-300ms, so I put 1 second on the sleep when each 100 entities upserted.
And when I checked the metric, the search performance also fell whenever a compaction occurred in the data coordinator.

1. What is the connection between these two?
2. How can I lower the compaction latency? Do I just need to increase the data node?
   
   
   

[yhmo]

1. double-check the "Segment Loaded Num" and "Collection Loaded Num" in the "Query Node" section
   the "Collection Loaded Num" should be 1
   the "Segment Loaded Num" should be a const value
   

2. double-check the "Search Vector Count Rate" in the "Proxy" section, it should be almost a const value
   

[hjlee9182]

Use Milvus 2.3.4-gpu

Query Node - Segment Loaded Num, Collection Loaded Num

Proxy - Search Vecotr Count Rate
I read it using locust

[yhmo]

Looks like the "Search Vector Count Rate" is affected by upsert at 14:18, 14:28 and 14:54

Check the "Mutation Latency" and "Insert Vector Count Rate" in "Proxy" section.

[hjlee9182]

here!

But In my metric, upsert is not show in insert vector count Rate only insert is show.

[yhmo]

Yes, upsert is not counted in the "Insert Vector Count Rate".
Search latency is affected by upsert requests in proxy node. There are three peaks of "Stored Rows Rate" at 14:18, 14:28 and 14:54. Proxy node needs to handle more requests(search + upsert) at these points.

You have at least 3 proxy nodes, check your client code, use different proxy nodes for different requests. For example, all search requests are sent to proxy_1 or proxy_2, all upsert requests are sent to proxy_3.

[hjlee9182]

I tested use different proxy nodes for different requests. But that's not effect. Same situation again occured.
I used the code below.

Host  = '{ proxy pod IP }'
port = 19530

connections.connect(host=Host, port=Port)

[xiaofan-luan]

what milvus version are u using && what consistency are you using?

did you check the cpu usage or all your node on metrics?

[hjlee9182]

data node

index node

proxy

query node

[hjlee9182]

But this phenomenon also happened in version 2.3.4milvus ( no gpu).

[hjlee9182]

and test in 2.3.9, also occured.

[barzan-hayati]

@hjlee9182 how you generated these charts?

[yhmo]

@barzan-hayati https://milvus.io/docs/monitor.md

[xiaofan-luan]

/assign @Presburger

[xiaofan-luan]

index build take GPU which slow the search?
Did you deploy index and querynode on same GPU card?

[hjlee9182]

diffrent card

[xiaofan-luan]

The cpu usgae of both proxy and querynode seems to be high.

How many qps are you running during the test and what's the target QPS/latency?

If you are using GPU I guess cpu should't be taken too much?

Maybe we can have a quick online meeting sync?

[hjlee9182]

I tested about 1000 qps,.
I don't have an exact target qps, but I'm testing it to see how well it performs.
Maybe I have GPU, but not using gpu index(ex. GPU_IV_FLAT) so cpu was used too much.

I'm Ok, but what purpose of quick online meeting?

[hjlee9182]

Is this not normally situation?

[xiaofan-luan]

If you need to any futher help of improving performance, we are open to do so.

From what all you described, there must be some shared resources of insert of seaerch. My guess

1. proxy cpu
2. proxy network bandwith(very likely)
3. GPU card, you said index and search are not sharing resources so this is not likely.

Maybe check network bandwidth. This happens especially when you retrieve fields out