High RAM Usage by Milvus Proxy Service

[kietheros]

I deploy Milvus cluster v2.4.6. Now, milvus proxy in one node is using a large amount of RAM. Is this normal or abnormal?
node-1

node-2

node-3

Some logs

024/11/05 02:53:26 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:31 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:36 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:41 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:46 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:51 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:53:56 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
[2024/11/05 02:53:58.033 +00:00] [WARN] [proxy/hook_interceptor.go:79] ["fail to get current user"] [error="fail to get authorization from the md, authorization:[token]"]
2024/11/05 02:54:01 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:06 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:11 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:16 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:21 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:26 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:31 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:36 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:41 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:46 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:51 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:54:56 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
[2024/11/05 02:54:59.034 +00:00] [WARN] [proxy/hook_interceptor.go:79] ["fail to get current user"] [error="fail to get authorization from the md, authorization:[token]"]
2024/11/05 02:55:01 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused
2024/11/05 02:55:06 Post "http://192.168.50.110:14268/api/traces": dial tcp 192.168.50.110:14268: connect: connection refused

[yhmo]

There are two fixes for memory leak/usage of proxy in v2.4.7 and v2.4.11:

• Resolved memory leak in proxy meta cache (fix: memory leak in proxy meta cache #36076)
• Implemented a dynamic CGO pool for proxy CGO calls, optimizing resource usage and performance. (enhance: [2.4] Add dynamic cgo pool for proxy CGO call (#34768) #34842)

Use pprof to export the heap state:
go tool pprof -http=:10002 http://[the-high-ram-proxy-address]:9091/debug/pprof/heap

[xiaofan-luan]

query node is designed for many memories because it host all data into memory.

proxy is not, this ususally means you have some request or response accumulate in proxy. You can do a pprof as @yhmo said so we can get more details about what is going on here

[kietheros]

@yhmo @xiaofan-luan Thank you. Since the milvus proxy was causing the server's RAM to fill up, I upgraded to v2.4.14, so I haven't run pprof yet. Now, everything is OK.