[observers/target_observer.go:432] ["failed to get collection info"]

[xiaobingxia-at]

Hello team! I'm trying to create partition under collections, but somehow the process gets stucked, it was hanging there.
I check the logs from mix coordinator and see the following logs:
[2024/11/07 15:41:58.740 +00:00] [WARN] [observers/target_observer.go:432] ["failed to get collection info"] [leaderID=14] [collectionID=453757715823438596] [channel=milvus-5m-partitions-rootcoord-dml_2_453757715823438596v0] [error="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n/workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n/workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n/workspace/source/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n/workspace/source/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:83 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeCollection\n/workspace/source/internal/querycoordv2/observers/target_observer.go:430 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n/workspace/source/internal/querycoordv2/observers/target_observer.go:409 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\n/workspace/source/internal/querycoordv2/observers/target_observer.go:227 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n/workspace/source/internal/querycoordv2/observers/task_dispatcher.go:101 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*taskDispatcher[...]).schedule.func1.1: rpc error: code = DeadlineExceeded desc = context deadline exceeded"] [errorVerbose="stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace: rpc error: code = DeadlineExceeded desc = context deadline exceeded\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | \t/workspace/source/internal/util/grpcclient/client.go:555\n | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | \t/workspace/source/internal/util/grpcclient/client.go:569\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n | \t/workspace/source/internal/distributed/rootcoord/client/client.go:183\n | github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeCollection\n | \t/workspace/source/internal/querycoordv2/meta/coordinator_broker.go:83\n | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:430\n | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\n | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:409\n | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n | \t/workspace/source/internal/querycoordv2/observers/target_observer.go:227\n | github.com/milvus-io/milvus/internal/querycoordv2/observers.(*taskDispatcher[...]).schedule.func1.1\n | \t/workspace/source/internal/querycoordv2/observers/task_dispatcher.go:101\n | github.com/milvus-io/milvus/pkg/util/conc.(*Pool[...]).Submit.func1\n | \t/workspace/source/pkg/util/conc/pool.go:81\n | github.com/panjf2000/ants/v2.(*goWorker).run.func1\n | \t/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:67\n | runtime.goexit\n | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) stack trace: /workspace/source/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace\n | /workspace/source/internal/util/grpcclient/client.go:555 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n | /workspace/source/internal/util/grpcclient/client.go:569 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n | /workspace/source/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n | /workspace/source/internal/distributed/rootcoord/client/client.go:183 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).DescribeCollection\n | /workspace/source/internal/querycoordv2/meta/coordinator_broker.go:83 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).DescribeCollection\n | /workspace/source/internal/querycoordv2/observers/target_observer.go:430 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).sync\n | /workspace/source/internal/querycoordv2/observers/target_observer.go:409 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).shouldUpdateCurrentTarget\n | /workspace/source/internal/querycoordv2/observers/target_observer.go:227 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*TargetObserver).check\n | /workspace/source/internal/querycoordv2/observers/task_dispatcher.go:101 github.com/milvus-io/milvus/internal/querycoordv2/observers.(*taskDispatcher[...]).schedule.func1.1\nWraps: (3) rpc error: code = DeadlineExceeded desc = context deadline exceeded\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *status.Error"]
May I know which component is a blocker here? Is it getting stucked on etcd?

[xiaofan-luan]

if you get too many collections that becomes a problem. because all the decribe is actually executing in serial.
2.5 improved the issue but for 2.4 we don't recommend to have 1000+ collections or partitions.

The only way to do that is to use partitionkey and that is the recommend way for milvus to do multi tenancy.