Can't set up milvus cluster - RootCoordClient mess key not exist

[zeionara]

Hey, I am trying to set up a milvus instance in single-node kubernetes cluster, but only minio and etcd services are ready, the other pods are not starting. That's what I've done:

Install nvidia device plugin

sudo helm repo add nvdp https://nvidia.github.io/k8s-device-plugin
sudo helm repo update
sudo helm search repo nvdp --devel  # remember the chart version
sudo helm upgrade -i nvdp nvdp/nvidia-device-plugin --namespace nvidia-device-plugin --create-namespace --version 0.17.0  # substitute chart version here from the result of previous command execution
sudo kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml

Install milvus

sudo helm repo add milvus https://zilliztech.github.io/milvus-helm/
sudo helm install playground milvus/milvus -f custom-values.yaml

Configuration

The content of custom-values.yaml:

indexNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"
queryNode:
  resources:
    requests:
      nvidia.com/gpu: "1"
    limits:
      nvidia.com/gpu: "1"

The node description:

Name:               k3d-milvus-server-0
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k3d-milvus-server-0
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 172.20.0.2
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"02:6f:db:c1:1e:f6"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 172.20.0.2
                    k3s.io/hostname: k3d-milvus-server-0
                    k3s.io/internal-ip: 172.20.0.2
                    k3s.io/node-args: ["server","--tls-san","0.0.0.0","--tls-san","k3d-milvus-serverlb"]
                    k3s.io/node-config-hash: D6S6K6U2QTGBIHYTK6DJU7CJYEYKOQY6JWYNORMVSPH5D6UFNU7Q====
                    k3s.io/node-env: {"K3S_KUBECONFIG_OUTPUT":"/output/kubeconfig.yaml","K3S_TOKEN":"********"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 21 Nov 2024 16:54:55 +0300
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k3d-milvus-server-0
  AcquireTime:     <unset>
  RenewTime:       Fri, 22 Nov 2024 13:32:28 +0300
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Fri, 22 Nov 2024 13:28:23 +0300   Thu, 21 Nov 2024 16:54:55 +0300   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.20.0.2
  Hostname:    k3d-milvus-server-0
Capacity:
  cpu:                32
  ephemeral-storage:  1920641288Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131743276Ki
  nvidia.com/gpu:     1
  pods:               110
Allocatable:
  cpu:                32
  ephemeral-storage:  1868399843502
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             131743276Ki
  nvidia.com/gpu:     1
  pods:               110
System Info:
  Machine ID:
  System UUID:                cbeddc58-6586-ac17-a6fe-d843ae9e92a7
  Boot ID:                    02a24c10-0252-4833-97ee-9609d04ecae9
  Kernel Version:             6.8.0-47-generic
  OS Image:                   K3s v1.31.2+k3s1
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.22-k3s1
  Kubelet Version:            v1.31.2+k3s1
  Kube-Proxy Version:         v1.31.2+k3s1
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://k3d-milvus-server-0
Non-terminated Pods:          (19 in total)
  Namespace                   Name                                            CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                            ------------  ----------  ---------------  -------------  ---
  default                     playground-etcd-0                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
  default                     playground-etcd-1                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m16s
  default                     playground-etcd-2                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-datanode-5cfb6cf5dc-znbcc     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-datanode-7d874ddfc6-kwqhc     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-indexnode-5b6988cd54-5rkmq    0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-mixcoord-85c4888ff8-7ztw4     0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-proxy-68478c4f79-qj5nm        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-milvus-proxy-6f67b4bb6b-zqmqh        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m36s
  default                     playground-minio-0                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-1                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-2                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m35s
  default                     playground-minio-3                              0 (0%)        0 (0%)      2Gi (1%)         0 (0%)         4m34s
  kube-system                 coredns-56f6fc8fd7-wrjsb                        100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     4m35s
  kube-system                 local-path-provisioner-5cf85fd84d-4hvct         0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
  kube-system                 metrics-server-5985cbc9d7-ck9h2                 100m (0%)     0 (0%)      70Mi (0%)        0 (0%)         4m35s
  kube-system                 nvidia-device-plugin-daemonset-b4rt2            0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h
  kube-system                 svclb-traefik-f9b9f340-9q8lw                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h
  kube-system                 traefik-57b79cf995-dwvnf                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         4m35s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests     Limits
  --------           --------     ------
  cpu                200m (0%)    0 (0%)
  memory             8332Mi (6%)  170Mi (0%)
  ephemeral-storage  0 (0%)       0 (0%)
  hugepages-1Gi      0 (0%)       0 (0%)
  hugepages-2Mi      0 (0%)       0 (0%)
  nvidia.com/gpu     1            1

Error description from mixcoord pod, the same error is printed in other pods' logs which are not starting:

[2024/11/22 10:34:30.985 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/22 10:34:30.985 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 10:34:30.985 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 10:34:31.786 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ListDatabases\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:639\n  | github.com/milvus-io/milvus/internal/datacoord/broker.(*coordinatorBroker).ListDatabases\n  | \t/workspace/source/internal/datacoord/broker/coordinator_broker.go:123\n  | github.com/milvus-io/milvus/internal/datacoord.(*meta).reloadCollectionsFromRootcoord\n  | \t/workspace/source/internal/datacoord/meta.go:234\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1.1\n  | \t/workspace/source/internal/datacoord/server.go:671\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/datacoord.(*Server).initMeta.func1.1\n  | \t/workspace/source/internal/datacoord/server.go:670\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

Can you please help me to find what what's wrong and how to fix it?

[zeionara]

After I've replaced custom-values.yaml with this file, in which I've disabled mixCoord and enabled rootCoord, dataCoord, queryCoord, indexCoord, the following error is started to be printed in the datanode:

[2024/11/22 11:07:58.086 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]
[2024/11/22 11:07:58.642 +00:00] [DEBUG] [datanode/data_node.go:567] ["DataNode current state"] [State=Abnormal]
[2024/11/22 11:07:58.788 +00:00] [DEBUG] [datanode/data_node.go:567] ["DataNode current state"] [State=Abnormal]
{"level":"warn","ts":"2024-11-22T11:08:03.087Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000698700/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}
[2024/11/22 11:08:06.088 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]

Similar error is printed in the proxy pod:

[2024/11/22 11:12:40.145 +00:00] [INFO] [sessionutil/session_util.go:201] ["Session try to connect to etcd"]
{"level":"warn","ts":"2024-11-22T11:12:45.146Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000316a80/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing dial tcp [::1]:2379: connect: connection refused\""}

And RootCoord error is printed in the rootcoord pod:

[2024/11/22 11:14:35.189 +00:00] [WARN] [client/client.go:90] ["RootCoordClient mess key not exist"] [key=rootcoord]
[2024/11/22 11:14:35.189 +00:00] [WARN] [grpcclient/client.go:248] ["failed to get client address"] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 11:14:35.189 +00:00] [WARN] [grpcclient/client.go:460] ["fail to get grpc client in the retry state"] [client_role=rootcoord] [error="find no available rootcoord, check rootcoord state"]
[2024/11/22 11:14:35.189 +00:00] [WARN] [retry/retry.go:106] ["retry func failed"] [retried=4] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]
[2024/11/22 11:14:38.390 +00:00] [WARN] [grpcclient/client.go:474] ["grpc client is nil, maybe fail to get client in the retry state"] [client_role=rootcoord] [error="empty grpc client: find no available rootcoord, check rootcoord state"] [errorVerbose="empty grpc client: find no available rootcoord, check rootcoord state\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call.func2\n  | \t/workspace/source/internal/util/grpcclient/client.go:473\n  | github.com/milvus-io/milvus/pkg/util/retry.Handle\n  | \t/workspace/source/pkg/util/retry/retry.go:104\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).call\n  | \t/workspace/source/internal/util/grpcclient/client.go:466\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call\n  | \t/workspace/source/internal/util/grpcclient/client.go:553\n  | github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall\n  | \t/workspace/source/internal/util/grpcclient/client.go:569\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:107\n  | github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).GetComponentStates\n  | \t/workspace/source/internal/distributed/rootcoord/client/client.go:121\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...].func1\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:39\n  | github.com/milvus-io/milvus/pkg/util/retry.Do\n  | \t/workspace/source/pkg/util/retry/retry.go:44\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentStates[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:64\n  | github.com/milvus-io/milvus/internal/util/componentutil.WaitForComponentHealthy[...]\n  | \t/workspace/source/internal/util/componentutil/componentutil.go:85\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).init\n  | \t/workspace/source/internal/distributed/querycoord/service.go:181\n  | github.com/milvus-io/milvus/internal/distributed/querycoord.(*Server).Run\n  | \t/workspace/source/internal/distributed/querycoord/service.go:113\n  | github.com/milvus-io/milvus/cmd/components.(*QueryCoord).Run\n  | \t/workspace/source/cmd/components/query_coord.go:59\n  | github.com/milvus-io/milvus/cmd/roles.runComponent[...].func1\n  | \t/workspace/source/cmd/roles/roles.go:126\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) empty grpc client\nWraps: (3) find no available rootcoord, check rootcoord state\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *errors.errorString"]

Rootcoord pod is not starting. Events from the pod description:

Events:
  Type     Reason            Age                    From               Message
  ----     ------            ----                   ----               -------
  Warning  FailedScheduling  27m                    default-scheduler  0/1 nodes are available: 1 node(s) were unschedulable. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Warning  FailedScheduling  27m                    default-scheduler  0/1 nodes are available: 1 node(s) were unschedulable. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal   Scheduled         27m                    default-scheduler  Successfully assigned default/playground-milvus-rootcoord-5796cd4bb-m9sr2 to k3d-milvus-server-0
  Normal   Pulled            27m (x2 over 27m)      kubelet            Container image "milvusdb/milvus:v2.4.17" already present on machine
  Normal   Created           27m (x2 over 27m)      kubelet            Created container rootcoord
  Normal   Started           27m (x2 over 27m)      kubelet            Started container rootcoord
  Warning  Unhealthy         21m (x6 over 25m)      kubelet            Liveness probe failed: HTTP probe failed with statuscode: 500
  Warning  Unhealthy         7m17s (x85 over 25m)   kubelet            Readiness probe failed: HTTP probe failed with statuscode: 500
  Warning  BackOff           2m19s (x6 over 2m55s)  kubelet            Back-off restarting failed container rootcoord in pod playground-milvus-rootcoord-5796cd4bb-m9sr2_default(697b0a2f-b6a4-4b09-9891-57a85ea16d5b)

Rootcoord logs:

open pid file: /run/milvus/rootcoord.pid
lock pid file: /run/milvus/rootcoord.pid
[2024/11/22 11:15:14.582 +00:00] [INFO] [roles/roles.go:333] ["starting running Milvus components"]
[2024/11/22 11:15:14.582 +00:00] [INFO] [roles/roles.go:182] ["Enable Jemalloc"] ["Jemalloc Path"=/milvus/lib/:/milvus/lib/libjemalloc.so]
[2024/11/22 11:15:14.587 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/11/22 11:15:14.588 +00:00] [DEBUG] [config/etcd_source.go:52] ["init etcd source"] [etcdInfo="{\"UseEmbed\":false,\"EnableAuth\":false,\"UserName\":\"\",\"PassWord\":\"\",\"UseSSL\":false,\"Endpoints\":[\"playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379\"],\"KeyPrefix\":\"by-dev\",\"CertFile\":\"/path/to/etcd-client.pem\",\"KeyFile\":\"/path/to/etcd-client-key.pem\",\"CaCertFile\":\"/path/to/ca.pem\",\"MinVersion\":\"1.3\",\"RefreshInterval\":5000000000}"]
[2024/11/22 11:15:14.588 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.589 +00:00] [DEBUG] [config/etcd_source.go:92] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"]
[2024/11/22 11:15:14.589 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=EtcdSource]
[2024/11/22 11:15:14.590 +00:00] [INFO] [paramtable/component_param.go:4286] ["DeployModeEnv is not set, use default"] [default=0.5]
[2024/11/22 11:15:14.591 +00:00] [INFO] [paramtable/hook_config.go:21] ["hook config"] [hook={}]
[2024/11/22 11:15:14.591 +00:00] [DEBUG] [config/refresher.go:67] ["start refreshing configurations"] [source=FileSource]
[2024/11/22 11:15:14.591 +00:00] [INFO] [logutil/logutil.go:163] ["Log directory"] [configDir=]
[2024/11/22 11:15:14.591 +00:00] [INFO] [logutil/logutil.go:164] ["Set log file to "] [path=]
[2024/11/22 11:15:14.591 +00:00] [INFO] [roles/roles.go:282] [setupPrometheusHTTPServer]
[2024/11/22 11:15:14.591 +00:00] [INFO] [http/server.go:160] ["management listen"] [addr=:9091]
[2024/11/22 11:15:14.591 +00:00] [INFO] [gc/gc_tuner.go:137] ["GC Helper initialized."] ["Initial GoGC"=100] [minimumGOGC=30] [maximumGOGC=200] [memoryThreshold=121414603161]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Abnormal]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/service.go:154] ["RootCoord listen on"] [address="[::]:53100"] [port=53100]
[2024/11/22 11:15:14.591 +00:00] [INFO] [rootcoord/service.go:196] ["init params done.."]
[2024/11/22 11:15:14.591 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.592 +00:00] [INFO] [rootcoord/service.go:216] ["etcd connect done ..."]
[2024/11/22 11:15:14.592 +00:00] [INFO] [rootcoord/service.go:276] ["start grpc "] [port=53100]
[2024/11/22 11:15:14.693 +00:00] [INFO] [rootcoord/service.go:233] ["grpc init done ..."]
[2024/11/22 11:15:14.693 +00:00] [INFO] [rootcoord/service.go:236] ["RootCoord start to create DataCoord client"]
[2024/11/22 11:15:14.693 +00:00] [INFO] [etcd/etcd_util.go:47] ["create etcd client"] [useEmbedEtcd=false] [useSSL=false] [endpoints="[playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379,playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379]"] [minVersion=1.3]
[2024/11/22 11:15:14.699 +00:00] [INFO] [rootcoord/service.go:245] ["RootCoord start to create QueryCoord client"]
[2024/11/22 11:15:14.701 +00:00] [INFO] [dependency/factory.go:85] ["try to init mq"] [standalone=false] [mqType=pulsar]
[2024/11/22 11:15:14.709 +00:00] [INFO] [sessionutil/session_util.go:296] ["start server"] [name=rootcoord] [address=10.42.0.170:53100] [id=194] [server_labels={}]
[2024/11/22 11:15:14.709 +00:00] [INFO] [rootcoord/root_coord.go:154] ["update rootcoord state"] [state=Initializing]
[2024/11/22 11:15:14.715 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/11/22 11:11:17.699 +00:00] [save=2024/11/22 11:15:17.710 +00:00] [next=2024/11/22 11:15:14.710 +00:00]
[2024/11/22 11:15:14.715 +00:00] [INFO] [rootcoord/root_coord.go:398] ["id allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/22 11:15:14.718 +00:00] [INFO] [tso/tso.go:122] ["sync and save timestamp"] [last=2024/11/22 11:11:17.702 +00:00] [save=2024/11/22 11:15:17.716 +00:00] [next=2024/11/22 11:15:14.716 +00:00]
[2024/11/22 11:15:14.718 +00:00] [INFO] [rootcoord/root_coord.go:422] ["tso allocator initialized"] [root_path=by-dev/kv] [sub_path=gid] [key=idTimestamp]
[2024/11/22 11:15:14.718 +00:00] [INFO] [rootcoord/root_coord.go:341] ["Using etcd as meta storage."]
[2024/11/22 11:15:14.720 +00:00] [INFO] [rootcoord/meta_table.go:157] ["recover databases"] ["num of dbs"=1]
[2024/11/22 11:15:14.721 +00:00] [INFO] [rootcoord/kv_catalog.go:741] ["unmarshal all collection details cost"] [db=0] [cost=826ns]
[2024/11/22 11:15:14.721 +00:00] [INFO] [rootcoord/kv_catalog.go:745] ["append partition and fields info cost"] [db=0] [cost=639ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/kv_catalog.go:741] ["unmarshal all collection details cost"] [db=1] [cost=633ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/kv_catalog.go:745] ["append partition and fields info cost"] [db=1] [cost=200ns]
[2024/11/22 11:15:14.724 +00:00] [INFO] [rootcoord/meta_table.go:200] ["collections recovered from db"] [db_name=default] [collection_num=0] [partition_num=0]
[2024/11/22 11:15:14.726 +00:00] [INFO] [rootcoord/meta_table.go:216] ["RootCoord meta table reload done"] [duration=7.600315ms]
[2024/11/22 11:15:14.727 +00:00] [INFO] [msgstream/mq_msgstream.go:118] ["Msg Stream state"] [can_produce=true]
[2024/11/22 11:17:06.184 +00:00] [WARN] [retry/retry.go:46] ["retry func failed"] [retried=0] [error="connection error"]
[2024/11/22 11:18:44.242 +00:00] [WARN] [roles/roles.go:312] ["Get signal to exit"] [signal=terminated]

[LoveEachDay]

@zeionara Could you double check your deployment?
It's very strong that proxy and datanode access to localhost:2379 for etcd, while rootcoord use playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379\",\"playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379

[zeionara]

I was also surprised by that, so that I even tried to enable externalEtcd in milvus config, and hard-coded endpoint to playground-etcd:2379, which didn't quite help

[LoveEachDay]

Could you paste all the details of the configmap which milvus use?

[zeionara]

Could you paste all the details of the configmap which milvus use?

Sure, here is my configmap (kubectl describe configmap playground-milvus):

Name:         playground-milvus
Namespace:    default
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: playground
              meta.helm.sh/release-namespace: default

Data
====
default.yaml:
----
# Copyright (C) 2019-2021 Zilliz. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software distributed under the License
# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express
# or implied. See the License for the specific language governing permissions and limitations under the License.

etcd:
  endpoints:
  - playground-etcd-0.playground-etcd-headless.default.svc.cluster.local:2379
  - playground-etcd-1.playground-etcd-headless.default.svc.cluster.local:2379
  - playground-etcd-2.playground-etcd-headless.default.svc.cluster.local:2379

metastore:
  type: etcd

minio:
  address: playground-minio
  port: 9000
  accessKeyID: minioadmin
  secretAccessKey: minioadmin
  useSSL: false
  bucketName: milvus-bucket
  rootPath: file
  useIAM: false
  useVirtualHost: false

mq:
  type: pulsar

messageQueue: pulsar

pulsar:
  address: playground-pulsar-proxy
  port: 6650
  maxMessageSize: 5242880

rootCoord:
  address: playground-milvus-rootcoord
  port: 53100
  enableActiveStandby: false  # Enable rootcoord active-standby

proxy:
  port: 19530
  internalPort: 19529

queryCoord:
  address: playground-milvus-querycoord
  port: 19531

  enableActiveStandby: false  # Enable querycoord active-standby

queryNode:
  port: 21123
  enableDisk: true # Enable querynode load disk index, and search on disk index

indexCoord:
  address: playground-milvus-indexcoord
  port: 31000
  enableActiveStandby: false  # Enable indexcoord active-standby

indexNode:
  port: 21121
  enableDisk: true # Enable index node build disk vector index

dataCoord:
  address: playground-milvus-datacoord
  port: 13333
  enableActiveStandby: false  # Enable datacoord active-standby

dataNode:
  port: 21124

log:
  level: info
  file:
    rootPath: ""
    maxSize: 300
    maxAge: 10
    maxBackups: 20
  format: text


user.yaml:
----
#    For example enable rest http for milvus proxy
#    proxy:
#      http:
#        enabled: true
#      maxUserNum: 100
#      maxRoleNum: 10
##  Enable tlsMode and set the tls cert and key
#  tls:
#    serverPemPath: /etc/milvus/certs/tls.crt
#    serverKeyPath: /etc/milvus/certs/tls.key
#   common:
#     security:
#       tlsMode: 1


BinaryData
====

Events:  <none>

[zeionara]

NAME                                         READY   STATUS    RESTARTS        AGE
milvus-cluster-minio-0                       1/1     Running   26 (45d ago)    340d
milvus-cluster-minio-3                       1/1     Running   26 (45d ago)    340d
milvus-cluster-minio-1                       1/1     Running   26 (45d ago)    340d
milvus-cluster-minio-2                       1/1     Running   26 (45d ago)    340d
milvus-cluster-pulsar-zookeeper-0            1/1     Running   25 (45d ago)    340d
milvus-cluster-etcd-1                        1/1     Running   23 (45d ago)    257d
milvus-cluster-etcd-2                        1/1     Running   23 (45d ago)    257d
milvus-cluster-pulsar-zookeeper-2            1/1     Running   25 (45d ago)    340d
milvus-cluster-pulsar-zookeeper-1            1/1     Running   25 (45d ago)    340d
milvus-cluster-etcd-0                        1/1     Running   21 (45d ago)    257d
milvus-cluster-pulsar-recovery-0             1/1     Running   44 (45d ago)    340d
milvus-cluster-indexcoord-84f8d66597-77rvh   1/1     Running   21 (45d ago)    257d
milvus-cluster-pulsar-bookie-2               1/1     Running   27 (45d ago)    340d
milvus-cluster-querynode-54f8969dc7-2mss4    1/1     Running   0               45d
milvus-cluster-pulsar-broker-0               1/1     Running   49 (45d ago)    340d
milvus-cluster-indexnode-5bf84b587f-zz7v8    1/1     Running   0               45d
milvus-cluster-pulsar-bookie-1               1/1     Running   27 (45d ago)    340d
milvus-cluster-pulsar-bookie-0               1/1     Running   27 (45d ago)    340d
milvus-cluster-pulsar-proxy-0                1/1     Running   26 (45d ago)    340d
milvus-cluster-rootcoord-555b6f8b6b-27bz7    1/1     Running   42 (45d ago)    257d
milvus-cluster-datacoord-7fdc6cd9c4-zvpln    1/1     Running   107 (45d ago)   257d
milvus-cluster-querycoord-9bfd75988-tmgsf    1/1     Running   44 (45d ago)    257d
milvus-cluster-proxy-c69895-l992k            1/1     Running   51 (45d ago)    257d
milvus-cluster-datanode-859799f49-jjxw5      1/1     Running   0               52m

Disabling gpu for datanode helped:

indexNode:
  resources:
    requests:
      cpu: "0"
      memory: "4Gi"
      nvidia.com/gpu: "1"
    limits:
      cpu: "0"
      memory: "16Gi"
      nvidia.com/gpu: "1"
  extraEnv:
  - name: CUDA_VISIBLE_DEVICES
    value: "0"
queryNode:
  resources:
    requests:
      cpu: "0"
      memory: "4Gi"
      nvidia.com/gpu: "1"
    limits:
      cpu: "0"
      memory: "16Gi"
      nvidia.com/gpu: "1"
  extraEnv:
  - name: CUDA_VISIBLE_DEVICES
    value: "0"
extraConfigFiles:
  user.yaml: |+
    common:
      security:
        authorizationEnabled: true