rootcoord cannot connect etcd

[rere950303]

• milvus version: 2.4.14

{"level":"warn","ts":"2024-11-06T11:46:29.088+0900","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000286a80/localhost:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}

Why are the etcd-endpoints being set to localhost in the logs above? I've confirmed that etcd itself is healthy.

[yhmo]

I think the etcd address is not properly configured.

Configure etcd on docker-compose:
https://milvus.io/docs/deploy_etcd.md#Configure-etcd-with-Docker-Compose

Configure etcd on k8s:
https://milvus.io/docs/deploy_etcd.md#Configure-etcd-on-K8s

[rere950303]

@yhmo
If we set etcd.enabled to true and externalEtcd.enabled to false, wouldn't we not need to set up a separate address?

[yhmo]

I think the etcd.enabled should always be true because currently we only support etcd as metadata server. (TiKV is experimental)

If externalEtcd.enabled is false, the milvus-helm will deploy an etcd pod for you.
If externalEtcd.enabled is true, the milvus-helm will not deploy the etcd pod, you need to provide an external etcd server address for milvus.

[rere950303]

@yhmo
In the /milvus/configs directory, there are default.yaml, milvus.yaml, and user.yaml files, and it seems to operate by reading milvus.yaml, so I didn't touch the customConfigMap settings.

this is the helm chart

        volumeMounts:
        {{- if .Values.customConfigMap }}
        - name: milvus-config
          mountPath: /milvus/configs/user.yaml
          subPath: milvus.yaml
          readOnly: true
        {{- else }}
        - name: milvus-config
          mountPath: /milvus/configs/default.yaml
          subPath: default.yaml
          readOnly: true
        - name: milvus-config
          mountPath: /milvus/configs/user.yaml
          subPath: user.yaml
          readOnly: true
        {{- end }}

etcd:
{{- if .Values.externalEtcd.enabled }}
  endpoints:
  {{- range .Values.externalEtcd.endpoints }}
    - {{ . }}
  {{- end }}
{{- else }}
  endpoints:
{{- range $i := until ( .Values.etcd.replicaCount | int ) }}
  - {{ $etcdReleaseName }}-{{ $i }}.{{ $etcdReleaseName }}-headless.{{ $namespace }}.svc.{{ $.Values.etcd.clusterDomain }}:{{ $etcdPort }}
{{- end }}
{{- end }}

i dont know why rootcoord try connecting to localhost

[rere950303]

i identified that the cause was the removal of milvus-config-tool usage when upgrading the helm chart version.