[BUG] driver name csi.juicefs.com not found in the list of registered CSI drivers #1191

YaAYadeer · 2024-11-22T09:24:17Z

What happened:
I deployed CSI successfully and created PV and PVC. However, I encountered an error while creating a pod:
kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers.
But I use the command "kubectl get csidrivers.storage.k8s.io". I find csi.juicefs.com is in the list.
Besides,I use kubectl logs -n kube-system juicefs-csi-node-xxx juicefs-plugin --tail 100
kubectl logs -n kube-system juicefs-csi-node-xxx juicefs-plugin --tail 100 to checked the logs,but find nothing wrong.

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?

Environment:

JuiceFS CSI Driver version (which image tag did your CSI Driver use):
Kubernetes version (e.g. kubectl version):
Object storage (cloud provider and region):
Metadata engine info (version, cloud provider managed or self maintained):
Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage):
Others:

The text was updated successfully, but these errors were encountered:

YaAYadeer · 2024-11-22T09:25:33Z

There exists the same issue #751 is also in unclosed state

YaAYadeer · 2024-11-22T09:28:25Z

@zwwhdls

YaAYadeer · 2024-11-22T09:30:27Z

kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver
NAME READY STATUS RESTARTS AGE
juicefs-csi-controller-0 3/3 Running 0 84m
juicefs-csi-controller-1 3/3 Running 0 83m
juicefs-csi-dashboard-686b87466f-vkh4m 1/1 Running 0 84m
juicefs-csi-node-8t5df 3/3 Running 0 84m
juicefs-csi-node-lkgl2 3/3 Running 0 84m
juicefs-csi-node-lsd4p 3/3 Running 0 84m
juicefs-csi-node-slrlb 3/3 Running 0 84m
juicefs-csi-node-sw8xn 3/3 Running 0 84m
juicefs-csi-node-t85hs 3/3 Running 0 84m
juicefs-csi-node-txdqp 3/3 Running 0 84m
juicefs-csi-node-vcvnx 3/3 Running 0 84m

juicefs-csi-node-slrlb is deployed on the machine where the pod is scheduled

YaAYadeer · 2024-11-22T09:35:48Z

Here are some details:
1.The version of 2.I1122 08:05:28.125702 I1122 08:05:28.138855 I1122 08:05:28.167414 I1122 08:05:28.167459 I1122 08:05:28.188651 3. 7 grace.go:56] "grace: Serve gracefully shutdown is listening" addr="/tmp/juicefs-csi-shutdown.sock" 7 node.go:207] "main: Pod Reconciler Started" 7 driver.go:53] "driver: get version info" driver="csi.juicefs.com" verison="v0.25.2-dirty" commit="95f5b7ca4e2554b05c54604126d27471e539beb5" date="2024-11-04T07:56:36Z" 7 driver.go:118] "driver: Listening for connection on address" address="/csi/csi.sock" slate">kubectl logs -n kube-system juicefs-csi-node-slrlb node-driver-registrar --tail 50 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded 1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock" 1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock 1 node_register.go:141] Calling node registrar to check if it still responds 1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded 1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock" 1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock 1 node_register.go:141] Calling node registrar to check if it still responds 1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded 1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock" 1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock 1 node_register.go:141] Calling node registrar to check if it still responds 1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded 1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock" 1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock 1 node_register.go:141] Calling node registrar to check if it still responds 1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded 1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock" 1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock 1 node_register.go:141] Calling node registrar to check if it still responds 1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo 1 connection.go:243] GRPC request: {} 1 main.go:90] Received GetInfo call: &InfoRequest{} 1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"} 1 connection.go:250] GRPC error: 1 node_register.go:106] health check succeeded

zwwhdls · 2024-11-25T02:28:02Z

Can you check if the CSI Node started(1122 08:05:28.125702) after application pod? Also, can you provide the log of kubelet? @YaAYadeer

YaAYadeer · 2024-11-25T03:13:03Z

Yes,the CSI Node is running
kubectl -n kube-system get pods -l app.kubernetes.io/name=juicefs-csi-driver
juicefs-csi-node-8w4p9 3/3 Running 0

kubectl logs -n kube-system juicefs-csi-node-8w4p9 -c liveness-probe --tail 100

# kubectl logs -n kube-system juicefs-csi-node-8w4p9  -c  liveness-probe   --tail 100
I1125 03:00:48.507406       1 main.go:133] "Calling CSI driver to discover driver name"
I1125 03:00:48.511523       1 main.go:141] "CSI driver name" driver="csi.juicefs.com"
I1125 03:00:48.511547       1 main.go:170] "ServeMux listening" address="0.0.0.0:9909"

kubectl logs -n kube-system juicefs-csi-node-8w4p9 -c node-driver-registrar --tail 100

I1125 03:05:24.993838       1 node_register.go:106] health check succeeded
I1125 03:05:34.993064       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:05:34.993117       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:05:34.993536       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:05:34.993557       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:05:34.993562       1 connection.go:243] GRPC request: {}
I1125 03:05:34.993747       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:05:34.993961       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:05:34.993978       1 connection.go:250] GRPC error: <nil>
I1125 03:05:34.994078       1 node_register.go:106] health check succeeded
I1125 03:05:44.992503       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:05:44.992556       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:05:44.993518       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:05:44.993553       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:05:44.993557       1 connection.go:243] GRPC request: {}
I1125 03:05:44.994204       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:05:44.994554       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:05:44.994587       1 connection.go:250] GRPC error: <nil>
I1125 03:05:44.994750       1 node_register.go:106] health check succeeded
I1125 03:05:54.993007       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:05:54.993057       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:05:54.993533       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:05:54.993549       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:05:54.993553       1 connection.go:243] GRPC request: {}
I1125 03:05:54.993754       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:05:54.993937       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:05:54.993970       1 connection.go:250] GRPC error: <nil>
I1125 03:05:54.994072       1 node_register.go:106] health check succeeded
I1125 03:06:04.993087       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:04.993135       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:04.993585       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:04.993600       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:04.993604       1 connection.go:243] GRPC request: {}
I1125 03:06:04.993787       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:04.993940       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:04.993951       1 connection.go:250] GRPC error: <nil>
I1125 03:06:04.994031       1 node_register.go:106] health check succeeded
I1125 03:06:14.992564       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:14.992609       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:14.993075       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:14.993098       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:14.993103       1 connection.go:243] GRPC request: {}
I1125 03:06:14.993316       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:14.993500       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:14.993510       1 connection.go:250] GRPC error: <nil>
I1125 03:06:14.993605       1 node_register.go:106] health check succeeded
I1125 03:06:24.992030       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:24.992076       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:24.993176       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:24.993229       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:24.993249       1 connection.go:243] GRPC request: {}
I1125 03:06:24.993708       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:24.994006       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:24.994024       1 connection.go:250] GRPC error: <nil>
I1125 03:06:24.994114       1 node_register.go:106] health check succeeded
I1125 03:06:34.992651       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:34.992711       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:34.993269       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:34.993287       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:34.993291       1 connection.go:243] GRPC request: {}
I1125 03:06:34.993524       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:34.993646       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:34.993656       1 connection.go:250] GRPC error: <nil>
I1125 03:06:34.993726       1 node_register.go:106] health check succeeded
I1125 03:06:44.993036       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:44.993084       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:44.993545       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:44.993574       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:44.993579       1 connection.go:243] GRPC request: {}
I1125 03:06:44.993753       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:44.993937       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:44.993951       1 connection.go:250] GRPC error: <nil>
I1125 03:06:44.994028       1 node_register.go:106] health check succeeded
I1125 03:06:54.992465       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:06:54.992515       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:06:54.993031       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:06:54.993055       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:06:54.993061       1 connection.go:243] GRPC request: {}
I1125 03:06:54.993304       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:06:54.993493       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:06:54.993511       1 connection.go:250] GRPC error: <nil>
I1125 03:06:54.993566       1 node_register.go:106] health check succeeded
I1125 03:07:04.992619       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:07:04.992666       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:07:04.993556       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:07:04.993590       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:07:04.993600       1 connection.go:243] GRPC request: {}
I1125 03:07:04.994273       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:07:04.994464       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:07:04.994483       1 connection.go:250] GRPC error: <nil>
I1125 03:07:04.994575       1 node_register.go:106] health check succeeded
I1125 03:07:14.992962       1 node_register.go:133] Attempting to open a gRPC connection with: "/registration/csi.juicefs.com-reg.sock"
I1125 03:07:14.993018       1 connection.go:213] Connecting to unix:///registration/csi.juicefs.com-reg.sock
I1125 03:07:14.993569       1 node_register.go:141] Calling node registrar to check if it still responds
I1125 03:07:14.993584       1 connection.go:242] GRPC call: /pluginregistration.Registration/GetInfo
I1125 03:07:14.993589       1 connection.go:243] GRPC request: {}
I1125 03:07:14.993759       1 main.go:90] Received GetInfo call: &InfoRequest{}
I1125 03:07:14.993942       1 connection.go:249] GRPC response: {"endpoint":"/var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock","name":"csi.juicefs.com","supported_versions":["1.0.0"],"type":"CSIPlugin"}
I1125 03:07:14.993964       1 connection.go:250] GRPC error: <nil>
I1125 03:07:14.994075       1 node_register.go:106] health check succeeded

4.kubectl logs -n kube-system juicefs-csi-node-8w4p9 -c juicefs-plugin --tail 100

I1125 03:00:48.122576       7 main.go:127] "main: Run CSI node"
I1125 03:00:48.137262       7 grace.go:56] "grace: Serve gracefully shutdown is listening" addr="/tmp/juicefs-csi-shutdown.sock"
I1125 03:00:48.170080       7 node.go:207] "main: Pod Reconciler Started"
I1125 03:00:48.170129       7 driver.go:53] "driver: get version info" driver="csi.juicefs.com" verison="v0.25.2-dirty" commit="95f5b7ca4e2554b05c54604126d27471e539beb5" date="2024-11-04T07:56:36Z"
I1125 03:00:48.189343       7 driver.go:118] "driver: Listening for connection on address" address="/csi/csi.sock"

kubectl describe csinode 10.0.68.xxx 10.0.68.xxx is the node that pod on
but I find there exists no csi.juicefs.com in the list

Name:               10.0.68.xxx
Labels:             <none>
Annotations:        storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/cinder
CreationTimestamp:  Mon, 25 Sep 2023 09:56:32 +0800
Spec:
  Drivers:
    com.ksc.csi.nfsplugin:
      Node ID: xxx
    com.ksc.csi.diskplugin:
      Node ID:  xxx
      Allocatables:
        Count:        8
      Topology Keys:  [failure-domain.beta.kubernetes.io/region failure-domain.beta.kubernetes.io/zone]
    alluxio:
      Node ID:  10.0.68.xx
    fuse.csi.fluid.io:
      Node ID:  10.0.68.xx
Events:         <none>

YaAYadeer · 2024-11-25T03:15:39Z

Besides, when I use kubectl get csidrivers
the result contains juicefs
csi.juicefs.com false true false false Persistent 13m

zwwhdls · 2024-11-25T03:29:35Z

Can you provide the log of kubelet?

YaAYadeer · 2024-11-25T03:42:59Z

I grep juice of the kubelet log

Nov 25 11:13:34 vm10-0-68-194 kubelet: E1125 11:13:34.728567    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:15:36.728549051 +0800 CST m=+20475190.442266282 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:14:10 vm10-0-68-194 kubelet: E1125 11:14:10.384959    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[default-token-fhgzq data]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:14:10 vm10-0-68-194 kubelet: E1125 11:14:10.385006    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[default-token-fhgzq data]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2
Nov 25 11:15:36 vm10-0-68-194 kubelet: E1125 11:15:36.829167    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:17:38.829148989 +0800 CST m=+20475312.542866216 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:16:27 vm10-0-68-194 kubelet: E1125 11:16:27.386126    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:16:27 vm10-0-68-194 kubelet: E1125 11:16:27.386192    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2
Nov 25 11:17:38 vm10-0-68-194 kubelet: E1125 11:17:38.891685    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:19:40.891667034 +0800 CST m=+20475434.605384348 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:18:41 vm10-0-68-194 kubelet: E1125 11:18:41.384286    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:18:41 vm10-0-68-194 kubelet: E1125 11:18:41.384312    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2
Nov 25 11:19:40 vm10-0-68-194 kubelet: E1125 11:19:40.913138    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:21:42.91311729 +0800 CST m=+20475556.626834519 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:20:57 vm10-0-68-194 kubelet: E1125 11:20:57.385098    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[default-token-fhgzq data]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:20:57 vm10-0-68-194 kubelet: E1125 11:20:57.385120    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[default-token-fhgzq data]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2

zwwhdls · 2024-11-25T03:50:19Z

Can you grep the log around the time CSI Node started?

YaAYadeer · 2024-11-25T04:09:19Z

The status of the csi node is normal, can the sign of successful CSI registration not be judged only by get drivers? This problem really bothers me, because it hinders my other work. Could you please help me check the problem

YaAYadeer · 2024-11-25T04:28:08Z

grep  "Nov 25 11:0" /var/log/messages | grep fs
[root@vm10-0-68-194 ~]# grep  "Nov 25 11:0" /var/log/messages | grep fs
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067679    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"jfs-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-jfs-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067722    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-device-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067745    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"jfs-fuse-fd\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-jfs-fuse-fd\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067767    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-plugin-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067788    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"juicefs-config\" (UniqueName: \"kubernetes.io/configmap/554eeeab-05c5-4cf6-b32c-e5c80d9862db-juicefs-config\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067819    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"juicefs-csi-node-sa-token-c7kks\" (UniqueName: \"kubernetes.io/secret/554eeeab-05c5-4cf6-b32c-e5c80d9862db-juicefs-csi-node-sa-token-c7kks\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067846    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-registration-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067864    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-kubelet-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:45 vm10-0-68-194 kubelet: I1125 11:00:45.067928    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"jfs-root-dir\" (UniqueName: \"kubernetes.io/host-path/554eeeab-05c5-4cf6-b32c-e5c80d9862db-jfs-root-dir\") pod \"juicefs-csi-node-8w4p9\" (UID: \"554eeeab-05c5-4cf6-b32c-e5c80d9862db\") "
Nov 25 11:00:47 vm10-0-68-194 kubelet: I1125 11:00:47.903764   89244 event.go:278] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"juicef-csi-node-8w4p9", UID:"554eeeab-05c5-4cf6-b32c-e5c80d9862db", APIVersion:"v1", ResourceVersion:"386156630", FieldPath:""}): type: 'Normal' reason: 'AddedInterface' Add eth0 [10.1.3.54/25]
Nov 25 11:01:03 vm10-0-68-194 kubelet: I1125 11:01:03.028091    6584 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Nov 25 11:01:03 vm10-0-68-194 kubelet: E1125 11:01:03.051076    6584 goroutinemap.go:150] Operation for "/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock" failed. No retries permitted until 2024-11-25 11:03:05.051045171 +0800 CST m=+20474438.764762432 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Nov 25 11:03:05 vm10-0-68-194 kubelet: I1125 11:03:05.117096    6584 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Nov 25 11:03:05 vm10-0-68-194 kubelet: E1125 11:03:05.126105    6584 goroutinemap.go:150] Operation for "/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock" failed. No retries permitted until 2024-11-25 11:05:07.126065001 +0800 CST m=+20474560.839782291 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Nov 25 11:05:07 vm10-0-68-194 kubelet: I1125 11:05:07.197221    6584 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Nov 25 11:05:07 vm10-0-68-194 kubelet: E1125 11:05:07.198427    6584 goroutinemap.go:150] Operation for "/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock" failed. No retries permitted until 2024-11-25 11:07:09.198403475 +0800 CST m=+20474682.912120722 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Nov 25 11:05:20 vm10-0-68-194 kubelet: I1125 11:05:20.714583    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-token-fhgzq\" (UniqueName: \"kubernetes.io/secret/6c5b92ba-6993-48be-b66e-d66b30e59fc2-default-token-fhgzq\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") "
Nov 25 11:05:20 vm10-0-68-194 kubelet: I1125 11:05:20.714631    6584 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") "
Nov 25 11:05:20 vm10-0-68-194 kubelet: E1125 11:05:20.816283    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:21.316262591 +0800 CST m=+20474575.029979819 (durationBeforeRetry 500ms). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:21 vm10-0-68-194 kubelet: E1125 11:05:21.321497    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:22.321478189 +0800 CST m=+20474576.035195416 (durationBeforeRetry 1s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:22 vm10-0-68-194 kubelet: E1125 11:05:22.332095    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:24.332077266 +0800 CST m=+20474578.045794495 (durationBeforeRetry 2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:24 vm10-0-68-194 kubelet: E1125 11:05:24.354127    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:28.354108341 +0800 CST m=+20474582.067825569 (durationBeforeRetry 4s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:28 vm10-0-68-194 kubelet: E1125 11:05:28.394604    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:36.39458746 +0800 CST m=+20474590.108304688 (durationBeforeRetry 8s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:36 vm10-0-68-194 kubelet: E1125 11:05:36.467745    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:05:52.467727668 +0800 CST m=+20474606.181444894 (durationBeforeRetry 16s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:05:52 vm10-0-68-194 kubelet: E1125 11:05:52.556459    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:06:24.556434716 +0800 CST m=+20474638.270151948 (durationBeforeRetry 32s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:06:24 vm10-0-68-194 kubelet: E1125 11:06:24.572180    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:07:28.572155848 +0800 CST m=+20474702.285873084 (durationBeforeRetry 1m4s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:07:09 vm10-0-68-194 kubelet: I1125 11:07:09.279253    6584 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Nov 25 11:07:09 vm10-0-68-194 kubelet: E1125 11:07:09.280077    6584 goroutinemap.go:150] Operation for "/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock" failed. No retries permitted until 2024-11-25 11:09:11.280062148 +0800 CST m=+20474804.993779380 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Nov 25 11:07:23 vm10-0-68-194 kubelet: E1125 11:07:23.605911    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:07:23 vm10-0-68-194 kubelet: E1125 11:07:23.605968    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2
Nov 25 11:07:28 vm10-0-68-194 kubelet: E1125 11:07:28.597327    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:09:30.597301031 +0800 CST m=+20474824.311018268 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:09:11 vm10-0-68-194 kubelet: I1125 11:09:11.371218    6584 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock  <nil> 0 <nil>}] <nil> <nil>}
Nov 25 11:09:11 vm10-0-68-194 kubelet: E1125 11:09:11.372482    6584 goroutinemap.go:150] Operation for "/data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock" failed. No retries permitted until 2024-11-25 11:11:13.3724542 +0800 CST m=+20474927.086171455 (durationBeforeRetry 2m2s). Error: "RegisterPlugin error -- failed to get plugin info using RPC GetInfo at socket /data/kubelet/plugins_registry/com.ksc.csi.nfsplugin/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration"
Nov 25 11:09:30 vm10-0-68-194 kubelet: E1125 11:09:30.663491    6584 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/csi.juicefs.com^juicefs-pv podName: nodeName:}" failed. No retries permitted until 2024-11-25 11:11:32.66346434 +0800 CST m=+20474946.377181576 (durationBeforeRetry 2m2s). Error: "MountVolume.MountDevice failed for volume \"juicefs-pv\" (UniqueName: \"kubernetes.io/csi/csi.juicefs.com^juicefs-pv\") pod \"juicefs-app\" (UID: \"6c5b92ba-6993-48be-b66e-d66b30e59fc2\") : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name csi.juicefs.com not found in the list of registered CSI drivers"
Nov 25 11:09:38 vm10-0-68-194 kubelet: E1125 11:09:38.384707    6584 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app"
Nov 25 11:09:38 vm10-0-68-194 kubelet: E1125 11:09:38.384735    6584 pod_workers.go:190] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data default-token-fhgzq]: timed out waiting for the condition" pod="default/juicefs-app" podUID=6c5b92ba-6993-48be-b66e-d66b30e59fc2

zwwhdls · 2024-11-25T05:47:33Z

No useful information can be obtained from the logs you provided, because you grep fs. The info is as the follows:

As you described, the CSI Node pod runs well on the node that application pod runs on;
The node did not get the csi driver registration.

Let’s recheck the environment. Be sure to run the exact command:

get CSI Node pod on the node: kubectl -n kube-system get po -owide | grep 10.0.68.xxx
check the kubelet log to see if the csi driver has registered: grep "Nov 25 11:0" /var/log/messages

The normal kubelet should contain kubernetes.io/csi: Register new plugin with name: csi.juicefs.com at endpoint: /var/lib/kubelet/csi-plugins/csi.juicefs.com/csi.sock

YaAYadeer · 2024-11-25T06:07:01Z

kubectl -n kube-system get po -owide | grep 10.0.68.194
csi-diskplugin-kvlnh 3/3 Running 7 427d 10.0.68.194 10.0.68.194
filebeat-kcde-s8vnx 1/1 Running 0 89d 10.0.68.194 10.0.68.194
ip-masq-agent-sd7gv 1/1 Running 2 427d 10.0.68.194 10.0.68.194
juicefs-csi-node-8w4p9 3/3 Running 0 179m 10.1.3.54 10.0.68.194
ksc-metrics-pusher-0 1/1 Running 0 89d 10.1.3.86 10.0.68.194
ksc-node-exporter-nwmbh 1/1 Running 0 111d 10.0.68.194 10.0.68.194
kube-flannel-5wzt4 2/2 Running 0 89d 10.0.68.194 10.0.68.194
kube-multus-ds-amd64-8p9bc 1/1 Running 2 427d 10.0.68.194 10.0.68.194
kube-proxy-g9l2z 1/1 Running 0 103d 10.0.68.194 10.0.68.194
kwok-controller-57b7ddc6ff-pc54t 1/1 Running 1 124d 10.1.3.9 10.0.68.194
The kubectl log does not contain the item "Register new plugin"
[root@vm10-0-68-194 ~]# grep "Register new plugin" /var/log/messages
[root@vm10-0-68-194 ~]#
sock file alread exists

[root@vm10-0-68-194 ~]# ll /var/lib/kubelet/csi-plugins/csi.juicefs.com/
total 0
srwxr-xr-x 1 root root 0 Nov 25 11:00 csi.sock
[root@vm10-0-68-194 ~]# ll /var/lib/kubelet/plugins_registry/
total 0
srwx------ 1 root root 0 Nov 25 11:00 csi.juicefs.com-reg.sock

YaAYadeer · 2024-11-25T06:51:27Z

I use helm to deploy the pod, I must use kubectl so that the log contains item "Register new plugin with name: csi.juicefs.com" ?

YaAYadeer · 2024-11-25T08:10:27Z

@zwwhdls

zwwhdls · 2024-11-26T02:09:40Z

Please check Mount Pod, see: https://juicefs.com/docs/csi/troubleshooting/#check-mount-pod

closing

YaAYadeer added the kind/bug Something isn't working label Nov 22, 2024

zwwhdls closed this as completed Nov 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] driver name csi.juicefs.com not found in the list of registered CSI drivers #1191

[BUG] driver name csi.juicefs.com not found in the list of registered CSI drivers #1191

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading

YaAYadeer commented Nov 25, 2024

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

zwwhdls commented Nov 26, 2024

[BUG] driver name csi.juicefs.com not found in the list of registered CSI drivers #1191

[BUG] driver name csi.juicefs.com not found in the list of registered CSI drivers #1191

Comments

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

YaAYadeer commented Nov 22, 2024

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 • edited by zwwhdls Loading

YaAYadeer commented Nov 25, 2024

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 • edited by zwwhdls Loading

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024 • edited by zwwhdls Loading

zwwhdls commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

YaAYadeer commented Nov 25, 2024

zwwhdls commented Nov 26, 2024

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading

YaAYadeer commented Nov 25, 2024 •

edited by zwwhdls

Loading