[BUG]单节点集群创建mysql实例失败

[skatexgzhao]

Describe the bug

创建mysql单节点实例的名称：

kbcli cluster create mycluster --cluster-definition mysql --set replicas=1 -n dbdemo

Info: --cluster-version is not specified, ClusterVersion mysql-8.0.33 is applied by default
Cluster mycluster created

报错如下：

kbcli cluster list -n dbdemo

NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME
mycluster dbdemo mysql mysql-8.0.33 Delete Failed Dec 02,2024 02:10 UTC-0500

pod的错误信息

kubectl get pods -A

NAMESPACE NAME READY STATUS RESTARTS AGE
dbdemo mycluster-mysql-0 3/4 CrashLoopBackOff 5 (110s ago) 4m49s
kb-system kb-addon-snapshot-controller-56b4cb447d-zc6s2 1/1 Running 474 (28h ago) 12d
kb-system kubeblocks-79f98f4df9-cbdr4 1/1 Running 3 (28h ago) 12d
kb-system kubeblocks-dataprotection-76c46975fc-9rxb6 1/1 Running 3 (28h ago) 12d
kube-system cilium-b88p8 1/1 Running 516 (28h ago) 12d
kube-system cilium-operator-86666d88cb-xdbz5 1/1 Running 413 (28h ago) 12d
kube-system coredns-5d78c9869d-xkccc 1/1 Running 1 (28h ago) 12d
kube-system coredns-5d78c9869d-zrnc5 1/1 Running 1 (28h ago) 12d
kube-system etcd-host-10-0-174-25 1/1 Running 2 (25h ago) 12d
kube-system kube-apiserver-host-10-0-174-25 1/1 Running 0 12d
kube-system kube-controller-manager-host-10-0-174-25 1/1 Running 5 (25h ago) 12d
kube-system kube-proxy-tdtqn 1/1 Running 1 (28h ago) 12d
kube-system kube-scheduler-host-10-0-174-25 1/1 Running 5 (25h ago) 12d
kube-system nfs-client-provisione-5d465b56d6-h8426 1/1 Running 0 25h

mysql实例内部报错

kubectl logs -f mycluster-mysql-0 -n dbdemo

Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init), init-syncer (init), init-xtrabackup (init)
2024-12-02T07:10:21Z INFO Initialize DB manager
2024-12-02T07:10:21Z DEBUG KB_WORKLOAD_TYPE ENV not set
2024-12-02T07:10:21Z INFO HTTPServer Starting HTTP Server
2024-12-02T07:10:21Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/datasync"]}
2024-12-02T07:10:21Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/rebuild", "/v1.0/start", "/v1.0/stop", "/v1.0/switchover"]}
2024-12-02T07:10:21Z INFO HTTPServer http server {"listen address": "0.0.0.0", "port": 3601}
2024-12-02T07:10:21Z INFO HA HA starting
2024-12-02T07:10:21Z INFO pinger Waiting for dns resolution to be ready
2024-12-02T07:10:21Z INFO pinger dns resolution is ready {"dns": "mycluster-mysql-0.mycluster-mysql-headless.dbdemo.svc"}
2024-12-02T07:10:22Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=mycluster,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql
2024-12-02T07:10:22Z INFO DCS-K8S podlist: 1
2024-12-02T07:10:22Z INFO DCS-K8S Leader configmap is not found {"configmap": "mycluster-mysql-leader"}
2024-12-02T07:10:22Z DEBUG HA cluster info {"cluster": {"ClusterCompName":"mycluster-mysql","Namespace":"dbdemo","Replicas":1,"HaConfig":{"DeleteMembers":null},"Leader":null,"Members":[{"Index":"","Name":"mycluster-mysql-0","Role":"","PodIP":"10.0.0.107","DBPort":"3306","SyncerPort":"3601","UID":"5a90f62b-5055-473f-a58c-33c5a490c908","ComponentName":"mysql","UseIP":false}],"Switchover":null,"Extra":null,"Resource":{"metadata":{"name":"mycluster","namespace":"dbdemo","uid":"c632052b-29c5-4a04-b248-d0fa5f242d82","resourceVersion":"2985213","generation":1,"creationTimestamp":"2024-12-02T07:10:12Z","labels":{"clusterdefinition.kubeblocks.io/name":"mysql","clusterversion.kubeblocks.io/name":"mysql-8.0.33"},"finalizers":["cluster.kubeblocks.io/finalizer"],"managedFields":[{"manager":"kbcli","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-12-02T07:10:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:affinity":{".":{},"f:podAntiAffinity":{},"f:tenancy":{}},"f:clusterDefinitionRef":{},"f:clusterVersionRef":{},"f:componentSpecs":{},"f:resources":{".":{},"f:cpu":{},"f:memory":{}},"f:storage":{".":{},"f:size":{}},"f:terminationPolicy":{}}}},{"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-12-02T07:10:12Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:finalizers":{".":{},"v:"cluster.kubeblocks.io/finalizer"":{}},"f:labels":{".":{},"f:clusterdefinition.kubeblocks.io/name":{},"f:clusterversion.kubeblocks.io/name":{}}}}},{"manager":"kubeblocks","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-12-02T07:10:13Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:clusterDefGeneration":{},"f:components":{".":{},"f:mysql":{".":{},"f:phase":{},"f:podsReady":{}}},"f:conditions":{},"f:observedGeneration":{},"f:phase":{}}},"subresource":"status"}]},"spec":{"clusterDefinitionRef":"mysql","clusterVersionRef":"mysql-8.0.33","terminationPolicy":"Delete","componentSpecs":[{"name":"mysql","componentDefRef":"mysql","enabledLogs":["auditlog","error","slow"],"replicas":1,"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"1","memory":"1Gi"}},"volumeClaimTemplates":[{"name":"data","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"20Gi"}}}}],"switchPolicy":{"type":"Noop"},"serviceAccountName":"kb-mycluster"}],"affinity":{"podAntiAffinity":"Preferred","tenancy":"SharedNode"},"resources":{"cpu":"0","memory":"0"},"storage":{"size":"0"},"monitor":{}},"status":{"observedGeneration":1,"phase":"Creating","components":{"mysql":{"phase":"Creating","podsReady":false}},"clusterDefGeneration":3,"conditions":[{"type":"ProvisioningStarted","status":"True","observedGeneration":1,"lastTransitionTime":"2024-12-02T07:10:12Z","reason":"PreCheckSucceed","message":"The operator has started the provisioning of Cluster: mycluster"},{"type":"ApplyResources","status":"True","observedGeneration":1,"lastTransitionTime":"2024-12-02T07:10:12Z","reason":"ApplyResourcesSucceed","message":"Successfully applied for resources"},{"type":"ReplicasReady","status":"False","lastTransitionTime":"2024-12-02T07:10:12Z","reason":"ReplicasNotReady","message":"pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components"},{"type":"Ready","status":"False","lastTransitionTime":"2024-12-02T07:10:12Z","reason":"ComponentsNotReady","message":"pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components"}]}}}}
2024-12-02T07:10:22Z INFO Hypervisor Starting Hypervisor
2024-12-02T07:10:22Z INFO Hypervisor Start DB Service {"command": "/usr/bin/bash -c mv /var/lib/mysql/plugin/audit_log.so /usr/lib64/mysql/plugin/\nrm -rf /var/lib/mysql/plugin\nchown -R mysql:root /var/lib/mysql\nskip_replica_start="OFF"\nif [ -f /var/lib/mysql/data/.restore_new_cluster ]; then\n skip_replica_start="ON"\nfi\ndocker-entrypoint.sh mysqld --server-id $(( ${KB_POD_NAME##*-} + 1)) \\n--plugin-load-add=rpl_semi_sync_source=semisync_source.so \\n--plugin-load-add=rpl_semi_sync_replica=semisync_replica.so \\n--plugin-load-add=audit_log=audit_log.so \\n--log-bin=/var/lib/mysql/binlog/mycluster-mysql-0-bin \\n--skip-replica-start=$skip_replica_start\n"}
2024-12-02T07:10:22Z INFO MySQL there is no leader member
2024-12-02T07:10:22Z INFO HA check if DB Service is running
2024-12-02T07:10:22Z INFO Hypervisor Starting watcher on dbService
2024-12-02T07:10:22Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"}
== DB ERR == mv: cannot stat '/var/lib/mysql/plugin/audit_log.so': No such file or directory
== DB ERR == chown: changing ownership of '/var/lib/mysql/log': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/binlog': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/auditlog': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/data': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql': Operation not permitted
== DB == 2024-12-02 07:10:22+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.33-1.el8 started.
== DB ERR == chown: changing ownership of '/var/lib/mysql/log': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/data/': Operation not permitted
2024-12-02T07:10:22Z INFO Hypervisor dbService exists {"state": "exit status 1"}
panic: dbService startup failed. Exiting...

goroutine 65 [running]:
github.com/apecloud/syncer/hypervisor.(*Watcher).Start(0xc000123d80)
/src/hypervisor/watcher.go:60 +0x3e8
created by github.com/apecloud/syncer/hypervisor.NewHypervisor in goroutine 1
/src/hypervisor/hypevisor.go:57 +0x10e

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[shanshanying]

Hi @skatexgzhao ,

1. what is the KB and Addons versions you are working on ?

kbcli version # get kbcli , kb version
kbcli addon list # show the list of Addons.

2. which storage class are you using?
   Noticed there are many permssion erros:

== DB ERR == chown: changing ownership of '/var/lib/mysql/log': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/binlog': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/auditlog': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/data': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql': Operation not permitted
== DB == 2024-12-02 07:10:22+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.33-1.el8 started.
== DB ERR == chown: changing ownership of '/var/lib/mysql/log': Operation not permitted
== DB ERR == chown: changing ownership of '/var/lib/mysql/data/': Operation not permitted

[shanshanying]

And you can report the bug using command:

kbcli report cluster <clusterName>  --with-logs

To pack info into a zp file and attach it while reporting an issue.