metadata_store_server_main.cc:236] Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error: [mysql-error-info='']) MetadataStore cannot be created with the given connection config.

[922tech]

I followed the exact instruction for installing kubeflow pipeline on version 1.9.1
After creation of the namespaces and pods:

NAME                                               READY   STATUS             RESTARTS         AGE
cache-deployer-deployment-55458bb9bf-d4vxf         1/1     Running            1 (16h ago)      21h
cache-server-568d98d56f-mxgck                      1/1     Running            34 (58s ago)     21h
metadata-envoy-deployment-b4bc46768-mmx8j          1/1     Running            1 (16h ago)      21h
metadata-grpc-deployment-65fc97b764-6h7k4          0/1     CrashLoopBackOff   99 (3m35s ago)   21h
metadata-writer-86b4945bd5-78tjx                   0/1     CrashLoopBackOff   57 (25s ago)     21h
minio-7c77bc59b8-sks9f                             1/1     Running            1 (16h ago)      21h
ml-pipeline-7f6bcb764-b9d4n                        0/1     Running            120 (6m8s ago)   21h
ml-pipeline-persistenceagent-d9c785cdb-hvxpj       0/1     CrashLoopBackOff  67 (4m56s ago)   21h
ml-pipeline-scheduledworkflow-6d5fc7cb6d-phftf     1/1     Running            1 (16h ago)      21h
ml-pipeline-ui-5785c9d69d-gf2mk                    1/1     Running            1 (107m ago)     21h
ml-pipeline-viewer-crd-5869ccf789-np2wx            1/1     Running            1 (16h ago)      21h
ml-pipeline-visualizationserver-65df54899f-9zzph   1/1     Running            1 (107m ago)     21h
mysql-758cd66576-4q56v                             1/1     Running            1 (16h ago)      21h
workflow-controller-5584bbc656-dlgmr               1/1     Running            2 (106m ago)     21h

As you see some pods do not run.
And I think the main cause of the problem is that metadata-grpc-deployment cannot start working and it crashes with this error log:

WARNING: Logging before InitGoogleLogging() is written to STDERR
F1105 08:28:41.229691     1 metadata_store_server_main.cc:236] Check failed: absl::OkStatus() == status (OK vs. INTERNAL: mysql_real_connect failed: errno: , error:  [mysql-error-info='']) MetadataStore cannot be created with the given connection config.
*** Check failure stack trace: ***

I don't know why on earth it cannot connect to mysql!
There was also a rather similar issue #8224

[922tech]

Deploying it in kind cluster did solve my problem.

[922tech]

But how can I run kind on multiple nodes?