Creating a new local cluster fails as minio pods crashloops

[General-Meow]

Which version of kubefirst are you using?

kubefirst-cli golang utility version: v2.3.7

Which cloud provider?

k3d (local)

Which DNS?

None specific

Which installation type?

CLI

Which distributed Git provider?

GitHub

Did you use a fork of gitops-template?

Yes

Which Operating System?

Linux

What is the issue?

I was attempting to setup the cluster by following the steps in https://docs.kubefirst.io/k3d/quick-start/install first off i diverged slightly from the steps in that I exported my personal github token so no temp one was need. Also, my specs on my setup don't quite meet the requirements (4 thread 11th gen Intel CPU nuc and on a 40meg internet connection, memory, storage and swap are met though) so please take that into account when looking at this.

Running the command kubefirst k3d create goes on well until the output becomes:

$ kubefirst k3d create
------------------------------------------------
Follow your logs in a new terminal with:
   tail -f -n +1 /home/paul/.k1/logs/log_1704730580.log
------------------------------------------------
Running preflight checks                ... done! [5 in 823ms]
Cloning and formatting git repositories ... done! [1 in 0s]
Applying github Terraform               ... done! [1 in 0s]
Pushing git repositories                ... done! [1 in 0s]
Creating k3d cluster                    ... done! [1 in 0s]
Bootstrapping Kubernetes resources      ... done! [2 in 22ms]
Verifying Kubernetes cluster is ready   ... done! [3 in 1m20.049s]
Installing and configuring Argo CD      ... done! [3 in 2m30.101s]
Configuring Vault                       ...  0.0% [......................................] [0 in 2m0.002981s]
[##############################################################################..........] [5m51s]
Error: the StatefulSet was not created within the timeout period

If a detailed error message was available, please make the necessary corrections before retrying.
You can re-run the last command to try the operation again.

The last few lines of the log looks like this:

2024-01-08T15:54 INF cmd/k3d/create.go:926 > Setting argocd username and password credentials
2024-01-08T15:54 INF cmd/k3d/create.go:939 > argocd username and password credentials set successfully
2024-01-08T15:54 INF cmd/k3d/create.go:940 > Getting an argocd auth token
2024-01-08T15:54 INF pkg/argocd/auth.go:75 > using argocd url https://argocd.kubefirst.dev
2024-01-08T15:54 INF cmd/k3d/create.go:978 > argocd admin auth token set
2024-01-08T15:54 INF cmd/k3d/create.go:1010 > applying the registry application to argocd
2024-01-08T15:54 INF pkg/k8s/exec.go:276 > waiting for vault StatefulSet to be created using label app.kubernetes.io/instance=vault
2024-01-08T15:56 ERR pkg/k8s/exec.go:306 > the StatefulSet was not created within the timeout period
2024-01-08T15:56 ERR cm d/k3d/create.go:1035 > Error finding Vault StatefulSet: the StatefulSet was not created within the timeout period

Getting help from Fred we first looked at the status of the pods with kuibectl get pods -A which output:

$ kubectl get pods -A
NAMESPACE      NAME                                                READY   STATUS             RESTARTS       AGE
kube-system    coredns-7c444649cb-xpsdw                            1/1     Running            0              19m
kube-system    metrics-server-7b67f64457-9nwk5                     1/1     Running            0              19m
kube-system    local-path-provisioner-5d56847996-mh4br             1/1     Running            0              19m
kube-system    helm-install-traefik-crd-sr2bm                      0/1     Completed          0              19m
kube-system    helm-install-traefik-z4mg4                          0/1     Completed          1              19m
kube-system    svclb-traefik-131738d7-fsc9v                        2/2     Running            0              18m
kube-system    svclb-traefik-131738d7-mztwt                        2/2     Running            0              18m
kube-system    svclb-traefik-131738d7-242x7                        2/2     Running            0              18m
kube-system    svclb-traefik-131738d7-rks5d                        2/2     Running            0              18m
kube-system    traefik-56b8c5fb5c-q47h6                            1/1     Running            0              18m
argocd         argocd-redis-ha-haproxy-68cc86d566-xbvs6            1/1     Running            0              16m
argocd         argocd-applicationset-controller-84656db669-m9dp7   1/1     Running            0              16m
argocd         argocd-notifications-controller-5fd999cb79-krqgc    1/1     Running            0              16m
argocd         argocd-repo-server-7d86c96cd9-cgvv4                 1/1     Running            0              16m
argocd         argocd-server-7d6b8895b7-lxh5n                      1/1     Running            0              16m
argocd         argocd-application-controller-0                     1/1     Running            0              16m
argocd         argocd-dex-server-6df79688bc-g25bz                  1/1     Running            0              16m
argocd         argocd-server-7d6b8895b7-2w62k                      1/1     Running            0              16m
argocd         argocd-repo-server-7d86c96cd9-scz9g                 1/1     Running            0              16m
argocd         argocd-redis-ha-haproxy-68cc86d566-h89gv            1/1     Running            0              16m
argocd         argocd-redis-ha-server-0                            3/3     Running            0              16m
argocd         argocd-redis-ha-haproxy-68cc86d566-klm54            1/1     Running            0              16m
argocd         argocd-redis-ha-server-1                            3/3     Running            0              13m
cert-manager   cert-manager-cainjector-56bbdd5c47-thdtr            1/1     Running            0              11m
argocd         argocd-redis-ha-server-2                            3/3     Running            0              12m
cert-manager   cert-manager-64f9f45d6f-npc6x                       1/1     Running            0              11m
cert-manager   cert-manager-webhook-d4f4545d7-vkkmb                1/1     Running            0              11m
minio          minio-76cbb484b5-pmtrz                              1/1     Running            0              11m
minio          minio-make-bucket-job-qkn7f                         0/1     CrashLoopBackOff   5 (118s ago)   10m
minio          minio-make-user-job-bgx6w                           0/1     CrashLoopBackOff   5 (98s ago)    10m

which shows the minio related pods crashlooping. They do go into the Running but then go back into crashloop. Getting the logs from the pods have the following printed multiple times

mc: <ERROR> Unable to initialize new alias from the provided credentials. Server not initialized, please try again.
"Failed attempts: 30"

Fred has suggested that running the command again might fix it. It initially had the same issue but coming back to my terminal (6 hours later) I can see that all the pods are in the running state. So it might be the case of increasing the timeouts

Code of Conduct

• I agree to follow this project's Code of Conduct

[fharper]

Thanks for creating this issue @General-Meow (love the username!)

From some research, it may means that in some situation, MinIO need more time to be ready for the mc command, so we should add a sleep command to give it some time. See minio/mc#3599