Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Qdrant cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5678

Closed
ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5695
Closed

[BUG]Qdrant cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5678

ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5695
Assignees
@free6om
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

ahjing99 commented Oct 30, 2023 •
edited
Loading

  1. Install kb 0.6.4
  2. create clusters
     `kbcli cluster create  qdrant-osglbj             --termination-policy=WipeOut             --monitoring-interval=0 --enable-all-logs=false  --cluster-definition=qdrant --set cpu=100m,memory=0.5Gi,replicas=2,storage=1Gi  --namespace default `

Info: --cluster-version is not specified, ClusterVersion qdrant-1.1.0 is applied by default
Cluster qdrant-osglbj created

➜  ~ k get cluster
NAME              CLUSTER-DEFINITION   VERSION              TERMINATION-POLICY   STATUS    AGE
qdrant-osglbj     qdrant               qdrant-1.1.0         WipeOut              Running   7m49s
  1. Upgrade kb to 0.7
➜  ~ kbcli kubeblocks upgrade --version 0.7.0-beta.13
Current KubeBlocks version 0.6.4.
Kubernetes version 1.27.3
Kubernetes provider GKE
kbcli version 0.7.0-beta.13
Upgrade KubeBlocks from 0.6.4 to 0.7.0-beta.13
Please type 'Yes/yes' to confirm your operation: yes
Add and update repo kubeblocks                     OK
Stop KubeBlocks 0.6.4                              OK
Stop DataProtection                                OK
⢿ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes in /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
⣽ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes successfully, remove /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
Upgrading KubeBlocks to 0.7.0-beta.13              OK

KubeBlocks has been upgraded to 0.7.0-beta.13 SUCCESSFULLY!

-> Basic commands for cluster:
    kbcli cluster create -h     # help information about creating a database cluster
    kbcli cluster list          # list all database clusters
    kbcli cluster describe <cluster name>  # get cluster information

-> Uninstall KubeBlocks:
    kbcli kubeblocks uninstall
  1. Qdrant cluster is always updating for "create Pod qdrant-osglbj-qdrant-1 in StatefulSet qdrant-osglbj-qdrant failed error: Pod "qdrant-osglbj-qdrant-1" is invalid: spec.containers[0].image: Required value"
➜  ~ kbcli cluster describe qdrant-osglbj
Name: qdrant-osglbj	 Created Time: Oct 30,2023 16:40 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION        STATUS     TERMINATION-POLICY
default     qdrant               qdrant-1.1.0   Updating   WipeOut

Endpoints:
COMPONENT   MODE        INTERNAL                                              EXTERNAL
qdrant      ReadWrite   qdrant-osglbj-qdrant.default.svc.cluster.local:6333   <none>
                        qdrant-osglbj-qdrant.default.svc.cluster.local:6334
                        qdrant-osglbj-qdrant.default.svc.cluster.local:3000

Topology:
COMPONENT   INSTANCE                 ROLE     STATUS    AZ              NODE                                                CREATED-TIME
qdrant      qdrant-osglbj-qdrant-0   <none>   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28   Oct 30,2023 16:40 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
qdrant      false       100m / 100m          512Mi / 512Mi           data:1Gi       kb-default-sc

Images:
COMPONENT   TYPE     IMAGE
qdrant      qdrant   docker.io/qdrant/qdrant:latest

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default qdrant-osglbj

➜  ~ k describe cluster qdrant-osglbj
Name:         qdrant-osglbj
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=qdrant
              clusterversion.kubeblocks.io/name=qdrant-1.1.0
Annotations:  kubeblocks.io/reconcile: 2023-10-30T09:17:29.323446038Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-30T08:40:31Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:affinity:
          .:
          f:nodeLabels:
          f:podAntiAffinity:
          f:tenancy:
          f:topologyKeys:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"qdrant"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
        f:tolerations:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-30T08:40:31Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:qdrant:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:      manager
    Operation:    Update
    Subresource:  status
    Time:         2023-10-30T08:55:36Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
    Manager:         manager
    Operation:       Update
    Time:            2023-10-30T09:17:29Z
  Resource Version:  180712
  UID:               eba115e5-eb98-4af9-b3db-fbec91b0db9f
Spec:
  Affinity:
    Node Labels:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
  Cluster Definition Ref:  qdrant
  Cluster Version Ref:     qdrant-1.1.0
  Component Specs:
    Component Def Ref:  qdrant
    Monitor:            false
    Name:               qdrant
    No Create PDB:      false
    Replicas:           2
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-qdrant-osglbj
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:   1Gi
  Termination Policy:  WipeOut
  Tolerations:
Status:
  Cluster Def Generation:  2
  Components:
    Qdrant:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-10-30T08:41:30Z
  Conditions:
    Last Transition Time:  2023-10-30T08:40:32Z
    Message:               The operator has started the provisioning of Cluster: qdrant-osglbj
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-30T08:55:36Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-30T08:55:36Z
    Message:               pods are not ready in Components: [qdrant], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-30T08:55:36Z
    Message:               pods are unavailable in Components: [qdrant], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Updating
Events:
  Type     Reason                    Age                 From                Message
  ----     ------                    ----                ----                -------
  Normal   ComponentPhaseTransition  41m                 cluster-controller  Create a new component
  Normal   PreCheckSucceed           41m                 cluster-controller  The operator has started the provisioning of Cluster: qdrant-osglbj
  Normal   ApplyResourcesSucceed     41m                 cluster-controller  Successfully applied for resources
  Normal   ComponentPhaseTransition  40m                 cluster-controller  Running: true, PodsReady: true, PodsTimedout: false
  Normal   AllReplicasReady          40m                 cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              40m                 cluster-controller  Cluster: qdrant-osglbj is ready, current phase is Running
  Normal   Running                   40m                 cluster-controller  Cluster: qdrant-osglbj is ready, current phase is Running
  Warning  ApplyResourcesFailed      26m                 cluster-controller  BackupPolicy.dataprotection.kubeblocks.io "qdrant-osglbj-qdrant-backup-policy" is invalid: spec.backupMethods: Required value
  Normal   ComponentPhaseTransition  26m (x2 over 26m)   cluster-controller  component is Updating
  Normal   ApplyResourcesSucceed     26m                 cluster-controller  Successfully applied for resources
  Warning  ReplicasNotReady          26m                 cluster-controller  pods are not ready in Components: [qdrant], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        26m                 cluster-controller  pods are unavailable in Components: [qdrant], refer to related component message in Cluster.status.components
  Warning  FailedCreate              4m7s (x6 over 25m)  event-controller    create Pod qdrant-osglbj-qdrant-1 in StatefulSet qdrant-osglbj-qdrant failed error: Pod "qdrant-osglbj-qdrant-1" is invalid: spec.containers[0].image: Required value

2023-10-30T08:55:14.017Z	ERROR	UPDATE *v1alpha1.BackupPolicy error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"qdrant-osglbj","namespace":"default"}, "namespace": "default", "name": "qdrant-osglbj", "reconcileID": "da53249f-7afb-4624-afe7-4bfb99599b2d", "cluster": {"name":"qdrant-osglbj","namespace":"default"}, "error": "BackupPolicy.dataprotection.kubeblocks.io \"qdrant-osglbj-qdrant-backup-policy\" is invalid: spec.backupMethods: Required value"}
github.com/apecloud/kubeblocks/controllers/apps.(*clusterPlanBuilder).defaultWalkFuncWithLogging
	/src/controllers/apps/cluster_plan_builder.go:244
github.com/apecloud/kubeblocks/pkg/controller/graph.(*DAG).WalkReverseTopoOrder
	/src/pkg/controller/graph/dag.go:168
github.com/apecloud/kubeblocks/controllers/apps.(*clusterPlan).Execute
	/src/controllers/apps/cluster_plan_builder.go:198
github.com/apecloud/kubeblocks/controllers/apps.(*ClusterReconciler).Reconcile
	/src/controllers/apps/cluster_controller.go:208
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226
2023-10-30T08:55:14.038Z	INFO	BackupPolicy.dataprotection.kubeblocks.io "qdrant-osglbj-qdrant-backup-policy" is invalid: spec.backupMethods: Required value	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"qdrant-osglbj","namespace":"default"}, "namespace": "default", "name": "qdrant-osglbj", "reconcileID": "da53249f-7afb-4624-afe7-4bfb99599b2d", "cluster": {"name":"qdrant-osglbj","namespace":"default"}}
2023-10-30T08:55:14.038Z	ERROR	Reconciler error	{"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"qdrant-osglbj","namespace":"default"}, "namespace": "default", "name": "qdrant-osglbj", "reconcileID": "da53249f-7afb-4624-afe7-4bfb99599b2d", "error": "BackupPolicy.dataprotection.kubeblocks.io \"qdrant-osglbj-qdrant-backup-policy\" is invalid: spec.backupMethods: Required value"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:324
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:226

➜ ~ k logs kubeblocks-6b9684fccc-8xphr -n kb-system >kb.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kb.txt
@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 30, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 30, 2023
@wangyelei
Copy link
Contributor

Normal SuccessfulDelete 6m13s statefulset-controller delete Pod qdrant-osglbj-qdrant-1 in StatefulSet qdrant-osglbj-qdrant successful
Warning FailedCreate 55s (x16 over 3m39s) statefulset-controller create Pod qdrant-osglbj-qdrant-1 in StatefulSet qdrant-osglbj-qdrant failed error: Pod "qdrant-osglbj-qdrant-1" is invalid: spec.containers[0].image: Required value

@wangyelei wangyelei assigned free6om and unassigned iziang and wangyelei Oct 30, 2023
@free6om free6om mentioned this issue Oct 31, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@free6om free6om

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: wrong cluster phase after upgrading to 0.7 apecloud/kubeblocks
5 participants
@iziang @free6om @wangyelei @ahjing99 @apecloud-bot