Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]Mongo cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5677

Closed
ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5695
Closed

[BUG]Mongo cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5677

ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5695
Assignees
@xuriwuyun @free6om
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@ahjing99
Copy link
Collaborator

  1. Install kb 0.6.4
  2. create clusters
   `kbcli cluster create  mongo-kroyiu             --termination-policy=DoNotTerminate             --monitoring-interval=0 --enable-all-logs=false  --cluster-definition=mongodb --cluster-version=mongodb-4.0 --set cpu=100m,memory=0.5Gi,replicas=3,storage=3Gi  --namespace default `

Cluster mongo-kroyiu created

➜  ~ k get cluster
NAME              CLUSTER-DEFINITION   VERSION              TERMINATION-POLICY   STATUS    AGE
mongo-kroyiu      mongodb              mongodb-4.0          DoNotTerminate       Running   11m
  1. Upgrade kb to 0.7
➜  ~ kbcli kubeblocks upgrade --version 0.7.0-beta.13
Current KubeBlocks version 0.6.4.
Kubernetes version 1.27.3
Kubernetes provider GKE
kbcli version 0.7.0-beta.13
Upgrade KubeBlocks from 0.6.4 to 0.7.0-beta.13
Please type 'Yes/yes' to confirm your operation: yes
Add and update repo kubeblocks                     OK
Stop KubeBlocks 0.6.4                              OK
Stop DataProtection                                OK
⢿ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes in /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
⣽ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes successfully, remove /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
Upgrading KubeBlocks to 0.7.0-beta.13              OK

KubeBlocks has been upgraded to 0.7.0-beta.13 SUCCESSFULLY!

-> Basic commands for cluster:
    kbcli cluster create -h     # help information about creating a database cluster
    kbcli cluster list          # list all database clusters
    kbcli cluster describe <cluster name>  # get cluster information

-> Uninstall KubeBlocks:
    kbcli kubeblocks uninstall
  1. Mongo cluster status is always updating
➜  ~ kbcli cluster describe mongo-kroyiu
Name: mongo-kroyiu	 Created Time: Oct 30,2023 16:36 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION       STATUS     TERMINATION-POLICY
default     mongodb              mongodb-4.0   Updating   DoNotTerminate

Endpoints:
COMPONENT   MODE        INTERNAL                                               EXTERNAL
mongodb     ReadWrite   mongo-kroyiu-mongodb.default.svc.cluster.local:27017   <none>

Topology:
COMPONENT   INSTANCE                 ROLE        STATUS    AZ              NODE                                                CREATED-TIME
mongodb     mongo-kroyiu-mongodb-0   primary     Running   us-central1-c   gke-yijing-default-pool-3e14ea35-hqtr/10.128.0.30   Oct 30,2023 16:57 UTC+0800
mongodb     mongo-kroyiu-mongodb-1   secondary   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-wg54/10.128.0.35   Oct 30,2023 16:56 UTC+0800
mongodb     mongo-kroyiu-mongodb-2   secondary   Running   us-central1-c   gke-yijing-default-pool-3e14ea35-hxpl/10.128.0.28   Oct 30,2023 16:56 UTC+0800

Resources Allocation:
COMPONENT   DEDICATED   CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE-SIZE   STORAGE-CLASS
mongodb     false       100m / 100m          512Mi / 512Mi           data:3Gi       kb-default-sc

Images:
COMPONENT   TYPE      IMAGE
mongodb     mongodb   infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mongo:4.0

Data Protection:
BACKUP-REPO   AUTO-BACKUP   BACKUP-SCHEDULE   BACKUP-METHOD   BACKUP-RETENTION

Show cluster events: kbcli cluster list-events -n default mongo-kroyiu

➜  ~ k describe cluster mongo-kroyiu
Name:         mongo-kroyiu
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=mongodb
              clusterversion.kubeblocks.io/name=mongodb-4.0
Annotations:  <none>
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-30T08:36:41Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:spec:
        .:
        f:affinity:
          .:
          f:nodeLabels:
          f:podAntiAffinity:
          f:tenancy:
          f:topologyKeys:
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"mongodb"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:serviceAccountName:
            f:volumeClaimTemplates:
        f:terminationPolicy:
        f:tolerations:
    Manager:      kbcli
    Operation:    Update
    Time:         2023-10-30T08:36:41Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-30T08:36:41Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:mongodb:
            .:
            f:consensusSetStatus:
              .:
              f:followers:
              f:leader:
                .:
                f:accessMode:
                f:name:
                f:pod:
            f:membersStatus:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-30T08:57:48Z
  Resource Version:  168712
  UID:               d6dbf193-cfbe-4eac-a4c9-1946b931b62c
Spec:
  Affinity:
    Node Labels:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
  Cluster Definition Ref:  mongodb
  Cluster Version Ref:     mongodb-4.0
  Component Specs:
    Component Def Ref:  mongodb
    Monitor:            false
    Name:               mongodb
    No Create PDB:      false
    Replicas:           3
    Resources:
      Limits:
        Cpu:     100m
        Memory:  512Mi
      Requests:
        Cpu:               100m
        Memory:            512Mi
    Service Account Name:  kb-mongo-kroyiu
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:   3Gi
  Termination Policy:  DoNotTerminate
  Tolerations:
Status:
  Cluster Def Generation:  2
  Components:
    Mongodb:
      Consensus Set Status:
        Followers:
          Access Mode:  Readonly
          Name:         secondary
          Pod:          mongo-kroyiu-mongodb-2
          Access Mode:  Readonly
          Name:         secondary
          Pod:          mongo-kroyiu-mongodb-1
        Leader:
          Access Mode:  ReadWrite
          Name:         primary
          Pod:          mongo-kroyiu-mongodb-0
      Members Status:
        Pod Name:  mongo-kroyiu-mongodb-0
        Role:
          Access Mode:  ReadWrite
          Can Vote:     true
          Is Leader:    true
          Name:         primary
        Pod Name:       mongo-kroyiu-mongodb-2
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         secondary
        Pod Name:       mongo-kroyiu-mongodb-1
        Role:
          Access Mode:  Readonly
          Can Vote:     true
          Is Leader:    false
          Name:         secondary
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-10-30T08:38:15Z
  Conditions:
    Last Transition Time:  2023-10-30T08:36:41Z
    Message:               The operator has started the provisioning of Cluster: mongo-kroyiu
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-30T08:56:46Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-30T08:55:39Z
    Message:               pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-30T08:55:39Z
    Message:               pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Updating
Events:
  Type     Reason                    Age                From                Message
  ----     ------                    ----               ----                -------
  Normal   PreCheckSucceed           38m                cluster-controller  The operator has started the provisioning of Cluster: mongo-kroyiu
  Warning  ApplyResourcesFailed      38m                cluster-controller  Operation cannot be fulfilled on pods "mongo-kroyiu-mongodb-0": the object has been modified; please apply your changes to the latest version and try again
  Normal   ComponentPhaseTransition  38m                cluster-controller  Create a new component
  Normal   ApplyResourcesSucceed     38m (x2 over 38m)  cluster-controller  Successfully applied for resources
  Normal   AllReplicasReady          36m                cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   Running                   36m                cluster-controller  Cluster: mongo-kroyiu is ready, current phase is Running
  Normal   WaitingForProbeSuccess    36m (x8 over 36m)  cluster-controller  Waiting for probe success
  Normal   ComponentPhaseTransition  36m                cluster-controller  Running: true, PodsReady: true, PodsTimedout: false
  Normal   ClusterReady              36m                cluster-controller  Cluster: mongo-kroyiu is ready, current phase is Running
  Warning  ApplyResourcesFailed      19m (x2 over 19m)  cluster-controller  BackupPolicy.dataprotection.kubeblocks.io "mongo-kroyiu-mongodb-backup-policy" is invalid: spec.backupMethods: Required value
  Normal   ComponentPhaseTransition  19m (x3 over 19m)  cluster-controller  component is Updating
  Warning  ReplicasNotReady          19m                cluster-controller  pods are not ready in Components: [mongodb], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        19m                cluster-controller  pods are unavailable in Components: [mongodb], refer to related component message in Cluster.status.components
  Normal   ApplyResourcesSucceed     18m (x2 over 19m)  cluster-controller  Successfully applied for resources
  Warning  ApplyResourcesFailed      18m                cluster-controller  Operation cannot be fulfilled on pods "mongo-kroyiu-mongodb-1": the object has been modified; please apply your changes to the latest version and try again

➜ ~ k logs kubeblocks-6b9684fccc-8xphr -n kb-system >kb.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kb.txt
@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 30, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 30, 2023
@xuriwuyun
Copy link
Contributor

image
image
all pods are ready, but rsm.status.InitReplicas is not equal to 3

@free6om free6om mentioned this issue Oct 31, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@xuriwuyun xuriwuyun

@free6om free6om

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: wrong cluster phase after upgrading to 0.7 apecloud/kubeblocks
4 participants
@xuriwuyun @free6om @ahjing99 @apecloud-bot