backup mysql 8.0 fail on 1.15

[sekot]

Report

The backup fail on version 1.15 and mysql 8.0

More about the problem

│ xtrabackup +++ sort -r                                                                                                                                                             │
│ xtrabackup +++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary                                                                │
│ xtrabackup +++ cut -d : -f 2                                                                                                                                                       │
│ xtrabackup +++ cut -d . -f 1                                                                                                                                                       │
│ xtrabackup ++ FIRST_NODE=percona-cluster-pxc-0                                                                                                                                     │
│ xtrabackup ++ SKIP_FIRST_POD='|'                                                                                                                                                   │
│ xtrabackup ++ (( 2 > 1 ))                                                                                                                                                          │
│ xtrabackup ++ SKIP_FIRST_POD=percona-cluster-pxc-0                                                                                                                                 │
│ xtrabackup ++ grep wsrep_ready:ON:wsrep_connected:ON:wsrep_local_state_comment:Synced:wsrep_cluster_status:Primary                                                                 │
│ xtrabackup ++ cut -d . -f 1                                                                                                                                                        │
│ xtrabackup ++ cut -d : -f 2                                                                                                                                                        │
│ xtrabackup ++ grep -v percona-cluster-pxc-0                                                                                                                                        │
│ xtrabackup ++ /opt/percona/peer-list -on-start=/usr/bin/get-pxc-state -service=percona-cluster-pxc                                                                                 │
│ xtrabackup ++ tail -1                                                                                                                                                              │
│ xtrabackup ++ sort                                                                                                                                                                 │
│ xtrabackup + local NODE_NAME=percona-cluster-pxc-1                                                                                                                                 │
│ xtrabackup + '[' -z percona-cluster-pxc-1 ']'                                                                                                                                      │
│ xtrabackup + set +o errexit                                                                                                                                                        │
│ xtrabackup + log INFO 'Garbd was started'                                                                                                                                          │
│ xtrabackup 2024-09-26 16:24:53 [INFO] Garbd was started                                                                                                                            │
│ xtrabackup + garbd --address 'gcomm://percona-cluster-pxc-1.percona-cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567' --donor percona-cluster-pxc-1 --group percona-cluster-pxc - │
│ -options 'socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl_cipher=;pc.weight │
│ =0;' --sst xtrabackup-v2:10.244.1.62:4444/xtrabackup_sst//1 --recv-script=/usr/bin/run_backup.sh                                                                                   │
│ xtrabackup 2024-09-26 16:24:53.298  INFO: CRC-32C: using 64-bit x86 acceleration.                                                                                                  │
│ xtrabackup 2024-09-26 16:24:53.298  INFO: Read config:                                                                                                                             │
│ xtrabackup     daemon:      0                                                                                                                                                      │
│ xtrabackup     name:        garb                                                                                                                                                   │
│ xtrabackup     address:     gcomm://percona-cluster-pxc-1.percona-cluster-pxc?gmcast.listen_addr=tcp://0.0.0.0:4567                                                                │
│ xtrabackup     group:       percona-cluster-pxc                                                                                                                                    │
│ xtrabackup     sst:         xtrabackup-v2:10.244.1.62:4444/xtrabackup_sst//1                                                                                                       │
│ xtrabackup     donor:       percona-cluster-pxc-1                                                                                                                                  │
│ xtrabackup     options:     socket.ssl_ca=/etc/mysql/ssl-internal/ca.crt;socket.ssl_cert=/etc/mysql/ssl-internal/tls.crt;socket.ssl_key=/etc/mysql/ssl-internal/tls.key;socket.ssl │
│ _cipher=;pc.weight=0;; gcs.fc_limit=9999999; gcs.fc_factor=1.0; gcs.fc_single_primary=yes; socket.ssl=YES                                                                          │
│ xtrabackup     cfg:                                                                                                                                                                │
│ xtrabackup     log:                                                                                                                                                                │
│ xtrabackup     recv_script: /usr/bin/run_backup.sh                                                                                                                                 │
│ xtrabackup     workdir:                                                                                                                                                            │
│ xtrabackup                                                                                                                                                                         │
│ xtrabackup 2024-09-26 16:24:53.298  WARN: SSL compression is not effective. The option socket.ssl_compression is deprecated and will be removed in future releases.                │
│ xtrabackup 2024-09-26 16:24:53.298  WARN: Parameter 'socket.ssl_compression' is deprecated and will be removed in future versions                                                  │
│ xtrabackup 2024-09-26 16:24:53.302  INFO: protonet asio version 0                                                                                                                  │
│ xtrabackup 2024-09-26 16:24:53.303  INFO: Using CRC-32C for message checksums.
│ xtrabackup 2024-09-26 16:24:53.303  INFO: backend: asio                                                                                                                            │
│ xtrabackup 2024-09-26 16:24:53.303  INFO: gcomm thread scheduling priority set to other:0                                                                                          │
│ xtrabackup 2024-09-26 16:24:53.303  INFO: Fail to access the file (./gvwstate.dat) error (No such file or directory). It is possible if node is booting for first time or re-booti │
│ ng after a graceful shutdown                                                                                                                                                       │
│ xtrabackup 2024-09-26 16:24:53.303  INFO: Restoring primary-component from disk failed. Either node is booting for first time or re-booting after a graceful shutdown              │
│ xtrabackup 2024-09-26 16:24:53.303  INFO: GMCast version 0                                                                                                                         │
│ xtrabackup 2024-09-26 16:24:53.304  INFO: (dc45c1f8-9914, 'ssl://0.0.0.0:4567') listening at ssl://0.0.0.0:4567                                                                    │
│ xtrabackup 2024-09-26 16:24:53.304  INFO: (dc45c1f8-9914, 'ssl://0.0.0.0:4567') multicast: , ttl: 1                                                                                │
│ xtrabackup 2024-09-26 16:24:53.305  INFO: EVS version 1                                                                                                                            │
│ xtrabackup 2024-09-26 16:24:53.305  INFO: gcomm: connecting to group 'percona-cluster-pxc', peer 'percona-cluster-pxc-1.percona-cluster-pxc:'                                      │
│ xtrabackup 2024-09-26 16:24:53.314  INFO: (dc45c1f8-9914, 'ssl://0.0.0.0:4567') connection established to 5b14ee5a-b132 ssl://10.244.3.44:4567                                     │
│ xtrabackup 2024-09-26 16:24:53.314  INFO: (dc45c1f8-9914, 'ssl://0.0.0.0:4567') turning message relay requesting on, nonlive peers: ssl://10.244.6.71:4567                         │
│ xtrabackup 2024-09-26 16:24:53.660  INFO: (dc45c1f8-9914, 'ssl://0.0.0.0:4567') connection established to 7681f4de-b0d0 ssl://10.244.6.71:4567                                     │
│ xtrabackup 2024-09-26 16:24:53.809  INFO: EVS version upgrade 0 -> 1                                                                                                               │
│ xtrabackup 2024-09-26 16:24:53.809  INFO: declaring 5b14ee5a-b132 at ssl://10.244.3.44:4567 stable                                                                                 │
│ xtrabackup 2024-09-26 16:24:53.809  INFO: declaring 7681f4de-b0d0 at ssl://10.244.6.71:4567 stable                                                                                 │
│ xtrabackup 2024-09-26 16:24:53.809  INFO: PC protocol upgrade 0 -> 1                                                                                                               │
│ xtrabackup 2024-09-26 16:24:53.810  INFO: Node 5b14ee5a-b132 state primary                                                                                                         │
│ xtrabackup 2024-09-26 16:24:53.812  INFO: Current view of cluster as seen by this node                                                                                             │
│ xtrabackup view (view_id(PRIM,5b14ee5a-b132,125)                                                                                                                                   │
│ xtrabackup memb {                                                                                                                                                                  │
│ xtrabackup     5b14ee5a-b132,0                                                                                                                                                     │
│ xtrabackup     7681f4de-b0d0,0                                                                                                                                                     │
│ xtrabackup     dc45c1f8-9914,0                                                                                                                                                     │
│ xtrabackup     }                                                                                                                                                                   │
│ xtrabackup joined {                                                                                                                                                                │
│ xtrabackup     }                                                                                                                                                                   │
│ xtrabackup left {                                                                                                                                                                  │
│ xtrabackup     }                                                                                                                                                                   │
│ xtrabackup partitioned {                                                                                                                                                           │
│ xtrabackup     }                                                                                                                                                                   │
│ xtrabackup )                                                                                                                                                                       │
│ xtrabackup 2024-09-26 16:24:53.812  INFO: Save the discovered primary-component to disk                                                                                            │
│ xtrabackup 2024-09-26 16:24:53.812  WARN: open file(./gvwstate.dat.tmp) failed(Permission denied)                                                                                  │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: gcomm: connected                                                                                                                         │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: Changing maximum packet size to 64500, resulting msg size: 32636                                                                         │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: Shifting CLOSED -> OPEN (TO: 0)                                                                                                          │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: Opened channel 'percona-cluster-pxc'                                                                                                     │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3                                                                   │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: STATE EXCHANGE: Waiting for state UUID.                                                                                                  │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: STATE EXCHANGE: sent state msg: dc944e3c-7c23-11ef-b507-17f36ceb492c
│ xtrabackup 2024-09-26 16:24:54.305  INFO: STATE EXCHANGE: got state msg: dc944e3c-7c23-11ef-b507-17f36ceb492c from 0 (percona-cluster-pxc-1)                                       │
│ xtrabackup 2024-09-26 16:24:54.305  INFO: STATE EXCHANGE: got state msg: dc944e3c-7c23-11ef-b507-17f36ceb492c from 1 (percona-cluster-pxc-0)                                       │
│ xtrabackup 2024-09-26 16:24:54.307  INFO: STATE EXCHANGE: got state msg: dc944e3c-7c23-11ef-b507-17f36ceb492c from 2 (garb)                                                        │
│ xtrabackup 2024-09-26 16:24:54.307  INFO: Quorum results:                                                                                                                          │
│ xtrabackup     version    = 6,                                                                                                                                                     │
│ xtrabackup     component  = PRIMARY,                                                                                                                                               │
│ xtrabackup     conf_id    = 120,                                                                                                                                                   │
│ xtrabackup     members    = 2/3 (primary/total),                                                                                                                                   │
│ xtrabackup     act_id     = 209859,                                                                                                                                                │
│ xtrabackup     last_appl. = 209814,                                                                                                                                                │
│ xtrabackup     protocols  = 4/11/4 (gcs/repl/appl),                                                                                                                                │
│ xtrabackup     vote policy= 0,                                                                                                                                                     │
│ xtrabackup     group UUID = 02b77a89-eb7e-11ee-95e2-8622d953eed8                                                                                                                   │
│ xtrabackup 2024-09-26 16:24:54.307 FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/g │
│ cs_group.cpp:group_check_proto_ver():341: Group requested gcs_proto_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.                │
│ xtrabackup 2024-09-26 16:24:54.307  INFO: garbd: Terminated.                                                                                                                       │
│ xtrabackup /usr/bin/backup.sh: line 68:   227 Aborted                 garbd --address "gcomm://$NODE_NAME.$PXC_SERVICE?gmcast.listen_addr=tcp://0.0.0.0:4567" --donor "$NODE_NAME" │
│  --group "$PXC_SERVICE" --options "$GARBD_OPTS" --sst "xtrabackup-v2:$LOCAL_IP:4444/xtrabackup_sst//1" --recv-script="/usr/bin/run_backup.sh"                                      │
│ xtrabackup + EXID_CODE=134                                                                                                                                                         │
│ xtrabackup + '[' -f /tmp/backup-is-completed ']'                                                                                                                                   │
│ xtrabackup + log ERROR 'Backup was finished unsuccessfull'                                                                                                                         │
│ xtrabackup 2024-09-26 16:24:54 [ERROR] Backup was finished unsuccessfull                                                                                                           │
│ xtrabackup + exit 134

the error : xtrabackup 2024-09-26 16:24:54.307 FATAL: /mnt/jenkins/workspace/pxc80-autobuild-RELEASE/test/rpmbuild/BUILD/Percona-XtraDB-Cluster-8.0.36/percona-xtradb-cluster-galera/gcs/src/gcs_group.cpp:group_check_proto_ver():341: Group requested gcs_proto_ver: 4, max supported by this node: 2.Upgrade the node before joining this group.Need to abort.

Steps to reproduce

1. install
2. backup

Versions

1. Kubernetes 1.29.7
2. Operator : 1.15
3. Database : 8.0

Anything else?

https://forums.percona.com/t/percona-xtradb-operator-version-8-0-fail-to-backup-both-s3-and-local-storage/33084/5

No response

[hors]

Hi @sekot, I can't reproduce this issue. :( Could you please provide your CR. I will help to reproduce it.

[sekot]

kind: List
apiVersion: v1
items:
- apiVersion: pxc.percona.com/v1
  kind: PerconaXtraDBCluster
  metadata:
    annotations:
    finalizers:
    - delete-pxc-pods-in-order
    generation: 39
    name: percona-cluster
    namespace: mysql80
  spec:
    allowUnsafeConfigurations: true
    backup:
      image: percona/percona-xtradb-cluster-operator:1.15.0-pxc8.0-backup-pxb8.0.35
      pitr:
        enabled: false
        resources: {}
        storageName: STORAGE-NAME-HERE
        timeBetweenUploads: 60
        timeoutSeconds: 60
      schedule:
      - keep: 1
        name: daily-backup
        schedule: 0 0 * * *
        storageName: azure-blob
      storages:
        azure-blob:
          azure:
            container: mysql80-pxc-backups
            credentialsSecret: percona-azure-secret
            endpointUrl: ""
            storageClass: ""
          resources: {}
          type: azure
    crVersion: 1.15.0
    haproxy:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      annotations:
        fluentbit.io/exclude: "true"
      configuration: "global\n  maxconn 2048\n  external-check\n  insecure-fork-wanted\n
        \ stats socket /etc/haproxy/pxc/haproxy.sock mode 600 expose-fd listeners
        level admin\ndefaults\n  log global\n  mode tcp\n  retries 10\n  timeout client
        28800s\n  timeout connect 100500\n  timeout server 28800s\n  default-server
        check inter 10000 rise 2 fall 3 init-addr last,libc,none\nresolvers kubernetes\n
        \ parse-resolv-conf\nfrontend galera-admin-in\n  bind *:33062\n  mode tcp\n
        \ option clitcpka\n  default_backend galera-admin-nodes\nfrontend write-0\n
        \ bind *:3306\n  mode tcp\n  option clitcpka\n  default_backend write-0\nfrontend
        write-1\n  bind *:3307\n  mode tcp\n  option clitcpka\n  default_backend write-1\nfrontend
        read-0\n  bind *:3309\n  mode tcp\n  option clitcpka\n  default_backend read-0\nbackend
        write-0\n  mode tcp\n  option srvtcpka\n  balance leastconn\n  option external-check\n
        \ external-check command /usr/local/bin/check_pxc.sh\n\n  server percona-cluster-pxc-0
        percona-cluster-pxc-0.percona-cluster-pxc.mysql80.svc.cluster.local:3306 \n\n
        \ server percona-cluster-pxc-1 percona-cluster-pxc-1.percona-cluster-pxc.mysql80.svc.cluster.local:3306
        backup\n\nbackend write-1\n  mode tcp\n  option srvtcpka\n  balance leastconn\n
        \ option external-check\n  external-check command /usr/local/bin/check_pxc.sh\n\n
        \ server percona-cluster-pxc-0 percona-cluster-pxc-0.percona-cluster-pxc.mysql80.svc.cluster.local:3306
        backup\n\n  server percona-cluster-pxc-1 percona-cluster-pxc-1.percona-cluster-pxc.mysql80.svc.cluster.local:3306
        \n\nbackend read-0\n  mode tcp\n  option srvtcpka\n  balance leastconn\n  option
        external-check\n  external-check command /usr/local/bin/check_pxc.sh\n\n  server
        percona-cluster-pxc-0 percona-cluster-pxc-0.percona-cluster-pxc.mysql80.svc.cluster.local:3306
        backup\n\n  server percona-cluster-pxc-1 percona-cluster-pxc-1.percona-cluster-pxc.mysql80.svc.cluster.local:3306
        \n"
      enabled: true
      gracePeriod: 30
      image: percona/haproxy:2.8.5
      livenessProbes: {}
      nodeSelector:
        name: services
      podDisruptionBudget:
        maxUnavailable: 1
      readinessProbes: {}
      resources:
        limits:
          cpu: 200m
          memory: 1Gi
        requests:
          cpu: 40m
          memory: 100Mi
      sidecarResources:
        limits:
          cpu: 10m
          memory: 50Mi
        requests:
          cpu: 1m
          memory: 24Mi
      size: 2
    initContainer:
      image: percona/percona-xtradb-cluster-operator:1.15.0
      resources:
        limits:
          cpu: 200m
          memory: 200Mi
        requests:
          cpu: 100m
          memory: 100Mi
    logcollector:
      enabled: false
      image: percona/percona-xtradb-cluster-operator:1.15.0-logcollector-fluentbit3.1.4
      resources:
        requests:
          cpu: 200m
          memory: 100M
    pmm:
      enabled: false
      image: percona/pmm-client:2.42.0
      resources:
        requests:
          cpu: 300m
          memory: 150M
      serverHost: monitoring-service
    proxysql:
      affinity:
        antiAffinityTopologyKey: kubernetes.io/hostname
      enabled: false
      gracePeriod: 30
      image: percona/proxysql2:2.5.5
      livenessProbes: {}
      nodeSelector:
        name: services
      podDisruptionBudget:
        maxUnavailable: 1
      readinessProbes: {}
      resources:
        requests:
          cpu: 600m
          memory: 1G
      sidecarResources: {}
      size: 3
      volumeSpec:
        persistentVolumeClaim:
          resources:
            requests:
              storage: 2G
    pxc:
      affinity:
        advanced:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: tgAffinityPriority
                  operator: In
                  values:
                  - high
                  - medium
                - key: statefulset.kubernetes.io/pod-name
                  operator: In
                  values:
                  - percona-cluster-pxc-0
              namespaceSelector: {}
              topologyKey: kubernetes.io/hostname
        antiAffinityTopologyKey: kubernetes.io/hostname
      autoRecovery: true
      configuration: |
        [mysqld]
        log_error_verbosity             = 2
        wsrep_debug                     = 0
        wsrep_provider_options          = "gcache.size=1G; gcache.recover=yes"
        default_storage_engine          = InnoDB
        character-set-client-handshake  = FALSE
        character-set-server            = utf8mb4
        collation-server                = utf8mb4_unicode_ci
        binlog_format                   = ROW
        pxc_strict_mode                 = DISABLED
        sql_mode                        = "STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION"
        interactive_timeout             = 300
        wait_timeout                    = 300
        log_bin_trust_function_creators = 1
        innodb_flush_log_at_trx_commit  = 0
        innodb_flush_method             = O_DIRECT
        innodb_file_per_table           = 1
        innodb_log_file_size            = 512M
        innodb_autoinc_lock_mode        = 2
        innodb_buffer_pool_instances    = 8
        innodb_stats_on_metadata        = OFF
        skip_name_resolve               = 1
        max_allowed_packet              = 128M

        wsrep_auto_increment_control    = OFF
        wsrep_slave_threads             = 1
        explicit_defaults_for_timestamp = 1
        innodb_temp_data_file_path      = ibtmp1:12M:autoextend:max:5G
        log_slave_updates               = 1
        log_bin                         = /var/lib/mysql/mysql-bin.log
        expire_logs_days                = 4
        sync_binlog                     = 1
        gtid_mode                       = ON
        enforce_gtid_consistency        = true
      expose: {}
      forceUnsafeBootstrap: false
      gracePeriod: 600
      image: percona/percona-xtradb-cluster:8.0
      labels:
        tgAffinityPriority: medium
      livenessDelaySec: 300
      livenessProbes: {}
      nodeSelector:
        name: services
      podDisruptionBudget:
        maxUnavailable: 1
      priorityClassName: tg-high-priority
      readinessDelaySec: 15
      readinessProbes: {}
      resources:
        limits:
          cpu: 1000m
          memory: 1Gi
        requests:
          cpu: 100m
          memory: 25Mi
      sidecarResources: {}
      size: 2
      volumeSpec:
        persistentVolumeClaim:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: default
    secretsName: percona-secrets
    sslInternalSecretName: percona-ssl-internal
    sslSecretName: percona-ssl
    updateStrategy: SmartUpdate
    upgradeOptions:
      apply: disabled
      schedule: 0 4 * * *
      versionServiceEndpoint: https://check.percona.com

[sekot]

it s not a new install, it s an upgrade from 1.14

[hors]

@sekot, please do not use images without minor tags. The image with the following tagpercona/percona-xtradb-cluster:8.0 can be based on PXC 8.0.36 two days ago but today it can be with 8.0.37. In your case it is 8.0.37 and the following backup image percona/percona-xtradb-cluster-operator:1.15.0-pxc8.0-backup-pxb8.0.35 does not work with PXC 8.0.37. You can start the cluster with percona/percona-xtradb-cluster:8.0.35 and backups will work. The PXCO 1.15.0 was not tested with PXC 8.0.37 at all.

[sekot]

yes i have understand that, but it s too late.
and .37 normally it s not a minor but a patch version.
So it s not related to pxc version.
Thanks for the rapid update.