Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]Pulsar cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5681

Closed
ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5685
Closed

[BUG]Pulsar cluster status is always updating after upgrade kb from 0.6.4 to 0.7 #5681

ahjing99 opened this issue Oct 30, 2023 · 1 comment · Fixed by #5685
Assignees
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone

Comments

@ahjing99
Copy link
Collaborator

1.Install kb 0.6.4
2.create clusters

apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: pulsar-jxogeq
  namespace: default
spec:
  clusterDefinitionRef: pulsar
  clusterVersionRef: pulsar-2.11.2
  terminationPolicy: Halt
  affinity:
    topologyKeys:
      - kubernetes.io/hostname
  componentSpecs:
    - name: pulsar-broker
      componentDefRef: pulsar-broker
      monitor: false
      replicas: 1
      resources:
        requests:
          cpu: 500m
          memory: 0.5Gi
        limits:
          cpu: 500m
          memory: 0.5Gi
    - name: bookies
      componentDefRef: bookies
      monitor: false
      replicas: 3
      resources:
        requests:
          cpu: 500m
          memory: 0.5Gi
        limits:
          cpu: 500m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: ledgers
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
        - name: journal
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: zookeeper
      componentDefRef: zookeeper
      monitor: false
      replicas: 3
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 1Gi
    - name: pulsar-proxy
      componentDefRef: pulsar-proxy
      monitor: false
      replicas: 1
      resources:
        requests:
          cpu: 100m
          memory: 0.5Gi
        limits:
          cpu: 100m
          memory: 0.5Gi

      `kubectl apply -f test_create_pulsar-jxogeq.yaml`

cluster.apps.kubeblocks.io/pulsar-jxogeq created

➜  ~ k get cluster
NAME              CLUSTER-DEFINITION   VERSION              TERMINATION-POLICY   STATUS    AGE
pulsar-jxogeq     pulsar               pulsar-2.11.2        Halt                 Running   9m11s

3.Upgrade kb to 0.7

➜  ~ kbcli kubeblocks upgrade --version 0.7.0-beta.13
Current KubeBlocks version 0.6.4.
Kubernetes version 1.27.3
Kubernetes provider GKE
kbcli version 0.7.0-beta.13
Upgrade KubeBlocks from 0.6.4 to 0.7.0-beta.13
Please type 'Yes/yes' to confirm your operation: yes
Add and update repo kubeblocks                     OK
Stop KubeBlocks 0.6.4                              OK
Stop DataProtection                                OK
⢿ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes in /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
⣽ Upgrading KubeBlocks to 0.7.0-beta.13
Transform breaking changes successfully, remove /Users/mjingz/kubeblocks-0.6.4-0.7.0-beta.13
Upgrading KubeBlocks to 0.7.0-beta.13              OK

KubeBlocks has been upgraded to 0.7.0-beta.13 SUCCESSFULLY!

-> Basic commands for cluster:
    kbcli cluster create -h     # help information about creating a database cluster
    kbcli cluster list          # list all database clusters
    kbcli cluster describe <cluster name>  # get cluster information

-> Uninstall KubeBlocks:
    kbcli kubeblocks uninstall
  1. Pulsar cluster is always updating for 2 pods are always in Init
➜  ~ k get pod | grep pulsar
pulsar-jxogeq-bookies-0                       2/2     Running     0          34m
pulsar-jxogeq-bookies-1                       2/2     Running     0          35m
pulsar-jxogeq-bookies-2                       2/2     Running     0          38m
pulsar-jxogeq-pulsar-broker-0                 0/3     Init:0/4    0          38m
pulsar-jxogeq-pulsar-proxy-0                  0/2     Init:0/1    0          38m
pulsar-jxogeq-zookeeper-0                     2/2     Running     0          38m
pulsar-jxogeq-zookeeper-1                     2/2     Running     0          38m
pulsar-jxogeq-zookeeper-2                     2/2     Running     0          38m

➜  ~ k describe sts pulsar-jxogeq-pulsar-broker
Name:               pulsar-jxogeq-pulsar-broker
Namespace:          default
CreationTimestamp:  Mon, 30 Oct 2023 16:39:13 +0800
Selector:           app.kubernetes.io/instance=pulsar-jxogeq,app.kubernetes.io/managed-by=kubeblocks,app.kubernetes.io/name=pulsar,apps.kubeblocks.io/component-name=pulsar-broker
Labels:             app.kubernetes.io/component=pulsar-broker
                    app.kubernetes.io/instance=pulsar-jxogeq
                    app.kubernetes.io/managed-by=kubeblocks
                    app.kubernetes.io/name=pulsar
                    apps.kubeblocks.io/component-name=pulsar-broker
                    rsm.workloads.kubeblocks.io/controller-generation=2
Annotations:        config.kubeblocks.io/tpl-agamotto-configuration: pulsar-jxogeq-pulsar-broker-agamotto-configuration
                    config.kubeblocks.io/tpl-broker-config: pulsar-jxogeq-pulsar-broker-broker-config
                    config.kubeblocks.io/tpl-broker-env: pulsar-jxogeq-pulsar-broker-broker-env
                    config.kubeblocks.io/tpl-client-config: pulsar-jxogeq-pulsar-broker-client-config
                    config.kubeblocks.io/tpl-pulsar-scripts: pulsar-jxogeq-pulsar-broker-pulsar-scripts
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app.kubernetes.io/component=pulsar-broker
           app.kubernetes.io/instance=pulsar-jxogeq
           app.kubernetes.io/managed-by=kubeblocks
           app.kubernetes.io/name=pulsar
           app.kubernetes.io/version=pulsar-2.11.2
           apps.kubeblocks.io/component-name=pulsar-broker
           apps.kubeblocks.io/workload-type=Stateful
  Init Containers:
   init-broker-cluster:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Port:       <none>
    Host Port:  <none>
    Command:
      /kb-scripts/init-broker.sh
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      brokerServiceUrl:          pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
      clusterName:               $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
      webServiceUrl:             http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
    Mounts:
      /kb-scripts from scripts (rw)
   init-sysctl:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Port:       <none>
    Host Port:  <none>
    Command:
      /kb-scripts/init-broker-sysctl.sh
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    Mounts:
      /kb-scripts from scripts (rw)
   init-pulsar-tools:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsartools:2.11
    Port:       <none>
    Host Port:  <none>
    Command:
      cp
      /bin/pulsar-admin
      /kb/tools/pulsar-admin
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      TOOLS_SCRIPTS_PATH:        /opt/kb-tools/reload/broker-config
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/client-conf from pulsar-client-config (rw)
      /opt/pulsar/conf from pulsar-broker-config (rw)
      /opt/pulsar/none from broker-env (rw)
   init-pulsar-client-config:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Port:       <none>
    Host Port:  <none>
    Command:
      $(TOOLS_SCRIPTS_PATH)/install-pulsar-client-config.sh
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      TOOLS_SCRIPTS_PATH:        /opt/kb-tools/reload/broker-config
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/client-conf from pulsar-client-config (rw)
      /opt/pulsar/conf from pulsar-broker-config (rw)
      /opt/pulsar/none from broker-env (rw)
  Containers:
   broker:
    Image:       infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Ports:       6650/TCP, 8080/TCP, 9092/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Command:
      sh
      -c
    Args:
      set -x
      /kb-scripts/merge_pulsar_config.py conf/client.conf /opt/pulsar/client-conf/client.conf && \
      /kb-scripts/merge_pulsar_config.py conf/broker.conf /opt/pulsar/conf/broker.conf && \
      bin/apply-config-from-env.py conf/broker.conf && \
      bin/apply-config-from-env.py conf/client.conf && \
      echo 'OK' > status;exec bin/pulsar broker

    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:      500m
      memory:   512Mi
    Liveness:   http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
    Readiness:  http-get http://:http/status.html delay=30s timeout=100s period=10s #success=1 #failure=3
    Startup:    http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=30
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env                 ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-broker-env-envfrom  ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-rsm-env             ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                                (v1:metadata.name)
      KB_POD_UID:                                 (v1:metadata.uid)
      KB_NAMESPACE:                               (v1:metadata.namespace)
      KB_SA_NAME:                                 (v1:spec.serviceAccountName)
      KB_NODENAME:                                (v1:spec.nodeName)
      KB_HOST_IP:                                 (v1:status.hostIP)
      KB_POD_IP:                                  (v1:status.podIP)
      KB_POD_IPS:                                 (v1:status.podIPs)
      KB_HOSTIP:                                  (v1:status.hostIP)
      KB_PODIP:                                   (v1:status.podIP)
      KB_PODIPS:                                  (v1:status.podIPs)
      KB_CLUSTER_NAME:                           pulsar-jxogeq
      KB_COMP_NAME:                              pulsar-broker
      KB_CLUSTER_COMP_NAME:                      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:                  76b834ba
      KB_POD_FQDN:                               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:                              8080
      POD_NAME:                                  $(KB_POD_NAME)
      PULSAR_PREFIX_brokerServicePort:           6650
      PULSAR_PREFIX_internalListenerName:        cluster
      PULSAR_PREFIX_advertisedListeners:         cluster:pulsar://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:6650
      PULSAR_PREFIX_allowAutoTopicCreationType:  partitioned
      PULSAR_PREFIX_kafkaListeners:              PLAINTEXT://0.0.0.0:9092
      PULSAR_PREFIX_kafkaAdvertisedListeners:    PLAINTEXT://$(POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc.cluster.local:9092
      brokerServiceUrl:                          pulsar://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:6650
      clusterName:                               $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
      webServiceUrl:                             http://$(KB_CLUSTER_COMP_NAME).$(KB_NAMESPACE).svc.cluster.local:80
    Mounts:
      /kb-scripts from scripts (rw)
      /kb/tools from kb-tools (rw)
      /opt/pulsar/client-conf from pulsar-client-config (rw)
      /opt/pulsar/conf from pulsar-broker-config (rw)
      /opt/pulsar/none from broker-env (rw)
   metrics:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2
    Port:       1234/TCP
    Host Port:  0/TCP
    Command:
      /bin/agamotto
      --config=/opt/agamotto/agamotto-config.yaml
      --feature-gates=-pkg.translator.prometheus.NormalizeName
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env      ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      JOB_NAME:                  broker
      SERVICE_PORT:              8080
      POD_NAME:                  $(KB_POD_NAME)
      POD_NAMESPACE:             $(KB_NAMESPACE)
      LOG_LEVEL:                 info
    Mounts:
      /opt/agamotto from agamotto-configuration (rw)
   config-manager:
    Image:      registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.7.0-beta.13
    Port:       <none>
    Host Port:  <none>
    Command:
      env
    Args:
      PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH)
      /bin/reloader
      --log-level
      info
      --operator-update-enable
      --tcp
      9901
      --config
      /opt/config-manager/config-manager.yaml
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-broker-env      ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-broker-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-broker
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-broker
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      CONFIG_MANAGER_POD_IP:      (v1:status.podIP)
      DB_TYPE:                   pulsar-broker
      TOOLS_PATH:                /opt/kb-tools/reload/broker-config:/opt/config-manager:/kb/tools
    Mounts:
      /kb/tools from kb-tools (rw)
      /opt/config-manager from config-manager-config (rw)
      /opt/kb-tools/reload/broker-config from cm-script-broker-config (rw)
      /opt/pulsar/client-conf from pulsar-client-config (rw)
      /opt/pulsar/conf from pulsar-broker-config (rw)
      /opt/pulsar/none from broker-env (rw)
  Volumes:
   agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-broker-agamotto-configuration
    Optional:  false
   pulsar-broker-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-broker-broker-config
    Optional:  false
   broker-env:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-broker-broker-env
    Optional:  false
   pulsar-client-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-broker-client-config
    Optional:  false
   scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-broker-pulsar-scripts
    Optional:  false
   cm-script-broker-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-pulsar-tools-script-pulsar-jxogeq
    Optional:  false
   config-manager-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      sidecar-pulsar-jxogeq-pulsar-broker-config-manager-config
    Optional:  false
   kb-tools:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
Volume Claims:  <none>
Events:
  Type     Reason               Age                From                    Message
  ----     ------               ----               ----                    -------
  Normal   SuccessfulDelete     39m (x4 over 40m)  statefulset-controller  delete Pod pulsar-jxogeq-pulsar-broker-0 in StatefulSet pulsar-jxogeq-pulsar-broker successful
  Normal   SuccessfulCreate     38m (x3 over 55m)  statefulset-controller  create Pod pulsar-jxogeq-pulsar-broker-0 in StatefulSet pulsar-jxogeq-pulsar-broker successful
  Warning  RecreatingFailedPod  38m (x4 over 39m)  statefulset-controller  StatefulSet default/pulsar-jxogeq-pulsar-broker is recreating failed Pod pulsar-jxogeq-pulsar-broker-0
  Warning  FailedDelete         38m (x2 over 38m)  statefulset-controller  delete Pod pulsar-jxogeq-pulsar-broker-0 in StatefulSet pulsar-jxogeq-pulsar-broker failed error: pods "pulsar-jxogeq-pulsar-broker-0" not found
➜  ~

➜  ~ k describe sts pulsar-jxogeq-pulsar-proxy
Name:               pulsar-jxogeq-pulsar-proxy
Namespace:          default
CreationTimestamp:  Mon, 30 Oct 2023 16:39:13 +0800
Selector:           app.kubernetes.io/instance=pulsar-jxogeq,app.kubernetes.io/managed-by=kubeblocks,app.kubernetes.io/name=pulsar,apps.kubeblocks.io/component-name=pulsar-proxy
Labels:             app.kubernetes.io/component=pulsar-proxy
                    app.kubernetes.io/instance=pulsar-jxogeq
                    app.kubernetes.io/managed-by=kubeblocks
                    app.kubernetes.io/name=pulsar
                    apps.kubeblocks.io/component-name=pulsar-proxy
                    rsm.workloads.kubeblocks.io/controller-generation=3
Annotations:        config.kubeblocks.io/tpl-agamotto-configuration: pulsar-jxogeq-pulsar-proxy-agamotto-configuration
                    config.kubeblocks.io/tpl-proxy-config: pulsar-jxogeq-pulsar-proxy-proxy-config
                    config.kubeblocks.io/tpl-proxy-env: pulsar-jxogeq-pulsar-proxy-proxy-env
                    config.kubeblocks.io/tpl-pulsar-scripts: pulsar-jxogeq-pulsar-proxy-pulsar-scripts
Replicas:           1 desired | 1 total
Update Strategy:    RollingUpdate
  Partition:        0
Pods Status:        0 Running / 1 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels:  app.kubernetes.io/component=pulsar-proxy
           app.kubernetes.io/instance=pulsar-jxogeq
           app.kubernetes.io/managed-by=kubeblocks
           app.kubernetes.io/name=pulsar
           app.kubernetes.io/version=pulsar-2.11.2
           apps.kubeblocks.io/component-name=pulsar-proxy
           apps.kubeblocks.io/workload-type=Stateful
  Init Containers:
   check-broker:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Port:       <none>
    Host Port:  <none>
    Command:
      /kb-scripts/init-proxy.sh
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-proxy-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-proxy
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-proxy
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
    Mounts:
      /kb-scripts from scripts (rw)
  Containers:
   proxy:
    Image:       infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/pulsar:2.11.2
    Ports:       6650/TCP, 8080/TCP
    Host Ports:  0/TCP, 0/TCP
    Command:
      /kb-scripts/start-proxy.sh
    Limits:
      cpu:     100m
      memory:  512Mi
    Requests:
      cpu:      100m
      memory:   512Mi
    Liveness:   http-get http://:http/status.html delay=30s timeout=100s period=10s #success=1 #failure=3
    Readiness:  http-get http://:http/status.html delay=30s timeout=100s period=10s #success=1 #failure=3
    Startup:    http-get http://:http/status.html delay=5s timeout=100s period=10s #success=1 #failure=20
    Environment Variables from:
      pulsar-jxogeq-pulsar-proxy-env                ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-proxy-proxy-env-envfrom  ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-proxy-rsm-env            ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-proxy
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-proxy
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      SERVICE_PORT:              8080
      brokerWebServiceURL:       http://$(brokerSVC):80
      brokerServiceURL:          pulsar://$(brokerSVC):6650
      clusterName:               $(KB_NAMESPACE)-$(KB_CLUSTER_COMP_NAME)
      webServicePort:            8080
    Mounts:
      /kb-scripts from scripts (rw)
      /opt/pulsar/conf from pulsar-proxy-config (rw)
      /opt/pulsar/none from proxy-env (rw)
   metrics:
    Image:      infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.2
    Port:       1234/TCP
    Host Port:  0/TCP
    Command:
      /bin/agamotto
      --config=/opt/agamotto/agamotto-config.yaml
      --feature-gates=-pkg.translator.prometheus.NormalizeName
    Limits:
      cpu:     0
      memory:  0
    Environment Variables from:
      pulsar-jxogeq-pulsar-proxy-env      ConfigMap  Optional: false
      pulsar-jxogeq-pulsar-proxy-rsm-env  ConfigMap  Optional: false
    Environment:
      KB_POD_NAME:                (v1:metadata.name)
      KB_POD_UID:                 (v1:metadata.uid)
      KB_NAMESPACE:               (v1:metadata.namespace)
      KB_SA_NAME:                 (v1:spec.serviceAccountName)
      KB_NODENAME:                (v1:spec.nodeName)
      KB_HOST_IP:                 (v1:status.hostIP)
      KB_POD_IP:                  (v1:status.podIP)
      KB_POD_IPS:                 (v1:status.podIPs)
      KB_HOSTIP:                  (v1:status.hostIP)
      KB_PODIP:                   (v1:status.podIP)
      KB_PODIPS:                  (v1:status.podIPs)
      KB_CLUSTER_NAME:           pulsar-jxogeq
      KB_COMP_NAME:              pulsar-proxy
      KB_CLUSTER_COMP_NAME:      pulsar-jxogeq-pulsar-proxy
      KB_CLUSTER_UID_POSTFIX_8:  76b834ba
      KB_POD_FQDN:               $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc
      JOB_NAME:                  proxy
      SERVICE_PORT:              8080
      POD_NAME:                  $(KB_POD_NAME)
      POD_NAMESPACE:             $(KB_NAMESPACE)
      LOG_LEVEL:                 info
    Mounts:
      /opt/agamotto from agamotto-configuration (rw)
  Volumes:
   agamotto-configuration:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-proxy-agamotto-configuration
    Optional:  false
   pulsar-proxy-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-proxy-proxy-config
    Optional:  false
   proxy-env:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      pulsar-jxogeq-pulsar-proxy-proxy-env
    Optional:  false
   scripts:
    Type:       ConfigMap (a volume populated by a ConfigMap)
    Name:       pulsar-jxogeq-pulsar-proxy-pulsar-scripts
    Optional:   false
Volume Claims:  <none>
Events:
  Type     Reason               Age                From                    Message
  ----     ------               ----               ----                    -------
  Warning  FailedDelete         40m                statefulset-controller  delete Pod pulsar-jxogeq-pulsar-proxy-0 in StatefulSet pulsar-jxogeq-pulsar-proxy failed error: pods "pulsar-jxogeq-pulsar-proxy-0" not found
  Normal   SuccessfulCreate     39m (x3 over 56m)  statefulset-controller  create Pod pulsar-jxogeq-pulsar-proxy-0 in StatefulSet pulsar-jxogeq-pulsar-proxy successful
  Normal   SuccessfulDelete     39m (x5 over 40m)  statefulset-controller  delete Pod pulsar-jxogeq-pulsar-proxy-0 in StatefulSet pulsar-jxogeq-pulsar-proxy successful
  Warning  RecreatingFailedPod  39m (x4 over 40m)  statefulset-controller  StatefulSet default/pulsar-jxogeq-pulsar-proxy is recreating failed Pod pulsar-jxogeq-pulsar-proxy-0

➜  ~ k describe cluster pulsar-jxogeq
Name:         pulsar-jxogeq
Namespace:    default
Labels:       clusterdefinition.kubeblocks.io/name=pulsar
              clusterversion.kubeblocks.io/name=pulsar-2.11.2
Annotations:  kubeblocks.io/reconcile: 2023-10-30T08:56:16.769991853Z
API Version:  apps.kubeblocks.io/v1alpha1
Kind:         Cluster
Metadata:
  Creation Timestamp:  2023-10-30T08:39:09Z
  Finalizers:
    cluster.kubeblocks.io/finalizer
  Generation:  1
  Managed Fields:
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:affinity:
          .:
          f:podAntiAffinity:
          f:tenancy:
          f:topologyKeys:
            .:
            v:"kubernetes.io/hostname":
        f:clusterDefinitionRef:
        f:clusterVersionRef:
        f:componentSpecs:
          .:
          k:{"name":"bookies"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:volumeClaimTemplates:
          k:{"name":"pulsar-broker"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
          k:{"name":"pulsar-proxy"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
          k:{"name":"zookeeper"}:
            .:
            f:componentDefRef:
            f:monitor:
            f:name:
            f:noCreatePDB:
            f:replicas:
            f:resources:
              .:
              f:limits:
                .:
                f:cpu:
                f:memory:
              f:requests:
                .:
                f:cpu:
                f:memory:
            f:volumeClaimTemplates:
        f:terminationPolicy:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-10-30T08:39:09Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:kubeblocks.io/reconcile:
        f:finalizers:
          .:
          v:"cluster.kubeblocks.io/finalizer":
        f:labels:
          .:
          f:clusterdefinition.kubeblocks.io/name:
          f:clusterversion.kubeblocks.io/name:
    Manager:      manager
    Operation:    Update
    Time:         2023-10-30T08:56:16Z
    API Version:  apps.kubeblocks.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:clusterDefGeneration:
        f:components:
          .:
          f:bookies:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:pulsar-broker:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:pulsar-proxy:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
          f:zookeeper:
            .:
            f:phase:
            f:podsReady:
            f:podsReadyTime:
        f:conditions:
        f:observedGeneration:
        f:phase:
    Manager:         manager
    Operation:       Update
    Subresource:     status
    Time:            2023-10-30T09:00:26Z
  Resource Version:  170399
  UID:               85e8e141-68d9-4272-b408-160876b834ba
Spec:
  Affinity:
    Pod Anti Affinity:  Preferred
    Tenancy:            SharedNode
    Topology Keys:
      kubernetes.io/hostname
  Cluster Definition Ref:  pulsar
  Cluster Version Ref:     pulsar-2.11.2
  Component Specs:
    Component Def Ref:  pulsar-broker
    Monitor:            false
    Name:               pulsar-broker
    No Create PDB:      false
    Replicas:           1
    Resources:
      Limits:
        Cpu:     500m
        Memory:  0.5Gi
      Requests:
        Cpu:            500m
        Memory:         0.5Gi
    Component Def Ref:  bookies
    Monitor:            false
    Name:               bookies
    No Create PDB:      false
    Replicas:           3
    Resources:
      Limits:
        Cpu:     500m
        Memory:  0.5Gi
      Requests:
        Cpu:     500m
        Memory:  0.5Gi
    Volume Claim Templates:
      Name:  ledgers
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:  1Gi
      Name:           journal
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    1Gi
    Component Def Ref:  zookeeper
    Monitor:            false
    Name:               zookeeper
    No Create PDB:      false
    Replicas:           3
    Resources:
      Limits:
        Cpu:     100m
        Memory:  0.5Gi
      Requests:
        Cpu:     100m
        Memory:  0.5Gi
    Volume Claim Templates:
      Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:    1Gi
    Component Def Ref:  pulsar-proxy
    Monitor:            false
    Name:               pulsar-proxy
    No Create PDB:      false
    Replicas:           1
    Resources:
      Limits:
        Cpu:     100m
        Memory:  0.5Gi
      Requests:
        Cpu:           100m
        Memory:        0.5Gi
  Termination Policy:  Halt
Status:
  Cluster Def Generation:  2
  Components:
    Bookies:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-30T09:00:25Z
    Pulsar - Broker:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-10-30T08:43:24Z
    Pulsar - Proxy:
      Phase:            Updating
      Pods Ready:       false
      Pods Ready Time:  2023-10-30T08:44:58Z
    Zookeeper:
      Phase:            Running
      Pods Ready:       true
      Pods Ready Time:  2023-10-30T08:56:27Z
  Conditions:
    Last Transition Time:  2023-10-30T08:39:09Z
    Message:               The operator has started the provisioning of Cluster: pulsar-jxogeq
    Observed Generation:   1
    Reason:                PreCheckSucceed
    Status:                True
    Type:                  ProvisioningStarted
    Last Transition Time:  2023-10-30T08:56:25Z
    Message:               Successfully applied for resources
    Observed Generation:   1
    Reason:                ApplyResourcesSucceed
    Status:                True
    Type:                  ApplyResources
    Last Transition Time:  2023-10-30T08:55:43Z
    Message:               pods are not ready in Components: [pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
    Reason:                ReplicasNotReady
    Status:                False
    Type:                  ReplicasReady
    Last Transition Time:  2023-10-30T08:55:43Z
    Message:               pods are unavailable in Components: [pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
    Reason:                ComponentsNotReady
    Status:                False
    Type:                  Ready
  Observed Generation:     1
  Phase:                   Updating
Events:
  Type     Reason                    Age                 From                Message
  ----     ------                    ----                ----                -------
  Normal   ComponentPhaseTransition  57m (x4 over 57m)   cluster-controller  Create a new component
  Normal   PreCheckSucceed           57m                 cluster-controller  The operator has started the provisioning of Cluster: pulsar-jxogeq
  Normal   ApplyResourcesSucceed     57m                 cluster-controller  Successfully applied for resources
  Warning  ReplicasNotReady          55m                 cluster-controller  pods are not ready in Components: [bookies pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        55m                 cluster-controller  pods are unavailable in Components: [bookies pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ReplicasNotReady          54m                 cluster-controller  pods are not ready in Components: [pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        54m                 cluster-controller  pods are unavailable in Components: [pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ReplicasNotReady          53m                 cluster-controller  pods are not ready in Components: [pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        53m                 cluster-controller  pods are unavailable in Components: [pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  Unhealthy                 52m (x4 over 52m)   event-controller    Pod pulsar-jxogeq-pulsar-proxy-0: Startup probe failed: Get "http://10.32.2.128:8080/status.html": dial tcp 10.32.2.128:8080: connect: connection refused
  Normal   ComponentPhaseTransition  51m (x4 over 55m)   cluster-controller  Running: true, PodsReady: true, PodsTimedout: false
  Normal   AllReplicasReady          51m                 cluster-controller  all pods of components are ready, waiting for the probe detection successful
  Normal   ClusterReady              51m                 cluster-controller  Cluster: pulsar-jxogeq is ready, current phase is Running
  Normal   Running                   51m                 cluster-controller  Cluster: pulsar-jxogeq is ready, current phase is Running
  Warning  Unhealthy                 41m                 event-controller    Pod pulsar-jxogeq-pulsar-proxy-0: Startup probe failed: Get "http://10.32.2.128:8080/status.html": dial tcp 10.32.2.128:8080: connect: connection refused
  Warning  ApplyResourcesFailed      41m                 cluster-controller  Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io "pulsar-jxogeq-zookeeper": the object has been modified; please apply your changes to the latest version and try again
  Warning  ApplyResourcesFailed      41m                 cluster-controller  Operation cannot be fulfilled on replicatedstatemachines.workloads.kubeblocks.io "pulsar-jxogeq-pulsar-broker": the object has been modified; please apply your changes to the latest version and try again
  Normal   ComponentPhaseTransition  41m (x12 over 41m)  cluster-controller  component is Updating
  Warning  ReplicasNotReady          40m                 cluster-controller  pods are not ready in Components: [bookies pulsar-broker pulsar-proxy zookeeper], refer to related component message in Cluster.status.components
  Warning  ComponentsNotReady        40m                 cluster-controller  pods are unavailable in Components: [bookies pulsar-broker pulsar-proxy zookeeper], refer to related component message in Cluster.status.components
  Warning  RecreatingFailedPod       40m (x2 over 40m)   event-controller    StatefulSet default/pulsar-jxogeq-bookies is recreating failed Pod pulsar-jxogeq-bookies-2
  Warning  RecreatingFailedPod       40m (x2 over 40m)   event-controller    StatefulSet default/pulsar-jxogeq-pulsar-broker is recreating failed Pod pulsar-jxogeq-pulsar-broker-0
  Warning  RecreatingFailedPod       40m                 event-controller    StatefulSet default/pulsar-jxogeq-pulsar-proxy is recreating failed Pod pulsar-jxogeq-pulsar-proxy-0
  Warning  ApplyResourcesFailed      40m                 cluster-controller  Operation cannot be fulfilled on pods "pulsar-jxogeq-pulsar-broker-0": the object has been modified; please apply your changes to the latest version and try again
  Warning  ApplyResourcesFailed      40m                 cluster-controller  Operation cannot be fulfilled on pods "pulsar-jxogeq-bookies-2": the object has been modified; please apply your changes to the latest version and try again
  Normal   ApplyResourcesSucceed     40m (x2 over 41m)   cluster-controller  Successfully applied for resources
  Warning  ComponentsNotReady        40m (x2 over 41m)   cluster-controller  pods are unavailable in Components: [bookies pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Warning  ReplicasNotReady          40m (x2 over 41m)   cluster-controller  pods are not ready in Components: [bookies pulsar-broker pulsar-proxy], refer to related component message in Cluster.status.components
  Normal   ComponentPhaseTransition  36m (x2 over 40m)   cluster-controller  component is Running

➜ ~ k logs kubeblocks-6b9684fccc-8xphr -n kb-system >kb.txt
Defaulted container "manager" out of: manager, tools (init), datascript (init)
kb.txt

@ahjing99 ahjing99 added kind/bug Something isn't working severity/major Great chance user will encounter the same problem labels Oct 30, 2023
@ahjing99 ahjing99 added this to the Release 0.7.0 milestone Oct 30, 2023
@wangyelei
Copy link
Contributor

after upgrading to 0.7, zookeeperServers missing port in broker env configmap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants