Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] tdengine cluster create CrashLoopBackOff: dnode can not be initialized #5438

Closed
JashBook opened this issue Oct 13, 2023 · 0 comments · Fixed by #5674
Closed

[BUG] tdengine cluster create CrashLoopBackOff: dnode can not be initialized #5438

JashBook opened this issue Oct 13, 2023 · 0 comments · Fixed by #5674
Assignees
@sophon-zt
Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Milestone
Release 0.7.0

Comments

@JashBook
Copy link
Collaborator

Describe the bug
tdengine cluster create CrashLoopBackOff:
UTL ERROR failed to get ip from fqdn:tdenginecluster-tdengine-0.tdenginecluster-tdengine-headless.default.svc since Success, dnode can not be initialized
DND ERROR failed to start since read config error

kbcli version
Kubernetes: v1.26.7-gke.500
KubeBlocks: 0.7.0-beta.3
kbcli: 0.7.0-beta.3

To Reproduce
Steps to reproduce the behavior:

  1. kbcli addon enable tdengine
  2. create tdengine cluster
kubectl apply -f -<<EOF
apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
  name: tdenginecluster
  labels: 
    app.kubernetes.io/version: "3.0.5.0"
    app.kubernetes.io/instance: tdenginecluster
spec:
  clusterVersionRef: tdengine-3.0.5.0
  terminationPolicy: Delete  
  affinity:
    podAntiAffinity: Preferred
    topologyKeys:
      - kubernetes.io/hostname
    tenancy: SharedNode
  clusterDefinitionRef: tdengine
  componentSpecs:
    - name: tdengine
      componentDefRef: tdengine
      monitor: true      
      replicas: 3
      serviceAccountName: kb-tdenginecluster      
      resources:
        limits:
          cpu: "0.5"
          memory: "1Gi"
        requests:
          cpu: "0.5"
          memory: "1Gi"      
      volumeClaimTemplates:
        - name: data
          spec:
            accessModes:
              - ReadWriteOnce
            resources:
              requests:
                storage: 10Gi      
      services:
EOF
  1. See error
➜  ~ kubectl get cluster 
NAME              CLUSTER-DEFINITION   VERSION            TERMINATION-POLICY   STATUS   AGE
tdenginecluster   tdengine             tdengine-3.0.5.0   Delete               Failed   114s
➜  ~ kubectl get pod
NAME                                            READY   STATUS             RESTARTS     AGE
tdenginecluster-tdengine-0                      1/3     CrashLoopBackOff   8 (5s ago)   119s

logs error pod

kubectl logs tdenginecluster-tdengine-0 
Defaulted container "tdengine" out of: tdengine, metrics, config-manager
sysctl: setting key "kernel.core_pattern": Read-only file system
10/13 16:30:06.237701 00000045 taos_ADAPTER info "start server:00000045" model=main
10/13 16:30:06.237864 00000045 taos_ADAPTER info "init plugin influxdb/v1" model=plugin
10/13 16:30:06.238109 00000045 taos_ADAPTER info "init plugin node_exporter/v1" model=plugin
10/13 16:30:06.240018 00000045 taos_ADAPTER info "node_exporter disabled" model=NodeExporter
10/13 16:30:06.240036 00000045 taos_ADAPTER info "init plugin opentsdb/v1" model=plugin
10/13 16:30:06.240163 00000045 taos_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin
10/13 16:30:06.241422 00000045 taos_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet
10/13 16:30:06.241436 00000045 taos_ADAPTER info "init plugin prometheus/v1" model=plugin
10/13 16:30:06.241571 00000045 taos_ADAPTER info "init plugin statsd/v1" model=plugin
10/13 16:30:06.243517 00000045 taos_ADAPTER info "statsd disabled" model=statsd
10/13 16:30:06.243532 00000045 taos_ADAPTER info "init plugin collectd/v1" model=plugin
10/13 16:30:06.244378 00000045 taos_ADAPTER info "collectd disabled" model=collectd
10/13 16:30:06.244677 00000045 taos_ADAPTER info "all plugin init finish" model=plugin
10/13 16:30:06.244684 00000045 taos_ADAPTER info "all plugin start finish" model=plugin
10/13 16:30:06.244763 00000045 taos_ADAPTER info "Running in terminal." model=main
10/13 16:30:06.245517 00000045 taos_ADAPTER info "server on : 6041" model=main
10/13 16:30:06.486343 00000060 UTL apoll url not load
10/13 16:30:06.486427 00000060 UTL load from cfg file /etc/taos//taos.cfg success
10/13 16:30:06.486449 00000060 UTL env file:.env not load
10/13 16:30:06.486479 00000060 UTL load from env variables cfg success
10/13 16:30:06.486781 00000060 DND startup path:/root args:taosd
10/13 16:30:06.487212 00000060 UTL apoll url not load
10/13 16:30:06.487270 00000060 UTL load from cfg file /etc/taos//taos.cfg success
10/13 16:30:06.487287 00000060 UTL env file:.env not load
10/13 16:30:06.487325 00000060 UTL load from env variables cfg success
10/13 16:30:06.487530 00000060 UTL                      global config
10/13 16:30:06.487541 00000060 UTL =================================================================
10/13 16:30:06.487545 00000060 UTL env_var      firstEp                  tdenginecluster-tdengine-0.tdenginecluster-tdengine-headless.default.svc:6030
10/13 16:30:06.487548 00000060 UTL default      secondEp                 tdenginecluster-tdengine-0.tdenginecluster-tdengine-headless.default.svc:6030
10/13 16:30:06.487551 00000060 UTL env_var      fqdn                     tdenginecluster-tdengine-0.tdenginecluster-tdengine-headless.default.svc
10/13 16:30:06.487554 00000060 UTL default      serverPort               6030
10/13 16:30:06.487557 00000060 UTL default      tempDir                  /tmp/
10/13 16:30:06.487559 00000060 UTL default      minimalTmpDirGB          1.00
10/13 16:30:06.487565 00000060 UTL default      shellActivityTimer       3
10/13 16:30:06.487568 00000060 UTL default      compressMsgSize          -1
10/13 16:30:06.487570 00000060 UTL default      compressColData          -1
10/13 16:30:06.487573 00000060 UTL default      queryPolicy              1
10/13 16:30:06.487580 00000060 UTL default      enableQueryHb            0
10/13 16:30:06.487583 00000060 UTL default      enableScience            0
10/13 16:30:06.487585 00000060 UTL default      querySmaOptimize         0
10/13 16:30:06.487588 00000060 UTL default      queryPlannerTrace        0
10/13 16:30:06.487591 00000060 UTL default      queryNodeChunkSize       32768
10/13 16:30:06.487593 00000060 UTL default      queryUseNodeAllocator    1
10/13 16:30:06.487596 00000060 UTL default      keepColumnName           0
10/13 16:30:06.487599 00000060 UTL default      smlChildTableName        
10/13 16:30:06.487601 00000060 UTL default      smlTagName               _tag_null
10/13 16:30:06.487604 00000060 UTL default      maxInsertBatchRows       1000000
10/13 16:30:06.487606 00000060 UTL default      maxRetryWaitTime         10000
10/13 16:30:06.487609 00000060 UTL default      useAdapter               0
10/13 16:30:06.487611 00000060 UTL default      crashReporting           1
10/13 16:30:06.487614 00000060 UTL default      queryMaxConcurrentTable  200
10/13 16:30:06.487616 00000060 UTL default      metaCacheMaxSize         -1
10/13 16:30:06.487619 00000060 UTL default      slowLogThreshold         3
10/13 16:30:06.487622 00000060 UTL default      slowLogScope             
10/13 16:30:06.487624 00000060 UTL default      numOfRpcThreads          2
10/13 16:30:06.487627 00000060 UTL default      numOfRpcSessions         10000
10/13 16:30:06.487629 00000060 UTL default      timeToGetAvailableConn   500000
10/13 16:30:06.487632 00000060 UTL default      numOfTaskQueueThreads    4
10/13 16:30:06.487634 00000060 UTL default      dataDir                  /var/lib/taos/
10/13 16:30:06.487638 00000060 UTL default      minimalDataDirGB         2.00
10/13 16:30:06.487641 00000060 UTL default      supportVnodes            8
10/13 16:30:06.487643 00000060 UTL default      maxShellConns            50000
10/13 16:30:06.487646 00000060 UTL default      statusInterval           1
10/13 16:30:06.487649 00000060 UTL default      minSlidingTime           1
10/13 16:30:06.487651 00000060 UTL default      minIntervalTime          1
10/13 16:30:06.487653 00000060 UTL default      maxNumOfDistinctRes      10000000
10/13 16:30:06.487656 00000060 UTL default      countAlwaysReturnValue   1
10/13 16:30:06.487658 00000060 UTL default      queryBufferSize          -1
10/13 16:30:06.487661 00000060 UTL default      printAuth                0
10/13 16:30:06.487664 00000060 UTL default      queryRspPolicy           0
10/13 16:30:06.487667 00000060 UTL default      numOfRpcThreads          2
10/13 16:30:06.487669 00000060 UTL default      numOfRpcSessions         10000
10/13 16:30:06.487671 00000060 UTL default      timeToGetAvailableConn   10000
10/13 16:30:06.487674 00000060 UTL default      numOfCommitThreads       2
10/13 16:30:06.487676 00000060 UTL default      numOfMnodeReadThreads    1
10/13 16:30:06.487679 00000060 UTL default      numOfVnodeQueryThreads   8
10/13 16:30:06.487682 00000060 UTL default      ratioOfVnodeStreamThrea  2.00
10/13 16:30:06.487684 00000060 UTL default      numOfVnodeFetchThreads   4
10/13 16:30:06.487687 00000060 UTL default      numOfVnodeRsmaThreads    4
10/13 16:30:06.487689 00000060 UTL default      numOfQnodeQueryThreads   8
10/13 16:30:06.487692 00000060 UTL default      numOfSnodeSharedThreads  2
10/13 16:30:06.487694 00000060 UTL default      numOfSnodeUniqueThreads  2
10/13 16:30:06.487697 00000060 UTL default      rpcQueueMemoryAllowed    1678372864
10/13 16:30:06.487699 00000060 UTL default      syncElectInterval        25000
10/13 16:30:06.487702 00000060 UTL default      syncHeartbeatInterval    1000
10/13 16:30:06.487704 00000060 UTL default      syncHeartbeatTimeout     20000
10/13 16:30:06.487707 00000060 UTL default      vndCommitMaxInterval     600000
10/13 16:30:06.487709 00000060 UTL default      mndSdbWriteDelta         200
10/13 16:30:06.487712 00000060 UTL default      mndLogRetention          2000
10/13 16:30:06.487714 00000060 UTL cfg_file     monitor                  1
10/13 16:30:06.487717 00000060 UTL default      monitorInterval          30
10/13 16:30:06.487722 00000060 UTL cfg_file     monitorFqdn              localhost
10/13 16:30:06.487725 00000060 UTL default      monitorPort              6043
10/13 16:30:06.487728 00000060 UTL default      monitorMaxLogs           100
10/13 16:30:06.487730 00000060 UTL default      monitorComp              0
10/13 16:30:06.487733 00000060 UTL default      crashReporting           1
10/13 16:30:06.487735 00000060 UTL default      telemetryReporting       1
10/13 16:30:06.487738 00000060 UTL default      telemetryInterval        43200
10/13 16:30:06.487740 00000060 UTL default      telemetryServer          telemetry.tdengine.com
10/13 16:30:06.487743 00000060 UTL default      telemetryPort            80
10/13 16:30:06.487745 00000060 UTL default      transPullupInterval      2
10/13 16:30:06.487748 00000060 UTL default      mqRebalanceInterval      2
10/13 16:30:06.487750 00000060 UTL default      ttlUnit                  86400
10/13 16:30:06.487753 00000060 UTL default      ttlPushInterval          3600
10/13 16:30:06.487755 00000060 UTL default      uptimeInterval           300
10/13 16:30:06.487758 00000060 UTL default      queryRsmaTolerance       1000
10/13 16:30:06.487760 00000060 UTL default      walFsyncDataSizeLimit    104857600
10/13 16:30:06.487763 00000060 UTL default      udf                      1
10/13 16:30:06.487765 00000060 UTL default      udfdResFuncs             
10/13 16:30:06.487768 00000060 UTL default      udfdLdLibPath            
10/13 16:30:06.487770 00000060 UTL default      disableStream            0
10/13 16:30:06.487773 00000060 UTL default      streamBufferSize         134217728
10/13 16:30:06.487775 00000060 UTL default      checkpointInterval       10800000
10/13 16:30:06.487778 00000060 UTL default      cacheLazyLoadThreshold   500
10/13 16:30:06.487780 00000060 UTL default      filterScalarMode         0
10/13 16:30:06.487783 00000060 UTL default      maxStreamBackendCache    128
10/13 16:30:06.487786 00000060 UTL default      configDir                /etc/taos/
10/13 16:30:06.487789 00000060 UTL default      scriptDir                /etc/taos/
10/13 16:30:06.487791 00000060 UTL default      logDir                   /var/log/taos/
10/13 16:30:06.487794 00000060 UTL default      minimalLogDirGB          1.00
10/13 16:30:06.487796 00000060 UTL default      numOfLogLines            10000000
10/13 16:30:06.487799 00000060 UTL default      asyncLog                 1
10/13 16:30:06.487801 00000060 UTL default      logKeepDays              0
10/13 16:30:06.487804 00000060 UTL default      debugFlag                0
10/13 16:30:06.487807 00000060 UTL default      simDebugFlag             143
10/13 16:30:06.487809 00000060 UTL default      tmrDebugFlag             131
10/13 16:30:06.487812 00000060 UTL default      uDebugFlag               131
10/13 16:30:06.487814 00000060 UTL default      rpcDebugFlag             131
10/13 16:30:06.487816 00000060 UTL default      jniDebugFlag             131
10/13 16:30:06.487819 00000060 UTL default      qDebugFlag               131
10/13 16:30:06.487821 00000060 UTL default      cDebugFlag               131
10/13 16:30:06.487824 00000060 UTL default      dDebugFlag               131
10/13 16:30:06.487827 00000060 UTL default      vDebugFlag               131
10/13 16:30:06.487829 00000060 UTL default      mDebugFlag               131
10/13 16:30:06.487832 00000060 UTL default      wDebugFlag               131
10/13 16:30:06.487834 00000060 UTL default      sDebugFlag               131
10/13 16:30:06.487837 00000060 UTL default      tsdbDebugFlag            131
10/13 16:30:06.487839 00000060 UTL default      tqDebugFlag              131
10/13 16:30:06.487841 00000060 UTL default      fsDebugFlag              131
10/13 16:30:06.487844 00000060 UTL default      udfDebugFlag             131
10/13 16:30:06.487846 00000060 UTL default      smaDebugFlag             131
10/13 16:30:06.487849 00000060 UTL default      idxDebugFlag             131
10/13 16:30:06.487851 00000060 UTL default      tdbDebugFlag             131
10/13 16:30:06.487856 00000060 UTL default      metaDebugFlag            131
10/13 16:30:06.487859 00000060 UTL default      timezone                 Asia/Shanghai (CST, +0800)
10/13 16:30:06.487862 00000060 UTL default      locale                   en_US.UTF-8
10/13 16:30:06.487865 00000060 UTL default      charset                  UTF-8
10/13 16:30:06.487867 00000060 UTL default      assert                   1
10/13 16:30:06.487870 00000060 UTL default      enableCoreFile           1
10/13 16:30:06.487872 00000060 UTL default      numOfCores               4.00
10/13 16:30:06.487875 00000060 UTL default      SSE42                    0
10/13 16:30:06.487877 00000060 UTL default      AVX                      0
10/13 16:30:06.487880 00000060 UTL default      AVX2                     0
10/13 16:30:06.487882 00000060 UTL default      FMA                      0
10/13 16:30:06.487885 00000060 UTL default      SIMD-builtins            0
10/13 16:30:06.487888 00000060 UTL default      tagFilterCache           0
10/13 16:30:06.487890 00000060 UTL default      openMax                  1048576
10/13 16:30:06.487893 00000060 UTL default      streamMax                16
10/13 16:30:06.487895 00000060 UTL default      pageSizeKB               4
10/13 16:30:06.487898 00000060 UTL default      totalMemoryKB            16390360
10/13 16:30:06.487900 00000060 UTL default      os sysname               Linux
10/13 16:30:06.487903 00000060 UTL default      os nodename              tdenginecluster-tdengine-0
10/13 16:30:06.487905 00000060 UTL default      os release               5.15.107+
10/13 16:30:06.487908 00000060 UTL default      os version               #1 SMP Thu Jun 29 09:19:06 UTC 2023
10/13 16:30:06.487910 00000060 UTL default      os machine               x86_64
10/13 16:30:06.487913 00000060 UTL default      version                  3.0.5.0
10/13 16:30:06.487915 00000060 UTL default      compatible_version       3.0.0.0
10/13 16:30:06.487918 00000060 UTL default      gitinfo                  e6ddaa7d4e7d0a7909ad16bf15f237bab25dd860
10/13 16:30:06.487920 00000060 UTL default      buildinfo                Built at 2023-06-07 20:22
10/13 16:30:06.487923 00000060 UTL =================================================================
10/13 16:30:06.502959 00000060 UTL ERROR failed to get ip from fqdn:tdenginecluster-tdengine-0.tdenginecluster-tdengine-headless.default.svc since Success, dnode can not be initialized
10/13 16:30:06.503174 00000060 DND ERROR failed to start since read config error

Expected behavior
tdengine cluster create success.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

  • OS: [e.g. iOS]
  • Browser [e.g. chrome, safari]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.
@JashBook JashBook added kind/bug Something isn't working severity/normal User may encounter the same problem labels Oct 13, 2023
@JashBook JashBook added this to the Release 0.7.0 milestone Oct 13, 2023
@JashBook JashBook added severity/major Great chance user will encounter the same problem and removed severity/normal User may encounter the same problem labels Oct 13, 2023
sophon-zt added a commit that referenced this issue Oct 30, 2023
@sophon-zt
fix: failed to get ip from fqdn (#5438)
58e7074
@sophon-zt sophon-zt linked a pull request Oct 30, 2023 that will close this issue
fix: failed to get ip from fqdn (#5438) #5674
Merged
sophon-zt added a commit that referenced this issue Oct 30, 2023
@sophon-zt
fix: failed to get ip from fqdn (#5438) (#5674)
90761cd
github-actions bot pushed a commit that referenced this issue Oct 30, 2023
@sophon-zt
fix: failed to get ip from fqdn (#5438) (#5674)
b638586 
(cherry picked from commit 90761cd)
ldming pushed a commit that referenced this issue Nov 21, 2023
@sophon-zt @ldming
fix: failed to get ip from fqdn (#5438) (#5674)
640d9d6 
(cherry picked from commit 90761cd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sophon-zt sophon-zt

Labels
bug kind/bug Something isn't working severity/major Great chance user will encounter the same problem
Projects
None yet
Milestone
Release 0.7.0
Development

Successfully merging a pull request may close this issue.

fix: failed to get ip from fqdn (#5438) apecloud/kubeblocks
3 participants
@JashBook @sophon-zt @apecloud-bot