Amazon MSK (provisioned) cluster is Offline. #667

githubeto · 2024-11-22T12:13:20Z

githubeto
Nov 22, 2024

Describe

We are building MSK (Managed Streaming for Apache Kafka) and trying to connect to MSK from EKS (Elastic Kubernetes Service) pods using IRSA (IAM Roles for Service Accounts). No authentication method is configured for MSK.
However, the cluster appears offline, and brokers and other elements are not displayed on the screen. Additionally, error logs are being output.

Installation details

App Version = v1.0.0
Helm chart version = kafka-ui-1.4.9
Your application config : none.
Any IAAC configs : none.

Steps to reproduce

helm upgrade --install kafbat-ui -f kafka-ui-values.yaml kafbat-ui/kafka-ui

port-forward or create virtualservice(istio)

kafka-ui-values.yaml

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
    KAFKA_CLUSTERS_0_NAME: spdkube-stg-cdc-msk
    KAFKA_CLUSTERS_0_READONLY: "true"
    KAFKA_CLUSTERS_0_ZOOKEEPER: z-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'
    LOGGING_LEVEL_ROOT: info
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi

IAM role (arn:aws:iam::xxxxxxxxxx:role/odc-stg-kafka-ui) policy.

{
    "Statement": [
        {
            "Action": [
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:Connect"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>",
            "Sid": "VisualEditor0"
        },
        {
            "Action": [
                "kafka-cluster:DeleteGroup",
                "kafka-cluster:DescribeCluster",
                "kafka-cluster:ReadData",
                "kafka-cluster:DescribeTopicDynamicConfiguration",
                "kafka-cluster:AlterTopicDynamicConfiguration",
                "kafka-cluster:AlterGroup",
                "kafka-cluster:AlterClusterDynamicConfiguration",
                "kafka-cluster:AlterTopic",
                "kafka-cluster:CreateTopic",
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:AlterCluster",
                "kafka-cluster:DescribeGroup",
                "kafka-cluster:DescribeClusterDynamicConfiguration",
                "kafka-cluster:Connect",
                "kafka-cluster:DeleteTopic",
                "kafka-cluster:WriteData"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor1"
        },
        {
            "Action": [
                "kafka-cluster:AlterGroup",
                "kafka-cluster:DescribeGroup"
            ],
            "Effect": "Allow",
            "Resource": "<cluster-arn>/*",
            "Sid": "VisualEditor2"
        }
    ],
    "Version": "2012-10-17"
}

Logs

2024-11-22 11:24:48,175 ERROR [parallel-2] c.p.k.u.s.StatisticsService: Failed to collect cluster spdkube-stg-cdc-msk info
java.lang.IllegalStateException: Error while creating AdminClient for Cluster spdkube-stg-cdc-msk
	at com.provectus.kafka.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)
	at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onError(MonoPeekTerminal.java:258)
	at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onError(FluxMap.java:265)
	at reactor.core.publisher.Operators$MonoSubscriber.onError(Operators.java:1886)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.signalCached(MonoCacheTime.java:340)
	at reactor.core.publisher.MonoCacheTime$CoordinatorSubscriber.onError(MonoCacheTime.java:363)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:180)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onError(MonoIgnoreThen.java:278)
	at reactor.core.publisher.MonoPublishOn$PublishOnSubscriber.run(MonoPublishOn.java:187)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68)
	at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes

githubeto · 2024-11-22T12:35:23Z

githubeto
Nov 22, 2024
Author

I also tried this pattern.

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::735099427143:role/odc-stg-kafka-ui
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
    KAFKA_CLUSTERS_0_NAME: spdkube-stg-cdc-msk
#    KAFKA_CLUSTERS_0_READONLY: "true"
#    KAFKA_CLUSTERS_0_ZOOKEEPER: z-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181,z-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:2181
#    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: AWS_MSK_IAM
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'software.amazon.msk.auth.iam.IAMLoginModule required;'
#    KAFKA_CLUSTERS_0_PROPERTIES_SASL_CLIENT_CALLBACK_HANDLER_CLASS: 'software.amazon.msk.auth.iam.IAMClientCallbackHandler'
    LOGGING_LEVEL_ROOT: info
yamlApplicationConfig:
  kafka:
    clusters:
      - name: yaml
        bootstrapServers: b-1.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.xxxxxxxxxx.r31ee1.c2.kafka.ap-northeast-1.amazonaws.com:9092
        properties:
          security.protocol: SASL_SSL
          sasl.mechanism: AWS_MSK_IAM
          sasl.jaas.config: software.amazon.msk.auth.iam.IAMLoginModule required;
          sasl.client.callback.handler.class: software.amazon.msk.auth.iam.IAMClientCallbackHandler
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi

0 replies

Haarolean · 2024-11-24T08:04:22Z

Haarolean
Nov 24, 2024
Maintainer

Timed out waiting for a node assignment

this is 99% IAM/firewall/access issue.
Please see this, might help.

0 replies

githubeto · 2024-11-29T06:27:51Z

githubeto
Nov 29, 2024
Author

this is 99% IAM/firewall/access issue.
Please see this, might help.

Yes, I have already checked.
My MSK cluster is restricted to SSL connections, but there is no authentication, including IAM authentication.

One thing I'm concerned about is where should I obtain the mounted /ssl/cacerts from? (What steps should I follow to create it?)
I am not sure where to obtain a JKS file from, or what steps to follow to create one.

Here is my latest values.yaml

replicaCount: 1
autoscaling:
  enabled: false
  minReplicas: 1
  maxReplicas: 2
  targetCPUUtilizationPercentage: 80
serviceAccount: 
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::xxxxxxxxxx:role/xxxxxxxxxx
  name: "kafka-ui"
ingress:
  enabled: false
envs:
  config:
    KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: b-1.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9094,b-2.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9094,b-3.xxxxxxxxxx.c2.kafka.ap-northeast-1.amazonaws.com:9094
    KAFKA_CLUSTERS_0_NAME: Staging
    KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SSL
    KAFKA_CLUSTERS_0_PROPERTIES_SSL_TRUSTSTORE_LOCATION: /ssl/cacerts
    KAFKA_CLUSTERS_0_PROPERTIES_SSL_KEYSTORE_LOCATION: /ssl/cacerts
    LOGGING_LEVEL_ROOT: info
resources:
  limits:
    memory: 512Mi
  requests:
    cpu: 300m
    memory: 512Mi
volumeMounts:
 - name: ssl-config
   mountPath: /ssl
volumes:
 - name: ssl-config
   configMap:
     name: ssl-file

Error:

2024-11-29 06:17:57,121 ERROR [parallel-3] i.k.u.s.StatisticsService: Failed to collect cluster spdkube-stg-cdc-msk info
java.lang.IllegalStateException: Error while creating AdminClient for Cluster spdkube-stg-cdc-msk
	at io.kafbat.ui.service.AdminClientServiceImpl.lambda$createAdminClient$5(AdminClientServiceImpl.java:56)
	at reactor.core.publisher.Mono.lambda$onErrorMap$28(Mono.java:3783)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)
	at reactor.core.publisher.Operators.error(Operators.java:198)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:135)
	at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4480)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onComplete(FluxSwitchIfEmpty.java:82)
	at reactor.core.publisher.Operators.complete(Operators.java:137)
	at reactor.core.publisher.MonoEmpty.subscribe(MonoEmpty.java:46)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4495)
	at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:427)
	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:446)
	at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:533)
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
	at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:551)
	at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:488)
	at org.apache.kafka.clients.admin.Admin.create(Admin.java:134)
	at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:39)
	at io.kafbat.ui.service.AdminClientServiceImpl.lambda$createAdminClient$2(AdminClientServiceImpl.java:53)
	at reactor.core.publisher.MonoSupplier.call(MonoSupplier.java:67)
	at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:127)
	... 16 common frames omitted
Caused by: org.apache.kafka.common.errors.InvalidConfigurationException: SSL key store is specified, but key store password is not specified.

1 reply

Haarolean Dec 4, 2024
Maintainer

PTAL at this #483 (reply in thread)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Amazon MSK (provisioned) cluster is Offline. #667

{{title}}

Replies: 3 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Amazon MSK (provisioned) cluster is Offline. #667

githubeto Nov 22, 2024

Describe

Installation details

Steps to reproduce

Logs

Replies: 3 comments · 1 reply

githubeto Nov 22, 2024 Author

Haarolean Nov 24, 2024 Maintainer

githubeto Nov 29, 2024 Author

Haarolean Dec 4, 2024 Maintainer

githubeto
Nov 22, 2024

Replies: 3 comments 1 reply

githubeto
Nov 22, 2024
Author

Haarolean
Nov 24, 2024
Maintainer

githubeto
Nov 29, 2024
Author

Haarolean Dec 4, 2024
Maintainer