Getting "javax.crypto.BadPaddingException" exception in MirrorMaker2

[tussinha]

Although I am able to open the truststore with the password mentioned in the mm2.properties using keytool on same pod. Getting below exception in Mirror Maker.
What could be the issue?

[2023-11-22 09:05:12,592] ERROR Scheduler for MirrorHeartbeatConnector caught exception in scheduled task: creating internal topics (org.apache.kafka.connect.mirror.Scheduler:102) org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:546) at org.apache.kafka.clients.admin.Admin.create(Admin.java:143) at org.apache.kafka.connect.util.TopicAdmin.<init>(TopicAdmin.java:277) at org.apache.kafka.connect.mirror.MirrorUtils.createCompactedTopic(MirrorUtils.java:108) at org.apache.kafka.connect.mirror.MirrorUtils.createSinglePartitionCompactedTopic(MirrorUtils.java:114) at org.apache.kafka.connect.mirror.MirrorHeartbeatConnector.createInternalTopics(MirrorHeartbeatConnector.java:83) at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93) at org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112) at org.apache.kafka.connect.mirror.Scheduler.lambda$execute$2(Scheduler.java:63) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /keystores/kafka-client-keystore.jks of type JKS at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:377) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.<init>(DefaultSslEngineFactory.java:349) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.createKeystore(DefaultSslEngineFactory.java:299) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory.configure(DefaultSslEngineFactory.java:161) at org.apache.kafka.common.security.ssl.SslFactory.instantiateSslEngineFactory(SslFactory.java:140) at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:97) at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:73) at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:192) at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) at org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:517) ... 14 more Caused by: java.io.IOException: keystore password was incorrect at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2090) at java.base/sun.security.util.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:243) at java.base/java.security.KeyStore.load(KeyStore.java:1479) at org.apache.kafka.common.security.ssl.DefaultSslEngineFactory$FileBasedStore.load(DefaultSslEngineFactory.java:374) ... 24 more Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption. ... 28 more

Below is the content of mm2.properties from the pod.

Properties from environment

bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443
clusters=source,target
emit.checkpoints.enabled=false
emit.heartbeats.enabled=false
groups=
kafka.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443
kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks
kafka.ssl.truststore.location=/keystores/truststore.jks
port=tcp://10.97.51.255:7071
port.7071.tcp=tcp://10.97.51.255:7071
port.7071.tcp.addr=10.97.51.255
port.7071.tcp.port=7071
port.7071.tcp.proto=tcp
replication.factor=3
security.protocol=SSL
service.host=10.97.51.255
service.port=7071
service.port.metrics=7071
source->target.emit.heartbeats.enabled=false
source->target.enabled=true
source->target.sync.group.offsets=false
source->target.topics=topic1
source.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.yyz-dataplane.prod.consul:443
source.config.storage.topic=namespace1.mm2.source.config.storage
source.group.id=service1.namespace1.health-aggregator-service.mm.source
source.kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks
source.kafka.ssl.truststore.location=/keystores/truststore.jks
source.offset.storage.topic=namespace1.mm2.source.offset.storage
source.security.protocol=SSL
source.status.storage.topic=namespace1.mm2.source.status.storage
ssl.keystore.location=/keystores/kafka-client-keystore.jks
ssl.truststore.location=/keystores/truststore.jks
sync.topic.acls.enabled=false
sync.topic.configs.enabled=false
target->source.enabled=false
target.bootstrap.servers=service1-kafka-bootstrap-namespace1.service.iad-dataplane.prod.consul:443
target.config.storage.topic=namespace1.mm2.target.config.storage
target.group.id=service1.namespace1.health-aggregator-service.mm.target
target.kafka.ssl.keystore.location=/keystores/kafka-client-keystore.jks
target.kafka.ssl.truststore.location=/keystores/truststore.jks
target.offset.storage.topic=namespace1.mm2.target.offset.storage
target.security.protocol=SSL
target.status.storage.topic=namespace1.mm2.target.status.storage
tasks.max=3
topic.replication.map=topic1=topic1

Connect workers (config topics)

ssl.truststore.password=HNIXZk7wsp
ssl.keystore.password=kK9EEGs18B
ssl.key.password=kK9EEGs18B

Connect sink connectors

consumer.ssl.truststore.password=HNIXZk7wsp
consumer.ssl.keystore.password=kK9EEGs18B
consumer.ssl.key.password=kK9EEGs18B

Connect source connectors

producer.ssl.truststore.password=HNIXZk7wsp
producer.ssl.keystore.password=kK9EEGs18B
producer.ssl.key.password=kK9EEGs18B