Issue running kminion with TLS enabled

[mrandreyeff]

Hi there,
I'm trying to enable TLS, so I've added TLS files and configs:

      tls:
        enabled: true
        caFilepath: /opt/certs/kafka/ca.crt
        certFilepath: /opt/certs/kafka/tls.crt
        keyFilepath: /opt/certs/kafka/tls.key

After applied, I get error:

{"level":"warn","ts":"2022-07-22T12:40:40.460Z","logger":"main.kgo_client","msg":"unable to open connection to broker","addr":"kafka1:9093","broker":"seed 1","err":"tls: server sent two HelloRetryRequest messages"}
{"level":"warn","ts":"2022-07-22T12:40:40.465Z","logger":"main.kgo_client","msg":"unable to open connection to broker","addr":"kafka2:9093","broker":"seed 0","err":"tls: server sent two HelloRetryRequest messages"}
{"level":"fatal","ts":"2022-07-22T12:40:40.465Z","logger":"main.kgo_client","msg":"failed to test connectivity to Kafka cluster","error":"failed to request metadata: unable to dial: tls: server sent two HelloRetryRequest messages"}

There are multiple other clients using java and python, also tried the same TLS credentials with python (PEM format) - worked well. Kafka brokers use Java 11, so according to docs it shall be TLS 1.3.
As far as I see, after some googling shows the error shall come from https://github.com/golang/go/blob/master/src/crypto/tls/handshake_client_tls13.go#L296

Tried also with TLS 1.2 and 1.3 via openssl client - works well:

docker run --rm -v /path/to/certs/:/mycerts -it alpine/openssl s_client -debug -connect kafka1:9093 -tls1_3 -CAfile /mycerts/ca.crt -key /mycerts/tls.key -cert /mycerts/tls.crt

Does anybody use TLS 1.3 with kminion (or underlying go Dial library)?
Are there any configs that may help from broker/client side?

[weeco]

Hello,
TLS 1.3 should be supported. Could you please provide additional information:

• What kMinion version do you use?
• Logs with debug level enabled (react the sensitive parts such as broker addresses)

[mrandreyeff]

• Version: https://github.com/redpanda-data/kminion/tree/30f84f50ebba412cf4194444f8e82f907a801d0d
  (from then I see only possibility to provide TLS as bytes)

• with debug nearly the same:

{"level":"info","ts":"2022-07-22T15:42:19.469Z","logger":"main","msg":"started kminion","version":""}
{"level":"info","ts":"2022-07-22T15:42:19.470Z","logger":"main.kafka_service","msg":"connecting to Kafka seed brokers, trying to fetch cluster metadata","seed_brokers":"kafka1:9093"}
{"level":"debug","ts":"2022-07-22T15:42:19.470Z","logger":"main.kgo_client","msg":"opening connection to broker","addr":"kafka1:9093","broker":"seed 0"}
{"level":"warn","ts":"2022-07-22T15:42:19.520Z","logger":"main.kgo_client","msg":"unable to open connection to broker","addr":"kafka1:9093","broker":"seed 0","err":"tls: server sent two HelloRetryRequest messages"}
{"level":"debug","ts":"2022-07-22T15:42:19.520Z","logger":"main.kgo_client","msg":"opening connection to broker","addr":"kafka1:9093","broker":"seed 0"}
{"level":"fatal","ts":"2022-07-22T15:42:19.520Z","logger":"main.kgo_client","msg":"failed to test connectivity to Kafka cluster","error":"failed to request metadata: unable to dial: tls: server sent two HelloRetryRequest messages"}

[mrandreyeff]

From broker side, where nodeid1, kafka1-ip and pod-ip - some obfuscated values:

[2022-07-22 14:47:03,722] INFO [SocketServer listenerType=ZK_BROKER, nodeId=nodeid1] Failed authentication with /pod-ip (SSL handshake failed) (org.apache.kafka.common.network.Selector)
[2022-07-22 14:47:03,722] TRACE [SocketServer listenerType=ZK_BROKER, nodeId=nodeid1] clients: Removed a reference to ClientInformation(softwareName=unknown, softwareVersion=unknown).  0 reference(s) remaining. (org.apache.kafka.common.network.Selector)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, handshakeResult Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 261, netWriteBuffer pos 0 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,725] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_WRAP status OK (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, handshakeResult Status = OK HandshakeStatus = NEED_TASK bytesConsumed = 261 bytesProduced = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_WRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 0 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeWrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_WRAP channelId kafka1-ip:9093-pod-ip:44586-238, handshakeResult Status = OK HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 166 sequenceNumber = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status BUFFER_UNDERFLOW (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, handshakeResult Status = BUFFER_UNDERFLOW HandshakeStatus = NEED_UNWRAP bytesConsumed = 0 bytesProduced = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 373, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,726] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_UNWRAP status OK (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,727] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeUnwrap: handshakeStatus NEED_WRAP status OK (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,727] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_UNWRAP channelId kafka1-ip:9093-pod-ip:44586-238, handshakeResult Status = OK HandshakeStatus = NEED_TASK bytesConsumed = 367 bytesProduced = 0, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,727] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake NEED_WRAP channelId kafka1-ip:9093-pod-ip:44586-238, appReadBuffer pos 0, netReadBuffer pos 0, netWriteBuffer pos 166 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,727] TRACE [SslTransportLayer channelId=kafka1-ip:9093-pod-ip:44586-238 key=channel=java.nio.channels.SocketChannel[connected local=/kafka1-ip:9093 remote=/pod-ip:44586], selector=sun.nio.ch.EPollSelectorImpl@9fc5047, interestOps=1, readyOps=0] SSLHandshake handshakeWrap kafka1-ip:9093-pod-ip:44586-238 (org.apache.kafka.common.network.SslTransportLayer)
[2022-07-22 14:47:03,727] INFO [SocketServer listenerType=ZK_BROKER, nodeId=nodeid1] Failed authentication with /pod-ip (SSL handshake failed) (org.apache.kafka.common.network.Selector)

Upd: The timestamps are different, since pods are restarting in crashloop, so taken some random

[weeco]

@mrandreyeff I'm sorry, I haven't seen this error before. As you've already figured out this error messages comes from Go. The kafka client we use just uses Go's underlying TLS dialer, so there's not much that we could do about it I think. Maybe you want to submit an issue at the Go repository and see what additional information you could provide there to figure out what's going on?

I'm afraid I can't be of help here otherwise.

[mrandreyeff]

@weeco , do you know if someone uses/used TLS with kminion, or if there were successful tests using SSL?

[weeco]

@mrandreyeff Yes, KMinion as well as Redpanda Console where the same kafka library and bootstrapping code is used against a lot of TLS secured clusters with all kinds of different distributions (Redpanda, Amazon MSK, Apache Kafka, Confluent Cloud, ...)

[mrandreyeff]

Thanks, then it is probably worth to dig deeper and search on my side

[hdhoang]

hi, we have some clusters with custom CA. those regressed recently w/ kminion, while older version & kafka-minion work. the ca file is in pem format. we will give some log soon.

Sorry to disappoint, our problem is different.

To support old kafka-minion in paraller, we set both KAFKA_TLS_CAFILEPATH & KAFKA_TLS_CA_FILE_PATH envvars (to the same path value).

Since rev a066, KAFKA_TLS_CA envvar has special meaning, and prefix-matched setting KAFKA_TLS_CA_FILE_PATH confuses config parser with error:

{
  "level": "fatal",
  "ts": 1658812168.1207764,
  "caller": "kminion/main.go:43",
  "msg": "failed to parse config",
  "error": "1 error(s) decoding:\n\n* 'kafka.tls.ca[0]' expected type 'uint8', got unconvertible type 'map[string]interface {}', value: 'map[file:map[path:kafka_chain.pem]]'",
  "stacktrace": "main.main\n\t/home/hdhoang/github/kminion/main.go:43\nruntime.main\n\t/usr/lib/golang/src/runtime/proc.go:250"
}

in our case, kminion hasn't reached network conn stage yet.

Sorry for mixing things up, we'll work around this locally.