F-KCE-4: Error polling for Kafka topic

[dev2vv]

Hi guys,

Trying to connect Exasol (on prem, 7.1.25) with Kafka (some local docker), got an error:

[2024-12-03 11:29:31] [22002] VM error: F-UDF-CL-LIB-1127: F-UDF-CL-SL-JAVA-1002: F-UDF-CL-SL-JAVA-1013:
[2024-12-03 11:29:31] com.exasol.ExaUDFException: F-UDF-CL-SL-JAVA-1080: Exception during run
[2024-12-03 11:29:31] com.exasol.cloudetl.kafka.KafkaConnectorException: F-KCE-4: Error polling for Kafka topic 'ticker' data. It occurs for partition '0' in node '0' and vm '140236459338560'. This is an internal error that should not happen. Please report it by opening a GitHub issue.

To reproduce:

create schema KAFKA;

OPEN SCHEMA KAFKA;
CREATE OR REPLACE JAVA SET SCRIPT KAFKA_CONSUMER(...) EMITS (...) AS
  %scriptclass com.exasol.cloudetl.kafka.KafkaConsumerQueryGenerator;
  %jar /buckets/bucketfs1/bucket1/exasol-kafka-connector-extension-1.7.9.jar;
/

CREATE OR REPLACE JAVA SET SCRIPT KAFKA_IMPORT(...) EMITS (...) AS
  %scriptclass com.exasol.cloudetl.kafka.KafkaTopicDataImporter;
  %jar /buckets/bucketfs1/bucket1/exasol-kafka-connector-extension-1.7.9.jar;
/

CREATE OR REPLACE JAVA SET SCRIPT KAFKA_METADATA(
  params VARCHAR(2000),
  kafka_partition DECIMAL(18, 0),
  kafka_offset DECIMAL(36, 0)
)
EMITS (partition_index DECIMAL(18, 0), max_offset DECIMAL(36,0)) AS
  %scriptclass com.exasol.cloudetl.kafka.KafkaTopicMetadataReader;
  %jar /buckets/bucketfs1/bucket1/exasol-kafka-connector-extension-1.7.9.jar;
/

create or replace table KAFKA.WIKIPEDIA_STREAM(
    bot varchar(128),
    comment varchar(512),
    id int,


    KAFKA_PARTITION DECIMAL(18, 0),
    KAFKA_OFFSET DECIMAL(36, 0));

IMPORT INTO KAFKA.WIKIPEDIA_STREAM
FROM SCRIPT KAFKA_CONSUMER WITH
  BOOTSTRAP_SERVERS   = '192.168.123.123:9092'
  SCHEMA_REGISTRY_URL = 'http://192.168.123.123:8085'
  TOPIC_NAME          = 'ticker'
  TABLE_NAME          = 'KAFKA.WIKIPEDIA_STREAM'
  GROUP_ID            = 'exasol-kafka-udf-consumers'
  CONNECTION_NAME     = 'KAFKA_123_123_CONNECTION';
;

CREATE OR REPLACE CONNECTION KAFKA_123_123_CONNECTION
TO ''
USER ''
IDENTIFIED BY 'SECURITY_PROTOCOL=SASL_PLAINTEXT;SASL_MECHANISM=PLAIN;SASL_USERNAME=admin;SASL_PASSWORD=secret;'

Any ideas?

Regars,
VV

[Shmuma]

Hi!

It might happen in situations when kafka cluster is configured to use internally-resolvable node names or internal addresses which are not reachable from exasol DB.

The process of Exasol-Kafka communication looks like this:

1. kafka-connector-extension udf is started inside the exasol db (in a container)
2. it connects to bootstrap nodes of kafka cluster and receives cluster configuration (list of kafka nodes)
3. then it connects to those nodes to get the partition data or write into partitions

If kafka cluster is configured in a way that list of nodes it provides on a step 2 cannot be resolved by exasol (records are not present in DNS, or internal network is used), you can observe the above situation.

So, I'd start with kafka configuration check.

@tkilias we had similar problem recently, do you have more suggestions how to troubleshoot it?