Kafka ingester to MySQL

[theprobugmaker]

Is it possible to create an ingester that reads from a Kafka topic and writes to MySQL?

The thing is that I want the Jaeger to have the Elasticsearch but also Kafka so I can read from a topic later and persist on MySQL. What is the best way to accomplish this?

Thank you.

[yurishkuro]

You can deploy Jaeger in the collector-Kafka-ingester configuration. The ingester can only write to supported storage backends (mysql is not one of them so far, although there's an open ticket for it). However, once you have data in kafka you could find various ways to ingest it into MySQL or other stores. Eg in Uber we were able to feed that data into Hive using standard pipeline, and use presto/sql to query spans.

[theprobugmaker]

@yurishkuro First of all, thank you very much for answering.
In the "collector-kafka-ingester" configuration you mean like having the collector with Kafka as "storage" that is going to be buffering information to ES (using the Ingester)? I also need a way to read from this Kafka and put on MySQL, I'm wondering if the Ingester once it reads from Kafka and puts on ES if it's going to commit the message and I won't be able to read it again to put on MySQL...
What you mean by "standard pipeline" by the way?

Thank you.

[theprobugmaker]

I just found out that Kafka has a connector called MySQL Sink that perhaps is able to do what I want but the only thing left to know if the Ingestor isn't going to remove the thing from Kafka after sending to ES.

[yurishkuro]

you can have multiple independent consumers on Kafka, they are not going to interfere with each other

[yurishkuro]

What you mean by "standard pipeline" by the way?

I meant Uber has a standard way of ingesting data from Kafka into Hive, but it required that the data in Kafka was in Avro format (whereas Jaeger collector will use protobuf to store in Kafka)

[theprobugmaker]

Thank you for the answer @yurishkuro. The only possible "solution" I could find so far was using the MySQL Sink (JDBC) Connector to get from Kafka and put on MySQL. The thing tho is that in order for the connect to get from Kafka and put on MySQL it needs the schema. The question here is: does Jaeger puts on Kafka with that schema in place? For example, does Jaeger puts on Kafka with the AVRO schema and stuff? The connector supports AVRO schema as well.

Thank you.

[theprobugmaker]

Sorry for my ignorance. Does it mean I won't be able to do what I want or it's still possible?

[joe-elliott]

What you want to do is possible. Use the proto defs in the jaeger-idl repo to generate code to parse the data stored in Kafka.

[theprobugmaker]

@joe-elliott It sounds quite challenging and I liked that. What you mean with:

generate code to parse the data stored in Kafka?

Jaeger is the one writing to Kafka. MySQL Sink is the one that reads from Kafka and writes on MySQL. The MySQL Sink Connector expects that the message on the topic has a schema and payload so it knows how to write on MySQL. That is why I'm kinda confused.

[joe-elliott]

Data has to be encoded to bytes in some manner. In this case it's proto as defined in the referenced repo.

If the MySQL Sink Connector is expecting json then it will not work out of the box and you will need to make something custom.

[theprobugmaker]

@joe-elliott I'm guessing it's getting kinda tricky. I found this article that seemed interesting: https://www.confluent.io/blog/fault-tolerance-distributed-systems-tracing-with-apache-kafka-jaeger/

Perhaps it can help me, I don't know yet. Is there an example or something similar on doing what you pointed out with the jaeger-idl proto?

Thank you for helping me out on this.