From be475b457193609067140f9afe2b88df6ec82243 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Sat, 18 May 2024 14:18:35 +0200 Subject: [PATCH 1/4] Update readme with quickstart steps --- README.md | 16 +++++++++++++++- ...operties => quickstart-opensearch.properties} | 0 2 files changed, 15 insertions(+), 1 deletion(-) rename config/{quickstart-elasticsearch.properties => quickstart-opensearch.properties} (100%) diff --git a/README.md b/README.md index 4e70e115..cbb24714 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Aiven's OpenSearch® Connector for Apache Kafka® +# Aiven's OpenSearch® Sink Connector for Apache Kafka® [![Build Status](https://github.com/aiven/opensearch-connector-for-apache-kafka/actions/workflows/master_push_workflow.yml/badge.svg)](https://github.com/aiven/opensearch-connector-for-apache-kafka/actions) @@ -75,6 +75,20 @@ curl http://localhost:8083/connector-plugins | jq . [OpenSearch® Sink Connector Configuration Options](docs/opensearch-sink-connector-config-options.rst) +## QuickStart guide + +1. Install and Run Kafka: Ensure Kafka is properly installed and running. +2. Run Kafka Connect: Make sure Kafka Connect is up and running. +3. Add OpenSearch Sink to Kafka Connect: Follow the 'How to install' instructions to add the OpenSearch sink connector to Kafka Connect. Example worker config is located here https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-opensearch.properties +4. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed +5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. +6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. If SSL is enabled, configure the credentials in the connector worker properties. +7. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. +8. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. +9. Create Index Pattern: Create an index pattern in OpenSearch. +10. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. +11. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. + # Contribute [Source Code](https://github.com/aiven/aiven-kafka-connect-opensearch) diff --git a/config/quickstart-elasticsearch.properties b/config/quickstart-opensearch.properties similarity index 100% rename from config/quickstart-elasticsearch.properties rename to config/quickstart-opensearch.properties From 8cf417a8bbe71ed95c350b913f8114cb239cf373 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Sat, 18 May 2024 14:29:13 +0200 Subject: [PATCH 2/4] Update readme with quickstart steps --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index cbb24714..d39be32c 100644 --- a/README.md +++ b/README.md @@ -77,16 +77,16 @@ curl http://localhost:8083/connector-plugins | jq . ## QuickStart guide -1. Install and Run Kafka: Ensure Kafka is properly installed and running. -2. Run Kafka Connect: Make sure Kafka Connect is up and running. +1. Install and Run Kafka: Ensure Kafka is properly installed and running. +2. Run Kafka Connect: Make sure Kafka Connect is up and running. 3. Add OpenSearch Sink to Kafka Connect: Follow the 'How to install' instructions to add the OpenSearch sink connector to Kafka Connect. Example worker config is located here https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-opensearch.properties -4. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed -5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. -6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. If SSL is enabled, configure the credentials in the connector worker properties. -7. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. -8. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. -9. Create Index Pattern: Create an index pattern in OpenSearch. -10. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. +4. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed +5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. +6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. If SSL is enabled, configure the credentials in the connector worker properties. +7. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. +8. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. +9. Create Index Pattern: Create an index pattern in OpenSearch. +10. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. 11. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. # Contribute From e6b07ba47af8a71e0dc1312e1bfc620e4d714fd3 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Tue, 21 May 2024 11:06:31 +0200 Subject: [PATCH 3/4] Adding step for required software in readme --- README.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index d39be32c..c54c3dd0 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,14 @@ The code was forked and all classes were renamed. # Documentation -## How to install +## Install and configure required software: +1. Install and Run Kafka (https://kafka.apache.org/quickstart) +2. Install and Run Kafka Connect (https://kafka.apache.org/documentation/#connect_running) +3. Install and Run OpenSearch (https://opensearch.org/docs/latest/install-and-configure/install-opensearch/index/) + - If OpenSearch is running on SSL, make sure to copy the credentials and copy them to woker properties as mentioned in Quickstart guide below +4. Install Aiven's OpenSearch® Sink Connector (see below) + +## How to install Aiven's OpenSearch® Sink Connector 1. Connector plugins are packaged in zip/tar format to be released 2. Users download plugins from GitHub releases or build binaries from source @@ -81,13 +88,14 @@ curl http://localhost:8083/connector-plugins | jq . 2. Run Kafka Connect: Make sure Kafka Connect is up and running. 3. Add OpenSearch Sink to Kafka Connect: Follow the 'How to install' instructions to add the OpenSearch sink connector to Kafka Connect. Example worker config is located here https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-opensearch.properties 4. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed -5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. -6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. If SSL is enabled, configure the credentials in the connector worker properties. -7. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. -8. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. -9. Create Index Pattern: Create an index pattern in OpenSearch. -10. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. -11. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. +5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. Example of adding acls : https://kafka.apache.org/documentation/#security_authz_examples +6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. +7. If SSL is enabled on OpenSearch, configure the credentials in the connector worker properties connection.username & connection.password. +8. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. +9. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. +10. Create Index Pattern: Create an index pattern in OpenSearch. +11. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. +12. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. # Contribute From 9f01015e7d622c3c137e6c23a244daf2bb539c42 Mon Sep 17 00:00:00 2001 From: Muralidhar Basani Date: Tue, 21 May 2024 16:44:19 +0200 Subject: [PATCH 4/4] Updating quickstart --- README.md | 21 +++++++++------------ 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index c54c3dd0..0fd0408e 100644 --- a/README.md +++ b/README.md @@ -84,18 +84,15 @@ curl http://localhost:8083/connector-plugins | jq . ## QuickStart guide -1. Install and Run Kafka: Ensure Kafka is properly installed and running. -2. Run Kafka Connect: Make sure Kafka Connect is up and running. -3. Add OpenSearch Sink to Kafka Connect: Follow the 'How to install' instructions to add the OpenSearch sink connector to Kafka Connect. Example worker config is located here https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-opensearch.properties -4. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed -5. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. Example of adding acls : https://kafka.apache.org/documentation/#security_authz_examples -6. Install and Run OpenSearch: Ensure OpenSearch is installed and running. -7. If SSL is enabled on OpenSearch, configure the credentials in the connector worker properties connection.username & connection.password. -8. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. -9. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. -10. Create Index Pattern: Create an index pattern in OpenSearch. -11. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. -12. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. +1. Ensure that the required software is installed and running +2. Add OpenSearch Sink connector to Kafka Connect: Follow the 'How to install' instructions to add the OpenSearch sink connector to Kafka Connect. Example worker config is located here https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/blob/main/config/quickstart-opensearch.properties +3. Verify plugin installation : Visit http://localhost:8083/connectors to confirm that the OpenSearch sink connector is listed +4. Check ACLs (If Enabled): If ACLs are enabled on Kafka, ensure there are no authorization exceptions for the topic and group resources. Example of adding acls : https://kafka.apache.org/documentation/#security_authz_examples +5. Produce Events: Produce JSON-formatted events to the Kafka topic specified in the worker properties. +6. Index Creation: An index will be created in OpenSearch with the same name as the Kafka topic. +7. Create Index Pattern: Create an index pattern in OpenSearch. +8. Discover Events: Events produced to the Kafka topic can now be discovered in OpenSearch. +9. Trouble shooting: If there are any deserialization errors in the connector logs, try setting schema.ignore to true. # Contribute