Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide example of properties in JSON format #85

Open
peetkes opened this issue Oct 5, 2022 · 2 comments
Open

Provide example of properties in JSON format #85

peetkes opened this issue Oct 5, 2022 · 2 comments

Comments

@peetkes
Copy link

peetkes commented Oct 5, 2022

If you want to add a connector via the rest endpoint you have to provide the configuration in yaml format.
Could we also provide a set of configurations in yaml format?

example of selecting connctor via rest api:

curl -d @./connect-mqtt-source.json -H "Content-Type: application/json" -X POST http://localhost:8083/connectors

whereas connect-mqtt-source.json contains the properties in json format

@rjrudin
Copy link
Contributor

rjrudin commented Oct 5, 2022

Thanks @peetkes - can you include a link to an example of a YAML config file or the Kafka reference for it? Should be straightforward to implement this and we'll almost certainly include it in the 1.8.0 release. If it's simple enough, I can see if we can squeeze it into the upcoming 1.7.0 release.

@peetkes
Copy link
Author

peetkes commented Oct 6, 2022

Kafka Configuration description can be found here: https://docs.confluent.io/platform/current/installation/configuration/connect/index.html
See below for a sample I cooked up whilst testing kafka. it is btw a json format as that is the only format you can use when configuring via kafka's rest-api.

{
  "name": "marklogic-sink",
  "config": {
    "group.id":"marklogic-connector",
    "connector.class": "com.marklogic.kafka.connect.sink.MarkLogicSinkConnector",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.storage.StringConverter",
    "offset.storage.file.filename":"/tmp/connect.offsets",
    "offset.flush.interval.ms":10000,
    "tasks.max": 1,
    "topics": "marklogic,connect-custom",
    "ml.connection.host": "marklogic.kafka",
    "ml.connection.port": "8000",
    "ml.connection.securityContextType": "DIGEST",
    "ml.connection.username": "admin",
    "ml.connection.password": "admin",
    "ml.connection.simpleSsl":false,
    "ml.dmsdk.batchSize":100,
    "ml.dmsdk.threadCount":8,
    "ml.document.collections": "purchases",
    "ml.document.format": "json",
    "ml.document.uriPrefix": "/purchases/",
    "ml.document.uriSuffix": ".json",
    "ml.document.addTopicToCollections": false,
    "ml.document.permissions": "rest-reader,read,rest-writer,update",
    "ml.dmsdk.transformParamsDelimiter": ",",
    "ml.datahub.flow.logResponse": true,
    "ml.log.record.key": false,
    "ml.log.record.headers": false,
    "ml.connection.enableCustomSsl": false,
    "ml.connection.customSsl.mutualAuth": false,
    "confluent.topic.bootstrap.servers": "kafka:9092",
    "confluent.topic.replication.factor": 1
  }
}

@rjrudin rjrudin changed the title provide properties in yaml format so you can use them with the REST API Provide example of properties in JSON format Feb 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants