Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support webhook sink #18606

Open
lmatz opened this issue Sep 19, 2024 · 2 comments
Open

feat: support webhook sink #18606

lmatz opened this issue Sep 19, 2024 · 2 comments

Comments

@lmatz
Copy link
Contributor

lmatz commented Sep 19, 2024

Is your feature request related to a problem? Please describe.

Webhook Sink let users send data to a downstream endpoint via standard HTTP post request.

These downstream in practice are mostly user-facing applications, e.g. Slack, Email, and all kinds of CRM tools, etc. Popular ones can be found here: https://www.getcensus.com/integrations?search=&integration-type=Destination&category=
Some of the detinations also appear in the sink connectors RW supports.

Take an example, once we enable the webhook of Slack: https://api.slack.com/messaging/webhooks
we hope that the user can do in RW:

CREATE TABLE slack (
    user_id varchar,
    notification varchar
) WITH (
    connector = 'webhook',
    endpoint = 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
);
  1. format: in theory it can be any. json is the most popular one, but nothing prevents it from using xml.
    We can either put it into With as format = 'json', or reuse the format plain encode json. xml support looks unnecessary. Reusing format plain encode json is a bit abusive but it seems there is no conflict with any other semantics regarding webhook sink.

  2. request type: in theory it can use request types other than post, e.g. put delete. But, in practice it seems only necessary to use post, i.e. append-only. We can discard all the deletes and updatedeletes if the upstream of the sink is non append-only.

  3. header: such as 'content-type: application/json', Authorization: Basic dXNlcjEyMzpwYXNzMTIz where the last part is the base64-encoded username:password string. (This is just one most common authentication method, aka basic authentication, there are many others that we can support on demand in the future). Users can put anything into it. I suggest we leave a field for user to fill in.

We expect users to define as follows.

CREATE TABLE slack (
    user_id varchar,
    notification varchar
) WITH (
    connector = 'webhook',
    endpoint = 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX',
    header = '......',
) format plain encode json;

user_id and notification will be translated to two fields in HTTP json payload.

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

Quite many popular modern stream processing systems support Webhook sink, either natively or via plugin.

@github-actions github-actions bot added this to the release-2.1 milestone Sep 19, 2024
@tabVersion
Copy link
Contributor

tabVersion commented Sep 20, 2024

header: such as 'content-type: application/json', Authorization: Basic dXNlcjEyMzpwYXNzMTIz where the last part is the base64-encoded username:password string. (This is just one most common authentication method, aka basic authentication, there are many others that we can support on demand in the future). Users can put anything into it. I suggest we leave a field for user to fill in.

Actually header is a hashmap with type string -> string, shall we enforce the users to do it with a jsonb column or a map column?

Similar to this request #17850, @xiangjinwu and I plan to enforce such a column as message header.

@lmatz
Copy link
Contributor Author

lmatz commented Sep 20, 2024

shall we enforce the users to do it with a jsonb column or a map column?

It enables dynamically setting the header, which is more powerful. Not a bad idea.

I think we can provide both options, static definitions in the with clause for common cases, and a dynamic one for advanced/premium usage.

@lmatz lmatz modified the milestones: release-2.1, release-2.2 Oct 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants