Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support dynamic index for elasticsearch sink #15626

Closed
neverchanje opened this issue Mar 12, 2024 · 0 comments
Closed

support dynamic index for elasticsearch sink #15626

neverchanje opened this issue Mar 12, 2024 · 0 comments

Comments

@neverchanje
Copy link
Contributor

neverchanje commented Mar 12, 2024

Is your feature request related to a problem? Please describe.

The user wants to create one sink that writes to multiple indexes dynamically. The sink decides which index to write to based on a column.

Describe the solution you'd like

index in ES is like tables in other relational databases. It's strange to automatically export data to different tables within one sink. But it might be reasonable in the context of ES. Flink supports it and they call this feature "dynamic index" https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/elasticsearch/#dynamic-index

The user doesn't need RisingWave to create indexes because indexes are prepared beforehand.

CREATE SINK s AS
SELECT '(' || a || ',' || b || ')' AS idx
WITH (
  index_column = 'idx'
)

My proposal (if we finally decide to work on it) is to use a index_column field to specify the column that indicates the destination "index".

Describe alternatives you've considered

No response

Additional context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants