Kafka MV multiple consumers 1 thread details #62
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
filimonov
reviewed
Dec 4, 2023
|
|
||
| So the basic pipeline is depicted in the schema. | ||
|
|
||
|  |
filimonov
reviewed
Dec 4, 2023
| 2022.11.09 17:49:34.282077 [ 2385 ] {} <Debug> StorageKafka (kafka_destination): Started streaming to 2 attached views | ||
| 2022.11.09 17:49:34.282778 [ 73593 ] {} <Debug> StorageKafka (kafka_destination): [rdk:CGRPOP] [thrd:main]: Group "test-group-2022-11-08" received op GET_SUBSCRIPTION in state wait-broker-transport (join-state init) | ||
| 2022.11.09 17:49:34.333031 [ 2385 ] {} <Warning> StorageKafka (kafka_destination): Can't get assignment. Will keep trying. | ||
| 2022.11.09 17:49:34.333598 [ 73593 ] {} <Debug> StorageKafka (kafka_destination): [rdk:CGRPOP] [thrd:main]: Group "test-group-2022-11-08" received op GET_ASSIGNMENT in state wait-broker-transport (join-state init) |
There was a problem hiding this comment.
strange fragment of logs: it has errors. Not related to the article subject.
There was a problem hiding this comment.
Yes put them there to show how streams were 'seen' in logs but after your explanation it does not make sense, also warnings are because of offline broker probably. I'll remove them.
filimonov
reviewed
Dec 4, 2023
| These streams are not threads so this is how it works: | ||
|
|
||
| * ClickHouse will create a INSERT AST for streaming the data. | ||
| * Create two streams and join them in a union stream. Only insert into dependent views and expect that input blocks contain virtual columns |
There was a problem hiding this comment.
No. 1 stream is one consumer. No matter how many MVs you have
There was a problem hiding this comment.
After checking your diagrams I'll use them and update explanation. Thanks a lot!. 🙇🏼
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Explain more in detail how kafka_thread_per_consumer = 0 and kafka_num_consumers = 1
works