Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

batch: support for setting scan parallelism #15627

Closed
zwang28 opened this issue Mar 12, 2024 · 1 comment
Closed

batch: support for setting scan parallelism #15627

zwang28 opened this issue Mar 12, 2024 · 1 comment

Comments

@zwang28
Copy link
Contributor

zwang28 commented Mar 12, 2024

Is your feature request related to a problem? Please describe.

The current batch_parallelism doesn't affect batch scan parallelism. The batch scan parallelism always equals to the stream parallelism of the table.
It's not always proper to change stream parallelism in order to get an ideal batch scan parallelism. Some downsides are:

  • it brings more memory footprints in compute node.
  • it results larger checkpoint size which impair storage compaction performance.

Describe the solution you'd like

Add a session variable batch_scan_parallelism. If set, it takes precedence.

Describe alternatives you've considered

No response

Additional context

No response

@github-actions github-actions bot added this to the release-1.8 milestone Mar 12, 2024
@zwang28
Copy link
Contributor Author

zwang28 commented Mar 12, 2024

Just don't limit max batch scan parallelism by stream parallelism.

// Set max serving parallelism to `streaming_parallelism`. It's not a must.

@zwang28 zwang28 closed this as not planned Won't fix, can't repro, duplicate, stale Mar 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant