Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance test of YdbSpliterator #42

Open
lavrukov opened this issue Mar 6, 2024 · 1 comment · May be fixed by #69
Open

Performance test of YdbSpliterator #42

lavrukov opened this issue Mar 6, 2024 · 1 comment · May be fixed by #69
Assignees
Labels

Comments

@lavrukov
Copy link
Contributor

lavrukov commented Mar 6, 2024

YdbSpliterator uses ArrayBlockingQueue<>(1) which produce overhead on threads synchronisation. This decision was made based on the desire to minimize memory usage, but it could be too slow.

We have to know numbers. We need to make a performance test on 1kk amount of data with:

  • current spliterator
  • spliterator with ArrayBlockingQueue<>(N) where N in [100, 1000, 10000]
  • spliterator with non-blocking queue or something which could work better by time
@lavrukov lavrukov added feature New feature or request refactoring Internal improvement that does not change the API or library behavior labels Mar 6, 2024
@lavrukov lavrukov self-assigned this Mar 6, 2024
@lavrukov lavrukov added research and removed feature New feature or request refactoring Internal improvement that does not change the API or library behavior labels Mar 6, 2024
@nvamelichev
Copy link
Collaborator

nvamelichev commented Mar 6, 2024

If current ArrayBlockingQueue(1) implementation turns out to be too slow in many practical applications, then the buffer size could be made adjustable in ReadTableParams :-)

@nvamelichev nvamelichev linked a pull request May 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants