Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Buffered reading #18

Open
waralex opened this issue Sep 4, 2020 · 0 comments
Open

Buffered reading #18

waralex opened this issue Sep 4, 2020 · 0 comments

Comments

@waralex
Copy link
Collaborator

waralex commented Sep 4, 2020

Currently, when reading each block, arrays with columns are created again. And this looks like an extra allocation. There is a suggestion to create them in advance and just do resize when reading the block. Given that blocks are usually about the same size + resize to a smaller size does not cause reallocation this should greatly reduce the number of allocations.

As a bonus, the read and write functions will get the same signatures.

read_col_data(sock::ClickHouseSock, data::T, ::Val{N}, args...) where {T, N}
write_col_data(sock::ClickHouseSock, data::T, ::Val{N}, args...) where {T, N}

And we will be able to read one Сlickhouse type to different Julia types using read_col_data overloads by T and N.
This in turn will give the advanced users great flexibility and the ability to create their own types and fill them directly from the clickhouse without intermediate transformations and arrays.

What do you think of this idea?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant