-
-
Notifications
You must be signed in to change notification settings - Fork 18.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: expose to_pandas_kwargs
in read_parquet
with pyarrow backend
#59654
ENH: expose to_pandas_kwargs
in read_parquet
with pyarrow backend
#59654
Conversation
to_pandas_kwargs
to_pandas_kwargs
in read_parquet
with pyarrow backend
@jorisvandenbossche would you be able to review? Tagging you since you mentioned this in #56842. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be better if also add to_pandas_kwargs
to read_parquet
explicitly.
Co-authored-by: Xiao Yuan <[email protected]>
I wasn't sure if I should do this since it is an engine specific option. Happy to add it though. |
Looking forward to upgrading to this fix. Due to this issue, I am currently using pickles, which is not ideal. |
This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this. |
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the slow follow-up!
Could you also add a parameter description to the docstring?
(I see that in #57044, we had some discussion if we actually want this .. Will move that part of the discussion to the issue) |
@jorisvandenbossche I think I've addressed your comments and resolved merge conflicts. |
to_pandas_kwargs
in read_parquet
with pyarrow backendto_pandas_kwargs
in read_parquet
with pyarrow backend
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update! The code looks good, just two small comments on the docs
Co-authored-by: Joris Van den Bossche <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
Co-authored-by: Joris Van den Bossche <[email protected]>
Thanks @kleinhenz! |
to_pandas_kwargs
inread_parquet
for pyarrow engine #49236, BUG: parquet serialization/deserialization adds all dict keys into column #56842doc/source/whatsnew/vX.X.X.rst
file if fixing a bug or adding a new feature.