Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable all json reader options in pylibcudf read_json #17563

Open
wants to merge 6 commits into
base: branch-25.02
Choose a base branch
from

Conversation

karthikeyann
Copy link
Contributor

@karthikeyann karthikeyann commented Dec 10, 2024

Description

This PR exposes all json reader options in pylibcudf and enables it via kwargs in cudf.read_json
since kwargs cannot be used in cython, kwargs is passed as dict to cython.
These options are hidden in docs intentionally, as these options are mostly used for testing feature requests from spark json reader now. These options are expected to change.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@karthikeyann karthikeyann added 2 - In Progress Currently a work in progress 4 - Needs Review Waiting for reviewer to review or respond cuIO cuIO issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change cudf.pandas Issues specific to cudf.pandas pylibcudf Issues specific to the pylibcudf package labels Dec 10, 2024
@karthikeyann karthikeyann requested a review from vyasr December 10, 2024 04:13
@karthikeyann karthikeyann requested a review from a team as a code owner December 10, 2024 04:13
@github-actions github-actions bot added Python Affects Python cuDF API. and removed cudf.pandas Issues specific to cudf.pandas labels Dec 10, 2024
@@ -198,6 +193,7 @@ def read_json(
mixed_types_as_string=mixed_types_as_string,
prune_columns=prune_columns,
recovery_mode=c_on_bad_lines,
extra_parameters=kwargs,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an FYI. I'm updating this API to use json reader options classes to match the other IO functions (eg. https://github.com/rapidsai/cudf/blob/branch-25.02/python/pylibcudf/pylibcudf/io/parquet.pyx#L309). So it will look like

plc.io.json.read_json(
    plc.io.json.JsonReaderOptions.builder(
        plc.io.SourceInfo(file_paths_or_buffers)
    )
    .byte_range_size(...)
    ...
)

Copy link
Contributor Author

@karthikeyann karthikeyann Dec 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These options will change because most of options are for spark, and they will change until all spark json feature requests are completed. These options are not intended for python users. It's exposed for quicker testing.

@karthikeyann karthikeyann removed the 2 - In Progress Currently a work in progress label Dec 11, 2024
@karthikeyann
Copy link
Contributor Author

/ok to test

@karthikeyann karthikeyann requested review from a team and Matt711 December 11, 2024 21:56
@karthikeyann
Copy link
Contributor Author

/ok to test

@karthikeyann karthikeyann requested a review from Matt711 December 13, 2024 21:36
@vyasr
Copy link
Contributor

vyasr commented Dec 13, 2024

I'm OK with the dict as a short-term experimental solution but I wouldn't want to merge it without a plan for removing and replacing with an options struct the way that the other I/O APIs work. Will this be removed by 25.02? Can we open an issue and tag it as required for the release?

@vyasr
Copy link
Contributor

vyasr commented Dec 14, 2024

To clarify, I'm referring to json_reader_options which should wind up with a corresponding Cython type like what @Matt711 showed above. We need to implement a wrapper type for the options in Cython like we do for other file formats, e.g. the CsvReaderOptions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4 - Needs Review Waiting for reviewer to review or respond cuIO cuIO issue improvement Improvement / enhancement to an existing function non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
Status: In Progress
Development

Successfully merging this pull request may close these issues.

3 participants