Skip to content

Commit

Permalink
Bug fix: restrict lines=True to JSON format in Kafka read_gdf method (#…
Browse files Browse the repository at this point in the history
…17333)

This pull request modifies the read_gdf method in kafka.py to pass the lines parameter only when the message_format is "json". This prevents lines from being passed to other formats (e.g., CSV, Avro, ORC, Parquet), which do not support this parameter.

Authors:
  - Hirota Akio (https://github.com/a-hirota)
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Vyas Ramasubramani (https://github.com/vyasr)

URL: #17333
  • Loading branch information
a-hirota authored Nov 20, 2024
1 parent 7158ee0 commit 05365af
Showing 1 changed file with 8 additions and 3 deletions.
11 changes: 8 additions & 3 deletions python/custreamz/custreamz/kafka.py
Original file line number Diff line number Diff line change
Expand Up @@ -151,9 +151,14 @@ def read_gdf(
"parquet": cudf.io.read_parquet,
}

result = cudf_readers[message_format](
kafka_datasource, engine="cudf", lines=True
)
if message_format == "json":
result = cudf_readers[message_format](
kafka_datasource, engine="cudf", lines=True
)
else:
result = cudf_readers[message_format](
kafka_datasource, engine="cudf"
)

# Close up the cudf datasource instance
# TODO: Ideally the C++ destructor should handle the
Expand Down

0 comments on commit 05365af

Please sign in to comment.