Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Making column names lowercase to make output table schema compatible with the Lakehouse #979

Merged
merged 2 commits into from
Jan 29, 2025

Conversation

pankajskku
Copy link
Member

@pankajskku pankajskku commented Jan 27, 2025

Why are these changes needed?

The column names must be in lowercase in Lakehouse. Otherwise, it throws the following error:

(RayTransformFileProcessor pid=11614) --- Logging error ---

(RayTransformFileProcessor pid=11614) Traceback (most recent call last):

(RayTransformFileProcessor pid=11614)   File "/home/ray/anaconda3/lib/python3.10/site-packages/lakehouse/catalog.py", line 178, in create_table_from_pyarrow_and_input_table

(RayTransformFileProcessor pid=11614)     self.create_table(table_name, location, new_schema, new_partition_spec)

(RayTransformFileProcessor pid=11614)   File "/home/ray/anaconda3/lib/python3.10/site-packages/lakehouse/catalog.py", line 243, in create_table

(RayTransformFileProcessor pid=11614)     raise ValueError(f"Invalid column names. Column names must be lowercase and can contains numbers or underscore ('_').")

(RayTransformFileProcessor pid=11614) ValueError: Invalid column names. Column names must be lowercase and can contains numbers or underscore ('_').

Related issue number (if any).

@pankajskku
Copy link
Member Author

pankajskku commented Jan 28, 2025

@touma-I Can you please have a look and merge this?
Please let me know the tag to refer to this release in the DPK-inner?

Signed-off-by: Pankaj Thorat <[email protected]>
@touma-I touma-I merged commit 5b32d2e into IBM:dev Jan 29, 2025
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants