Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Delta lake sink does not support timestamp data type #13153

Closed
wenym1 opened this issue Oct 30, 2023 · 0 comments · Fixed by #13420
Closed

Delta lake sink does not support timestamp data type #13153

wenym1 opened this issue Oct 30, 2023 · 0 comments · Fixed by #13420
Assignees
Labels
type/bug Something isn't working
Milestone

Comments

@wenym1
Copy link
Contributor

wenym1 commented Oct 30, 2023

Describe the bug

Sinking to a delta lake table with timestamp data type column will gets the following validation error

Sink error: Remote sink error: sink cannot pass validation: INT96 is deprecated. As interim enable READ_INT96_AS_FIXED flag to read as byte array.

Error message/log

No response

To Reproduce

Enable minio and compactor in risedev.yml, and run ./risedev d

Go to minio UI and create a bucket called yiming-test

Download spark 3.3.1 distribution, extract and run

./spark-3.3.1-bin-hadoop3/bin/spark-sql --packages io.delta:delta-core_2.12:2.2.0,org.apache.hadoop:hadoop-aws:3.3.2 \
--conf 'spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension' \
--conf 'spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog' \
--conf 'spark.hadoop.fs.s3a.access.key=hummockadmin' \
--conf 'spark.hadoop.fs.s3a.secret.key=hummockadmin' \
--conf 'spark.hadoop.fs.s3a.endpoint=http://localhost:9301' \
--conf 'spark.hadoop.fs.s3a.path.style.access=true' 

in the shell run

create table delta.`s3a://yiming-test1/deltalake-time`(id int, time timestamp) using delta;

In psql run

create table t(id int, time timestamp);
create sink delta_lake_sink_time as select id, cast(time as string) as time from t_with_timestamp
with (
    connector = 'deltalake',
    type = 'append-only', 
    force_append_only = 'true',
    location = 's3a://yiming-test1/deltalake-time',
    s3.access.key = 'hummockadmin',
    s3.secret.key = 'hummockadmin',
    s3.endpoint = 'http://localhost:9301'
);

and then we will get the validation error.

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

@wenym1 wenym1 added the type/bug Something isn't working label Oct 30, 2023
@github-actions github-actions bot added this to the release-1.4 milestone Oct 30, 2023
@hzxa21 hzxa21 modified the milestones: release-1.4, release-1.5 Nov 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants