Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Native writer should respect table properties for spark 3.2/3.3 #8039

Open
yikf opened this issue Nov 25, 2024 · 0 comments · May be fixed by #8040
Open

[VL] Native writer should respect table properties for spark 3.2/3.3 #8039

yikf opened this issue Nov 25, 2024 · 0 comments · May be fixed by #8040
Labels
bug Something isn't working triage

Comments

@yikf
Copy link
Contributor

yikf commented Nov 25, 2024

Backend

VL (Velox)

Bug description

Native writer should respect table properties for spark 3.2/3.3, like compress codec.

Let's say have a test suite,

Seq(true, false).foreach { enableNativeWrite =>
      withSQLConf("spark.gluten.sql.native.writer.enabled" -> enableNativeWrite.toString) {
        withTable("t") {
          withSQLConf(
            "spark.sql.hive.convertMetastoreParquet" -> "false",
            "spark.sql.parquet.compression.codec" -> "gzip") {
            checkNativeWrite(
              "CREATE TABLE t STORED AS PARQUET TBLPROPERTIES ('parquet.compression'='zstd') AS SELECT 1 as c",
              checkNative = enableNativeWrite)
            val warehouse = conf.getConf(StaticSQLConf.WAREHOUSE_PATH)
          }
        }
      }
    }

vanilla spark will write parquet file with ZSTD compress codec, but the current native writer use GZIP

Spark version

Spark-3.2.x

Spark configurations

No response

System information

No response

Relevant logs

No response

@yikf yikf added bug Something isn't working triage labels Nov 25, 2024
@yikf yikf changed the title [VL] Native writer should respect table properties for spark 3.3/3.4 [VL] Native writer should respect table properties for spark 3.2/3.3 Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant