Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

run spark+tpcds 1T with blaze, execution error happened #634

Open
shaileneF opened this issue Oct 28, 2024 · 4 comments
Open

run spark+tpcds 1T with blaze, execution error happened #634

shaileneF opened this issue Oct 28, 2024 · 4 comments

Comments

@shaileneF
Copy link

shaileneF commented Oct 28, 2024 •
edited
Loading

env：

host
os：CentOS Linux release 8.2.2004 (Core)
kernel：4.18.0-193.el8.x86_64
container：
os：CentOS Linux release 7.4.1708 (Core)
kernel：4.18.0-193.el8.x86_64
cpu cores：128
spark: 3.4.3

spark-default.conf:

spark.master local
spark.eventLog.enabled true
spark.eventLog.dir /home/blaze-test/spark-eventlog
spark.history.fs.logDirectory /home/blaze-test/spark-eventlog
spark.externalBlockStore.url.list /home/blaze-test/platform
spark.driver.extraJavaOptions -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/home/blaze-test/spark/ -Djava.io.tmpdir=/home/blaze-test/tmp -Dlog4j2.formatMsgNoLookups=true
spark.local.dir /home/blaze-test/localdir
spark.shuffle.service.enabled true
spark.shuffle.service.port 7337
spark.driver.memory 10g
spark.driver.memoryOverhead 4096
spark.executor.instances 12
spark.dynamicallocation.maxExecutors 12
spark.executor.cores 8
spark.io.compression.codec zstd
spark.sql.parquet.compression.codec zstd
spark.executor.memory 10g
spark.executor.memoryOverhead 4096
spark.blaze.enable true
spark.sql.extensions org.apache.spark.sql.blaze.BlazeSparkSessionExtension
spark.shuffle.manager org.apache.spark.sql.execution.blaze.shuffle.BlazeShuffleManager
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 40g

description ：

hello，I run spark+blaze+tpcds in local model. There is no errors with 1GB and 10GB. However, with 100GB, a few queries resulted in errors, and with 1TB, more than half of the queries encountered errors.

errors:

output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Project] error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
(+6895.714s) [ERROR] Blaze - [partition=1] native execution panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
24/10/24 21:34:37 WARN TaskSetManager: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
24/10/24 21:34:37 INFO DAGScheduler: ShuffleMapStage 3403 (main at NativeMethodAccessorImpl.java:0) failed in 0.136 s due to Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
case: q96, round: 2, error: Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 3403.0 failed 1 times, most recent failure: Lost task 1.0 in stage 3403.0 (TID 31521) (22aa3cf9e3ca executor driver): java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
Caused by: java.lang.RuntimeException: [partition=1] panics: Execution error: Execution error: output_with_sender[Shuffle] error: Execution error: output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] error: Execution error: output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] error: Execution error: index out of bounds: the len is 0 but the index is 0
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[ParquetScan] canceled due to task finished/killed
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
output_with_sender[BroadcastJoin]: output() returns error: Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
output_with_sender[Shuffle]: output() returns error: Arrow error: External error: agg: execute_agg_no_grouping() error
Execution error: Execution error: output_with_sender[BroadcastJoin] canceled due to task finished/killed
@shaileneF shaileneF changed the title run spark+tpcds 10T with blaze error run spark+tpcds 1T with blaze error Oct 28, 2024
@shaileneF shaileneF changed the title run spark+tpcds 1T with blaze error run spark+tpcds 1T with blaze, execution error happened Oct 28, 2024
@richox
Copy link
Collaborator

richox commented Oct 28, 2024

this should have been fixed in #616 , you can try compiling with the latest code.

@shaileneF
Copy link
Author

this should have been fixed in #616 , you can try compiling with the latest code.

thank you，I am trying.

@shaileneF
Copy link
Author

shaileneF commented Oct 31, 2024 •
edited
Loading

this should have been fixed in #616 , you can try compiling with the latest code.

**I got some compiling error with the latest main branch code. ** 🙏

rust version: rustc 1.82.0 (f6e511eec 2024-10-15)
cargo version: cargo 1.82.0 (8f40fc59f 2024-08-21)

[ERROR] [main] find: 'target/release/libblaze.so': No such file or directory
[INFO] [main] old build-checksum: cat: ./.build-checksum_release-so.cache: No such file or directory
[INFO] [main] new build-checksum: 2e1f0422f62271e27a5f4a2b923d44e1  -
[INFO] [main] Running cargo fix...
[ERROR] [main]  Downloading crates ...
[ERROR] [main]   Downloaded lzokay-native v0.1.0
[ERROR] [main]    Compiling proc-macro2 v1.0.87
[ERROR] [main]    Compiling unicode-ident v1.0.13
[ERROR] [main]    Compiling libc v0.2.159
[ERROR] [main]    Compiling autocfg v1.4.0
[ERROR] [main]     Checking cfg-if v1.0.0
[ERROR] [main]    Compiling libm v0.2.8
[ERROR] [main]    Compiling shlex v1.3.0
[ERROR] [main]    Compiling version_check v0.9.5
[ERROR] [main]     Checking bytes v1.7.2
[ERROR] [main]    Compiling regex-syntax v0.8.5
[ERROR] [main]     Checking byteorder v1.5.0
[ERROR] [main]    Compiling serde v1.0.210
[ERROR] [main]    Compiling siphasher v0.3.11
[ERROR] [main]    Compiling rand_core v0.6.4
[ERROR] [main]    Compiling pkg-config v0.3.31
[ERROR] [main]     Checking once_cell v1.20.2
[ERROR] [main]     Checking memchr v2.7.4
[ERROR] [main]    Compiling heck v0.5.0
[ERROR] [main]     Checking bitflags v2.6.0
[ERROR] [main]     Checking iana-time-zone v0.1.61
[ERROR] [main]     Checking static_assertions v1.1.0
[ERROR] [main]     Checking allocator-api2 v0.2.18
[ERROR] [main]    Compiling rustversion v1.0.17
[ERROR] [main]     Checking pin-project-lite v0.2.14
[ERROR] [main]     Checking ryu v1.0.18
[ERROR] [main]    Compiling semver v1.0.23
[ERROR] [main]     Checking futures-core v0.3.31
[ERROR] [main]     Checking futures-sink v0.3.31
[ERROR] [main]     Checking unicode-width v0.1.14
[ERROR] [main]    Compiling parking_lot_core v0.9.10
[ERROR] [main]     Checking futures-io v0.3.31
[ERROR] [main]     Checking base64 v0.22.1
[ERROR] [main]     Checking itoa v1.0.11
[ERROR] [main]     Checking pin-utils v0.1.0
[ERROR] [main]     Checking smallvec v1.13.2
[ERROR] [main]     Checking tinyvec_macros v0.1.1
[ERROR] [main]     Checking scopeguard v1.2.0
[ERROR] [main]     Checking futures-task v0.3.31
[ERROR] [main]    Compiling serde_json v1.0.96 (https://github.com/blaze-init/json?branch=v1.0.96-blaze#95fa6cba)
[ERROR] [main]     Checking bitflags v1.3.2
[ERROR] [main]     Checking log v0.4.22
[ERROR] [main]    Compiling zstd-safe v7.2.1
[ERROR] [main]     Checking equivalent v1.0.1
[ERROR] [main]     Checking hashbrown v0.15.0
[ERROR] [main]     Checking adler2 v2.0.0
[ERROR] [main]     Checking alloc-no-stdlib v2.0.4
[ERROR] [main]    Compiling snap v1.1.1
[ERROR] [main]     Checking unicode-bidi v0.3.17
[ERROR] [main]     Checking percent-encoding v2.3.1
[ERROR] [main]     Checking same-file v1.0.6
[ERROR] [main]     Checking lazy_static v1.5.0
[ERROR] [main]    Compiling paste v1.0.15
[ERROR] [main]     Checking either v1.13.0
[ERROR] [main]     Checking integer-encoding v3.0.4
[ERROR] [main]     Checking humantime v2.1.0
[ERROR] [main]    Compiling rustix v0.38.37
[ERROR] [main]    Compiling seq-macro v0.3.5
[ERROR] [main]    Compiling crossbeam-utils v0.8.20
[ERROR] [main]    Compiling typenum v1.17.0
[ERROR] [main]     Checking linux-raw-sys v0.4.14
[ERROR] [main]     Checking fastrand v2.1.1
[ERROR] [main]     Checking fixedbitset v0.4.2
[ERROR] [main]     Checking hex v0.4.3
[ERROR] [main]     Checking subtle v2.6.1
[ERROR] [main]    Compiling thiserror v1.0.64
[ERROR] [main]    Compiling anyhow v1.0.89
[ERROR] [main]    Compiling syn v1.0.109
[ERROR] [main]     Checking constant_time_eq v0.3.1
[ERROR] [main]     Checking cpufeatures v0.2.14
[ERROR] [main]     Checking arrayref v0.3.9
[ERROR] [main]     Checking arrayvec v0.7.6
[ERROR] [main]     Checking unicode-segmentation v1.12.0
[ERROR] [main]    Compiling radium v0.7.0
[ERROR] [main]     Checking jni-sys v0.3.0
[ERROR] [main]     Checking glob v0.3.1
[ERROR] [main]     Checking cesu8 v1.1.0
[ERROR] [main]     Checking tap v1.0.1
[ERROR] [main]     Checking funty v2.0.0
[ERROR] [main]     Checking radsort v0.1.1
[ERROR] [main]     Checking unchecked-index v0.2.2
[ERROR] [main]     Checking simdutf8 v0.1.5
[ERROR] [main]    Compiling prettyplease v0.2.22
[ERROR] [main]    Compiling zstd-safe v6.0.6
[ERROR] [main]     Checking bumpalo v3.16.0
[ERROR] [main]    Compiling multimap v0.10.0
[ERROR] [main]    Compiling gxhash v3.4.1
[ERROR] [main]     Checking fallible-streaming-iterator v0.1.9
[ERROR] [main]     Checking panic-message v0.3.0
[ERROR] [main]     Checking count-write v0.1.0
[ERROR] [main]     Checking bytesize v1.3.0
[ERROR] [main]    Compiling phf_shared v0.11.2
[ERROR] [main]    Compiling rand v0.8.5
[ERROR] [main]     Checking lexical-util v0.8.5
[ERROR] [main]     Checking futures-channel v0.3.31
[ERROR] [main]     Checking twox-hash v1.6.3
[ERROR] [main]     Checking tinyvec v1.8.0
[ERROR] [main]     Checking miniz_oxide v0.8.0
[ERROR] [main]     Checking crc32fast v1.4.2
[ERROR] [main]     Checking alloc-stdlib v0.2.2
[ERROR] [main]     Checking form_urlencoded v1.2.1
[ERROR] [main]     Checking tracing-core v0.1.32
[ERROR] [main]     Checking walkdir v2.5.0
[ERROR] [main]     Checking wyz v0.5.1
[ERROR] [main]     Checking itertools v0.13.0
[ERROR] [main]    Compiling itertools v0.12.1
[ERROR] [main]     Checking brotli-decompressor v4.0.1
[ERROR] [main]     Checking phf v0.11.2
[ERROR] [main]     Checking lz4_flex v0.11.3
[ERROR] [main]     Checking aho-corasick v1.1.3
[ERROR] [main]     Checking csv-core v0.1.11
[ERROR] [main]     Checking combine v4.6.7
[ERROR] [main]     Checking indexmap v2.6.0
[ERROR] [main]     Checking flate2 v1.0.34
[ERROR] [main]     Checking lexical-parse-integer v0.8.6
[ERROR] [main]     Checking lexical-write-integer v0.8.5
[ERROR] [main]    Compiling ahash v0.8.11
[ERROR] [main]    Compiling generic-array v0.14.7
[ERROR] [main]    Compiling phf_generator v0.11.2
[ERROR] [main]     Checking lexical-parse-float v0.8.5
[ERROR] [main]     Checking lexical-write-float v0.8.5
[ERROR] [main]    Compiling num-traits v0.2.19
[ERROR] [main]    Compiling slab v0.4.9
[ERROR] [main]    Compiling lock_api v0.4.12
[ERROR] [main]    Compiling bigdecimal v0.4.5
[ERROR] [main]    Compiling phf_codegen v0.11.2
[ERROR] [main]     Checking unicode-normalization v0.1.24
[ERROR] [main]    Compiling jni v0.20.0
[ERROR] [main]     Checking brotli v6.0.0
[ERROR] [main]     Checking petgraph v0.6.5
[ERROR] [main]     Checking lexical-core v0.8.5
[ERROR] [main]     Checking bitvec v1.0.1
[ERROR] [main]     Checking idna v0.5.0
[ERROR] [main]    Compiling regex-automata v0.4.8
[ERROR] [main]    Compiling rustc_version v0.4.1
[ERROR] [main]    Compiling quote v1.0.37
[ERROR] [main]    Compiling jobserver v0.1.32
[ERROR] [main]    Compiling flatbuffers v24.3.25
[ERROR] [main]     Checking getrandom v0.2.15
[ERROR] [main]     Checking num_cpus v1.16.0
[ERROR] [main]    Compiling syn v2.0.79
[ERROR] [main]     Checking threadpool v1.8.1
[ERROR] [main]     Checking uuid v1.10.0
[ERROR] [main]     Checking url v2.5.2
[ERROR] [main]     Checking parking_lot v0.12.3
[ERROR] [main]    Compiling cc v1.1.28
[ERROR] [main]     Checking num-integer v0.1.46
[ERROR] [main]     Checking num-complex v0.4.6
[ERROR] [main]     Checking half v2.4.1
[ERROR] [main]     Checking chrono v0.4.38
[ERROR] [main]     Checking atoi v2.0.0
[ERROR] [main]     Checking ordered-float v2.10.1
[ERROR] [main]     Checking thrift v0.17.0
[ERROR] [main]     Checking num-bigint v0.4.6
[ERROR] [main]     Checking num-iter v0.1.45
[ERROR] [main]     Checking crypto-common v0.1.6
[ERROR] [main]     Checking block-buffer v0.10.4
[ERROR] [main]     Checking digest v0.10.7
[ERROR] [main]     Checking tempfile v3.13.0
[ERROR] [main]     Checking md-5 v0.10.6
[ERROR] [main]     Checking sha2 v0.10.8
[ERROR] [main]     Checking blake2 v0.10.6
[ERROR] [main]    Compiling zstd-sys v2.0.13+zstd.1.5.6
[ERROR] [main]    Compiling lzma-sys v0.1.20
[ERROR] [main]    Compiling blake3 v1.5.4
[ERROR] [main]    Compiling bzip2-sys v0.1.11+1.0.8
[ERROR] [main]    Compiling jemalloc-sys v0.5.4+5.3.0-patched
[ERROR] [main]    Compiling regex v1.11.0
[ERROR] [main]     Checking num-rational v0.4.2
[ERROR] [main]    Compiling parse-zoneinfo v0.3.1
[ERROR] [main]     Checking num v0.4.3
[ERROR] [main]     Checking arrow-buffer v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]    Compiling chrono-tz-build v0.3.0
[ERROR] [main]     Checking bzip2 v0.4.4
[ERROR] [main]     Checking xz2 v0.1.7
[ERROR] [main]    Compiling chrono-tz v0.9.0
[ERROR] [main]    Compiling ptr_meta_derive v0.2.0
[ERROR] [main]    Compiling derivative v2.2.0
[ERROR] [main]    Compiling serde_derive v1.0.210
[ERROR] [main]    Compiling zerocopy-derive v0.7.35
[ERROR] [main]    Compiling strum_macros v0.26.4
[ERROR] [main]    Compiling tokio-macros v2.4.0
[ERROR] [main]    Compiling futures-macro v0.3.31
[ERROR] [main]    Compiling snafu-derive v0.8.5
[ERROR] [main]    Compiling async-trait v0.1.83
[ERROR] [main]    Compiling tracing-attributes v0.1.27
[ERROR] [main]    Compiling sqlparser_derive v0.2.2
[ERROR] [main]    Compiling thiserror-impl v1.0.64
[ERROR] [main]    Compiling prost-derive v0.13.3
[ERROR] [main]    Compiling prost-derive v0.12.6
[ERROR] [main]     Checking ptr_meta v0.2.0
[ERROR] [main]     Checking slimmer_box v0.6.5
[ERROR] [main]     Checking zstd v0.13.2
[ERROR] [main]     Checking zstd v0.12.4
[ERROR] [main]     Checking sqlparser v0.50.0
[ERROR] [main]     Checking tokio v1.40.0
[ERROR] [main]     Checking futures-util v0.3.31
[ERROR] [main]     Checking zerocopy v0.7.35
[ERROR] [main]     Checking lzokay-native v0.1.0
[ERROR] [main]     Checking tracing v0.1.40
[ERROR] [main]     Checking ppv-lite86 v0.2.20
[ERROR] [main]     Checking prost v0.12.6
[ERROR] [main]    Compiling prost v0.13.3
[ERROR] [main]     Checking hashbrown v0.14.5
[ERROR] [main]     Checking strum v0.26.3
[ERROR] [main]     Checking snafu v0.8.5
[ERROR] [main]     Checking comfy-table v7.1.1
[ERROR] [main]     Checking rand_chacha v0.3.1
[ERROR] [main]    Compiling prost-types v0.13.3
[ERROR] [main]     Checking dashmap v6.1.0
[ERROR] [main]    Compiling prost-build v0.13.3
[ERROR] [main]     Checking futures-executor v0.3.31
[ERROR] [main]    Compiling tonic-build v0.12.3
[ERROR] [main]     Checking futures v0.3.31
[ERROR] [main]     Checking object_store v0.11.0
[ERROR] [main]     Checking datafusion-common-runtime v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking async-compression v0.4.13
[ERROR] [main]     Checking tokio-util v0.7.12
[ERROR] [main]    Compiling blaze-serde v0.1.0 (/home/blaze/native-engine/blaze-serde)
[ERROR] [main]     Checking arrow-schema v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking csv v1.3.0
[ERROR] [main]     Checking faststr v0.2.23
[ERROR] [main]     Checking sonic-rs v0.3.13
[ERROR] [main]     Checking arrow-data v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-array v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-select v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-arith v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-row v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-cast v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-ord v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-string v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-ipc v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-csv v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow-json v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking arrow v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking parquet v53.0.0 (https://github.com/blaze-init/arrow-rs.git?rev=9dbfd9018e#9dbfd901)
[ERROR] [main]     Checking orc-rust v0.3.1 (https://github.com/blaze-init/datafusion-orc.git?rev=c54bfb5#c54bfb5b)
[ERROR] [main]     Checking datafusion-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-expr-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-expr-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-aggregate-common v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-expr v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-execution v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-sql v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-window v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-expr v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-aggregate v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-optimizer v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-physical-plan v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-functions-nested v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking jemallocator v0.5.4
[ERROR] [main]     Checking datafusion-physical-optimizer v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion-catalog v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking datafusion v42.0.0 (https://github.com/blaze-init/arrow-datafusion.git?rev=9a09e14#9a09e14f)
[ERROR] [main]     Checking blaze-jni-bridge v0.1.0 (/home/blaze/native-engine/blaze-jni-bridge)
[ERROR] [main]     Checking datafusion-ext-commons v0.1.0 (/home/blaze/native-engine/datafusion-ext-commons)
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:15:1
[ERROR] [main]    |
[ERROR] [main] 15 | #![feature(new_uninit)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:16:1
[ERROR] [main]    |
[ERROR] [main] 16 | #![feature(slice_swap_unchecked)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] error[E0554]: `#![feature]` may not be used on the stable release channel
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:17:1
[ERROR] [main]    |
[ERROR] [main] 17 | #![feature(vec_into_raw_parts)]
[ERROR] [main]    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ERROR] [main]
[ERROR] [main] warning: the feature `new_uninit` has been stable since 1.82.0 and no longer requires an attribute to enable
[ERROR] [main]   --> native-engine/datafusion-ext-commons/src/lib.rs:15:12
[ERROR] [main]    |
[ERROR] [main] 15 | #![feature(new_uninit)]
[ERROR] [main]    |            ^^^^^^^^^^
[ERROR] [main]    |
[ERROR] [main]    = note: `#[warn(stable_features)]` on by default
[ERROR] [main]
[ERROR] [main] For more information about this error, try `rustc --explain E0554`.
[ERROR] [main] warning: `datafusion-ext-commons` (lib) generated 1 warning
[ERROR] [main] error: could not compile `datafusion-ext-commons` (lib) due to 3 previous errors; 1 warning emitted
[ERROR] [main] warning: build failed, waiting for other jobs to finish...
[ERROR] [main] warning: `datafusion-ext-commons` (lib test) generated 1 warning (1 duplicate)
[ERROR] [main] error: could not compile `datafusion-ext-commons` (lib test) due to 3 previous errors; 1 warning emitted
[ERROR] Command execution failed.
org.apache.commons.exec.ExecuteException: Process exited with an error: 101 (Exit value: 101)
    at org.apache.commons.exec.DefaultExecutor.executeInternal (DefaultExecutor.java:404)
    at org.apache.commons.exec.DefaultExecutor.execute (DefaultExecutor.java:166)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:982)
    at org.codehaus.mojo.exec.ExecMojo.executeCommandLine (ExecMojo.java:929)
    at org.codehaus.mojo.exec.ExecMojo.execute (ExecMojo.java:452)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
    at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
    at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
    at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
    at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
    at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
    at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:498)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for blaze-engine 4.0.0-SNAPSHOT:
[INFO]
[INFO] blaze-engine ....................................... SUCCESS [  1.228 s]
[INFO] proto .............................................. SUCCESS [  2.456 s]
[INFO] spark-extension .................................... SUCCESS [  8.056 s]
[INFO] spark-extension-shims-spark3 ....................... SUCCESS [  2.189 s]
[INFO] assembly ........................................... FAILURE [01:36 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

@wangfann
Copy link

wangfann commented Nov 1, 2024

The problem was solved by using the jar package verified in the following link, and the owner above can also try it. https://github.com/kwai/blaze/actions/runs/11595556860

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@richox @wangfann @shaileneF