nightly-20240126 deleting data from backfill table stuck

[cyliu0]

Describe the bug

The deleting operation stuck during backfill test.

| 2024-01-30 04:32:31 | deleting data from backfill table
| 2024-01-30 04:32:31| Timing is on.

| 2024-01-30 05:24:17 | ❌ backfill test timeout

https://buildkite.com/risingwave-test/backfill/builds/140#018d56d0-5910-4d1e-b146-55090520c4df/570-1004

You can check the test code via https://github.com/risingwavelabs/kube-bench/blob/4d46161d17cbddaadb36370cb807de97da2f468e/manifests/benchmarks/backfill/start.template.yaml#L39-L72

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

nightly-20240129

Additional context

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240126

nightly-20240126

• bb3be1091bc00311428dde002f9898355f6f9926 chore: Update README.md (#14808)
• dad438783aafa2f942d8882056c437b9e98c233f fix: patch rquickjs-sys and remove clang dependency (#14817)
• 5576c2ccea03abf5438da7ec5f8d1bb0f4299e85 feat: support sequential exchange (#14795)
• 4793c519df3774d8d6c1bd2ccb24c14bb830b66d fix: extract decimal correctly when bytes len >= 5 (#14810)
• 51e00fac6bef9d6a7621d24bd5e2f4b2947d991b fix(metric): fix checkpoint size use small buckets (#14809)
• 57d0b0e903d31d3870e43d530481bfdb25a6e8b4 feat(meta): add get back pressure RPC for UI dashboard (#14790)
• a4f0ec2bff516d6ab9b8a562d466d87be2ef79bb feat(sql-udf): introduce udf_binding_flag (#14782)
• e5c6558f8d54d64566d71a4fd138b4ff85d3094e fix(avro): fix parsing of avro ref type (#14781)
• 8201fbc5559088b77578c63e529e4ec11e18a05a refactor(storage): remove writing delete ranges in flush (#14776)
• 996a4ba83698eb539c3f3b9fcb53392083931fae fix(obserivation): fix missed hitmap updates, fix drop order (#14788)
• 372c2d70a303f994fb52d3e603ce05c9a910b03c refactor: deprecate StreamChunkWithState (#14524)
• 7cf63dfa202befd29f5d80b926c8374b959d49a4 fix(error): log error field in stream reader retry loop (#14792)
• 90a2139ec9dd8d4ca8af0d721cc807182efd4ce5 feat: add rw_fragment_parallelism and re-define rw_streaming_parallelism (#14789)
• c8030fc0f2179c563452afb1983411a70267c455 fix(storage): replace imms with merged imms in any position of staging (#14756)
• 2559b2307aae4548e8ba21dcf63a33decc6ffe8a fix: fix unneccessary copy in block decode (#14786)
• df87c2d6e7249633a984ecd1f08a9fd7f96dbbf7 feat: only ingest key-ed value in additional header column (#14628)
• fa9e75471143e4835d17b1886aa4113cf57fb842 refactor(error): dedicated error type for dml crate (#14768)
• 7b900e02a92428cc04d715d4c6f75c1bd1d20bda fix(hummock): fix gc delete keys incorrectly (#14751)
• 38785f7e81239d641e43976be1a58f1feec92529 refactor: Remove unused iceberg java (#14779)
• 6c149615bae30bbdb28c7b5279a67c7ffaf63186 feat: do constant folding and timezone transformation before create funtional indexes (#14770)
• 040ea58ec544f8a2ce468d6f0dfd5f831b038f02 fix: add clang to dockerfile (#14777)
• b27d25283d8f7e8cee7110e0ee97500cd45fa7ad fix(stream): arrangement backfill background ddl (#14563)
• 76656e0b7780dd84090f35410e28f33465d058e8 chore(deps): Bump serde_with from 3.4.0 to 3.5.0 (#14730)
• 317b36108cd41b767ee376e91c8bfabe6174591e fix: missing columns in format_encode_options (#14771)
• c67e4e44ea88382071843da2d98bf70aa75c1534 chore: add assertion to ensure monotonically decreasing user key epoch (#14488)
• d563f1eb6aa478464e9f0c6e5de1f017e400adca chore(storage): remove cached entry value, use deref (#14769)
• ee08ac5cbd9e2f1042ea8b61a7c9958d81840823 fix: update arrow-udf to fix compile error on arm64 linux (#14761)
• e8822edd5dea933276b314260512637f880bf222 fix: fix block hitmap clone, fix unused pub (#14767)
• c2c20328b22c10315e241ebb69dadbd3054cf139 refactor: avoid RwError in schema registry client (#14603)
• 9382565dd3a32737864141013de2b5839723234e feat: report the error causes cdc connector exit to grafana (#14611)
• bf777a0fb180af90ad088681068186604c3f9ec0 feat: support ISO 8601 interval parsing (#14747)
• a9ce6c7afebb4a9d7c75c778631482e5eccae9cc fix(stream):hash join: check degree & row iterator have same length (#14509)
• 603f5ea3f09a83953f5280f38ab9a2b98834c4d8 feat(stream): add subscription executor (#14684)
• c19ad8ee8d023f531d0622b2541a142c742a2dc2 chore(risedev): increment api-requests-deadline to avoid SlowDown in no_shuffle_backfill (#14758)
• 30fcc93024579a417e4459f81c7c7d877e30ade2 feat(lints): refine help message when formatting error in tracing::event (#14752)
• 6b7f37a79ab1a4d122e064ca00be294360455e4d fix(optimizer): ban using subquery in non-table-in-out function. (#14748)
• 7853be6cd209c71cb4235c8ac106d5c2fbefb009 fix: only fill state table id of materialize node in table fragments (#14754)
• f804ed09529340f5131484956a6db328ff5c5113 feat(object_store): add retry for Minio SlowDown and TooManyRequests errors (#14739)
• 4c953ebc89e1c4a9e501dcf68d9d1e80a348150d fix: dedup staging sst on storage table read (#14664)
• c75db2ff1ed19e36ba458da5b10450acad17df17 chore(stream): rename ActorContext creation methods (#14750)
• 2e16c264863f9cdc95802e462f728abfe5499cfe feat(stats): introduce block efficiency metrics (#14654)
• 5436acfa5074b57e198ac8b8d66b2d7675fb52d8 feat(ci): integrate with cargo-dylint (#14713)
• 5625492704ec4ee410963f67669983a668de1b76 refactor(connector): rename source crate to dml (#14746)
• 33937b3e3b9362ddc346799fa35bd2e311e2ff1a feat: add heap profiling for frontend (#14744)
• 2fd1c9a0c9baad128dad9340fdf038bfc6122238 refactor(udf-context): assemble sql udf related functions inside UdfContext (#14732)
• 092923d69b43939399320548cd0fb46797d41615 chore(deps): bump lru to 95f347b (#14728)
• 8c721c4c93c707538699a9ad7ad862dd23027fae feat(stream): disable conflict check for stand alone table (#14655)
• 54492085ac652b4e972ffcfa2d5621b34ba1e2d3 feat(risedev): add configurations to rate limit minio request rate (#14726)
• 330fd335089603c6258a96fc4e3614c3e3c4c4db chore(deps): Bump shlex from 1.2.0 to 1.3.0 (#14733)
• 657b8ab5139c036ce548a903e815cc353a0e0ad4 feat(sql-udf): add semantic check when creating sql udf (#14549)
• 641483214521f9570332311ca93aff99752fd264 chore(ci/typos): ignore checks on Cargo files (#14729)
• 705be197c920639b06aa8baf1f74872d5062dd40 feat(udf): add initial support for JavaScript UDF (#14513)

[hzxa21]

Backfill uses the DELETE FROM to trigger deletion. https://github.com/risingwavelabs/kube-bench/blob/4d46161d17cbddaadb36370cb807de97da2f468e/manifests/backfill/backfill-sql.template.yaml#L50

@Little-Wallace Please help take a look.

[Little-Wallace]

But, as I know, risingwave does not use delete range for DELETE FROM tableA

[hzxa21]

But, as I know, risingwave does not use delete range for DELETE FROM tableA

Yes, I think it will just do key deletion.

[hzxa21]

The retry run cannot start benachmark-risingwave component. Can there be any infra issue here?
https://buildkite.com/risingwave-test/backfill/builds/140#018d5d66-bf23-4773-a486-e4ceb77ec4d1/586-600

[cyliu0]

The retry run cannot start benachmark-risingwave component. Can there be any infra issue here?
https://buildkite.com/risingwave-test/backfill/builds/140#018d5d66-bf23-4773-a486-e4ceb77ec4d1/586-600

Need to specify the RW_VERSION

[cyliu0]

Verified by nightly-20240201