nightly-20240312 pipeline test failed

[huangjw806]

sysbench pipeline：

https://buildkite.com/risingwave-test/sysbench/builds/706

---

longevity test：

================================================================================
longevity-test Result
================================================================================
Result               FAIL                
Pipeline Message     run all nexmark (8 sets of nexmark queries) with 10k throughput daily
Namespace            reglngvty-20240312-150216
TestBed              medium-arm-3cn-all-affinity
RW Version           nightly-20240312    
Test Start time      2024-03-12 15:08:04 
Test End time        2024-03-12 15:09:19 
Test Queries         nexmark_q0,nexmark_q1,nexmark_q2,nexmark_q3,nexmark_q4,nexmark_q5,nexmark_q6_group_top1,nexmark_q7,nexmark_q8,nexmark_q9,nexmark_q10,nexmark_q12,nexmark_q14,nexmark_q15,nexmark_q16,nexmark_q17,nexmark_q18,nexmark_q19,nexmark_q20,nexmark_q21,nexmark_q22,nexmark_q101,nexmark_q102,nexmark_q103,nexmark_q104,nexmark_q105
Grafana Metric       https://grafana.test.risingwave-cloud.xyz/d/EpkBw5W4k/risingwave-dev-dashboard?orgId=1&var-datasource=Prometheus:%20test-useast1-eks-a&var-namespace=reglngvty-20240312-150216&from=1710256084000&to=1710256159000
Grafana Logs         https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?orgId=1&var-data_source=Logging:%20test-useast1-eks-a&var-namespace=reglngvty-20240312-150216&from=1710256084000&to=1710256159000
Memory Dumps         https://s3.console.aws.amazon.com/s3/buckets/test-useast1-mgmt-bucket-archiver?region=us-east-1&bucketType=general&prefix=k8s/reglngvty-20240312-150216/&showversions=false
Buildkite Job        https://buildkite.com/risingwave-test/longevity-test/builds/1162

---

customer-usecase:
https://buildkite.com/risingwave-test/customer-usecases/builds/502

chaos-mesh:
https://buildkite.com/risingwave-test/chaos-mesh/builds/679

[cyliu0]

In addition, most of them failed to execute the first query right after the RW cluster is started in the pipeline. In the sysbench pipeline, it failed at creating sysbench tables. In the nexmark pipelines, it failed at creating first query.

[lmatz]

For longevity test:

2024-03-12T15:07:13.220474715Z ERROR handle_query{mode="simple query" session_id=0 sql=CREATE MATERIALIZED VIEW bid AS
SELECT (bid).auction, (bid).bidder, (bid).price, (bid).channel, (bid).url, (bid).date_time, (bid).extra, p_time, _row_id FROM nexmark WHERE event_type = 2;}: risingwave_frontend::session: failed to handle sql error=gRPC request to meta service failed: Internal error: failed to inject barrier sql=CREATE MATERIALIZED VIEW bid AS SELECT (bid).auction, (bid).bidder, (bid).price, (bid).channel, (bid).url, (bid).date_time, (bid).extra, p_time, _row_id FROM nexmark WHERE event_type = 2

This is the first error.

https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?from=1710255962403&orgId=1&to=1710256040669&var-data_source=PE59595AED52CF917&var-namespace=reglngvty-20240312-150216&var-pod=benchmark-risingwave-frontend-f-7b664fbf9d-886bs&var-search=

https://github.com/risingwavelabs/rw-commits-history?tab=readme-ov-file#nightly-20240312

If it is not a random error, then
72d37ef #14887 looks suspicious

Edit:
if multiple pipelines run into the same problem, then it seems not random

[lmatz]

The same happens to customer use cases:

https://grafana.test.risingwave-cloud.xyz/d/liz0yRCZz1/log-search-dashboard?from=now-24h&orgId=1&to=now&var-data_source=PE59595AED52CF917&var-namespace=rwc-g1hopmf4ntfukpldrscr1jri8n-customer-usercases-20240312&var-pod=risingwave-frontend-default-67f774b7df-4hzh8&var-search=

[lmatz]

#15653

[cyliu0]

Looks like we hit this again.

https://buildkite.com/risingwave-test/sysbench/builds/709#018e3ec5-0159-4d33-b164-165e2c5261ed

https://buildkite.com/risingwave-test/nexmark-benchmark/builds/3291#018e3db2-6127-4447-8d27-8fd458dd82f8

[fuyufjh]

Will continue to track in #15730