source parser/sink writer errors may lead to standalone cluster being entirely unavailable during recovery

[kwannoel]

source/sink parser errors should only take down Compute nodes. It shouldn't affect meta nodes / fe nodes if there's no panic.

Needs investigation to reproduce it.

[xxchan]

#16813 Here's a bug for you to try 🤪

[st1page]

sink a null value into a PG with NOT NULL constraint

[kwannoel]

Before this, we can also try run the recover command, and run cluster commands which do not depend on CN, only meta and fe, to see if it's a recovery in single node mode problem, or a source / sink problem.

[fuyufjh]

Yesterday I was trying to reproduce the case but failed. I created a PG sink without sink decouple and ingested some bad data (nulls). The Meta node kept recovering forever, which is expected, but nothing was down. 🙁

[fuyufjh]

Also notice that it's standalone mode instead of single-node mode. Here is the command I used to start the process.

CONNECTOR_LIBS_PATH=/Users/eric/Workspace/risingwave/.risingwave/bin/connector-node/libs cargo run --bin risingwave -- standalone --meta-opts="--listen-addr 127.0.0.1:5690  --advertise-addr 127.0.0.1:5690  --dashboard-host 127.0.0.1:5691  --prometheus-host 127.0.0.1:1250  --backend etcd  --etcd-endpoints 127.0.0.1:2388  --state-store hummock+minio://hummockadmin:[email protected]:9301/hummock001  --data-directory hummock_001"      --compute-opts="--listen-addr 127.0.0.1:5688  --prometheus-listener-addr 127.0.0.1:1222  --advertise-addr 127.0.0.1:5688  --async-stack-trace verbose  --parallelism 4  --total-memory-bytes 8589934592  --role both  --meta-address http://127.0.0.1:5690"      --frontend-opts="--listen-addr 127.0.0.1:4566 --advertise-addr 127.0.0.1:4566 --prometheus-listener-addr 127.0.0.1:2222 --health-check-listener-addr 127.0.0.1:6786 --meta-addr http://127.0.0.1:5690"      --compactor-opts="--listen-addr 127.0.0.1:6660   --prometheus-listener-addr 127.0.0.1:1260   --advertise-addr 127.0.0.1:6660   --meta-address http://127.0.0.1:5690"

[kwannoel]

Another user encounters:

    "message": "librdkafka: Global error: Authentication (Local: Authentication failure): <REDACTED> SASL authentication error: invalid username or password for <REDACTED>

This error also causes standalone cluster to crash loop.