Help Wanted: Not able to upgrade to 1.6.0

[sheltonsuen]

Describe the bug

When tring to upgrade from 1.5.4 to 1.6.0

the compute node keep crash due to probe failed, but no other error logs in compute node and meta node

Error message/log

No response

To Reproduce

No response

Expected behavior

No response

How did you deploy RisingWave?

helm upgrade

The version of RisingWave

1.6.0

Additional context

No response

[sheltonsuen]

and k8s version:

[sheltonsuen]

It's a cluster issue not related to risingwave, close it

[sheltonsuen]

Still have this issue, the compute pods keep crash with exist code 1 after startup, no other message collected

[sheltonsuen]

I have try to enlarge the probe timeout and delay time, but not help

[lmatz]

the compute pods keep crash with exist code 1 after startup

what's the log of RW
If RW crashes, there must be some error or panic logs.

what's the status of pods, what does it say in the REASON field?

[lmatz]

I just rerun the upgrade test for v1.5.4 and v1.6.0, and it works as expected.

Normally, if the upgrade fails, there should be some logs about certain inconsistencies, e.g.:

1. key values in the old meta cannot be recognized by the new meta
2. key values in the state of old operators cannot be recognized by the new operators

They must trigger some panic and will be shown in the log.

[sheltonsuen]

I just rerun the upgrade test for v1.5.4 and v1.6.0, and it works as expected.

Normally, if the upgrade fails, there should be some logs about certain inconsistencies, e.g.:

1. key values in the old meta cannot be recognized by the new meta
2. key values in the state of old operators cannot be recognized by the new operators

They must trigger some panic and will be shown in the log.

Thank you for the test,

I didn't collect any panic log from compute node, maybe it's another new instance up, or somthing

Based on your test, later today(once our tester not using current env) I will start another upgrade to see if there any usefull log

[lmatz]

Hi @sheltonsuen

could you check if "connector node" is still set to replica "1" in the yaml file?

If it is, please delete it or set the replica to 0.

The connector node is embedded into compute node, and no longer need to be set explicitly.

[sheltonsuen]

Hi @sheltonsuen

could you check if "connector node" is still set to replica "1" in the yaml file?

If it is, please delete it or set the replica to 0.

The connector node is embedded into compute node, and no longer need to be set explicitly.

It's working after setting replica to 0, thanks

[zwang28]

FYI, connector node should be disabled in helm via

connectorComponent:
  enabled: false