Bug: There may be phantom fragments present in the e2e test

[shanicky]

Describe the bug

https://buildkite.com/risingwavelabs/pull-request/builds/39536#018cfd06-0bce-4b29-adb4-35d724237d5c

In this test, alter parallelism used rw_fragments to check the parallelism of fragments after scaling (which hadn't been used before), resulting in two excess fragments and leading to a test error. A hot fix from pr #14542 has been applied to avoid this issue, but we should continue to investigate the specific cause.

Error message/log

-08-12T11:53:09.950280Z DEBUG risingwave_simulation::slt: Record Statement { loc: Location { file: "e2e_test/ddl/alter_parallelism.slt", line: 16, upper: None }, conditions: [], connection: Default, expected_error: None, sql: "create table t (v int);", expected_count: None } finished in 204.53815ms
2022-08-12T11:53:24.287452Z DEBUG risingwave_simulation::slt: Running cmd=Others
2022-08-12T11:53:24.287452Z DEBUG risingwave_simulation::slt: retry count iteration=0
2022-08-12T11:53:24.300087Z DEBUG risingwave_simulation::slt: Record Query { loc: Location { file: "e2e_test/ddl/alter_parallelism.slt", line: 19, upper: None }, conditions: [], connection: Default, expected_types: [Text], sort_mode: None, label: None, expected_error: None, sql: "select parallelism from table_parallelism where name = 't';", expected_results: ["AUTO"] } finished in 12.634383ms
2022-08-12T11:53:25.000779Z  INFO risingwave_simulation::cluster: kill frontend-1
2022-08-12T11:53:25.000779Z ERROR risingwave_simulation::client: postgres connection error: connection closed
2022-08-12T11:53:25.171745Z  INFO risingwave_simulation::cluster: kill frontend-2
2022-08-12T11:53:25.273469Z  INFO risingwave_simulation::cluster: restart frontend-2
2022-08-12T11:53:25.945804Z  INFO risingwave_simulation::cluster: restart frontend-1
2022-08-12T11:53:40.945804Z DEBUG risingwave_simulation::slt: Running cmd=Alter
2022-08-12T11:53:41.151871Z DEBUG risingwave_simulation::slt: Record Statement { loc: Location { file: "e2e_test/ddl/alter_parallelism.slt", line: 24, upper: None }, conditions: [], connection: Default, expected_error: None, sql: "alter table t set parallelism = 2;", expected_count: None } finished in 206.067ms
2022-08-12T11:53:41.151871Z DEBUG risingwave_simulation::slt: Running cmd=Others
2022-08-12T11:53:41.151871Z DEBUG risingwave_simulation::slt: retry count iteration=0
2022-08-12T11:53:41.170372Z DEBUG risingwave_simulation::slt: Record Query { loc: Location { file: "e2e_test/ddl/alter_parallelism.slt", line: 27, upper: None }, conditions: [], connection: Default, expected_types: [Text], sort_mode: None, label: None, expected_error: None, sql: "select parallelism from table_parallelism where name = 't';", expected_results: ["FIXED(2)"] } finished in 18.500955ms
2022-08-12T11:53:41.529664Z  INFO risingwave_simulation::cluster: kill compactor-2
2022-08-12T11:53:41.531752Z  INFO risingwave_simulation::cluster: kill frontend-1
2022-08-12T11:53:41.558412Z  INFO risingwave_simulation::cluster: kill compactor-1
2022-08-12T11:53:41.623950Z  INFO risingwave_simulation::cluster: restart frontend-1
2022-08-12T11:53:41.807413Z  INFO risingwave_simulation::cluster: kill frontend-2
2022-08-12T11:53:41.807413Z ERROR risingwave_simulation::client: postgres connection error: connection closed
2022-08-12T11:53:41.947542Z  INFO risingwave_simulation::cluster: restart compactor-1
2022-08-12T11:53:42.190175Z  INFO risingwave_simulation::cluster: restart compactor-2
2022-08-12T11:53:42.579259Z  INFO risingwave_simulation::cluster: restart frontend-2
2022-08-12T11:53:57.579259Z DEBUG risingwave_simulation::slt: Running cmd=Others
2022-08-12T11:53:57.579259Z DEBUG risingwave_simulation::slt: retry count iteration=0
2022-08-12T11:53:57.649849Z DEBUG risingwave_simulation::slt: Record Query { loc: Location { file: "e2e_test/ddl/alter_parallelism.slt", line: 32, upper: None }, conditions: [], connection: Default, expected_types: [Integer], sort_mode: None, label: None, expected_error: None, sql: "select parallelism from rw_fragments;", expected_results: ["2", "2"] } finished in 70.590276ms
2022-08-12T11:53:57.649849Z ERROR risingwave_simulation::slt: failed to run test: query result mismatch:
[SQL] select parallelism from rw_fragments;
[Diff] (-expected|+actual)
+   6
+   6
    2
    2
at e2e_test/ddl/alter_parallelism.slt:32

To Reproduce

seed = 1
--
  | Configuration {
  | config_path: Regular(
  | "src/config/ci-sim.toml",
  | ),
  | frontend_nodes: 2,
  | compute_nodes: 3,
  | meta_nodes: 3,
  | compactor_nodes: 2,
  | compute_node_cores: 2,
  | etcd_timeout_rate: 0.0,
  | etcd_data_path: None,
  | per_session_queries: [],
  | }
  | e2e_test/ddl/alter_owner.slt
  | e2e_test/ddl/alter_parallelism.slt

Expected behavior

No response

How did you deploy RisingWave?

No response

The version of RisingWave

No response

Additional context

No response

[BugenZhao]

Is it possible that the fragment is from a table that's not dropped in another test case? 😕

[shanicky]

Is it possible that the fragment is from a table that's not dropped in another test case? 😕

Yes, that's what I think.