Upgrade clickhouse from v22.8.9.24 to v23.8.7.24

[citrus-it]

The tests all now pass and I've deployed this to a bench gimlet and done some basic smoke tests.

• Single node deployment;
• Clustered deployment;
• Upgrade from old version to new;
• More tests, TBD

[bnaecker]

Looks good to me, thanks for pushing on this!

[karencfv]

Thanks for looking into this, and also for cleaning up the tests!

[citrus-it]

I booted a fresh control plane with the following changes to the omicron repo:

a/sled-agent/src/rack_setup/plan/service.rs
-const CLICKHOUSE_COUNT: usize = 1;
+const CLICKHOUSE_COUNT: usize = 2;
-const CLICKHOUSE_KEEPER_COUNT: usize = 0;
+const CLICKHOUSE_KEEPER_COUNT: usize = 3;

a/smf/clickhouse/method_script.sh
-single_node=true
+single_node=false

and all five expected zones came up:

gimlet-sn06 # zoneadm list | grep -i click
oxz_clickhouse_keeper_abcc5b68-a928-430f-8d85-5022311cb2ec
oxz_clickhouse_keeper_bb33cccc-c12f-4c69-94a2-a192d49c4799
oxz_clickhouse_15a50998-0fb9-4f51-925c-7beab0bf64a0
oxz_clickhouse_72b39137-62fe-4771-8530-1de0ddd9d453
oxz_clickhouse_keeper_6a440ca6-04b9-441b-a718-482a9e326ba6

I logged into one of the keeper zones and connected to the keeper server, which looked good. I noticed that the PATH was not correctly set to find clickhouse so that's addressed in the next commit.

root@oxz_clickhouse_keeper_abcc5b68:~# /opt/oxide/clickhouse_keeper/clickhouse  keeper-client -p 9181 -h '[fd00:1122:3344:101::12]'
Connected to ZooKeeper at [fd00:1122:3344:101::12]:9181 with session_id 3
Keeper feature flag FILTERED_LIST: enabled
Keeper feature flag MULTI_READ: enabled
Keeper feature flag CHECK_NOT_EXISTS: disabled
/ :) ls
clickhouse keeper
/ :) ls clickhouse
tables task_queue

I then logged into one of the clickhouse zones and verified that the cluster existed.

oximeter_cluster node 1 :) show cluster 'oximeter_cluster' format Vertical;

SHOW CLUSTER oximeter_cluster
FORMAT Vertical

Query id: a0fc31bf-08bf-4e40-b61f-4585385278c4

Row 1:
──────
cluster:                 oximeter_cluster
shard_num:               1
shard_weight:            1
replica_num:             1
host_name:               15a50998-0fb9-4f51-925c-7beab0bf64a0.host.control-plane.oxide.internal.
host_address:            fd00:1122:3344:101::e
port:                    9000
is_local:                1
user:                    default
default_database:
errors_count:            0
slowdowns_count:         0
estimated_recovery_time: 0
database_shard_name:
database_replica_name:
is_active:               ᴺᵁᴸᴸ

Row 2:
──────
cluster:                 oximeter_cluster
shard_num:               1
shard_weight:            1
replica_num:             2
host_name:               72b39137-62fe-4771-8530-1de0ddd9d453.host.control-plane.oxide.internal.
host_address:            fd00:1122:3344:101::f
port:                    9000
is_local:                0
user:                    default
default_database:
errors_count:            0
slowdowns_count:         0
estimated_recovery_time: 0
database_shard_name:
database_replica_name:
is_active:               ᴺᵁᴸᴸ

2 rows in set. Elapsed: 0.004 sec.

After creating and deleting some instances, the metrics appeared to work correctly.