Graceful teardown

[PietroPasotti]

Issue

Fixes canonical/tempo-worker-k8s-operator#50

TLDR:
when the worker is happy and configured, but you want it to go, no event will tell it to stop all services. Instead, it will attempt to restart and keep failing at it, as some of the required resources (s3, a coordinator) might be gone already.
Juju will then see the error status and refuse to clean it up.

Solution

This PR changes the Worker logic to ensure that, in the reconcile flow, if the worker isn't ready:

• we wipe all configurations (strictly speaking not necessary, but it felt like the clean thing to do)
• we avoid triggering .restart() as we know it should fail
• we update the layer to disable service autorestart (as there is no way to 'remove the layer' altogether), and we stop the services one by one

Testing Instructions

deploy COS with this version of cosl
(the quickest way probably is to deploy COS from edge, then scp/sync this version of cosl into their venvs)

what I did:

deploy Tempo HA and get it to active/idle

# shell 1
cd cos-lib/src
jhack sync tempo-worker --source ./ --remote-root /var/lib/juju/agents/unit-tempo-worker-0/charm/venv/

# shell 2 
touch cos-lib/src/cosl/coordinated_workers/worker.py

juju destroy-model --destroy-storage --no-prompt

Case 1: teardown
then do a destroy-model or remove-application <tempo|loki|mimir> (and workers)
Wait for things to go down without ever setting error status as that will prevent juju from cleaning it up.

Case 2: upgrade
juju refresh any of the HA solutions (the worker alone should do)
Wait for things to come back up without ever setting error status as that will prevent juju from proceeding.

Upgrade Notes

To upgrade from an older revision of any of these charms, the user will need to juju resolve manually all units of tempo, mimir, loki that are in error until they're gone.

[mmkay]

Tested with Tempo Pro:

Case 1, model teardown: all is good ✔️

Case 2, juju refresh for the modified worker with a version of a charm that doesn't have the changes: it seems we still have a restart in the logs however the charm never goes to error state:

20 Nov 2024 12:30:17+01:00  juju-unit  executing    running tempo-cluster-relation-changed hook
20 Nov 2024 12:30:18+01:00  workload   active       (all roles) ready.
20 Nov 2024 12:30:19+01:00  juju-unit  idle         
20 Nov 2024 12:32:17+01:00  workload   maintenance  stopping charm software
20 Nov 2024 12:32:17+01:00  juju-unit  executing    running stop hook
20 Nov 2024 12:32:18+01:00  workload   maintenance  restarting... (attempt #1)
20 Nov 2024 12:32:23+01:00  workload   maintenance  restarting... (attempt #2)
20 Nov 2024 12:32:23+01:00  workload   waiting      waiting for resources patch to apply
20 Nov 2024 12:32:26+01:00  workload   maintenance  
20 Nov 2024 12:33:12+01:00  juju-unit  executing    running upgrade-charm hook