reconsider juju-systemd-notices functionality in the slurmd charm

[jamesbeedy]

The slurmd service has tight integration with systemd, and even tighter in upcoming versions of slurm. Slurm + cgroupsv2 requires systemd to create and manage the scope of slurmstepd (a process responsible for managing steps spawned from slurm jobs via slurmd). Slurm delegates the management of the cgroup tree to systemd via the option Delegate=yes ref. I'm not sure if there are other ways to get around this, but for me I remove the middle-man slurm driver juju-systemd-notices and things work with slurm 23.11 + cgroupsv2. Possibly we can still use the juju-systemd-notices middleman and we just need to investigate further how to properly do this given the increasingly tighter coupling of slurmd and systemd.

[NucciTheBoss]

💢 Cue angry man shaking fist at systemd... 💢

Makes sense that systemd wants to have a single writer to sys/fs/cgroup, and the Delegate=yes option seems simple enough. I'm wondering if this is a juju-systemd-notices problem, or if this is an issue with how we're currently overriding the slurmd service file within the slurmd charm? Reason saying is that upon installation, we're overriding the service file for slurmd to supply the "try and start" functionality that enabled us to remove our dependence on etcd for tracking the state of nodes:

https://github.com/charmed-hpc/slurm-charms/blob/246f1da25861766b0b42a020f3b40bff4bf20f32/charms/slurmd/src/utils/slurmd.py#L67-L96

Perhaps we just need to add the Delegate=yes option here to this override file? juju-systemd-notices itself is just waiting for messages over DBus, so a bit strange if it's messing with Slurm's ability to manage cgroups.