Cluster quickstart

[bnordgren]

Hi,

I've installed HPC clusters in the past with Rocks Cluster and OpenHPC (xCat+Slurm). I'm looking at constructing a new (small) cluster using charmed HPC because our contractor is starting to use MAAS+Juju to manage Openstack onsite. Juju is very new to me.

I made myself a login node to run slurmctld, mysql, slrumrestd and slurmdbd, and I'm running slurmd on two other nodes. The slurmdbd-mysql-router doesn't appear to have installed, but I can apparently contact the slurmctld from my login node. I appear to have a main partition "slurmd" with two nodes in it (and having the correct hostnames). Unfortunately, they are marked "down". Also I don't see any of the non-slurm parts of a cluster that I'm used to seeing.

bryce@maas:~$ juju status
Model        Controller          Cloud/Region        Version  SLA          Timestamp
charmed-hpc  maas-cloud-default  maas-cloud/default  3.5.3    unsupported  19:58:34Z

App                    Version          Status   Scale  Charm         Channel      Rev  Exposed  Message
mysql                  8.0.36-0ubun...  active       1  mysql         8.0/stable   240  no       
slurmctld              23.02.7-0ubu...  active       1  slurmctld     latest/edge   69  no       
slurmd                 23.02.7-0ubu...  active       2  slurmd        latest/edge   90  no       
slurmdbd               23.02.7-0ubu...  waiting      1  slurmdbd      latest/edge   61  no       Waiting on: MySQL
slurmdbd-mysql-router                   error        1  mysql-router  dpe/beta     198  no       hook failed: "install"
slurmrestd             23.02.7-0ubu...  active       1  slurmrestd    latest/edge   63  no       

Unit                        Workload  Agent  Machine  Public address  Ports           Message
mysql/0*                    active    idle   2        10.0.1.25       3306,33060/tcp  Primary
slurmctld/0*                active    idle   2        10.0.1.25                       
slurmd/0*                   active    idle   0        10.0.1.23                       
slurmd/1                    active    idle   1        10.0.1.24                       
slurmdbd/0*                 waiting   idle   2        10.0.1.25                       Waiting on: MySQL
  slurmdbd-mysql-router/0*  error     idle            10.0.1.25                       hook failed: "install"
slurmrestd/0*               active    idle   2        10.0.1.25                       

Machine  State    Address    Inst id    Base          AZ       Message
0        started  10.0.1.23  node-b     [email protected]  default  Deployed
1        started  10.0.1.24  node-d     [email protected]  default  Deployed
2        started  10.0.1.25  hpc-login  [email protected]  default  Deployed

How do I use the juju charm (or another juju charm?) to:

1. Mount a shared directory (home directory or otherwise).
2. Sync certain critical files (e.g. passwd).
3. Install libraries (Mellanox. etc) and software (singularity or apptainer) across the cluster.

Also, does anyone have some pointers as to what is going wrong with the slurmdbd-mysql-router?

[NucciTheBoss]

Hi there @bnordgren - thank you for trying out Charmed HPC! I will say that Charmed HPC is still "early alpha" - we're hoping to have an alpha release later this year - so the feedback collected here will help us determine where extra work must be done. Please feel free to share what is and isn't working the way you expect it to. It's a huge help for us!

Now, I can answer some of your questions here:

1. For mounting directories, you current have two options. You can use our NFS charms or CephFS charms. In both cases, they use a client operator which handles mounting the directory in the location that you want the mount on the filesystem. Let me know which one you're potentially interested in using.

2. Which files do you need to sync across the cluster? Are you trying to sync the /etc/passwd file so that users will exist on each node within the system? We have LDAP + SSSD charms that you could potentially use. As for the munge.key and slurm.conf files, those are automatically synced across the cluster.

3. For installing critical libraries like Mellanox drivers and other software packages, we're still working on those bits, so it will be a bit manual for the time being. You can use juju exec to run commands across your nodes to install the drivers. We're going to have this be automatic in the future, but we're still writing the relevant bits of Python & Go 😅

   • As for the software packages, you can use Spack. You can use the package I linked here (I can update it quickly if necessary), or you can just clone Spack directly from GitHub. As for the container runtime, you can use @jamesbeedy's apptainer subordinate charm. I can send you instructions for that if you're interested. Also, if there's an ideal solution that you prefer, let us know.

4. The slurmd nodes come up as down by default since they'll typically need additional provisioning before they're ready to start running user workloads. To make them active, i.e. take them out of DownNode in slurm.conf, you can use the node-configured action:

$ juju run slurmd/0 node-configured --background
$ juju run slurmd/1 node-configured --background

Let me know if you have any questions/comments. I'll start a separate thread regarding your issues with the mysql-router.

[bnordgren]

Hi Nucci!

1. I'd potentially be interested in NFS charms when the cluster is small, and grow into a GlusterFS scenario later. We have a commercial Ceph cluster onsite, but its pretty heavily tied into Openstack Cinder to farm out RBDs. I looked into the NFS charms, but it appears the most recent supported platform (according to the webpage) is 18.04LTS, whereas I have 22.04LTS. Is that project abandoned, or is it going to be available going forward?
2. I was indeed attempting to sync /etc/passwd to distribute users across the cluster. I've never dug deep into the details of munge, but I thought that the passwd file was necessary to ensure that UIDs/GIDs were consistent across all the machines. We do now have an Active Directory onsite that we could use for the new cluster, but the compute nodes have always been isolated on their own network segment that can't see the real world. If there's some way to just join the login node to the domain and have the compute nodes not need to contact AD, I could be happy with that!
3. I'm super new to juju so thanks for pointing me to juju exec. Is there any way to specify a range of units, or just drop the '/0' at the end? If I wanted to run the same command on all the compute nodes for instance could I do juju exec -u slurmd -- whoami? Sometimes stupid slurmd would die and I'd just preemtively restart it using xcat's psh command. What approach are you taking for cluster management? Configuring an xCat or Warewulf instance with the compute nodes preloaded and dhcp/tftp disabled? Or is there a native juju thing you have in mind?
   1. For user code, we're aiming for local compatibility with larger supercomputing centers, so that when our users get large tasks, they can just run as-is on a partner's computer. So far we're targeting the singularity/apptainer path mostly because that's what I've observed in partner machines. Spack may be new enough that it's just escaped my notice. For system drivers (e.g. Mellanox) there's usually OFED support for the popular base OSes. We tend not to stray very far from the RHELs/OpenSuses/Ubuntus of the world, and our IT department doesn't like us to fall behind on updates.
4. Ah, that makes sense! That may deserve a mention on the "how to install" page. Even just a thing at the end saying "go configure your nodes and run the magic command when you're done" would raise awareness of the need for this last step, I think,

[NucciTheBoss]

Re. your issues with slurmdbd-mysql-router, the juju debug-log command is your friend here. It will show you the log stream collected from your deployed machines. Can you execute the following command:

$ juju debug-log --replay --include slurmdbd-mysql-router/0

This will show you the logs collected from the router. Please share a copy of those logs here so I can see what's going on with the failed unit. cc @taurus-forever 👀👀

[bnordgren]

Here's the log output. I left it on overnight, so it looks like the install retried a buncha times and failed probably the same way each time.
slurmdbd-log.txt