Charmed HPC
slurmutils for slurm config parsing, editing, and rendering
#5
Replies: 3 comments 1 reply
-
|
Looking at the readme, I found an example that I think I can use to better illustrate my point with. Preface: Slurm has 6 different categories of configuration parameters:
Taking a look at the readme example: from slurmutils.editors import slurmconfig
from slurmutils.models import Node
with slurmconfig.edit("/etc/slurm/slurm.conf") as config:
node = Node(
NodeName="batch-[0-25]",
NodeAddr="12.34.56.78",
CPUs=1,
RealMemory=1000,
TmpDisk=10000,
)
config.nodes[node.node_name] = nodeWhat I would like to be able to do with the slurm configuration is assemble it in a category by category manner. So I'm imagining something like: from slurmutils.editors import slurmconfig
from slurmutils.models import DownNode, Frontend, Node, NodeSet, Parameter, Partition
slurm_conf = slurmconfig.SlurmConfig()
parameters = get_slurm_parameter_conifg()
partitions = get_slurm_partition_config()
nodes = get_slurm_node_config()
nodesets = get_slurm_nodeset_config()
frontend = get_slurm_frontend_config()
down_nodes = get_slurm_down_nodes_config()
if all(parameters, partitions, nodes, nodesets, frontend, down_nodes):
with slurm_conf as config:
for node_inventory in nodes:
node = Node(**node_inventory)
config.nodes[node.node_name] = node
for partition_inventory in partitions:
partition = Partition(**partition_inventory)
config.partitions[partition.partition_name] = partition
...
slurm_conf.write_slurm_conf()Possibly this is a little far out from what exists right now, but I think it conveys a mapping between config constructs in slurm and config constructs that will be assembled and accounted for by the charm- and would be a nice way to use this SlurmModel as a way to construct the slurm conf in a very nice way. Thoughts? |
Beta Was this translation helpful? Give feedback.
-
|
I'd like to see We should add alternative constructors to the data models to make it easier to compose configuration objects. We can just add in class methods that call out to each models from slurmutils.editors import slurmconfig
from slurmutils.models import SlurmConfig
config = SlurmConfig.from_dict(
{
"SlurmctldHost": [
"juju-c9fc6f-0(10.152.28.20)",
"juju-c9fc6f-1(10.152.28.100)"
],
"ClusterName": "charmed-hpc",
"AuthType": "auth/munge",
"Epilog": "/usr/local/slurm/epilog",
"Prolog": "/usr/local/slurm/prolog",
"FirstJobId": "65536",
"InactiveLimit": "120",
"JobCompType": "jobcomp/filetxt",
"JobCompLoc": "/var/log/slurm/jobcomp",
"KillWait": "30",
"MaxJobCount": "10000",
"MinJobAge": "3600",
"PluginDir": [
"/usr/local/lib",
"/usr/local/slurm/lib"
],
"ReturnToService": "0",
"SchedulerType": "sched/backfill",
"SlurmctldLogFile": "/var/log/slurm/slurmctld.log",
"SlurmdLogFile": "/var/log/slurm/slurmd.log",
"SlurmctldPort": "7002",
"SlurmdPort": "7003",
"SlurmdSpoolDir": "/var/spool/slurmd.spool",
"StateSaveLocation": "/var/spool/slurm.state",
"SwitchType": "switch/none",
"TmpFS": "/tmp",
"WaitTime": "30",
"nodes": {
"juju-c9fc6f-2": {
"NodeAddr": "10.152.28.48",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-3": {
"NodeAddr": "10.152.28.49",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-4": {
"NodeAddr": "10.152.28.50",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-5": {
"NodeAddr": "10.152.28.51",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
}
},
"frontend_nodes": {},
"down_nodes": [
{
"DownNodes": [
"juju-c9fc6f-5"
],
"State": "DOWN",
"Reason": "Maintenance Mode"
}
],
"node_sets": {},
"partitions": {
"DEFAULT": {
"MaxTime": "30",
"MaxNodes": "10",
"State": "UP"
},
"batch": {
"Nodes": [
"juju-c9fc6f-2",
"juju-c9fc6f-3",
"juju-c9fc6f-4",
"juju-c9fc6f-5"
],
"MinNodes": "4",
"MaxTime": "120",
"AllowGroups": [
"admin"
]
}
}
}
)
slurmconfig.dump(config, "/etc/slurm/slurm.conf")Having these alternative constructors would be pretty nice since it would greatly simplify unit testing in the HPC charms. What do you think @jamesbeedy? |
Beta Was this translation helpful? Give feedback.
-
|
100% dead on! |
Beta Was this translation helpful? Give feedback.
-
|
@jamesbeedy landed support for constructing the slurm and slurmdbd configuration files from a dictionary! charmed-hpc/slurmutils#15 Here's how you can do it: config = SlurmConfig.from_dict(
{
"SlurmctldHost": [
"juju-c9fc6f-0(10.152.28.20)",
"juju-c9fc6f-1(10.152.28.100)"
],
"ClusterName": "charmed-hpc",
"AuthType": "auth/munge",
"Epilog": "/usr/local/slurm/epilog",
"Prolog": "/usr/local/slurm/prolog",
"FirstJobId": "65536",
"InactiveLimit": "120",
"JobCompType": "jobcomp/filetxt",
"JobCompLoc": "/var/log/slurm/jobcomp",
"KillWait": "30",
"MaxJobCount": "10000",
"MinJobAge": "3600",
"PluginDir": [
"/usr/local/lib",
"/usr/local/slurm/lib"
],
"ReturnToService": "0",
"SchedulerType": "sched/backfill",
"SlurmctldLogFile": "/var/log/slurm/slurmctld.log",
"SlurmdLogFile": "/var/log/slurm/slurmd.log",
"SlurmctldPort": "7002",
"SlurmdPort": "7003",
"SlurmdSpoolDir": "/var/spool/slurmd.spool",
"StateSaveLocation": "/var/spool/slurm.state",
"SwitchType": "switch/none",
"TmpFS": "/tmp",
"WaitTime": "30",
"Nodes": {
"juju-c9fc6f-2": {
"NodeAddr": "10.152.28.48",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-3": {
"NodeAddr": "10.152.28.49",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-4": {
"NodeAddr": "10.152.28.50",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
},
"juju-c9fc6f-5": {
"NodeAddr": "10.152.28.51",
"CPUs": "1",
"RealMemory": "1000",
"TmpDisk": "10000"
}
},
"FrontendNodes": {},
"DownNodes": [
{
"DownNodes": [
"juju-c9fc6f-5"
],
"State": "DOWN",
"Reason": "Maintenance Mode"
}
],
"NodeSets": {},
"Partitions": {
"DEFAULT": {
"MaxTime": "30",
"MaxNodes": "10",
"State": "UP"
},
"batch": {
"Nodes": [
"juju-c9fc6f-2",
"juju-c9fc6f-3",
"juju-c9fc6f-4",
"juju-c9fc6f-5"
],
"MinNodes": "4",
"MaxTime": "120",
"AllowGroups": [
"admin"
]
}
}
}
)
slurmconfig.dump(config, "/etc/slurm/slurm.conf")I published the new version of |
Beta Was this translation helpful? Give feedback.
-
Hello,
I would like to discuss the potentiality of using the
slurmutilspackage to parse and render the slurm configuration in theslurmctld-operatorcharm.We currently assemble and render the slurm config file in an adhoc/hacky way in the slurmctld-operator charm by collecting all of the configs we care about and render the
slurm.confby passing jinja2 a dict containing the slurm configuration options to render to the template file.What I would like to do is populate the
SlurmConfigmodel fromslurmutilswith a dict or json string containing the slurm configuration values, and use theSlurmConfigmodel to validate the slurm configuration and render theslurm.confconfiguration file.Looking at
slurmutils, I see that theSlurmConfigmodel can read in aslurm.conffile and parse it to json or dict, but I don't see a way to populateSlurmConfigwith a json/dict and then render it to a file.I figure I'm probably just missing something, but maybe this is the way things are.
Is there a way to populate the
SlurmConfigmodel from a json or dict?Thanks in advance,
James
Beta Was this translation helpful? Give feedback.
All reactions