Splitting independent iterations across slots

[jvansanten]

Tasks can have multiple iterations, all of which are guaranteed to run in the same pilot/cluster job in order, and so can have access to files created in previous iterations. This made a great deal of sense for workloads where input files are processed piecewise, e.g. the old photonics simulation workflow where each iteration only loaded tables for a restricted slice of zenith angles.

There is however another, possibly more common, application for iterations, namely running the same simulation N times with different parameters, producing N different output files from either 1 or N input files. Examples of this are simulating the same events with different DOM efficiencies or ice models, or filtering the same simulation through the processing chain for different pole seasons. Each of these "iterations" is independent, and potentially much longer than the "root" photon propagation task when the photon propagation parameters are chosen to maximize GPU efficiency. In the concrete case I'm faced with at the moment, I have 1 photon propagation task feeding 15 different detector simulations. The photon propagation and a single iteration of the downstream processing each take 2 hours, meaning that the downstream processing needs 30 hours to complete all 15 iterations, which is longer than the lifetime of most of our glidein slots. This reduces throughput dramatically as tasks are killed until they happen to land on a site with long-lived slots.

It would be better if there were a way for pilots to only run a single iteration of a task in cases where those tasks are known to be independent. There are a couple of ways to attack this:

1. Split the dataset in two, and make the long-running iterations independent jobs in a dependent dataset, 15 of which depend on a single job in the photon propagation dataset. This is potentially tricky, because at the moment all task dependencies (including cross-dataset dependencies) assume a 1-to-1 mapping between job indices in the parent and child datasets. Changing that at the very least involves updating the db schema to track the parent job index for each task. It may also be necessary to allow parseable expressions in the depends field in order to specify job dependency graphs that do more than just branch and join at fixed ratios.

2. Make pilots request single iterations instead of entire tasks. This would require tracking which iterations have completed, as well as a way of forcing dependent iterations to run in order on the same slot if required.

(1) is potentially more work for the server, while (2) complicates the pilot and breaks the simple model of iterations. Which is the least bad?

[dsschult]

(1) just sounds like we need to work on it more. Right now you can set raw task_id dependencies in the database, but are missing convenience features in the config file. I like the idea of allowing an expression, so you can conceivably do a bit of math to shift the index.

I've specifically avoided (2) to reduce complexity, since this is part of what killed iceprod v1. If I had a choice, I would have killed iterations altogether.

One other solution is to have a tool make the dataset config for you, and make separate tasks that can run in parallel.