-
Notifications
You must be signed in to change notification settings - Fork 50
Smeared workflow execution #110
Comments
Without having thought too much about it, I would say it would make more sense to smear/throttle From our monitoring, it looks like the bursts appear on trigger boundaries (hours, days), but then also on a 10 minute interval, caused by the fixed 10 minute delay on "missing dependencies". By just throttling the dequeu event, we would solve both of those bursts right? |
Throttling/smearing dequeue sounds good to me 👍 |
#70 should have addressed this issue. Close? @danielnorberg |
@fabriziodemaria no #70 is about rate limiting pos submission in order to safe-guard the health of the k8s master and docker daemon. This issue is about smearing execution of workflows and by extension the data processing jobs they execute in order for users like Spotify to achieve smoother resource usage across the data processing platform. |
We would like to smear execution of scheduled workflows over time to achieve a smoother less bursty resource usage.
Would this imply smearing out the actual triggering (i.e. creation of active states) or would it make more sense to smear/throttle the actual execution of the workflows by e.g. some additional "smear-wait" state?
@rouzwawi WDYT?
The text was updated successfully, but these errors were encountered: