Skip to content
This repository has been archived by the owner on Jul 12, 2023. It is now read-only.

Smeared workflow execution #110

Open
danielnorberg opened this issue Mar 24, 2017 · 4 comments
Open

Smeared workflow execution #110

danielnorberg opened this issue Mar 24, 2017 · 4 comments

Comments

@danielnorberg
Copy link
Contributor

We would like to smear execution of scheduled workflows over time to achieve a smoother less bursty resource usage.

Would this imply smearing out the actual triggering (i.e. creation of active states) or would it make more sense to smear/throttle the actual execution of the workflows by e.g. some additional "smear-wait" state?

@rouzwawi WDYT?

@rouzwawi
Copy link
Member

Without having thought too much about it, I would say it would make more sense to smear/throttle
the dequeueing in Scheduler. The trigger handler should only be doing metadata bookkeeping and not care about actual executions.

From our monitoring, it looks like the bursts appear on trigger boundaries (hours, days), but then also on a 10 minute interval, caused by the fixed 10 minute delay on "missing dependencies". By just throttling the dequeu event, we would solve both of those bursts right?

@danielnorberg
Copy link
Contributor Author

Throttling/smearing dequeue sounds good to me 👍

@fabriziodemaria
Copy link
Member

#70 should have addressed this issue. Close? @danielnorberg

@danielnorberg
Copy link
Contributor Author

@fabriziodemaria no #70 is about rate limiting pos submission in order to safe-guard the health of the k8s master and docker daemon.

This issue is about smearing execution of workflows and by extension the data processing jobs they execute in order for users like Spotify to achieve smoother resource usage across the data processing platform.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants