GPU Load Balancing

[j0sh]

The current load balancing algorithm for GPUs is a simple, stateless round robin. For each segment:

gpu = gpus[i++ % len(gpus)`

With the LPMS bottleneck fix, transcode sessions become stateful. A naive approach to maintaining "sticky" sessions is this:

if sessions[id].nil ? sessions[id] = gpus[i++ % len(gpus)]
gpu = sessions[id]

Can we do better than the naive approach? Probably.

Some challenges here are:

• Segments for a given session may be sporadic
  • O / T may not necessarily be the “primary” for a given stream
• Session assignment on GPUs may grow unbalanced
  • Sessions come and go, some may be long-lived, others short-lived
• Workload varies per session
  • Different transcoding profiles have different performance characteristics, segment lengths, etc
• Hardware capabilities not necessarily known
  • Would be nice not to manually tune / configure

Need to determine:

• What might constitute an improved algorithm
• How to measure / prove its efficacy
• Success criteria for when to wrap this up

TODOs:

• Finish LPMS Fix LPMS Bottleneck Tasks #997 and LPMS bottleneck fix lpms#141
• Test Design / Spec GPU LB : Test Design / Spec #1107
• Tooling Development / Improvements GPU LB : Tooling Improvements #1109
• Algorithm Refinement GPU LB : Algorithm Improvements #1110
• Simulation GPU LB : Simulation #1112
• Test Execution
• Analysis
• Iteration
• Reproducible Environment Reproducible GPU Test Environment #1111