runner/live: Make sure to resize images outside of infer process

[victorges]

On the LiveKit PoC we resized the images on the agent code, and were missing that part from this infer.py piece. This is probably one reason for the inference to be lower on the ai-worker as the resize still happened, but it happened on the same process as we run the model so it was competing for the same Python GIL (and resizing on CPU might be some relevant time).

[j0sh]

We already resize as part of the video-to-image flow. Do we need to re-process it here?

[victorges]

@j0sh hmm I see. bad news that this doesn't explain the worse performance then :3

I think it might still be worth having this logic "self-contained" and not rely on an external logic to work (it doesn't just get slow, it actually explodes when using tensorrt).

Since I also have the checks to not reprocess in case it has the right size, I think it's beneficial to have it. WDYT?

[j0sh]

For now it is fine to try and get past this performance bottleneck. Does this (re-)encode the frame or are we only passing around raw frame data after this point?

For what it's worth, the ffmpeg resize tries to preserve the aspect ratio when rescaling which will probably trigger this extra work here unless the input is square. If we want to avoid that, feel free to adjust the ffmpeg CLI - cropping will might also need to be added before the rescale if we want to match fidelity. It's also still using CPU rather than GPU, although I am not sure if there is an equivalent to crop_cuda yet.

If this needs additional configuration for a specific model later, we can always pass it in to the ffmpeg command line from the initial params.

[victorges]

@j0sh yeah I don't think we should preserve the aspect ratio. The input has to be 512x512