SAM2 video-to-video real-time pipeline

[eliteprox]

Adds segment-anything-2 support to the live-video-to-video pipeline, supporting the models facebook/sam2-hiera-tiny, facebook/sam2-hiera-small and facebook/sam2-hiera-large.

Start stream:

curl --location 'http://localhost:8937/live/video-to-video/test/start' \
--form 'source_id="testout"' \
--form 'source_type="webrtcSession"' \
--form 'query="pipeline=segment_anything_2&params={\"point_coords\": [[200,300]], \"point_labels\": [1], \"show_points\": true, \"model_id\":\"facebook/sam2-hiera-tiny\"}"'

Update model parameters:

curl --location 'http://localhost:8937/live/video-to-video/test/update' \
--header 'Content-Type: application/json' \
--data '{
    "point_coords": [[200,300]],
    "point_labels": [1],
    "show_points": false,
    "model_id": "facebook/sam2-hiera-tiny"
}
'

[victorges]

Awesome! Only some minor nits

[seanhanca]

in segment_anything_2.py,

def update_params function, there is no exception handling or parmas validation, before applied the change, i.e recreate a model, with new params. what if it failed.

[eliteprox]

in segment_anything_2.py,

def update_params function, there is no exception handling or parmas validation, before applied the change, i.e recreate a model, with new params. what if it failed.

Params validation is handled by Sam2LiveParams which only accepts the fields and types defined there.

Errors originating from update_params or process_frame will be handled by the PipelineProcess class which loads the pipeline using a try/catch.

update_params is also used in infer.py to update model parameters from control api requests and uses a try/catch already.