libavcodec: v4l2m2m: include SPS/PPS headers in keyframes #10
Conversation
This adds an option "-repeat_seq_header", which tells the encoder to insert sequence headers repeatedly in the stream. This is required for clients to be able to join mid-stream in a H.264 stream from the Raspberry Pi. Without this option (the previous and default behavior), PPS/SPS headers are only added to the first IDR frame. When -repeat_seq_header is enabled, PPS/SPS is added to every IDR frame, allowing clients to decode the stream from that point. Signed-off-by: Marcus Better <[email protected]>
|
Fair comment. I'll see if it is sane to make that the default behaviour for contexts that don't ask for SPS/PPS in avctx (with an option to turn it off) as the overhead is low and the advantages noticeable. |
|
I've had a look at the code again and the encoder should prepend a SPS/PPS pair to every key frame if the AV_CODEC_FLAG_GLOBAL_HEADER (extract SPS/PPS and put it in the context) is not set. Trivial testing i.e. "ffmpeg -i test.mkv -c:v h264_v4l2m2m t.h264" shows that it works. Can you give me a failing ffmpeg command line please? |
|
I see what you mean, I can confirm it works with that command. In my case, I am pulling a H.265 stream from a camera, decoding it with the hardware HEVC decoder, and encoding into a H.264 RTSP stream, all in one ffmpeg command (in this case using go2rtc): In this setting, I can reproduce it 100% reliably. I verify with and it prints out the SPS/PPS headers. Without I don't know what exactly the difference is between the scenarios, and haven't been able to reproduce it in a self-contained way. |
|
I think the problem is that the rtsp encoder has AVFMT_GLOBALHEADER in its flags (end of rtspenc.c) which, I believe, requests SPS/PPS (or other similar global data) to be sent in extradata rather than inline in the stream. The v4l2m2m encoder respects this flag and doesn't send them inline. Removing the flag causes inline SPS/PPS to occur again and your problem goes away. My problem is that I don't understand the ffmpeg rtsp module very well & I can't work out whether removing this flag is a bad thing in other circumstances. |
|
Interesting that it doesn't seem to have that effect with libx264. At a minimum it should be possible to override that behavior with flags on the command line, but I wasn't able to make it work. Tried some combinations of |
|
libx264 doesn't appear to support the offending flag and that appears to be fine :-( I've kicked a query out onto ff-devel - maybe I'll get a response. This is something that the user shouldn't have to add flags for and I think that rtspenc is currently wrong so I'd prefer not to "fix" my code unless it becomes clear that is the only way forward. |
|
OK I now have a better understanding of how this is meant to work. Having got a reply on ff-devel it seems that the SPS/PPS packets aren't meant to turn up in the stream they are just meant to be part of SDP that is returned on stream join and my testing shows that they indeed turn up there. As they are constant there is no need to repeat - and this isn't a broadcast medium where joining a stream is just snooping on the data, the client has to actively connect and will get the SDP as part of that process. Snip from my ffprobe run showing this, using mediamtx as my RTSP server: ` |
|
Hmm I'm using go2rtc as RTSP server and I'm not seeing the Wonder if the problem is in go2rtc then? |
|
Might be - for testing purposes maybe try mediamtx? It is extraordinarily easy to set up for this test - just download a copy from |
This adds an option "-repeat_seq_header", which tells the encoder to insert sequence headers repeatedly in the stream. This is required for clients to be able to join mid-stream in a H.264 stream from the Raspberry Pi.
Without this option (the previous and default behavior), PPS/SPS headers are only added to the first IDR frame. When repeat_seq_header is enabled, PPS/SPS is added to every IDR frame, allowing clients to decode the stream from that point.