How to loop 29.97fps CTA-WAVE content and other content without accumulated drift?

[tobbee]

With livesim2, it is possible to support much more content than livesim1.
The loop duration is now an integral number of milliseconds as is the average segment duration, instead of using integral
seconds in livesim1. This makes it possible to loop a set of 2.002s 29.97fps video segments,
or any length of 1.92s segments for both audio and video.

I’m now looking into how to use sim2 targets was to support more general content including CTA-WAVE reference content.
However, it has not been totally clear what it means.

I’ve discussed with Daniel and Romain about what it available and none of them see that there is a big
benefit in trying to add much content from CTA-WAVE for livesim2.

The CTA-WAVE content is available at https://dash.akamaized.net/WAVE/vectors
All variants are short (around 30s) and the audio track I listened to is only white noise.

Furthermore, the video and audio segment durations do not have the same average segment duration so their start will drift over time.:

• 25fps video => 50frames/segments = 2s
• 29.97fps video => 60 frames/segment = 2.002s (last segment is 59 frames)
• 48kHz AAC-LC => 93 frames/segment = 1.984s (last segment is 23 frames)

Of course, it would be nice to be able to loop 29.97fps 2.002s video segments with synchronized AAC audio.
That would require an average audio segment duration of 2.002s as well. The shortest duration which makes
the AAC frame duration compatible with the video is then 160 segments = 320.32s.

Since there is no such CTA-WAVE content available, and I would like to make it possible to have much more
flexible loop duration, I intend to implement audio resegmentation so that the output audio segments always
end with the first frame that reaches the end time of the corresponding video segment.
At the loop boundary, there may be a possible shoot-over (less than a frame) which will be added as an audio
timeline offset to the next iteration. After a cycle (whose length depends on the framerate and segment duration), the
audio and video will be perfectly aligned at the start of the loop again. This scheme makes it possible to have loop durations
of any number of (fixed-duration) video segments without accumulated audio drift, at the cost of a small cyclic audio time shift
below the perception threshold (less than 21ms for AAC-LC) The cycle is 4 for 6s segments, and 120 for 2.002s segments.
Of course, for content that has a duration corresponding to (a multiple of) the cycle duration, and those
audio segments already end at the same time as the video or right after, there will be no modification.

Beyond this I do not intend to do any adaptation to the CTA-WAVE content, but just make sure to include some shortened
content (8s of 25fps video and 8.008s of 29.97fps video) in the test suite, and then longer variants for livesim2 online.
For audio, I intend to use the beep audio of testpic, unless there is some other audio with synchronization markers that
is preferrable?

Any feedback on this plan or on audio track to accompany the CTA-WAVE video is very welcome.

[jpiesing]

I have no background with 29.97 FPS content so can't help you with that. Perhaps a US based colleague?

The audio in the WAVE test content is pseudo-random noise carrying timecode and other machine-readable information. If someone records the output of the TV / phone then they can analyse the recording. The noise fills the same function for audio as the QR code for video.

[tobbee]

Hi Jon, Thanks for the information about the audio content. I have looked around a bit but haven’t found any description of the test content. The Github page https://github.com/cta-wave/Test-Content contains nothing as far as I can tell. Do you have a good entry point to learn more? If the audio contains timing information for automatic tuning I think it makes sense to loop it together with the video. But, for full alignment after looping the content, the assets need a duration of that is a multiple of 320ms, like 30.08s or 32s, instead of 30s. Even nicer would be to the same segment length of 1.92s segment for both audio and video. Kind regards, Tobbe From: Jon Piesing ***@***.***> Date: Monday, 2 October 2023 at 13:34 To: Dash-Industry-Forum/livesim2 ***@***.***> Cc: Torbjörn Einarsson ***@***.***>, Author ***@***.***> Subject: Re: [Dash-Industry-Forum/livesim2] How to loop 29.97fps CTA-WAVE content and other content without accumulated drift? (Discussion #104) I have no background with 29.97 FPS content so can't help you with that. Perhaps a US based colleague? The audio in the WAVE test content is pseudo-random noise carrying timecode and other machine-readable information. If someone records the output of the TV / phone then they can analyse the recording. The noise fills the same function for audio as the QR code for video. — Reply to this email directly, view it on GitHub<#104 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AABPMETDKFIRD5MVTAADIETX5KRETANCNFSM6AAAAAA5MDVU7U>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[tobbee]

Implemented a solution where audio segment starts happens as soon as possible after video, but always at an exact timestamps to avoid drift. In PR #109.