AssertionError: pts glitch while opening RGB .mp4 video

[niksirbi]

I'm opening this issue following up on the discussion started here.

I am trying to open this .mp4 video using the napari-pyav plugin, but end up with: AssertionError: pts glitch: 27729000 > 27726000.

Feel free to use the linked video for debugging.

I'm using Python 3.11.10, napari v0.5.4, and napari-pyav v0.0.6, on an M2 MacBook Pro.

The relevant section of the stack trace

File ~/.miniconda3/envs/movement-napari-pyav/lib/python3.11/site-packages/napari/layers/image/_slice.py:316, in _ImageSliceRequest._project_thick_slice(self=_ImageSliceRequest(slice_input=_SliceInput(ndisp...,    0,    0],
       [   0, 1027, 1299]]), id=2), data=<napari_pyav._reader.FastVideoReader object>, data_slice=_ThickNDSlice(point=(np.float64(9242.0), np.floa....float64(0.0), np.float64(nan), np.float64(nan))))
    313 if self.projection_mode == 'none':
    314     # early return with only the dims point being used
    315     slices = self._point_to_slices(data_slice.point)
--> 316     return np.asarray(data[slices])
        slices = (9242, slice(None, None, None), slice(None, None, None))
        np = <module 'numpy' from '/Users/nsirmpilatze/.miniconda3/envs/movement-napari-pyav/lib/python3.11/site-packages/numpy/__init__.py'>
        data = <napari_pyav._reader.FastVideoReader object at 0x15072e3d0>
    318 slices = self._data_slice_to_slices(
    319     data_slice, self.slice_input.displayed
    320 )
    322 return project_slice(
    323     data=np.asarray(data[slices]),
    324     axis=tuple(self.slice_input.not_displayed),
    325     mode=self.projection_mode,
    326 )

File ~/.miniconda3/envs/movement-napari-pyav/lib/python3.11/site-packages/napari_pyav/_reader.py:130, in FastVideoReader.__getitem__(self=<napari_pyav._reader.FastVideoReader object>, index=(9242, slice(None, None, None), slice(None, None, None)))
    128     return self.read_frame(index)
    129 elif isinstance(index, tuple) and isinstance(index[0], int):
--> 130     return self.read_frame(index[0])
        index = (9242, slice(None, None, None), slice(None, None, None))
        self = <napari_pyav._reader.FastVideoReader object at 0x15072e3d0>
        index[0] = 9242
    131 elif isinstance(index, slice):
    132     frames = [self.read_frame(i) for i in np.r_[index]]

File ~/.miniconda3/envs/movement-napari-pyav/lib/python3.11/site-packages/napari_pyav/_reader.py:114, in FastVideoReader.read_frame(self=<napari_pyav._reader.FastVideoReader object>, frame_idx=9242)
    112 frame_obj = next(self.framegenerator)
    113 while frame_obj.pts != target_pts:
--> 114     assert frame_obj.pts <= target_pts, f'pts glitch: {frame_obj.pts} > {target_pts}'
        target_pts = 27726000
        frame_obj = <av.VideoFrame, pts=27729000 yuv420p 1300x1028 at 0x1574fc6a0>
    115     frame_obj = next(self.framegenerator)
    116 frame = frame_obj.to_ndarray(format=self.read_format)

AssertionError: pts glitch: 27729000 > 27726000

I also tried transcoding the video following SLEAP's recommendations:

ffmpeg -y -i "single-mouse_EPM_video.mp4" -c:v libx264 -pix_fmt yuv420p -preset superfast -crf 23 "single-mouse_EPM_video_transcoded.mp4"

Following that, I could open the video, but the resulting layer appeared in grayscale.

Hopefully this info helps to improve/debug the plugin.

[bjudkewitz]

Hi @niksirbi thank you for opening this issue. I suspect this is now addressed with v0.0.7. Can you give it a try?

• it no longer raises an exception upon seek issues, but issues a warning (and suggestion to transcode). In case of seek overshoot (landing after the requested timeframe for some reason) it also tries to backtrack using a simple heuristic. Not sure yet how well that works in practice, but it probably won't harm. In any case, it will return a frame (with warning, if the time stamp is not what it expects).
• outputs are now rgb

[niksirbi]

Just gave v0.0.7 a try. A few weird things:

• Opening the original video (before transcoding) indeed emits a warning, as expected. But when playing the video in napari, that same warning keeps getting emitted over and over again, which I suspect is not the behaviour you intended.
• Opening the transcoded video emits the same warning, but only when opening the video, not when it's being played.
• If I try jumping to another frame during playback, the target frame starts "stuttering", and sometimes gets stuck in an infinite back-n-forth (see attached screen recording). I suspect this is becaus of

  In case of seek overshoot (landing after the requested timeframe for some reason) it also tries to backtrack using a simple heuristic.

2024-10-14.11-21-55.mp4

On a positive note, the outputs are now indeed RGB.

[bjudkewitz]

On a positive note, the outputs are now indeed RGB

:)

same warning keeps getting emitted over and over again, which I suspect is not the behaviour you intended.

I'm not so sure actually. I do kind of want to warn on every wrong time stamp (things don't have to fail that much as they do here).

If I try jumping to another frame during playback, the target frame starts "stuttering"

That's interesting. For the transcoded video or the original one?

Thank you for the video! It is is exactly the kind of example I've been looking for, because it has a bunch of issues we could test. Time stamps are irregular, not strictly increasing and some decoding time stamps are even negative.

[bjudkewitz]

Oh I believe the jitter is due to you clicking play but also jumping to a different point in the time line. I think that's a napari glitch for any video. It shouldn't happen if you press stop and then move the scroll bar.

[niksirbi]

That's interesting. For the transcoded video or the original one?

I had tried this on the transcoded version only.

Thank you for the video! It is is exactly the kind of example I've been looking for, because it has a bunch of issues we could test. Time stamps are irregular, not strictly increasing and some decoding time stamps are even negative.

Good to hear. Feel free to even use for unit testing/CI if you wish. It's part of a data repository we maintain for testing our Python package - movement.

[niksirbi]

Oh I believe the jitter is due to you clicking play but also jumping to a different point in the time line. I think that's a napari glitch for any video. It shouldn't happen if you press stop and then move the scroll bar.

You're right, jumping around while the video is paused works fine.

[bjudkewitz]

v0.0.8 seems to work with both versions of your video and generates a hopefully sensible amount of warnings.

[niksirbi]

It worked nicely now, with v0.08 installed from git.
I only got the warnings once, upon opening the non-transcoded video.
The transcoded video worked well without triggering any warnings.
Thanks 🎉

I'm closing this issue.

[bjudkewitz]

Thank you very much for the video and for testing!