VMAF score of 100 for pictures with artefacts

[loicSallegro]

Hi all,
When I'm trying to run vmaf executable on a test sequence I obtain a vmaf score of 100 which is not possible comparing the two following frames:

As you can see we have artifacts on the second frame. This cannot lead to a VMAF score of 100.

I'm trying another sequence with the same strange results on black artefacts where I have a score of VMAF = 94.608120:

Have you already see this kind of score using VMAF measure on sequence with corrupted pixels?

If needed I can send YUV files.
The version of the VMAF I use is 2.3.1 and the model is vmaf_v0.6.1.

BR.

[kylophone]

Can you share .yuvs/.y4ms and the command-line you used? I would like to try to reproduce.

[loicSallegro]

Find attached the first example.pedestrian_3.zip
The command line I used is :

./vmaf -r pedestrian_ref_3.yuv -d pedestrian_dist_3.yuv -w 1920 -h 1080 -p 420 -b 8 -o vmaf_pedestrian.txt

see below the result : vmaf_pedestrian.txt

[christosbampis]

Thanks for sharing your finding and the corresponding files. We will take a look on our end as a corner case of the metric. In this case, the distortions are quite localized distortions (especially the first one) so I doubt a spatial (i.e. frame-based) feature averaging can catch this well.

[nilfm]

Hi, closing this as part of a clean-up. Please feel free to re-open this issue or a new one if you have further questions.

[loicSallegro]

Hi,
Is there any update on this subject?

[nilfm]

Actually I'll leave this open to provide updates in the future, we are working on a version that will fix this edge case and similar ones but it's not ready to be open-sourced yet.