How to evaluate the MSVD-QA and MSRVTT-QA

[jhj7905]

@mmaaz60 @hanoonaR
Hello, Thank you for sharing excellent work.
Can you tell me how to evaluate the MSVD-QA and MSRVTT-QA in detail?

[hanoonaR]

Hi @jhj7905,

Apologies for the late reply. Kindly check the instruction given for Zero-shot ActivityNet inference here.
We follow the same steps - with very minimal changes to adapt to the MSVD-QA and MSRVTT-QA datasets. The evaluation protocol, however, remains the same across all datasets. In case you have a specific question or is stuck with a problem, please let us know.

[wcy1122]

Hello, may I know where to download video in MSRVTT-QA. It looks like the official website [https://ms-multimedia-challenge.com/2016/dataset] is out of maintained.

[hb-jw]

Hello, I've also been replicating related benchmarks recently, and these benchmarks are mostly based on GPT-assistant, which seems quite costly. I'd like to ask, approximately how much does each of your evaluations cost?