The 'mask' operation in the code

[taylor2success]

I've seen 'mask' in a lot of places in your code, such as functions called 'makeMaskfromLength' and 'makepadding', and I'd like to ask how 'mask' works in this task and what it does.

[xichenpan]

Hi,
these masks means padding mask in this task. you can refer to https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html for how padding mask works in Transfomer.
makeMaskfromLength function takes a longtensor as input, and output masked matrix. For example, the input is [3, 4, 7]
and the function output can be:
[[F, F, F, T, T, T, T],
[F, F, F, F, T, T, T],
[F, F, F, F, F, F, F]]
makepadding is a large function, it aims at padding the dowansampled audio and visual input stream at 25FPS.
You can try to debug the code to understand how each line works. (I'm sorry not giving enough comments to explain them in my code~

Xichen

[taylor2success]

Thanks for your reply