why is the visual transformer position embedding initializing to zeros

[matthewchung74]

I'm looking at the timm implementation of visual transformers and for the positional embedding, he is initializing his position embedding with zeros

self.pos_embed = nn.Parameter(torch.zeros(1, num_patches + 1, embed_dim))

https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/vision_transformer.py#L309

I'm not sure how this actually embeds anything about the position when it is later added to the patch?

x = x + self.pos_embed

any feedback is appreciated.

[rwightman]

@matthewchung74 it's actually init to trunc_normal_(self.pos_embed, std=.02) ... the zeros was just for defining the param but could have done it differently

[matthewchung74]

Thanks for responding @rwightman . This might be a silly question but what I'm not following is how can a truncated normal distribution give information about the position?

[rwightman]

It doesn't, but the gradients do once you start training. The pos embedding is learned.

[matthewchung74]

oh ic. thanks. FYI, I'm working on a blog post unpacking some of your visual transformer code. :)