What exactly is Ref_emb and how to use it?

[117CCC]

First of all, thanks for this excellent library and it really helps.

I am little bit confuse about the concept Ref_emb, the reason why I want to use it is the VICReg loss.

I noticed that there is a paragraph on the website explaining this concept, but I still don't understand how to get the Ref_emb. For instance, I can get the embedding by apply the code like embeddings = model(data), But I don't know how can I get the Ref_emb from the dataset?

I would really appreciate it if you could explain the concept。

[cwkeam]

The way VICReg works is that, if you have embeddings like you said:

embeddings = model(data)

You create an augmentation

aug_embs = model(AUG(data))

where AUG is some augmentation function, then use the loss:

loss = loss_fn(embeddings, ref_emb=aug_embs)

This AUG(data) could be something like flipping the image, cropping the image, color jittering the image, and such, in the example of computer vision. The whole idea is that you want your model to produce aug_embs as close as possible to embeddings.

In more abstract terms, let's say data is some RGB image. Then:

embeddings = model(image)
embeddings_2 = model(FLIP_IMAGE(image))

loss = loss_fn(embeddings, embeddings_2)
# where the loss represents how far the embeddings is from embeddings_2.

We want the model to create a similar embedding for the original image and the flipped image, because we want our model to extract deeper information about the image than its trivial features (like the orientation of the image).

Does that answer your question?

[cwkeam]

Might be helpful to refer to this issue: #372

[117CCC]

Thank you so much for your answer. Your answer is very clear and easy to understand. Thanks again.