Research : Embedding Aware Attention

[Optimox]

Main Problem

When training with large embedding dimensions, the mask size goes up.

One problem I see is that sparsemax does not know about which columns come from the same embedded columns, this could create something a bit difficult for the model to learn:

• create embeddings that make sense
• mask embeddings without destroying them, in fact since sparsemax is sparse it's very unlikely that all the columns from a same embedding are used, so you lose the power of your embedding

Proposed Solutions

It's an open problem but one way I see as promising is to create embedding aware attention.

The idea would be to mask all dimensions from a same embedding the same way, either by using the mean or the max of the initial mask.

I implemented a first version here : #92

If you feel like this is interesting and would like to contribute, please share your ideas in comments or open a PR!

[joseluismoreira]

Great issue! Reading the paper and the both original implementation and this one I made the same question about the embeddings from the same columns be used together. Have you benchmarked that already, somehow?

[joseluismoreira]

I guess if we take the mean or the max we will lose the sparcity property... I am not sure, but maybe we can torch.stack the features instead of torch.cat them, and apply the sparse max in this new dimension. I would like to test this idea and maybe we can evaluate together the approachs.

[Optimox]

Hello @joseluismoreira,

There are two PR on this topic already:

• a very old one from me that was definitely not ideal : feat: add attentive embeddings #92
• a more recent one and more promising one : feat: embedding-aware attention #217

I guess the sparsity is not very important since you will only break the sparsity for embeddings from the same feature, so in the end you'll still be looking at only a few original features.

[joseluismoreira]

Thank you, @Optimox . The PR 217 seems what I look for. 👍

[W-void]

After sparsemax, only 1% feature is nonzeros, is it normal？

[Optimox]

hello @W-void,

The goal of sparsemax activation is to get a sparse mask, with a lot of 0 values. So yes it is expected.

However if you think the masks are too sparse you can play with different parameters:

• switch from sparsemax to entmax (not sure this will have a big impact)
• set lambda_sparse to 0 to reduce the penalization on sparsity
• add more steps and set a larger value of gamma to be sure that the features used by each step are different (in the end the algorithm will use more features)

Let me know if this helped.

[W-void]

hello @W-void,

The goal of sparsemax activation is to get a sparse mask, with a lot of 0 values. So yes it is expected.

However if you think the masks are too sparse you can play with different parameters:

• switch from sparsemax to entmax (not sure this will have a big impact)
• set lambda_sparse to 0 to reduce the penalization on sparsity
• add more steps and set a larger value of gamma to be sure that the features used by each step are different (in the end the algorithm will use more features)

Let me know if this helped.

@Optimox ，thx！I have set lambda_sparse to 0，It has little improvement. I will switch from sparsemax to entmax as you said.
I'm still confused. If only 2 or 3 features are used, the following FeatTransformer will get little information. I don't think it's a good result.

[SeohuiPark]

I have a question about mask and "explain" method.

I checked on Tabmodel.py that there is an "explain" method of the model. I wonder how different masks are printed for each instance of the test data because the "explain" method does not train the model.

Do we use the mask that has been learned until the last epoch of the training data in each row of the test data?

[Optimox]

The model learns to use its attention layer. For each row, the model decides what column should be masked or not.
During inference, the model decides on its own where to put its attention, nothing is hard coded here and every test row gets a different attention mask. You can use the explain method to see where the model has been looking for every row at every step.

[Optimox]

@athewsey this has been around for way too long. I have been thinking of a more general way to deal with this and I created a PR here : #443

I'd be happy to have your thoughts on this!