In the prefilling stage, will any tokens be pruned?

[exhyy]

Hi, thank you for your wonderful work! Here is my question:
In the prefilling stage, that is, during the prompt processing stage, will any tokens be pruned?

[sanagno]

Hi!

Pruning tokens in pre-filling does not make sense unless you propagate the dropping mechanism across depth as well. This is definitely possible but led to suboptimal results for the models that we tested. I am sure that if you use some pre-trained models that have some inductive bias to use less and less context with depth (I do not currently know any such models) then you can easily drop during pre-filling in that way.