Question about 'attention bias not supported for flash attention'

[amitaie]

audiolm-pytorch/audiolm_pytorch/attend.py

Line 112 in 879f3bd

assert not exists(attn_bias), 'attention bias not supported for flash attention'

Why not to add the bias and the mask and create an attn_mask of type float and supply it to the scaled_dot_product_attention as attn_mask? is that not the same as we do where not using flash attention?

[amitaie]

actually I see that in the SoundStrom repo you started to do something like that:

https://github.com/lucidrains/soundstorm-pytorch/blob/22d257d6b5241583e84619b7af6a634158aba426/soundstorm_pytorch/attend.py#L96-L99

but yo left the assert there and also i didn't understand why you divide the value by 2?

[lucidrains]

@amitaie it is not possible to get bias gradients from flash attention