2024-04-18-choromanski24a.md

title	abstract	layout	series	publisher	issn	id	month	tex_title	firstpage	lastpage	page	order	cycles	bibtex_author	author	date	address	container-title	volume	genre	issued	pdf	extras
Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers	We propose a new class of linear Transformers called FourierLearner-Transformers (FLTs), which incorporate a wide range of relative positional encoding mechanisms (RPEs). These include regular RPE techniques applied for sequential data, as well as novel RPEs operating on geometric data embedded in higher-dimensional Euclidean spaces. FLTs construct the optimal RPE mechanism implicitly by learning its spectral representation. As opposed to other architectures combining efficient low-rank linear attention with RPEs, FLTs remain practical in terms of their memory usage and do not require additional assumptions about the structure of the RPE mask. Besides, FLTs allow for applying certain structural inductive bias techniques to specify masking strategies, e.g. they provide a way to learn the so-called local RPEs introduced in this paper and give accuracy gains as compared with several other linear Transformers for language modeling. We also thoroughly test FLTs on other data modalities and tasks, such as image classification, 3D molecular modeling, and learnable optimizers. To the best of our knowledge, for 3D molecular data, FLTs are the first Transformer architectures providing linear attention and incorporating RPE masking.	inproceedings	Proceedings of Machine Learning Research	PMLR	2640-3498	choromanski24a	0	Learning a {F}ourier Transform for Linear Relative Positional Encodings in Transformers	2278	2286	2278-2286	2278	false	Choromanski, Krzysztof and Li, Shanda and Likhosherstov, Valerii and Avinava Dubey, Kumar and Luo, Shengjie and He, Di and Yang, Yiming and Sarlos, Tamas and Weingarten, Thomas and Weller, Adrian	given family Krzysztof Choromanski given family Shanda Li given family Valerii Likhosherstov given family Kumar Avinava Dubey given family Shengjie Luo given family Di He given family Yiming Yang given family Tamas Sarlos given family Thomas Weingarten given family Adrian Weller	2024-04-18		Proceedings of The 27th International Conference on Artificial Intelligence and Statistics	238	inproceedings	date-parts 2024 4 18	https://proceedings.mlr.press/v238/choromanski24a/choromanski24a.pdf