Quantum-inspired quadratic attention with fourier-domain rank control in transformer architectures

Quadratic attention enhances interaction capacity in Transformer models but leads to rapid growth in computational demands as attention rank increases. This paper presents a bounded-rank quadratic attention mechanism where fixed-dimensional feature encodings determine the interaction space and enforce a strict upper bound on attention rank. A Fourier-domain formulation offers a spectral interpretation of the quadratic kernel via unitary transformation while maintaining exact attention computation. The proposed approach achieves bounded-rank attention with computational complexity that scales linearly with sequence length when the encoding dimension remains constant. Experimental validation on a real-world Tea pest image dataset yields 84.5% classification accuracy, surpassing the 82.1% achieved by a standard Vision Transformer under equivalent experimental conditions. Attention processing time per image declines from 14.8 ms to 5.6 ms. Peak memory usage declines from 820 MB to 430 MB. Although memory bandwidth and kernel launch overhead restrict the measured speedup to [Formula: see text], the accuracy loss under additive noise reaches 2.9% compared to 3.1% for the baseline, demonstrating comparable robustness. These findings indicate that explicit rank control in attention mechanisms can be realized through representational design, offering an efficient bounded-rank alternative to conventional full-rank attention.