Nonlocal Attention Operator: Materializing Hidden Knowledge Towards Interpretable Physics Discovery

Area of Science:

• Artificial Intelligence
• Computational Physics
• Machine Learning

Background:

• Attention-based neural networks are popular in NLP and CV but under-explored for physical systems.
• Modeling physical systems involves discovering operators between function spaces, often leading to ill-posed PDE inverse problems.

Purpose of the Study:

• To propose a novel neural operator architecture, the Nonlocal Attention Operator (NAO), for foundation physical modeling.
• To explore the capability of attention mechanisms in addressing ill-posed inverse PDE problems.

Main Methods:

• Developed the Nonlocal Attention Operator (NAO) based on the attention mechanism.
• Demonstrated that the attention mechanism acts as a double integral operator enabling nonlocal interactions.
• Showcased NAO's ability to encode regularization and achieve generalizability for inverse PDE problems.

Main Results:

• NAO effectively addresses ill-posedness and rank deficiency in inverse PDE problems.
• The attention mechanism in NAO extracts global prior information and suggests an exploratory space via a nonlinear kernel map.
• Empirically demonstrated NAO's superior generalizability to unseen data resolutions and system states compared to baseline neural models.

Conclusions:

• NAO offers a novel neural operator architecture for learning interpretable foundation models of physical systems.
• The study provides a new perspective on understanding the attention mechanism within the context of physical system modeling.