Transformers for modeling physical systems.

Area of Science:

• Physics
• Machine Learning
• Dynamical Systems

Background:

• Transformers excel at natural language processing tasks by capturing long-term dependencies.
• Their application beyond natural language processing has been limited.
• Dynamical systems modeling is crucial for understanding physical phenomena.

Purpose of the Study:

• To adapt transformer models for predicting dynamical systems.
• To leverage Koopman-based embeddings for vector representation of dynamical systems.
• To evaluate the performance of transformer models against classical methods in scientific machine learning.

Main Methods:

• Utilizing transformer architectures for time-series prediction.
• Employing Koopman operator theory to create embeddings of dynamical systems.
• Training and testing the transformer model on diverse dynamical systems.

Main Results:

• The proposed transformer model accurately predicts the behavior of various dynamical systems.
• The model demonstrates superior performance compared to established classical methods.
• Koopman-based embeddings effectively represent complex system dynamics for transformer prediction.

Conclusions:

• Transformer models, enhanced with Koopman embeddings, offer a powerful new tool for dynamical systems prediction.
• This approach extends the utility of transformers beyond natural language processing into physical sciences.
• The method shows significant potential for advancing scientific machine learning applications.