How Do Transformers Model Physics? Investigating the Simple Harmonic Oscillator.

Area of Science:

• Artificial Intelligence
• Computational Physics
• Machine Learning

Background:

• Transformers are powerful deep learning models.
• Understanding how transformers model physical systems is crucial for developing reliable AI.
• The simple harmonic oscillator (SHO) is a fundamental physics model.

Purpose of the Study:

• To investigate the methods transformers employ to model the simple harmonic oscillator (SHO).
• To determine if transformers create interpretable physics or an "alien physics".

Main Methods:

• Developed four criteria (correlational and causal) to analyze transformer hidden states.
• Evaluated methods by analyzing the encoding of intermediate variables.
• Used linear regression as a test bed, focusing on the intermediate 'w' in y=wx.

Main Results:

• Transformers model the SHO using the matrix exponential method, a known numerical technique.
• The analysis framework successfully identified the underlying physical modeling method.
• The study provides a method to analyze transformer "world models".

Conclusions:

• Transformers utilize established numerical methods to model physical systems like the SHO.
• The developed analysis framework can be extended to complex linear and nonlinear systems.
• This research contributes to demystifying the internal workings of transformers in physics modeling.