Transformations establishing equivalence across neural networks: When have two networks learned the same task?

Area of Science:

• Dynamical Systems
• Artificial Intelligence
• Machine Learning

Background:

• Transformations are crucial for analyzing dynamical systems, particularly for studying instabilities and bifurcations.
• Artificial neural networks (ANNs) are powerful tools, but understanding equivalences between different network instantiations is challenging.

Purpose of the Study:

• To develop and test a data-driven method for establishing equivalence classes between ANNs.
• To explore transformations between network outputs and internal neuron activations.
• To demonstrate the applicability of this method across various learning tasks.

Main Methods:

• Utilized diffusion maps with a Mahalanobis-like metric to construct data-driven transformations between ANNs.
• Considered transformations based on network outputs and intermediate neuron activations.
• Applied Whitney's theorem to determine the necessary measurements for reconstruction.

Main Results:

• Successfully established data-driven transformations, indicating equivalence between pairs of ANNs.
• Demonstrated the algorithm's effectiveness on tasks including scalar functions, vector fields, and image representations.
• Showcased the reconstruction of network inputs and outputs from partial observations of internal activations.

Conclusions:

• The developed method provides a framework for identifying equivalent artificial neural networks.
• This equivalence construction is relevant to transfer learning and comparing different machine learning tools.
• The findings contribute to a deeper understanding of neural network relationships and capabilities.