Physics-Informed Dataset Distillation

Dataset Distillation (DD) compresses a large real dataset into a substantially smaller synthetic set for training purposes while preserving the essential semantic information. Distribution-matching DD methods empirically design statistical alignment losses to optimize the synthetic set, aiming to preserve the crucial representativeness and diversity of the real dataset. However, relying solely on statistical information is difficult to truly reflect representativeness and diversity, making it impossible to obtain such exquisite synthetic sets. To address this limitation, motivated by free energy minimization principle that governs the physical system toward the equilibrium state, we first reveal the interesting similarity between the desired state of the synthetic set in DD and the equilibrium state of a particle system. It inspires us to derive a principled optimization objective in DD to enable the solved synthetic set toward the desired state. Based on this finding, we propose a Physics-Informed Dataset Distillation (PIDD) framework that optimizes the synthetic set by minimizing free energy. To implement the free energy minimization, we mathematically prove the free energy minimization is equivalent to minimizing the Kullback-Leibler (KL) divergence between the synthetic and real distributions. By applying random projection and then approximating the projected features as a Gaussian, we obtain a sliced KL surrogate for the free energy that admits a tractable closed-form expression. Extensive experiments on standard benchmarks demonstrate that PIDD achieves the first or second place in 18 out of 26 evaluation tasks, exhibits strong cross-architecture generalization, presents interpretable evolution process following the free energy minimization principle, and highlights the potential of physics principles in dataset distillation.