Streaming Molecular Dynamics Simulation Data for On-the-Fly Processing and Analysis

Only a small fraction of the data generated in state-of-the-art all-atom multimicrosecond molecular dynamics (MD) simulations is typically analyzed. With femtosecond integration steps, microsecond simulations generate billions of time steps with a complete set of atomic positions, velocities, and forces for all atoms, corresponding to petabytes of data and exceeding typical storage capacities. Consequently, only a fraction of the simulated time steps are usually written to a trajectory file for subsequent analysis, often at time intervals of 10-100 ps. Such a trajectory file allows for the analysis of ensemble averages and slow dynamics, but information on faster processes is lost. These fast processes include intra- and intermolecular vibrations, dynamics in nonglassy solvents, short-lived transition states, transport properties, etc., which encode physical information and are related to fundamental macroscopic properties and experimental observables. Here, we introduce a data streaming interface for MD simulations that provides easy access to all data generated during a running simulation. Instead of writing data to a storage medium, our interface enables user-defined analysis routines that access live simulation data via streaming. For this purpose, we build on existing implementations of the Interactive Molecular Dynamics (IMD) protocol and implement an enhanced protocol (termed IMD version 3 or 'IMDv3') in three popular MD packages: GROMACS, NAMD, and LAMMPS. Our new Python package imdclient receives an IMDv3 data stream and makes it available for other applications. To maximize usability, we added the capability to process streamed MD simulation data to the popular MDAnalysis software package. We demonstrate increased simulation performance for streaming compared to simulations that write data at high frequency to a trajectory output file. We include usage examples that illustrate live monitoring of custom variables during a running simulation, evaluation of velocity time correlation functions with fast fluctuations, and instantaneous analysis of currents through a membrane pore.