Data smashing: uncovering lurking order in data.

Area of Science:

• Data Science
• Machine Learning
• Signal Processing

Background:

• Automated discovery relies on comparing data streams to find connections and outliers.
• Current methods require human experts to define relevant data features, creating a bottleneck.
• This limits the pace of automated discovery despite abundant data.

Purpose of the Study:

• To propose a new principle for estimating similarity between arbitrary data streams.
• To enable automated data comparison without domain knowledge or machine learning.
• To address the limitations of feature-dependent data comparison algorithms.

Main Methods:

• Developed a principle for estimating data stream similarity.
• Applied the principle to real-world datasets without domain expertise.
• Tested on electro-encephalograph (EEG) patterns, cardiac sound recordings, and astronomical photometry.

Main Results:

• Achieved performance comparable to expert-designed algorithms in diverse applications.
• Successfully disambiguated epileptic seizures from EEG data.
• Detected anomalous cardiac activity and classified astronomical objects accurately.

Conclusions:

• The proposed data smashing principle facilitates automated discovery across various domains.
• It offers a powerful approach for analyzing complex data when expert knowledge is limited.
• This method may unlock new insights into complex observations by identifying unknown patterns.