Analyzing Particularities of Sensor Datasets for Supporting Data Understanding and Preparation.

Area of Science:

• Data Science
• Sensor Technology
• Statistical Analysis

Background:

• Data cleaning is time-consuming for data scientists, especially with sensor data where anomalies are common.
• Existing research on anomaly detection in sensor data is abundant but often overlooks sensor-specific data characteristics.

Purpose of the Study:

• To analyze sensor data particularities based on sensor type and measurement units.
• To understand the behavior and detection of outliers and their impact on sensor equivalence for anomaly detection.
• To provide guidance for handling sensor datasets and propose an efficient implementation.

Main Methods:

• Analysis of statistical distributions and their impact on mean and variance across different sensor types and units.
• Investigation of outlier detection methods and their influence on sensor equivalence.
• Development and evaluation of a parallelized implementation for sensor data processing.

Main Results:

• Sensor type and measurement units significantly affect basic statistics like mean and variance.
• Outlier analysis provides insights into sensor equivalence, crucial for anomaly detection.
• The proposed guidance and parallelized implementation improve sensor data analysis efficiency, achieving up to four times faster computation.

Conclusions:

• Understanding sensor data characteristics is vital for effective data cleaning and anomaly detection.
• The developed methodologies offer practical guidance for data scientists working with diverse sensor data.
• Parallelization enhances computational efficiency for sensor data processing, enabling faster insights.