Practical Considerations for Accuracy Evaluation in Sensor-Based Machine Learning and Deep Learning

Area of Science:

• Machine Learning
• Sensor Data Analysis
• Embedded Systems

Background:

• Standard machine learning accuracy evaluation uses train/test splits, which are insufficient for sensor-based problems.
• Real-world sensor data introduces challenges like thermal noise, quantization effects, and sensor failure impacting inference accuracy.
• Existing methods do not adequately prepare models for practical production environments.

Purpose of the Study:

• To propose and evaluate practical tests for comparing machine learning models used with sensor data.
• To assess the impact of thermal noise, lower inference quantization, and sensor failure on model accuracy.
• To provide a more robust method for selecting machine learning models for sensor-based applications.

Main Methods:

• Simulated the impact of sensor thermal noise on model inference accuracy.
• Compared model accuracy under lower inference quantization levels, mimicking reduced analog-to-digital converter (ADC) resolution.
• Evaluated and compared model tolerance to sensor failure.
• Utilized the UCI 'Daily and Sports Activities' dataset for presenting the practical tests.

Main Results:

• Machine learning algorithms exhibit varying resilience to thermal noise.
• Lowering inference quantization significantly affects model accuracy, with practical implications for embedded designs.
• Model performance degradation due to sensor failure differs across algorithms.
• The proposed practical tests reveal performance differences not apparent with standard train/test splits.

Conclusions:

• Standard train/test splits are inadequate for evaluating sensor-based machine learning models in production.
• Practical tests simulating thermal noise, quantization, and sensor failure are crucial for robust model selection.
• The proposed methodology enhances the reliability of machine learning models in real-world sensor applications.