MM-HAR: Multi-Modal Human Activity Recognition Using Consumer Smartwatch and Earbuds.

Area of Science:

• Digital Health
• Wearable Technology
• Machine Learning

Background:

• Human Activity Recognition (HAR) is crucial for digital health applications like fitness tracking and sedentary behavior monitoring.
• Consumer wearables, including smartwatches and earbuds, are increasingly used for HAR.
• Single-device HAR systems often lack the comprehensive data needed for accurate activity recognition.

Purpose of the Study:

• To propose and evaluate a multi-modal approach for Human Activity Recognition using both earbuds and smartwatches.
• To demonstrate the limitations of single-modality HAR and the benefits of a multi-modal strategy.
• To establish a benchmark for future HAR research using diverse device combinations.

Main Methods:

• Collected a large dataset from 44 subjects in both in-lab and in-home settings.
• Trained and evaluated five distinct machine learning classifiers.
• Compared performance across various device combinations: earbuds only, watch only, and both earbuds and watch.

Main Results:

• Single-modality HAR (earbuds or watch alone) showed significant limitations in activity tracking accuracy.
• The multi-modal approach, utilizing both earbuds and watch, demonstrated superior performance in HAR.
• Classifier performance varied depending on the device combination, highlighting the impact of multi-modal data.

Conclusions:

• A multi-modal approach integrating smartwatches and earbuds offers enhanced accuracy for Human Activity Recognition.
• The findings underscore the limitations of single wearable devices and the necessity of multi-modal systems for comprehensive health monitoring.
• This study provides a valuable benchmark for the research community in developing advanced HAR systems.