Real-Time Emotion Recognition Performance of Mobile Devices: A Detailed Analysis of Camera and TrueDepth Sensors...

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Affective Computing

Background:

• Facial features convey critical information regarding emotions, motor function, and genetic conditions.
• Mobile devices with cameras and depth sensors enable real-time facial analysis for diverse applications.
• Understanding on-device emotion recognition is crucial for developing valid mobile applications.

Purpose of the Study:

• To evaluate the efficacy of on-device emotion recognition using Apple's ARKit framework on an iPhone 14 Pro.
• To assess the performance of a cosine similarity metric for classifying facial emotions from ARKit blend shapes.
• To compare the accuracy of the on-device system against human emotion recognition capabilities.

Main Methods:

• A native iOS application was developed to capture facial expressions from 31 healthy adults.
• ARKit blend shapes representing 36 movements and 7 discrete emotions were extracted per frame.
• A prototype-based cosine similarity metric was employed to classify emotions, with performance measured by accuracy and AUC.

Main Results:

• The cosine similarity classifier achieved an overall accuracy of 68.3%, outperforming the average human rater accuracy (58.9%).
• High per-emotion accuracy was observed for joy, fear, sadness, and surprise, with competitive results for anger, disgust, and contempt.
• Area Under the Curve (AUC) values were consistently high (≥0.84) across all emotion classes.

Conclusions:

• On-device facial emotion recognition using ARKit blend shapes and cosine similarity is feasible and achieves human-comparable performance.
• The method operates in real-time using only vector operations, ensuring privacy and minimal computational overhead.
• This technology supports the development of privacy-preserving mobile applications for real-time facial analysis.