Event-Driven Multimodal Sensing and Computing for Context-Aware Home Monitoring Using Stereo Vision and Dietary Event...

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Real-world home monitoring faces challenges with privacy, user burden, and irregular domestic activities.
• Existing systems often require continuous data recording, raising privacy concerns and increasing storage demands.

Purpose of the Study:

• To develop an event-driven multimodal sensing and computing framework for privacy-aware, context-aware home monitoring.
• To integrate stereo vision and dietary event anchoring for synchronized behavioral episode reconstruction.

Main Methods:

• Utilized stereo RGB-based 3D human motion sensing and dining-zone-triggered meal image acquisition.
• Implemented event-driven orchestration, cross-modal synchronization, and privacy-aware local storage.
• Employed large-language-model (LLM)-assisted dietary context interpretation.

Main Results:

• The event-driven system concentrated motion data into behaviorally meaningful windows, with 81.5% of pose frames from 30.8% of active sessions.
• Achieved 87.5% overlap between meal events and motion sessions.
• LLM-assisted meal interpretation showed a mean absolute percentage error of 25.44% for coarse dietary context.

Conclusions:

• The developed framework demonstrates system-level feasibility for transforming irregular domestic observations into structured, temporally indexed, and privacy-aware multimodal behavioral records.
• The approach offers a promising direction for future home monitoring applications by balancing data capture with privacy preservation.