Semantic Event Fusion of Different Visual Modality Concepts for Activity Recognition

Area of Science:

• Computer Science
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Multimodal sensor fusion for activity recognition is challenging due to imperfect temporal synchronization.
• Existing methods often rely on problem-specific graphical models tied to training data and single sensors.

Purpose of the Study:

• To propose a hybrid knowledge-driven and probabilistic framework for robust event representation and recognition.
• To address limitations in current multimodal fusion techniques, particularly in real-world, noisy conditions.

Main Methods:

• Developed a framework separating semantic modeling from raw sensor data using an intermediate 'concept' representation.
• Introduced a sensor alignment algorithm leveraging concept similarity to overcome temporal inaccuracies.
• Combined ontology language for flexible model definition with probabilistic interpretation for handling ambiguous data.

Main Results:

• The proposed framework demonstrated superior performance in event recognition compared to baseline methods.
• Accurate delimitation of temporal boundaries for event instances was achieved.
• Effectiveness shown in multimodal recordings of elderly individuals performing Instrumental Activities of Daily Living (IADLs).

Conclusions:

• The hybrid framework offers a more flexible and robust approach to multimodal activity recognition.
• The use of concepts as an intermediate representation and probabilistic ontological models enhances handling of real-world data complexities.
• This approach shows significant promise for applications in elderly care and assisted living.