Exploring Semi-Supervised Methods for Labeling Support in Multimodal Datasets.

Area of Science:

• Human-Computer Interaction
• Data Science
• Sensor Data Analysis

Background:

• Multimodal dataset annotation, especially for video and sensor data, is labor-intensive and time-consuming.
• Existing methods often require extensive manual labeling, hindering efficient data processing.
• Integrating multiple data streams, like video and acceleration, presents unique annotation challenges.

Purpose of the Study:

• To develop an annotation tool that facilitates the labeling of multimodal datasets comprising video and inertial sensor data.
• To implement a semi-supervised labeling approach to reduce manual annotation effort.
• To leverage acceleration data from wrist-worn sensors to assist in video recording annotation.

Main Methods:

• Developed a novel annotation tool for video and inertial sensor data.
• Implemented semi-supervised learning to provide labeling recommendations after initial manual input.
• Utilized template matching on wrist-worn sensor acceleration data to identify activity intervals.
• Tested the approach on datasets involving warehouse picking, daily living activities, and meal preparation.

Main Results:

• The annotation tool successfully supports the labeling of multimodal datasets.
• Semi-supervised learning provided effective labeling recommendations, reducing annotation time.
• Template matching on acceleration data proved useful for identifying relevant time intervals for annotation.
• The method demonstrated its ability to assist annotators by suggesting potential labels.

Conclusions:

• The developed annotation tool and semi-supervised approach significantly streamline the annotation of multimodal datasets.
• Leveraging sensor data, such as acceleration, is a viable strategy to support video-based activity recognition and annotation.
• This method offers a practical solution for researchers dealing with large-scale multimodal data acquisition and analysis.