Evaluating interactive 2D visualization as a sample selection strategy for biomedical time-series data annotation

Area of Science:

• Biomedical Informatics
• Machine Learning
• Data Science

Background:

• Accurate data labeling is crucial for reliable machine learning in biomedical applications.
• Annotating complex biomedical time-series data presents significant challenges.
• Algorithmic sample selection methods can aid annotation but require validation with human annotators.

Purpose of the Study:

• To compare the effectiveness of three algorithmic sample selection methods for annotating biomedical time-series data.
• To evaluate random sampling (RND), farthest-first traversal (FAFT), and a 2D visualization (2DV) based method.
• To assess performance across infant motility assessment (IMA) and speech emotion recognition (SER) tasks with varying annotator expertise.

Main Methods:

• Three sample selection methods (RND, FAFT, 2DV) were compared across four classification tasks.
• Twelve human annotators (experts and non-experts) performed data annotation under a limited budget.
• Post-annotation experiments evaluated method performance and annotator experience.

Main Results:

• The 2DV method generally yielded the best aggregated labels across all tasks.
• In infant motility assessment, 2DV captured rare classes but showed high label variability, favoring FAFT for individual annotator models.
• For speech emotion recognition, 2DV excelled with expert annotators and matched expert performance with non-experts.
• Random sampling (RND) presented the lowest risk when annotator count or expertise was uncertain.

Conclusions:

• 2D visualization-based sampling is a promising approach for biomedical time-series data annotation, especially with adequate annotation budgets.
• The choice of method depends on task specifics, annotator characteristics, and budget constraints.
• The 2DV method enhanced annotator engagement, making the task more interesting and enjoyable.