Crowd-sourced annotation of ecg signals using contextual information

Area of Science:

• Biomedical Engineering
• Medical Informatics
• Signal Processing

Background:

• Accurate medical annotation is crucial, but manual labeling by experts suffers from inter-observer variability and human bias.
• Establishing a reliable ground truth for medical data is challenging due to inherent limitations in human and automated annotation methods.

Purpose of the Study:

• To develop and validate a novel probabilistic framework, the Probabilistic Label Aggregator (PLA), for generating a reliable ground truth from aggregated labels.
• To introduce contextual features (signal quality, physiology) for adaptive weighting of annotator performance, moving beyond simple aggregation methods.

Main Methods:

• The Probabilistic Label Aggregator (PLA) was developed to compare and aggregate human and automated labels without prior knowledge of individual performance.
• Novel contextual features, including signal quality and physiology, were incorporated to dynamically weight the contribution of each annotator (human or algorithm).
• The PLA was applied to QT interval estimation using crowd-sourced data from 20 humans and 48 algorithms from the 2006 PhysioNet/Computing in Cardiology Challenge.

Main Results:

• The PLA achieved a root mean square error of 13.97 ± 0.46 ms for automatic annotations, significantly outperforming the best challenge entry (16.36 ms) and mean/median voting strategies (17.67 ± 0.56 ms, 14.44 ± 0.52 ms).
• With only three annotators, the PLA improved annotation accuracy by 10.7% for humans and 14.4% for algorithms compared to median aggregation.
• Statistical analysis confirmed the significant improvement offered by the PLA (p < 0.05).

Conclusions:

• The Probabilistic Label Aggregator (PLA) offers a robust method for establishing a reliable ground truth in medical annotation tasks, even without pre-existing ground truth data.
• The PLA's adaptive weighting mechanism, utilizing contextual features, enhances accuracy and reliability over conventional aggregation techniques.
• This framework has the potential to serve as an improved "gold standard" for various medical annotation applications, mitigating issues of variability and bias.