Performance evaluation of human cough annotators: optimal metrics and sex differences

Area of Science:

• Medical Devices
• Biomedical Engineering
• Data Science

Background:

• Accurate cough quantification is lacking, hindering the development of automated monitoring systems.
• Human annotation is the gold standard but requires reproducible methods.
• Emerging automated techniques need validated annotation approaches.

Purpose of the Study:

• To assess human annotator agreement for cough sounds (intralabeller and interlabeller).
• To compare annotation agreement using cough counts versus cough seconds.
• To identify potential sex differences in cough characteristics.

Main Methods:

• 24 participants wore audiorecording smartwatches for 6-24 hours.
• 400 hours of audio were collected; 40 hours were analyzed.
• Cough counts and cough seconds were compared using linear and Bland-Altman analyses.

Main Results:

• Excellent intralabeller (Pearson's r=0.98) and interlabeller (Pearson's r=0.96) agreement was achieved.
• Using cough seconds reduced annotator discrepancies by 50% compared to cough counts.
• Significant sex differences were observed in cough sound length and epoch size.

Conclusions:

• Cough seconds are proposed as a more reliable unit for manual cough annotation, reducing variability.
• This method is crucial for evaluating automated cough monitoring systems.
• Observed sex differences highlight the need for equitable development of cough monitoring tools.