A Generic Human-Machine Annotation Framework Based on Dynamic Cooperative Learning.

Area of Science:

• Computational Linguistics
• Machine Learning
• Data Annotation

Background:

• Data annotation is costly and time-consuming, especially for subjective phenomena.
• Active learning and cooperative learning aim to reduce human annotation effort.
• Existing methods often lack efficiency in balancing label reliability and cost.

Purpose of the Study:

• To develop a novel generic annotation framework for optimal tradeoff between label reliability and cost reduction.
• To efficiently utilize both human and machine resources in the annotation process.
• To address limitations of existing confidence measures by extending applicability to regression problems.

Main Methods:

• Utilizing dropout for model uncertainty assessment to distinguish between machine-labelable and human-inspectable instances.
• Implementing an early stopping criterion based on inter-rater agreement to focus human effort on ambiguous instances.
• Developing new confidence measures applicable to both binary classification and regression tasks.

Main Results:

• The dynamic cooperative learning algorithm achieved a Spearman's correlation coefficient of 0.424, outperforming passive learning (0.413).
• The proposed method reduced the amount of human annotations required by 74%.
• Evaluated on benchmark datasets for non-native English prosody estimation.

Conclusions:

• The novel dynamic cooperative learning framework offers an efficient approach to data annotation.
• The method effectively reduces human annotation costs while ensuring high label reliability.
• The framework demonstrates broad applicability beyond binary classification to regression problems.