The Value of Corrective Feedback in the Online Active Learning Paradigm

Area of Science:

• Machine Learning
• Data Science
• Human-Computer Interaction

Background:

• Online Active Learning (OAL) classifies evolving data streams with limited expert annotations.
• The goal of OAL is to minimize classification errors and annotation costs via Active Learning (AL) queries.
• Existing OAL methods do not fully leverage expert insights for error correction during classification tasks.

Purpose of the Study:

• To introduce and define a novel paradigm, Online Active Learning with Corrective Feedback (OAL-CF).
• To integrate a secondary feedback channel for experts to identify and correct classifier errors.
• To evaluate the impact of Corrective Feedback (CF) on classification performance and annotation efficiency.

Main Methods:

• Formal definition of the OAL-CF paradigm.
• Integration of a Corrective Feedback (CF) channel alongside the Active Learning (AL) channel.
• Experimental validation on binary classification tasks: Spoken Language Verification and Voice-Type Discrimination.

Main Results:

• OAL-CF enables experts to provide corrective feedback without disrupting their workflow.
• Experimental results demonstrate the efficacy of OAL-CF in classification tasks.
• The addition of CF to OAL led to a 53% relative reduction in annotation cost compared to OAL alone.

Conclusions:

• OAL-CF is an effective extension of OAL, improving classification efficiency.
• Corrective Feedback significantly reduces the cost associated with expert annotations.
• The OAL-CF paradigm offers a more efficient approach to classifying dynamic data streams.