Active Learning of Multi-class Classification Models from Ordered Class Sets.

Area of Science:

• Machine Learning
• Computer Science
• Data Science

Background:

• Learning multi-class classification models typically requires extensive labeled data.
• Human annotators often have nuanced preferences beyond a single top choice, especially in complex domains like diagnostics.
• Existing methods may not fully leverage this richer annotator feedback.

Purpose of the Study:

• To develop a novel machine learning framework for multi-class classification using ordered class sets.
• To incorporate an active learning strategy that utilizes this ordered class set feedback.
• To demonstrate the effectiveness of the proposed framework in reducing annotation costs.

Main Methods:

• Developed strategies for training multi-class classification models with ordered class set information.
• Designed an active learning algorithm that leverages the ordered class feedback.
• Evaluated the framework's performance on multiple benchmark datasets.

Main Results:

• The proposed framework successfully learns from ordered class sets, capturing more nuanced annotator preferences.
• The active learning strategy effectively selects informative samples based on ordered class feedback.
• Both class-order feedback and active learning individually and jointly demonstrated a significant reduction in annotation cost.

Conclusions:

• Ordered class set feedback provides valuable information for multi-class classification.
• Integrating active learning with ordered class sets enhances learning efficiency and reduces data annotation burden.
• The framework offers a practical solution for scenarios with limited labeled data and complex classification tasks.