Active learning with human heuristics: an algorithm robust to labeling bias.

Area of Science:

• Machine Learning
• Cognitive Science
• Data Science

Background:

• Active learning enhances model training by adaptively selecting data for labeling.
• Human oracles, like doctors, can introduce labeling biases due to cognitive heuristics.

Purpose of the Study:

• To investigate the impact of human heuristics on active learning performance.
• To develop active learning algorithms robust to labeling bias.

Main Methods:

• Evaluated combinations of human heuristics (fast-and-frugal tree, tallying model) with active learning algorithms (entropy sampling, multi-view learning, information density, inverse information density) and classifiers.
• Tested on 15 real-world datasets across health, marketing, and transportation domains.

Main Results:

• Labeling by human heuristics significantly decreased active learning performance, sometimes below random chance.
• The proposed inverse information density algorithm improved performance by 87% compared to other methods.

Conclusions:

• Modeling human heuristics is crucial for designing effective active learning systems.
• Developing bias-robust active learning algorithms is essential for reliable prediction models.