Humans in the Loop: Incorporating Expert and Crowd-Sourced Knowledge for Predictions Using Survey Data

Area of Science:

• Social Sciences
• Computer Science
• Machine Learning

Background:

• Survey data often has more variables than observations, increasing overfitting risk.
• Variable selection is crucial for accurate prediction in such datasets.
• Integrating human knowledge into machine learning is an emerging area.

Purpose of the Study:

• To implement and evaluate a human-in-the-loop approach for variable selection in prediction models.
• To assess the impact of expert and layperson knowledge on predictive performance.
• To explore methods for incorporating human insights into machine learning workflows.

Main Methods:

• Utilized the Fragile Families Challenge dataset.
• Employed surveys to gather human knowledge on variable importance.
• Implemented strategies for data subsetting and incorporating human knowledge scores.
• Compared predictive performance of human-informed methods against baseline approaches.

Main Results:

• Human-informed data subsetting decreased predictive performance.
• Approaches using human knowledge scores alone performed marginally worse than those without.
• No significant overall improvement in prediction was observed with current human-in-the-loop methods.

Conclusions:

• Current methods of incorporating human knowledge into prediction models showed limited benefit.
• Further research is needed to develop more effective ways to integrate human insights.
• Human expertise may still hold potential for improving machine learning predictions.