Democratizing AI: non-expert design of prediction tasks.

Area of Science:

• Artificial Intelligence
• Human-Computer Interaction
• Machine Learning

Background:

• Non-experts have historically contributed to machine learning (ML) by providing training data and suggesting features.
• Previous research focused on non-expert contributions to pre-defined ML tasks, not task design.
• The potential for non-experts to independently design ML prediction tasks remains largely unexplored.

Purpose of the Study:

• To investigate how individuals without machine learning (ML) expertise can design prediction tasks.
• To categorize the types of prediction tasks designed by non-experts.
• To assess the feasibility of automatically training predictive models on crowdsourced data for these novel tasks.

Main Methods:

• Utilized a crowdsourcing platform for non-experts to propose and design prediction tasks.
• Employed crowd-based categorization and ranking of the designed tasks.
• Automatically collected data for top-ranked tasks and trained/evaluated predictive models.
• Conducted a randomized trial to evaluate the impact of instructions on task design.

Main Results:

• Individuals without ML experience successfully designed a diverse range of prediction tasks across various domains (e.g., politics, health, demographics).
• Predictive models could be automatically trained and evaluated on the crowdsourced data collected for these non-expert-designed tasks.
• The quality and type of tasks proposed by non-experts were influenced by the clarity and nature of the provided instructions.

Conclusions:

• Non-experts can collectively create valuable datasets and contribute to the ML pipeline through task design.
• Leveraging non-expert creativity in task design can enhance the scope and applicability of ML.
• Further research into optimizing instructions and addressing remaining challenges is crucial for maximizing non-expert contributions to ML and automation.