ICARUS: Minimizing Human Effort in Iterative Data Completion.

Area of Science:

• Data Science
• Database Management
• Human-Computer Interaction

Background:

• Data preparation is crucial, especially for incomplete datasets where missing values require domain expert input.
• Traditional imputation methods fail when missing values are domain-specific, necessitating manual rule-based completion.
• Manual rule formulation for large datasets is time-consuming and inefficient for experts.

Purpose of the Study:

• To develop an efficient system for domain experts to complete missing data in large datasets.
• To reduce the labor and time involved in manual data imputation through rule-based systems.
• To present users with impactful data subsets that facilitate informed data completion.

Main Methods:

• Introduction of ICARUS, a system employing a heuristic algorithm to identify and present critical data subsets.
• Utilizing a matrix-based interface for iterative data filling and rule suggestion.
• Leveraging database schema to infer hierarchies and amplify user edits into suggested rules.

Main Results:

• Simulations demonstrated an average improvement of 50% across three datasets compared to baseline systems.
• User studies showed naive users could complete 68% of missing data within an hour.
• ICARUS significantly outperforms manual rule specification, reducing completion time from weeks to hours.

Conclusions:

• ICARUS offers a computationally efficient and effective solution for expert-driven data imputation.
• The system empowers users to make significant progress in data completion tasks with minimal training.
• ICARUS represents a substantial advancement in interactive data preparation tools for large, incomplete datasets.