Into the Void: Mapping the Unseen Gaps in High Dimensional Data.

Area of Science:

• Data Mining and Analytics
• Machine Learning
• High-Dimensional Data Exploration

Background:

• Exploring high-dimensional datasets often leaves valuable 'empty' regions unexamined.
• Traditional methods struggle to systematically identify and exploit these uncharted data voids.

Purpose of the Study:

• To develop a comprehensive pipeline for exploring untapped opportunities in high-dimensional datasets.
• To identify and exploit novel configurations within the empty regions of data.

Main Methods:

• Utilizing a visual analytics system (GapMiner) and a novel Empty-Space Search Algorithm (ESA).
• Integrating user interaction with a deep neural network (DNN) for iterative dataset enhancement and configuration discovery.
• Employing gradient ascent and refined empty-space searches guided by a trained DNN for autonomous optimization.

Main Results:

• The pipeline successfully identifies center points of empty regions, representing potential valuable configurations.
• The system demonstrates consistent generation of superior novel configurations compared to randomization-based approaches.
• Effectiveness illustrated across multiple diverse case studies.

Conclusions:

• The developed methodology offers a systematic and effective approach to discovering novel configurations in high-dimensional data.
• The integration of visual analytics, user expertise, and DNNs enhances the exploration of data voids.
• This approach provides a significant advancement over conventional methods for data-driven discovery.