Finding Related Tables in Data Lakes for Interactive Data Science

Area of Science:

• Data Science
• Computer Science

Background:

• Data lakes offer vast data storage but lack organization, hindering data discovery for scientific applications.
• Modern data science workflows require efficient access to tables, schemas, and datasets within these repositories.

Purpose of the Study:

• To develop integrated search and management solutions for data lakes within the Jupyter Notebook environment.
• To enhance data scientists' ability to find and utilize relevant data for tasks like feature extraction and data augmentation.

Main Methods:

• Development of search functionalities tailored for schema-agnostic data lake repositories.
• Integration of data management tools within the Jupyter Notebook platform.
• Generalization of core methods for broader application in script-based computational tasks.

Main Results:

• Scientists can now more easily locate and leverage data lake assets within their existing workflows.
• Improved efficiency in augmenting training data, identifying features, and finding joinable tables.
• Demonstrated applicability of the developed methods beyond the initial Jupyter Notebook environment.

Conclusions:

• The developed search and management solutions significantly improve data discoverability and usability in data lake environments.
• These tools empower data scientists to more effectively utilize data lake resources for various analytical tasks.
• The generalized methods offer a scalable approach to data management in diverse computational settings.