HYPNOS: Interactive Data Lineage Tracing for Data Transformation Scripts

Area of Science:

• Data Science
• Information Visualization
• Software Engineering

Background:

• Data validation is crucial for reliable data analysis.
• Manual data lineage tracing is complex and time-consuming for analysts.
• Understanding data derivation is essential for debugging and validation.

Purpose of the Study:

• To introduce HYPNOS, a visualization system for interactive data lineage tracing.
• To support both schema-level and instance-level data lineage capture from transformation scripts.
• To provide intuitive views for data transformation process overview and detailed tracing.

Main Methods:

• Developed a lineage module to parse and adapt data transformation scripts.
• Implemented a visualization system with a lineage view and a detail view.
• Captured schema-level and instance-level data lineage.
• Conducted a use case, expert interviews, and a user study to evaluate the system.

Main Results:

• HYPNOS effectively captures and visualizes data lineage from transformation scripts.
• The system provides interactive views for exploring data relationships at different levels.
• Users can efficiently trace instance-level data lineage and inspect details.
• Demonstrated usability and effectiveness through user studies and expert feedback.

Conclusions:

• HYPNOS significantly aids data analysts in understanding and debugging data transformation pipelines.
• The system reduces the complexity and time required for data lineage tracing.
• Interactive visualization is a powerful approach for managing complex data workflows.