Temporal dependency modeling of data inconsistencies in AI-driven ETL systems using graph-based embeddings

Area of Science:

• Data Science
• Artificial Intelligence
• Machine Learning

Background:

• Modern AI-driven Extract, Transform, Load (ETL) systems face challenges with data consistency due to heterogeneous sources, high velocity, and complex inconsistencies like missing values and duplication.
• Traditional rule-based validation frameworks struggle to capture intricate relationships and temporal dependencies in contemporary data pipelines.

Purpose of the Study:

• To propose a novel Temporal Graph-Based Embedding Model (TGBEM) designed to enhance data consistency detection in AI-driven ETL processes.
• To address the limitations of conventional methods by integrating Graph Neural Networks (GNNs) and Temporal Convolutional Networks (TCNs).

Main Methods:

• The TGBEM framework transforms ETL data into graphical features to represent inter-entity dependencies.
• It utilizes Temporal Convolutional Networks (TCNs) to process time-series data and identify evolving inconsistencies.
• The model was evaluated on both real-world benchmark datasets and a synthetic ETL dataset simulating complex pipeline conditions.

Main Results:

• The TGBEM achieved superior performance compared to traditional and existing deep learning models, with an accuracy of 95.80%.
• Key performance metrics included precision (94.95%), recall (93.70%), F1-score (94.32%), and AUC (96.10%).
• The model demonstrated reduced detection latency, making it suitable for near-real-time ETL validation.

Conclusions:

• The proposed TGBEM effectively tackles data inconsistency challenges in next-generation AI-guided ETL systems.
• Hybrid graph-temporal learning models are crucial for improving the detection performance of complex data inconsistencies.
• The TGBEM offers a robust solution for maintaining data integrity in dynamic data pipelines.