An end-to-end framework for data lineage analysis covering link pattern recognition, fault diagnosis, and early...

Area of Science:

• Data Science
• Artificial Intelligence
• Network Engineering

Background:

• Increasing data platform complexity necessitates real-time failure detection.
• Traditional methods struggle with complex data lineage and fault tracing.

Purpose of the Study:

• To propose an End-to-End Full-Link (EEFL) intelligent analysis framework.
• To enable real-time prediction and tracing of data link failures.
• To enhance data governance through intelligent fault analysis.

Main Methods:

• Constructing a dynamic data lineage graph model using graph neural networks (GNN).
• Employing temporal convolutional networks (TCN) and GNN for hybrid fault diagnosis.
• Implementing a dynamic threshold warning mechanism with Bayesian optimization and online learning.

Main Results:

• EEFL achieved an average accuracy of 92.73% on enterprise and simulation data.
• The framework accurately classifies data outages, latency anomalies, and data contamination.
• Demonstrated superior performance compared to traditional fault detection methods.

Conclusions:

• The EEFL framework offers an effective solution for real-time data link failure analysis.
• EEFL provides intelligent decision support for data governance.
• The proposed methods significantly improve fault detection accuracy and reduce false alarms.