Dynamic multi-variant relational scheme-based intelligent ETL framework for healthcare management

Area of Science:

• Computer Science
• Data Engineering
• Information Systems

Background:

• Organizations manage increasing volumes and dimensions of data in diverse formats.
• Big data analytics requires efficient data processing techniques like ETL.
• Current ETL methods face challenges in optimizing real-time data analysis.

Purpose of the Study:

• To present an efficient dynamic multi-variant relational intelligent ETL framework.
• To improve the performance and reduce the time complexity of ETL processes.
• To enable real-time data analysis through optimized ETL.

Main Methods:

• A distributed ETL framework utilizing ontologies and dynamic data dictionaries.
• Extraction of data from various sources, verified against data dictionaries.
• Computation of multi-variant relational similarity (MVRS) for data sources.
• MapReduce and data merging based on MVRS values for efficient data warehousing.

Main Results:

• The framework dynamically updates ontologies and data dictionaries using multiple ETL threads.
• Relational scores and MVRS are computed to guide data merging and selection.
• Improved ETL performance with minimized time complexity and enhanced efficiency.

Conclusions:

• The proposed intelligent ETL framework significantly enhances data processing performance.
• Dynamic updates and relational similarity measures are key to optimizing ETL.
• This approach facilitates efficient real-time big data analysis and integration.