Scalable Big Data Platform With End-to-End Traceability for Health Data Monitoring in Older Adults: Development and...

Area of Science:

• Health Informatics
• Big Data Analytics
• Digital Health

Background:

• Real-time health data from wearables and questionnaires offer preventive care opportunities for aging populations.
• Current platforms lack data/model traceability, version control, and management of heterogeneous data streams, hindering reproducibility and accountability.
• Essential features for clinical accountability, regulatory compliance, and reproducibility are missing in existing health data platforms.

Purpose of the Study:

• Introduce DeltaTrace, a unified big data health platform with traceability as a core architectural feature.
• Demonstrate that embedding traceability enables scalable, auditable, and version-controlled processing of health data.
• Facilitate reproducible analytics and long-term maintenance of health monitoring systems.

Main Methods:

• DeltaTrace utilizes a medallion architecture with Delta Lake for version-controlled data transformations.
• Employs Apache Spark for distributed computation, Apache Kafka for data ingestion, and Apache Airflow for workflow orchestration.
• Integrates MLflow for machine learning model lifecycle management and Grafana for data visualization.

Main Results:

• DeltaTrace supports continuous processing for ~1500 users with end-to-end delays under 10 minutes.
• Ingestion, aggregation, and anomaly detection latencies are within acceptable ranges (e.g., < 7.5 min for visualization).
• Increasing CPU cores from 8 to 24 improved ingestion/cleaning latency by 25% and anomaly detection by 50%.

Conclusions:

• DeltaTrace offers a scalable, modular architecture with integrated traceability, model management, orchestration, and visualization.
• The platform provides complete version control for data and models, maintaining performance under limited hardware.
• DeltaTrace is suitable for continuous monitoring and preventive healthcare in aging populations due to reproducible and auditable processing.