Text Knows What, Tables Know When: Clinical Timeline Reconstruction via Retrieval-Augmented Multimodal Alignment

Area of Science:

• Biomedical Informatics
• Clinical Data Science
• Artificial Intelligence in Healthcare

Background:

• Precise clinical timelines are crucial for patient trajectory modeling and risk forecasting, especially in complex conditions like sepsis.
• Unstructured clinical narratives offer rich context but lack temporal precision, while structured electronic health record (EHR) data has precise timestamps but incomplete event capture.

Purpose of the Study:

• To develop and evaluate a retrieval-augmented multimodal alignment framework to improve the temporal precision of clinical timelines extracted from text.
• To bridge the gap between the semantic richness of narratives and the temporal accuracy of structured EHR data.

Main Methods:

• A graph-based, multistep process involving extraction of anchor events from narratives to form a temporal scaffold.
• Placement of non-central events relative to the scaffold and calibration using retrieved structured EHR data as external temporal evidence.
• Evaluation using instruction-tuned large language models on the i2m4 benchmark (MIMIC-III and MIMIC-IV).

Main Results:

• The multimodal pipeline consistently improved absolute timestamp accuracy (AULTC) and temporal concordance compared to text-only reconstruction across multiple models.
• Event match rates were not compromised by the multimodal approach.
• Analysis revealed 34.8% of text-derived events were absent from tabular records, highlighting the value of multimodal alignment.

Conclusions:

• Aligning unstructured narratives with structured EHR data produces more temporally faithful and clinically informative patient trajectory reconstructions than using either source alone.
• The developed framework enhances the temporal precision of clinical timelines, benefiting patient modeling and risk prediction.
• Multimodal data integration is essential for comprehensive clinical timeline reconstruction.