Using Multimodal Data to Improve Precision of Inpatient Event Timelines

Area of Science:

• Medical Informatics
• Natural Language Processing
• Machine Learning

Background:

• Clinical text often lacks precise event timing.
• Structured data provides timestamps but lacks context.
• Integrating both data types is crucial for accurate healthcare event analysis.

Purpose of the Study:

• To develop methods for precise event interval timing using unimodal and multimodal data.
• To train multimodal language models for locating healthcare events in time.
• To evaluate the performance of multimodal approaches against unimodal ones.

Main Methods:

• Clinician annotation of discharge summaries using a dashboard tool.
• Development of unimodal (text-only) and multimodal (text and tabular) data processing pipelines.
• Training and evaluation of multimodal BERT and Llama-2 encoder-decoder models.

Main Results:

• Multimodal timestamping reduced uncertainty in lower bounds (42%), upper bounds (36%), and duration (13%).
• Multimodal BERT model achieved higher F1 scores for upper bounds (10% and 61%) and lower bounds (8% and 56%) compared to unimodal BERT and Llama-2.
• Annotation procedures and tools yielded high-quality timestamps.

Conclusions:

• Multimodal data significantly enhances the precision of event timing in clinical text.
• Multimodal language models trained on annotated data outperform unimodal models for temporal event localization.
• The developed tools and models offer a promising approach for improving healthcare data analysis.