Multi-Modal Fusion of Routine Care Electronic Health Records (EHR): A Scoping Review | JoVE Visualize

Area of Science:

Artificial Intelligence in Healthcare
Health Informatics
Clinical Decision Support Systems

Background:

Electronic Health Records (EHR) are widely used in healthcare, collecting diverse patient data including demographics, diagnoses, medications, notes, vital signs, and lab results.
These multi-modal EHR data possess rich semantic, conceptual, and temporal information, offering potential for advanced clinical decision support.
Recent generative learning techniques have shown promise in fusing these heterogeneous EHR data elements to enhance healthcare decision-making.

Purpose of the Study:

To conduct a scoping review of techniques for fusing multi-modal routine care EHR data.
To synthesize variances in fusion architectures, input data elements, and application areas.
To identify research gaps and promote the re-use of fusion techniques for novel clinical outcomes.

Main Methods:

A comprehensive literature search was performed on Google Scholar for high-impact fusion architectures using multi-modal routine care EHR data from 2018-2023.
The review followed PRISMA extension guidelines for scoping reviews.
Findings were analyzed thematically and comparatively.

Main Results:

A lack of standardized definitions for EHR data elements used as input modalities was observed, often neglecting data source, encoding, and concept level.
A revised classification for fusion architectures is proposed, distinguishing fusion from learning and accounting for concurrent learning across encoding, representation, and decision layers.
Current pre-trained encoding models exhibit inconsistencies in handling temporal and semantic information, limiting their re-use.

Conclusions:

Current EHR fusion architectures predominantly rely on a design-by-example approach.
There is a need for guidelines to design efficient multi-modal models for diverse healthcare applications.
Guidelines should emphasize best practices for modality combination, transfer learning, co-learning, and semantic/temporal encoding to promote re-use.