Longitudinal and Multimodal Recording System to Capture Real-World Patient-Clinician Conversations for AI and...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Data Capture

Background:

• Current AI models in medicine rely heavily on electronic health records (EHRs), which lack crucial patient-clinician interaction data.
• This data gap limits AI's ability to understand the nuances of care delivery beyond biological measures.
• Multimodal data, including voice, text, and video, are essential for a comprehensive view of clinical encounters.

Purpose of the Study:

• To design, implement, and evaluate a system for longitudinal, multimodal capture of patient-clinician encounters.
• To create a foundational dataset for artificial intelligence (AI) research by linking 360° recordings, postvisit surveys, and EHR data.
• To assess the feasibility of collecting comprehensive clinical encounter data.

Main Methods:

• A single-site study in an academic outpatient specialty clinic.
• Adult patients and clinicians were invited to enroll; encounters were recorded using 360° video and dual-channel audio.
• Postvisit surveys captured patient perspectives, and data were linked with EHR information.

Main Results:

• High consent rates were achieved: 97% of clinicians and 75% of patients.
• Successful recording and survey completion rates were high (76% and 96%, respectively).
• Data linkage across modalities is ongoing, with further analyses planned.

Conclusions:

• A feasible protocol for a longitudinal, multimodal patient-clinician encounter capture system was established.
• The system links 360° audio/video, surveys, and EHR data for AI research.
• This approach supports the development of AI systems that better reflect the holistic nature of medical care.