Voice EHR: introducing multimodal audio data for health.

Area of Science:

• Biomedical Informatics
• Artificial Intelligence in Healthcare
• Health Equity

Background:

• Artificial intelligence (AI) models trained on audio data show promise for clinical tasks and early detection.
• Current AI technologies rely on limited datasets from high-income countries, hindering deployment in resource-constrained settings.
• Audio data has the potential to significantly impact health equity.

Purpose of the Study:

• Introduce a novel protocol for audio data collection.
• Present a corresponding application for capturing health information via guided questions.
• Demonstrate the potential of Voice EHR as a health biomarker.

Main Methods:

• Developed a novel protocol and application (HEAR) for audio electronic health record (Voice EHR) collection.
• Utilized guided questions to capture health information through voice.
• Employed large language models (LLMs) to compare Voice EHR data with conventionally collected data.

Main Results:

• Voice EHR data was consistently rated as equally or more relevant for health evaluation compared to traditional methods.
• Initial experiments demonstrated the data quality of the Voice EHR system.
• Case studies highlighted the potential of Voice EHR as a biomarker of health.

Conclusions:

• The HEAR application facilitates Voice EHR collection, capturing complex health biomarkers from voice and speech.
• Voice EHR may compensate for limitations of unimodal clinical datasets by incorporating diverse audio features and context.
• This approach holds promise for improving healthcare in resource-constrained settings and advancing health equity.