Natural Language Understanding Performance & Use Considerations in Virtual Medical Encounters

Area of Science:

• Medical Education Technology
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Traditional medical training relies on human standardized patients, which can be resource-intensive.
• Developing effective conversational artificial intelligence for medical training presents significant natural language understanding (NLU) challenges.
• Previous virtual patient systems have shown limitations in generalizability and performance.

Purpose of the Study:

• To evaluate the NLU performance of a novel virtual standardized patient (VSP) prototype.
• To assess the VSP's effectiveness and user interaction over multiple case repetitions.
• To determine if the VSP's performance is comparable to human-led training scenarios.

Main Methods:

• A controlled study involving 61 human subjects interacting with the VSP prototype.
• Subjects completed four repetitions of a standardized patient case.
• Performance was measured by the appropriate response rate of naïve users to the VSP's conversational inputs.

Main Results:

• The VSP prototype achieved an appropriate response rate exceeding 92% from users on their initial attempt.
• Performance remained stable across four case repetitions, indicating consistent NLU capabilities.
• The system's accuracy is comparable to human conversational patient training, with specific considerations.

Conclusions:

• The developed VSP prototype demonstrates high NLU performance suitable for medical training applications.
• A unified medical taxonomy is crucial for the VSP's ability to generalize training across diverse patient cases.
• This technology offers a scalable and effective alternative or supplement to traditional standardized patient training methods.