Validation of an Artificial Intelligence-Powered Virtual Assistant for Emergency Triage in Neurology

Lucas Alessandro ^1,2, Santiago Crema ¹, Juan I Castiglione ¹

⁰Departments of Neurology.

The Neurologist +

|

February 6, 2025

View abstract on PubMed

Summary

An AI virtual assistant shows high accuracy in diagnosing neurological emergencies, outperforming ChatGPT. This technology is promising for improving emergency care in resource-limited settings.

Area Of Science

Neurology
Medical Informatics
Artificial Intelligence in Healthcare

Background

Neurological emergencies present significant challenges in resource-limited countries.
Artificial intelligence (AI) health chatbots offer a potential solution for improving medical care.
Rigorous validation is crucial to ensure the safety and accuracy of AI tools in clinical settings.

Purpose Of The Study

To evaluate the diagnostic safety and effectiveness of an AI-powered virtual assistant (VA) for the triage of neurological pathologies.
To assess the VA's performance in emergency neurological triage.
To compare the VA's capabilities against ChatGPT 3.5 and 4.

Main Methods

The AI virtual assistant's performance was evaluated in two stages.
Stage 1 involved 9 neurologists assessing the VA's safety using clinical records.
Stage 2 assessed the VA's accuracy when used by patients, followed by comparison with ChatGPT versions.

Main Results

Neurologists achieved 98.5% agreement with the VA's syndromic diagnosis, with definitive diagnoses consistently within the top 5 differentials.
The VA demonstrated superiority over ChatGPT 3.5 and 4 in all diagnostic and safety aspects (P <0.0001).
Neurologists rated the VA as "excellent" in 57.8% of evaluations, indicating adequate utility.

Conclusions

The AI virtual assistant demonstrated promising diagnostic accuracy and user satisfaction in neurological emergency triage.
These findings support further development and a comprehensive trial to assess the VA's real-time application.
The study encourages proceeding to a phase 1/2 trial with 100 patients.

Keywords:

artificial intelligence neurology triage