Using generative AI for the objective assessment of language in healthcare

Area of Science:

• Neuroscience
• Artificial Intelligence
• Clinical Psychology

Background:

• Traditional language assessments in neurological and psychiatric disorders are subjective and resource-intensive.
• Limitations of current methods impact reliability, sensitivity, and tracking treatment efficacy in clinical trials.
• Objective language evaluation is crucial for diverse patient populations.

Purpose of the Study:

• To introduce an objective language assessment method using GPT-4.
• To evaluate GPT-4's ability to predict Vineland Adaptive Behaviour Scales (VABS-II) expressive communication scores in autistic individuals.
• To compare GPT-4's performance against benchmark linguistic features.

Main Methods:

• Utilized GPT-4 to analyze transcribed conversations from autistic participants in an observational clinical trial.
• Predicted VABS-II expressive communication scores using the AI model.
• Compared GPT-4 predictions with traditional linguistic features (e.g., sentence length).

Main Results:

• GPT-4 predictions showed strong correlation with actual VABS-II scores (Pearson's r > 0.65).
• The AI model demonstrated high test-retest reliability (ICC(2,1) = 0.97).
• GPT-4 accounted for significantly more variance in scores than benchmark linguistic features.

Conclusions:

• GPT-4 provides a holistic, reliable, objective, and time-efficient assessment of expressive communication.
• Generative AI models can enhance the evaluation of communicative abilities in clinical trials.
• AI offers a scalable tool for assessing patients in clinical practice and treatment efficacy.