Aiding Large Language Models Using Clinical Scoresheets for Neurobehavioral Diagnostic Classification From Text:...

Area of Science:

• Artificial Intelligence
• Psychiatry
• Behavioral Sciences

Background:

• Large language models (LLMs) demonstrate advanced capabilities but their diagnostic utility in psychiatry is under-explored.
• Automated diagnostics in behavioral sciences using LLMs require further investigation.

Purpose of the Study:

• Evaluate LLM chatbot diagnostic performance for neuropsychiatric conditions (autism, aphasia, depression).
• Compare direct diagnosis vs. code generation prompting strategies with and without clinical scales.
• Benchmark LLM performance against traditional machine learning classifiers.

Main Methods:

• Tested ChatGPT, Gemini, and Claude models with direct diagnosis and code generation prompting.
• Utilized ASDBank, AphasiaBank, and Distress Analysis Interview Corpus datasets.
• Assessed performance with and without structured clinical assessment scales, comparing to ML benchmarks.

Main Results:

• Clinical scales offered minimal performance improvement across datasets.
• LLM performance was inconsistent and generally below existing machine learning benchmarks.
• Code generation improved F1-scores for AphasiaBank (up to 86.5%) but direct diagnosis remained low for other datasets.

Conclusions:

• Current LLM chatbots, prompted naively, underperform specialized machine learning models in psychiatric diagnostics.
• Clinical assessment scales may offer slight improvements, but advanced prompt engineering is crucial for clinical utility.