Prompting and Fine-Tuning Large Language Models for Parkinson Disease Diagnosis: Comparative Evaluation Study Using...

Area of Science:

• Artificial Intelligence in Medicine
• Computational Neuroscience
• Clinical Informatics

Background:

• Parkinson disease (PD) diagnosis is challenging due to diverse symptoms.
• Traditional machine learning (ML) models analyze structured clinical data.
• The utility of large language models (LLMs) for PD diagnosis using natural language is largely unexplored.

Purpose of the Study:

• Evaluate LLM diagnostic performance using natural language prompts from structured clinical data.
• Compare LLM performance against traditional ML classifiers.
• Assess the impact of prompting strategies and fine-tuning on LLM diagnostic accuracy.

Main Methods:

• Structured clinical variables from the Parkinson's Progression Markers Initiative (PPMI) dataset were converted into natural language prompts.
• Shapley additive explanations (SHAP) identified top 10 features, removing multicollinear variables.
• LLM performance was evaluated using few-shot prompting, dual-output prompting, and supervised fine-tuning.
• Logistic regression (LR) and support vector machine (SVM) served as ML baselines, with F1-scores used for evaluation.

Main Results:

• LLMs achieved comparable or superior diagnostic performance (F1-scores up to 0.987) to LR and SVM (F1-scores 0.960) on the test set.
• LLMs demonstrated sustained high performance on a temporal validation set, outperforming SVM which degraded significantly.
• Supervised fine-tuning enhanced stability, enabling GPT-4o-mini to achieve 0.987 F1-score on the test set.

Conclusions:

• LLMs can effectively process structured clinical data in natural language for PD diagnosis.
• LLM performance is influenced by prompting techniques, model selection, and data distribution.
• Further research is needed to optimize LLM application for clinical data analysis and ensure reliable generalization.