Can Large Language Models Predict Patient Treatment Choices? A Discrete Choice Experiment Framework

Area of Science:

• Health economics
• Artificial intelligence in healthcare
• Patient-reported outcomes

Background:

• Predicting patient health preferences is crucial for personalized medicine.
• Discrete choice experiments (DCEs) are commonly used to elicit patient preferences.
• Large-language-models (LLMs) offer new possibilities for analyzing complex health data.

Purpose of the Study:

• To evaluate the accuracy of GPT-4, a large-language-model, in predicting patient health preference-consistent choices.
• To assess the viability of using LLMs within a discrete choice experiment (DCE) framework for preference prediction.

Main Methods:

• Generated synthetic patient data from real DCE responses of cancer patients.
• Used GPT-4 to predict choices on hold-out questions based on varying numbers of sample questions.
• Assessed prediction accuracy across four experiments, varying sample size and question characteristics.

Main Results:

• GPT-4 achieved an average prediction accuracy of approximately 70% across experiments.
• Prediction accuracy improved with an increasing number of sample questions, up to a plateau.
• GPT-4 showed higher accuracy for choice questions with more distinct attribute differences.

Conclusions:

• GPT-4 can effectively infer patient preferences from limited choice data.
• LLM performance in preference prediction is comparable to surrogate decision-makers.
• The number of sample questions influences LLM accuracy, with diminishing returns after a certain point.