"Can ChatGPT Answer Patient's Questions?": A Preliminary Analysis.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Consumer Health Information

Background:

• Public access to health information via AI tools like ChatGPT is increasing.
• Evaluating the accuracy and reliability of AI-generated medical answers is a significant challenge for non-experts.
• The trustworthiness of ChatGPT's medical advice for the general public remains largely unassessed.

Purpose of the Study:

• To evaluate the scientific accuracy and comprehensiveness of ChatGPT-4.0's responses to medical questions from the public.
• To assess the performance of GPT-4.0 (ChatGPT-4.0) using a dataset of consumer-generated health queries.
• To provide insights into the quality of AI-driven medical information available to non-specialists.

Main Methods:

• Utilized an existing dataset of consumer health questions from the NIH Genetic and Rare Diseases Information Center (GARD).
• Generated 1467 question-answer pairs using the GPT-4-0613 API.
• Randomly selected 100 pairs for evaluation based on Scientific Accuracy and Comprehensiveness using a 0-5 scale.

Main Results:

• ChatGPT-4.0 achieved high scores, with approximately 90% of answers rated 4 or 5 for Scientific Accuracy and 84% for Comprehensiveness.
• Around 7% and 14% of answers received an average score of 3 for Scientific Accuracy and Comprehensiveness, respectively.
• No significant difference in answer quality was observed based on whether the questions followed a specific framework.

Conclusions:

• ChatGPT-4.0 exhibits a high level of accuracy and comprehensiveness in answering public medical queries.
• Despite strong performance, the study underscores the necessity for healthcare consumers to consult healthcare providers for verification and personalized advice.
• Further research is recommended to explore question phrasing impact and comparative evaluations by consumers versus professionals.