Analysis of large-language model versus human performance for genetics questions

Area of Science:

• Biomedical informatics
• Artificial intelligence in medicine
• Human genetics education

Background:

• Large language models (LLMs) are gaining attention for potential biomedical applications.
• Assessing the performance of LLMs in specialized fields like human genetics is crucial.

Purpose of the Study:

• To compare the performance of ChatGPT against human respondents on human genetics multiple-choice questions.
• To evaluate the accuracy and consistency of LLM responses in a biomedical context.

Main Methods:

• A set of 85 multiple-choice questions on human genetics was administered.
• Performance was evaluated by comparing ChatGPT's answers to 13,642 human responses.
• Accuracy rates and response consistency were analyzed.

Main Results:

• ChatGPT achieved 68.2% accuracy, comparable to human respondents' 66.6% accuracy (p=0.8327).
• Both ChatGPT and humans performed better on memorization questions than critical thinking questions (p<0.0001).
• ChatGPT provided different answers to the same question 16% of the time, offering plausible explanations for both correct and incorrect responses.

Conclusions:

• ChatGPT demonstrates impressive capabilities in human genetics knowledge recall but has significant limitations.
• Inconsistencies in responses and shortcomings in critical thinking highlight areas needing improvement for clinical use.
• Further development is necessary to ensure the reliability and safety of LLMs in high-stakes biomedical scenarios.