Analysis of large-language model versus human performance for genetics questions

Area of Science:

• Genetics
• Artificial Intelligence
• Computational Biology

Background:

• The rapid advancement of large-language models (LLMs) necessitates evaluation of their capabilities in specialized scientific domains.
• Assessing the utility of AI tools like ChatGPT in genetics is crucial for understanding their potential applications and limitations.

Approach:

• A comparative analysis was conducted between ChatGPT's responses and human expert answers to genetics questions sourced from social media.
• The study involved a dataset of 13,636 genetics-related questions and responses posted online since 2021.

Key Points:

• ChatGPT's overall performance in answering genetics questions was statistically similar to that of human respondents.
• ChatGPT demonstrated superior performance on questions requiring factual recall (memorization) compared to those demanding analytical reasoning (critical thinking).
• The model exhibited variability, providing different answers to the same question upon repeated inquiries and generating plausible justifications for both accurate and inaccurate responses.

Conclusions:

• LLMs like ChatGPT can serve as a valuable resource in genetics, though with caveats regarding consistency and depth of understanding.
• Further research is warranted to refine AI models for complex problem-solving and critical analysis in scientific fields.
• The findings highlight the need for careful validation of AI-generated scientific information, particularly in educational and clinical genetics settings.