A Mixed-Methods Evaluation of LLM-Based Chatbots for Menopause.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Large Language Models (LLMs) show promise for healthcare question-answering.
• Ensuring accuracy and reliability of LLM-generated health content is critical to prevent adverse outcomes.
• Existing evaluation metrics for LLMs may not be adequate for high-stakes medical applications.

Purpose of the Study:

• To examine the performance of publicly available LLM-based chatbots for menopause-related queries.
• To evaluate these chatbots using a mixed-methods approach focusing on safety, consensus, objectivity, reproducibility, and explainability.
• To identify limitations of current evaluation metrics for sensitive health topics and propose improvements.

Main Methods:

• A mixed-methods approach was employed.
• Publicly available LLM-based chatbots were queried on menopause-related topics.
• Evaluation criteria included safety, consensus, objectivity, reproducibility, and explainability.

Main Results:

• Findings revealed both the potential and limitations of current LLM evaluation metrics in the context of sensitive health information.
• The performance of menopause chatbots indicated areas for improvement in reliability and safety.
• Traditional metrics demonstrated shortcomings when applied to nuanced medical queries.

Conclusions:

• There is a need for customized and ethically grounded evaluation frameworks for assessing LLMs in healthcare.
• Developing specialized metrics is essential for the safe and effective integration of LLMs into clinical practice.
• Further research is required to refine evaluation methodologies for AI in sensitive domains like women's health.