Evaluation of Large Language Model Performance in Assessing Health Economic Study Quality.

Area of Science:

• Health economics
• Artificial intelligence in research
• Quality assessment methodologies

Background:

• Economic evaluations are crucial for healthcare decisions but suffer from inconsistent reporting and complexity.
• Large Language Models (LLMs) present a scalable solution for evaluating adherence to reporting standards.
• Existing tools like Hileas provide a foundation for automated quality assessment.

Purpose of the Study:

• To assess the accuracy and reliability of LLM-generated evaluations of economic research quality.
• To compare LLM performance against human reviewers using the CHEERS checklist.
• To identify limitations and advance automated, assistive quality assessment in health economics.

Main Methods:

• 110 economic evaluation papers were assessed using the CHEERS checklist via LLM prompts.
• LLM outputs were compared against consensus ratings from two human reviewers.
• Performance metrics included Cohen's kappa, sensitivity, specificity, AUC, and System Usability Scale.

Main Results:

• Human reviewers disagreed on 25.3% of item-level evaluations, indicating low interrater reliability (kappa=-0.07 to 0.43).
• The LLM achieved 72.3%–94.7% agreement with human consensus, with AUC up to 0.96.
• LLM-assigned paper-level CHEERS scores were consistently lower than human scores.

Conclusions:

• LLMs offer a viable, albeit imperfect, proof-of-concept for evaluating research quality in health economics.
• LLM evaluations generally align well with human assessments, providing well-reasoned judgments.
• Further development is needed to address LLM limitations in fully supporting its quality assessments.